Implementing multi-GPUs Training for RecBole

[juyongjiang]

Hi there,

I am a fan of the RecBole framework. Considering the complexity of the RecBole framework, I provide an easy but feasible method to achieve multi-GPUs training.
The core implementation idea is that re-wrapping the internal data type of Interaction to the PyTorch Dataloader object. The more details in my pull request branch "fix_multi_gpus", please check it.

Note that it is just one of the promising ways to realize multi-GPUs training. Hoping this method can inspire you to come up with a more novelty method to do it.

To use multi-gpus training model (e.g. BERT4Rec), you just need to run the following command:

1. Set the multi_gpus: True in your config.yaml files.
2. $ python -m torch.distributed.launch --nproc_per_node=3 run_recbole.py --model=BERT4Rec --config_files recbole/properties/model/BERT4Rec.yaml

Best Regards,
John

[2017pxy]

@juyongjiang Hi, thanks for your PR and we will carefully check it.

[2017pxy]

I do not understand this loop, it seems batch_data will be the last 'data' of dis_loader, could you please explain it?

[2017pxy]

And did you test your code in some datasets like ml-100k? Could you provide us the performance results of models? I want to know if the model performance will change a lot compared with single-GPU training.

[juyongjiang]

Hi, Xingyu! Yeah, my pleasure! In our DataLoader class, I assign batch_size=dataset.shape[0] that means it extracts all data in current batch_size. So the length of dis_loader will be only one, i.e. like this for data in range(1).

https://github.com/juyongjiang/RecBole/blob/0d35771629f65a9a06ad7e66dd11bfbe06091971/recbole/trainer/trainer.py#L173-L180

[juyongjiang]

Yeah, of course! Please wait for a moment! I will provide a table to illustrate the performance compared with single GPU training.

[juyongjiang]

@2017pxy Hi, Xingyu! I have got the experimental results. It seems that it doesn't decrease the performance a lot but significantly reduces the training time by about 3.78 times. BTW, I just run the experiment only one time. So I think this performance drift can be ignored. : )
Note that the original item means I got the result through running your original RecBole code. And multi-GPUs item result is produced by 3 multi-GPUs.

[hunkim]

@2017pxy Any further questions and or comments? Thanks in advance.

[2017pxy]

Hi @juyongjiang @hunkim, sorry for late reply.

Following your implementation, our team modified the trainer and made some tests. We find your implementation works well for model training. Thanks for your contribution!

However, since the time cost of run_recbole is mainly from model evaluation, we want to implement the multi-GPUs evaluation as well and release together with the multi-GPUs training. Unfortunately, we face some problems when we apply your implementations to evaluation since the data organization for evaluation is different. Thus, I am sorry to tell you that it still takes some time to release this new feature, and even this new feature might not be added in next version.

Thanks again for your implementation, and if you have any idea or suggestions about multi-GPUs evaluation, please let us know.

[juyongjiang]

@2017pxy Okay, got it! Thanks for your reply. I will implement the multi-GPUs evaluation as well and pull a new request. : )

[hunkim]

@juyongjiang Cool!

[KlaineWei]

Hello， I used your methods to implement multi-gpus on kagt, but after setting multi-gpus: True, this parameter doesn't seem to work as it isn't printed on the log. Is there any other setting that I have missed?