# Help with running a sweep agent on a multi-gpu machine with pytorch DistributedDataParallel **URL:** \ **Category:** W&B Help\ **Tags:** sweeps\ **Created:** [April 4, 2024, 12:03am UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291 "2024-04-04T00:03:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![yanyiphei](https://avatars.discourse-cdn.com/v4/letter/y/7bcc69/32.png) [@yanyiphei](https://community.wandb.ai/u/yanyiphei)\ **Post date:** [April 4, 2024, 12:03am UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291/1 "2024-04-04T00:03:37Z")
Hello, I need to do sweep agent runs on a multi-gpu machine with pytorch DistributedDataParallel. This means that I prepend `torchrun --standalone --nproc_per_node=8` to the training script command. The desired behavior includes 1) only the master process calls `wandb.agent(...)`, 2) data populated in `wandb.config` needs to be passed from the master process to all other processes, and 3) when the end of the run is reached, the agent starts another run when sweep count \> 1. I was able to do 1) and 2) in a very hacky way, so I dont need help here but if there are some recommended ways of doing them, please let me know. 1. remains unsolved for me. The primary problem is that, because only the master process calls `wandb.agent()` and `wandb.agent()` calls the main training function where `init_process_group` and DDP are used, then at the end of the script, only the master process is re-run again (because all other processes did not go through wandb.agent). Would really appreciate any help or pointers. You can find my entire training code here: [yif-AI/utils/train.py at main · yiphei/yif-AI · GitHub](https://github.com/yiphei/yif-AI/blob/main/utils/train.py). Problem 3) should be self-evident in this block: [yif-AI/utils/train.py at main · yiphei/yif-AI · GitHub](https://github.com/yiphei/yif-AI/blob/main/utils/train.py#L718-L755) ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [April 9, 2024, 10:09am UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291/2 "2024-04-09T10:09:49Z")
Hi @yanyiphei, thanks for your question. [Here](https://docs.wandb.ai/guides/sweeps/parallelize-agents#parallelize-on-a-multi-gpu-machine) you have our reference docs about running sweeps in a multi-gpu environment and [here](https://wandb.ai/wandb_fc/articles/reports/Multi-GPU-Hyperparameter-Sweeps-in-Three-Simple-Steps--Vmlldzo1NDQ0MDEy) is a report that could be helpful. ---
**Author:** ![yanyiphei](https://avatars.discourse-cdn.com/v4/letter/y/7bcc69/32.png) [@yanyiphei](https://community.wandb.ai/u/yanyiphei)\ **Post date:** [April 9, 2024, 5:40pm UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291/3 "2024-04-09T17:40:48Z")
@luis_bergua did you even read my post? The links you sent are entirely irrelevant. Or was I not clear? i can help clarify. ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [April 17, 2024, 3:11pm UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291/4 "2024-04-17T15:11:10Z")
Hey @yanyiphei, apologies if I caused any kind of confusion. Currently, we don’t have any official examples to run multi-gpu sweeps with DDP so that’s why I shared that page from our docs and the report since the recommended way is to set `CUDA_VISIBLE_DEVICES` for each process. Another option would be to use our feature [Launch](https://docs.wandb.ai/guides/launch/sweeps-on-launch) to push the sweep to a Launch Queue with the specified hyperparameters to sweep over. ---
**Author:** ![kokocakes](https://avatars.discourse-cdn.com/v4/letter/k/ee59a6/32.png) [@kokocakes](https://community.wandb.ai/u/kokocakes)\ **Post date:** [January 8, 2025, 2:13am UTC](https://community.wandb.ai/t/help-with-running-a-sweep-agent-on-a-multi-gpu-machine-with-pytorch-distributeddataparallel/6291/5 "2025-01-08T02:13:23Z")
Hello, did you ever find a solution to this problem? I’m also stuck on problem (3) you described above. @luis_bergua Are there any examples on how to run a sweep where the training function is run on multiple GPUs?