Sweep in DDP mode

I wonder how Sweep works in multi-GPU mode? I want to initialize the parameters that need to be optimized in just one process, and then use Sweep for hyperparametric optimization. However, if I only initialize parameters in one process, other processes will report an error because they did not query parameters when loading the model. I didn’t find the answer to using Sweep in multi-GPU mode, thanks for answering!

Hey there, to use sweeps in multi-gpu setup you need to do the following:

  • Specify the hyperparameters you’re sweeping over in a YAML file, as detailed further in the sweep docs.

  • Get the sweep id by running the wandb sweep command and passing the yaml file as an argument

  • Run the wandb agent with the sweep id you just got. You will also need the to specify the GPU like this:

     CUDA_VISIBLE_DEVICES=0 wandb agent sweep_id
     CUDA_VISIBLE_DEVICES=1 wandb agent sweep_id
1 Like

Thank you ! I will have a try.

Feel free to message us if there are any issues.