I wonder how Sweep works in multi-GPU mode? I want to initialize the parameters that need to be optimized in just one process, and then use Sweep for hyperparametric optimization. However, if I only initialize parameters in one process, other processes will report an error because they did not query parameters when loading the model. I didn’t find the answer to using Sweep in multi-GPU mode, thanks for answering!
Hey there, to use sweeps in multi-gpu setup you need to do the following:
Specify the hyperparameters you’re sweeping over in a YAML file, as detailed further in the sweep docs.
Get the sweep id by running the wandb sweep command and passing the yaml file as an argument
Run the wandb agent with the sweep id you just got. You will also need the to specify the GPU like this:
CUDA_VISIBLE_DEVICES=0 wandb agent sweep_id CUDA_VISIBLE_DEVICES=1 wandb agent sweep_id
Thank you ! I will have a try.
Feel free to message us if there are any issues.