I wonder how Sweep works in multi-GPU mode? I want to initialize the parameters that need to be optimized in just one process, and then use Sweep for hyperparametric optimization. However, if I only initialize parameters in one process, other processes will report an error because they did not query parameters when loading the model. I didn’t find the answer to using Sweep in multi-GPU mode, thanks for answering!
Hey there, to use sweeps in multi-gpu setup you need to do the following:
-
Specify the hyperparameters you’re sweeping over in a YAML file, as detailed further in the sweep docs.
-
Get the sweep id by running the wandb sweep command and passing the yaml file as an argument
-
Run the wandb agent with the sweep id you just got. You will also need the to specify the GPU like this:
CUDA_VISIBLE_DEVICES=0 wandb agent sweep_id CUDA_VISIBLE_DEVICES=1 wandb agent sweep_id
1 Like
Thank you ! I will have a try.
Feel free to message us if there are any issues.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.