Initiate Sweep Agent with Specific Hyperparameter Value

W&B Help
1

I am running a sweep from several different machines. The machines have different GPU RAM, so I need to manually set the batch size on each machine. I would like to set batch size as a sweep parameter, however, I can not figure out how to specify that certain agents use a specific value for a given hyperparameter, and randomly assign the rest.

Thanks,
Edward

2

Hi @edwardwang1,

The W&B Sweep server sends out hyperparameter configurations agnostic of your system’s capabilities, so I don’t think this can be done from inside the sweep config.

I think the best way to achieve this would be to have your script find out the amount of GPU VRAM on the system it is running on (I think PyNVML could be useful here) and then calculate the batch size using that information.

Thanks,
Ramit

3

Hi Ramit, thanks for the response. I know a priori what batch size I should use for a given machine, and I just want to hardcode that in to a given agent.

I’ve attempted to change the config via wandb.config.update, but it seems that does not work (I get the following message: “Config item ‘batch_size’ was locked by ‘sweep’ (ignored update).”

Thanks,
Edward

4

Hey Edward,

Even if you know the batch sizes a priori, I don’t think this would be possible from inside the sweep config. We plan to release conditional sweep configurations in the future, through which this should be possible.

For now, I think the best method would still be to calculate the batch sizes from inside your script. If you know the batch sizes you want to use, you should be able to set up a conditional statement to assign batch sizes according to the machine the script is running on.

Thanks,
Ramit

5

Hi Ramit,

Gotcha. Thanks!

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.