Sweep parameters default `uniform` is not uniform param from min to max

W&B Help
,
1

I create a sweep by this python script.
I notice that in the documentation: W&B sets the following distributions based on the following conditions if a distribution is not specified: uniform if you specify max and min as floats

sweep_config = {
    'method': 'bayes',
    'name': 'lr_sweep',
    'metric': {
        'goal': 'minimize',
        'name': 'val/EPE_FD'
    },
    'parameters': {
        'lr': {'max': 1e-2, 'min': 1e-6},
    }
}
sweep_id=wandb.sweep(sweep_config, project="lr_sweep")
wandb.agent(sweep_id=sweep_id, function=main, count=10)

While as I set 1e-2 to 1e-6, if uniform should be 1e-2, 1e-3 … 1e-6 etc. However, in the wandb page as attached below, it is definitely not uniform as all most all are inside 1e-3 to 1e-4 only.

image
image486×602 45.2 KB

Could anyone provide a hint about why the uniform distribution in parameter is not working as it should be?

2

Hi @kinzhang , thank you for reaching out with your question.

As you have set the method for the hpo to bayes, the process will make an informed decision when picking the parameter values for successive runs so the uniform distribution won’t be the only criteria used to pick the next lr value, which will be chosen based on which previous values of lr minimised the value of val/EPE_FD.

For example, I set up a test sweep with val/EPE_FD decreasing when lr decreases (using the same range as you for lr) and you can see that values from the whole range are used for lr:

Screenshot 2024-07-31 at 13.18.17
Screenshot 2024-07-31 at 13.18.17984×906 74.5 KB

If you want to ensure all the values in that range are explored you may want to use grid or random method for the search.

Please let me know if you have any further questions

3

Hi @fmamberti-wandb

Thanks for your reply! I see. Here are two sweeps I did with metric added:

image
image874×505 48.3 KB

After your reminds, I checked and wondering is there anyway we can help expand the step at the beginning. (I mean like 1 is sweep at the end to find, and the 2 didn’t have a chance to go down.)

In another way, you are right about Bayes with uniform here maybe I should change to random etc.

Thanks.

4

Hi @kinzhang , the optimizer for the bayesian method is restarted twice to avoid finding a local optimum (see code here) however, currently is not possible to control this. If this is something you would like to be able to do, I’d be happy to raise a feature request for this - let me know.

1 Like
5

Hi @kinzhang , I wanted to follow up on this request. Please let us know if we can be of further assistance.

6

Hi @fmamberti-wandb Thanks for your sync. The previous discussed already helped me understand Bayes and it’s restart strategy. Thanks again!

Recently, I changed to random with 10 possible values provided directly. While I’m curious again, random is not random with the reset not running but random on all possible values even if it runs once?

image

7

Hi @kinzhang , the random method for sweep doesn’t prevent the same set of parameters from being reused so the same combination may be used for multiple runs.

8

Hi @kinzhang , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

1 Like