I create a sweep by this python script.

I notice that in the documentation: W&B sets the following distributions based on the following conditions if a distribution is not specified: `uniform if you specify max and min as floats`

```
sweep_config = {
'method': 'bayes',
'name': 'lr_sweep',
'metric': {
'goal': 'minimize',
'name': 'val/EPE_FD'
},
'parameters': {
'lr': {'max': 1e-2, 'min': 1e-6},
}
}
sweep_id=wandb.sweep(sweep_config, project="lr_sweep")
wandb.agent(sweep_id=sweep_id, function=main, count=10)
```

While as I set 1e-2 to 1e-6, if uniform should be 1e-2, 1e-3 … 1e-6 etc. However, in the wandb page as attached below, it is definitely not uniform as all most all are inside 1e-3 to 1e-4 only.

Could anyone provide a hint about why the uniform distribution in parameter is not working as it should be?

Hi @kinzhang , thank you for reaching out with your question.

As you have set the method for the hpo to `bayes`

, the process will make an informed decision when picking the parameter values for successive runs so the `uniform`

distribution won’t be the only criteria used to pick the next `lr`

value, which will be chosen based on which previous values of `lr`

minimised the value of `val/EPE_FD`

.

For example, I set up a test sweep with `val/EPE_FD`

decreasing when `lr`

decreases (using the same range as you for `lr`

) and you can see that values from the whole range are used for lr:

If you want to ensure all the values in that range are explored you may want to use `grid`

or `random`

method for the search.

Please let me know if you have any further questions

Hi @fmamberti-wandb

Thanks for your reply! I see. Here are two sweeps I did with metric added:

After your reminds, I checked and wondering is there anyway we can help expand the step at the beginning. (I mean like 1 is sweep at the end to find, and the 2 didn’t have a chance to go down.)

In another way, you are right about Bayes with uniform here maybe I should change to random etc.

Thanks.

Hi @kinzhang , the optimizer for the bayesian method is restarted twice to avoid finding a local optimum (see code here) however, currently is not possible to control this. If this is something you would like to be able to do, I’d be happy to raise a feature request for this - let me know.

1 Like

Hi @kinzhang , I wanted to follow up on this request. Please let us know if we can be of further assistance.

Hi @fmamberti-wandb Thanks for your sync. The previous discussed already helped me understand Bayes and it’s restart strategy. Thanks again!

Recently, I changed to random with 10 possible values provided directly. While I’m curious again, random is not random with the reset not running but random on all possible values even if it runs once?

Hi @kinzhang , the random method for sweep doesn’t prevent the same set of parameters from being reused so the same combination may be used for multiple runs.