I often perform several sweeps,because of several reasons (require computational resources, crashes, adjust hyperparameter ranges, etc)
If I start another bayes sweep, will it take the results of runs from the already finished or stopped sweeps into account? Do I need to narrow down the ranges based on my interpretation of the results from the last sweep, or will the bayes search automatically do that?
Also: does the run_cap influence how soon the search will switch from exploration to optimization?
hey @tim-kuipers - if you start up a new sweep, this would be independent of the previous. I will look into seeing if there’s a way to circumvent this or add a condition to prevent duplicate parameters from being tried on the new sweep, but I would recommend narrowing the ranges in the new sweep for now.
run_cap solely controls the maximum number of runs per sweep
Thanks for your help.
I’m not really concerned with duplicate parameters being tried out; my concern is that subsequent bayes searches don’t learn from previous results. Doing more sweeps doesn’t make the result any better, because a new sweep doesn’t take the results of the last one into account.
I thought that bayes search starts out with doing more exploration, but toward the run_cap it would switch to just trying to get the optimum. However, looking at the docs again I seem to have been mistaken.
hey @tim-kuipers - here is an article that details the bayes hyperparemeter optimization we have set up in our sdk, that is mostly based on this paper right here. please let me know if you have any more questions about this!
Did you ever find this out? Did you make a feature request for this?
The article you just linked doesn’t say which surrogate model is actually used. Is that confidential information?
BOHB needs a couple of random trials before we can fit the surrogate model. Generally people start out with 3 or 5 random trials. How many random runs does wandb do before employing the surrogate model?