I have a question about WandB’s Sweep.
If I set ‘bayes’ to the Sweep method, the controller selects new better parameters from the parameters and the training results using Bayesian optimization. In this case, is the controller referring to the value of the key [‘metric’][‘name’] in the Summary dictionary as the learning result?
(Here [‘metric’][‘name’] refers to the optimization metric specified in the config file.)
Also, if wandb.define_metric(‘val_loss’, summary=‘min’) is used, the value will be registered in the Summary dictionary as val_loss.min (not val_loss). In this case, does the parameter selection by the bayes controller still work correctly?
Hi @ohsawa, yes the Bayes method looks at the metric’s value in the summary dictionary. Are you seeing an error when using the define metric workflow with a sweep? This should work as expected as well but it may be looking at the final value only. Checking with the team now to confirm the behavior of this.
No errors, thank you.
I would like to know if the Bayes method correctly refers to the value even if it is registered in the Summary dictionary by executing define_metrics() with the key “val_loss.min” (not “val_loss”).
(Or does the fact that there are no errors mean that it is working correctly?)
After learning, I am currently running a process to re-register the keys of the best metric values recorded in the Summary dictionary by define_metrics() from “val_loss.min” to “val_loss”. This is because I am concerned that the Bayes method may not function properly with the key “val_loss.min”.
If the Bayes method functions properly when used with define_metrics(‘val_loss’, summary=‘min’), then this processing is unnecessary.
If you can confirm this behavior, I would appreciate it if you could let me know.
Thank you very much.
Hi @ohsawa as long as the metric you are tracking is logged in your experiment then this shouldn’t be a problem with sweeps. Example if you are tracking test.loss in your sweep config, then the metric test.loss must be logged as part of the experiment wandb.log({"test.loss": <val>"}). Regards