Average multiple runs over one parameter

[parisi]

I am doing a sweep over multiple parameters. For every configuration, I also have a seed to control randomness.
For instance, let’s say I am sweeping over learning rate in [0.1, 0.01] and batch size in [16, 32]. This will give me 4 runs, and for each I will have two runs with seed 1 and 2, treated as hyperparameter, for a total of 8 runs.
I’d like to average ALL plots over the 2 seeds. So, instead of having 8 curves per plot I’ll have only 4.
I can do this manually by grouping over learning rate and batch size. However, in practice, I have way more hyperparameters than 2, and I don’t want to manually add all of them one by one to the group.

Is there a way to, basically, “group over all but one” hyperparameter?

[raphael-sanandres]

Hello @parisi !

Thank you for bringing this up! We do not support this currently but I can make a feature request for you for our internal team to look at.

[parisi]

That would be great, thanks!

[jlperla]

Yes, I do this as well and more tooling would help.

More broadly, when doing a HPO it would be great to be able to tell it to run a set number of seeds at each hyperparameter value and then use the median (or whatever) as the HPO objective value.