Hi there! I’m looking to run a hyerparameter sweep across a wide range of resnet configurations. I would like to explore two relevant paramters - the number of resnet blocks, and the number of filters in each resnet block. I would like the option for both of these hyperparameters to be large, but not at the same time! As this model will just take too long to train, and so my sweep time will be dominated by just one or two modules. So in the 2D domain of n_resnet_blocks and n_conv_filters, I want to explore some subdomain of this. Is there some way this could be done within the sweep configuration? An obvious approach would be just to run consecutive sweeps, capping each dimension to some sensible value while exploring the other one, but it’d just be neater if the space could be explored within a single sweep. So interested to hear any ideas!
Hi @bill-morrisson - apologies for the late reply, thanks for getting back to me. I guess I’m describing some additional criterion. Imagine we have two hyperparameters, a and b, which we both want to vary between [0, 1]. I am looking to search this space, but exclude runs where both a>0.5 and b>0.5. I guess I could add a break clause within the train() function to skip such values, but wondering if this could be done within the sweep configuration itself.
Hi @chris-pedersen , apologies for the delay here. What you are describing above seems to be conditional sweeps or having conditional sweep parameters. Would that be correct? For instance, condition paramA on paramB and in this case, conditions: a < 0.5 with b < 0.5, a < 0.5 with b > 0.5, a > 0.5 with b < 0.5.
Also, note that this should become possible with the support for external sweeps controllers, you’ll be able to use Optuna / Hyperopt to do conditional searching to start. We are going to release this feature soon this quarter and i’ll surely keep you posted regarding the same.