How to build a sweeps model for different numbers of hidden layers?

I am making a simple network in PyTorch with linear units as a practise project. I’d like to use sweeps to find the best hyper parameters for the network. Some of these hyperparameters include batch_norm, dropout value, number of hidden layers, number of units in each hidden layer.

I can’t figure out how to set up the model and sweep config so that two different model structures can be swept without being confusing. For example, I want to use batch_norm OR have dropout values of [0, 0.2, 0.4, 0.5]. I never want batch_norm AND dropout to be used. If I use random search with wandb, it may choose both 0.4 dropout AND batch_norm which I don’t want.

I know how to set up the network class with simple if statements so it adds either batch_norm or dropout, but the wandb.config would still select a value for dropout and a boolean for batch_norm, and I don’t want the sweep report to show both these parameters if the network only uses one.

Another example is, I’d like 2, 3, 4 or 5 hidden layers. I’d also like each layer to have a randomly selected number of neurons from the range [64, 128, 256, 512].

I can forsee a problem where wandb will select the model to have 3 hidden layers but also pick say, 256 neurons for the 4th or 5th layer which will be misleading on the sweep parameter graph.

Hey Gary, at the moment this is not possible unfortunately. But we are planning to add conditional sweeps in the near future which will address your use-case.