HP Sweep: Conditional Sampling

W&B Help
1

Suppose I have a somewhat complicated hyper parameter distribution I’d like to sample:

For example, I have a hyperparameter called HP1 controlling normalization applied my dataset. If I sample HP1 ← maximum-eigenvalue-norm, then maybe I have another hyperparameter I must sample; in this case that could be how to compute maximum eigenvalue which could be in the set {fancy-eigenvalue-computation, torch-built-in-symeig}.

But suppose if normalization technique was sampled as HP1 ← Frobenius-norm, then I have no other hyper parameters to sample.

Optuna handles this nicely, and I was hoping W&B’s had a similar way of auto-magically handling it.

2

Hi @max_wasserman , thanks for your request and providing details for your use-case as well. We’ve a feature request filed for this and I’ve bumped up the priority for it too. We’ll keep you posted once there’s an update.

Current workaround: One way to address this problem for now is to pre-process the sweep.yaml and provide plausible combinations of pairs of (HP1, HP2) that satisfy your condition (which you can do because of your prior knowledge).

3

Hi @max_wasserman , I just wanted to let you know that conditional sweeps is possible using Launch, check out this colab.