Nested Sweep Configuration

W&B Help
,
1

I have been attempting to use Hydra configuration with a wandb sweeps. I have nested hierarchy of configuration. When defining the parameters I want to sweep over I would like to something like the following:

optimizer:
    parameters:
        learning_rate:
            values: [0.01, 0.001]
        momentum:
            value: 0.9

Source: Sweep Configuration
When I attempt this, I get a CommError with a message that sweep config must have a parameters section . I do not get this error if parameters is at the top level of the hierarchy, but then the configs are not read in properly. If I were to modify the above example to the following (only swapped parameters with optimizer):

parameters:
    optimizer:
        learning_rate:
            values: [0.01, 0.001]
        momentum:
            value: 0.9

I would get a CommError with message Invalid sweep config: invalid hyperparameter configuration: optimizer. I would like to keep the hierarchical structure of the YAML base config but add parameters which will indicate which parameters to sweep over. Any recommendation? or resources? A code example using a nested sweep.yaml would be ideal.

2

Hi @mjvolk3,

This is currently not available with our sweep configurations. We have a more customizable nested sweep config as you describe planned as a feature for the future, I can go ahead and increase the priority on it for you.

1 Like
3

It looks like this is supported. It works if you put parameters: between every level. With the previous example it looks like the following:

parameters:
    optimizer:
        parameters:
            learning_rate:
                values: [0.01, 0.001]
            momentum:
                value: 0.9

The original source I referenced (wandb docs) must be indicating that the block below works if it is put under parameters:

optimizer:
    parameters:
        learning_rate:
            values: [0.01, 0.001]
        momentum:
            value: 0.9

I have tested this pattern with deeper hierarchies and it works.

2 Likes
4

Hey @mjvolk3,

You are right - I confirmed and this has been implemented recently, just skipped my radar. Is there anything else I can assist with here?

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.