Combining grid search and bayesian optimization

W&B Help
1

Hello,
I would like to perform a sweep of models with different hyperparameters. However, some (say the number of layers, 3, 4 and 5) I would like to like to vary in a grid search and for each of those perform a bayesian optimization of a different subset of parameters (say the learning rate, optimizer settings etc.). Is there a clever way to set this up in such a way that I can use the analysis provided by the dashboard?
Thank you so much for your help!

2

Hi @hannes-stagge

Thank you for reaching out for support.

Weights & Biases sweeps allow you to define a search strategy for hyperparameter optimization. However, currently, you can only specify one search strategy per sweep. This means you can’t perform a grid search on certain hyperparameters and a Bayesian optimization on a different subset of parameters within the same sweep.
Here’s an example of how to set up a sweep with a grid search strategy:

python
sweep_config = {
    "method": "grid", # grid search
    "metric": {
        "name": "accuracy",
        "goal": "maximize"
    },
    "parameters": {
        "num_layers": {
            "values": [1, 2, 3, 4]
        },
        "optimizer": {
            "values": ["adam", "sgd"]
        }
    }
}

sweep_id = wandb.sweep(sweep_config)
wandb.agent(sweep_id, function=train)

And here’s an example of how to set up a sweep with a Bayesian search strategy:

python
sweep_config = {
    "method": "bayes", # bayesian optimization
    "metric": {
        "name": "accuracy",
        "goal": "maximize"
    },
    "parameters": {
        "learning_rate": {
            "min": 0.001,
            "max": 0.1
        },
        "batch_size": {
            "min": 32,
            "max": 256
        }
    }
}

sweep_id = wandb.sweep(sweep_config)
wandb.agent(sweep_id, function=train)

You can run these sweeps separately and compare the results on the Weights & Biases dashboard.
Sources:

Regards,
Carlo Argel

3

Hi @hannes-stagge ,

Reaching back from the support team, just want to follow up if you still need more assistance.

Regards,
Carlo Argel

4

Thank you for clarification. Unfortunate that it does not work. Guess I will have to start multiple sweeps one after the other by varying the parameters by hand.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.