Three points of improvement for Sweeps

W&B Help
1

Hi all,

I’ve started using Sweeps, and after having wrestled with it for some time, I see three points of improvement.

  1. A “repeat” parameter. It is pretty standard practice to run a particular hyperparameter choice for a couple different seeds. To do this now, we have to add a few different seeds to the optimized hyperparameters, and the results are not aggregated across different seeds.
  2. More generally, it would be great if there was a way to create multiple runs for one agent’s function call (i.e. one hyperparameter choice) . I am working in a decentralized setting with multiple clients training a collective model, and I’d like to track each client with a separate run. There is currently no way to create multiple runs for the same hyperparameters.
  3. Better handling of nested hyperparameters in the config. Currently, once-nesting is supported, but with an annoying caveat: all hp’s in the nested dictionary are frozen have to be specified in the sweep configuration, as the nested dictionary is frozen by the sweep. The way I do this now is by setting all parameters to a list with one value, because setting a constant throws an incorrect sweep configuration error (is this intentional?):
    sweep_configuration = {
        "entity": "username",
        "project": "projectname",
        "name": name,
        "method": "grid", 
        "metric": {"goal": "maximize", "name": "score"},
        "parameters": {
            "seed": {
                "values": generate_seeds(repeats)
            },
            "optim" : {
                "parameters" : {
                    "base_lr": { "values": [0.0001, 0.001, 0.01] },
                    "warm_up": {"values": [5] }
                }
            }
        }
    }

This is cumbersome with large configs, and it muddles the Sweep visualizations.

Some of these discussions have floated on the forum before, but seem to have been stagnant for a while.

Let me know what you think!