Sweep random search method behavior when the hyperparameter space is fixed

W&B Help
1

how would the random search method behave if the sweep config has 2 hyper-parameters H1 and H2, for example, and H1 has 2 values and H2 has 6 values, not a distribution but just an array of fixed values each! (2*6 = 12 total combinations)

  1. would the controller make agents repeat some runs? (a.k.a giving them the same hyperparamter combination for H1 and H2)
  2. if agents won’t repeat the run, they always get unique combination for H1 and H2, is providing count for the agent still important?
  3. is providing the run_cap in the sweep config enough to stop the random search if count is not give to the agent?
2

Hey @osama2o1ooo , thanks for asking these questions – answering them in the same order that you ask them below:

  1. Yes the random sweeps method can repeat combinations.
  2. Providing count is relevant as you can exceed the number of parameters in search space – eg in your example 12
  3. The run cap is sufficient to limit the number of runs an agent can conduct. For example, in this config, there would only be one run:
sweep_config = {
    'method': 'random',
    'run_cap': 1,
    'metric': {
      'name': 'accuracy',
      'goal': 'maximize'   
    },
    'parameters': {
        'H1': {
            'values': [1, 2]
        },
        'H2': {
            'values': [0.1, 0.2]
        }
    }
}

However this config:

sweep_config = {
    'method': 'random',
    'run_cap': 20,
    'metric': {
      'name': 'accuracy',
      'goal': 'maximize'   
    },
    'parameters': {
        'H1': {
            'values': [1, 2]
        },
        'H2': {
            'values': [0.1, 0.2]
        }
    }
}

would result in 20 runs (if not limited by count being passed to agent with wandb.agent(sweep_id, function=train, count=number_of_runs) ) even though there are only 4 possible permutations.

I hope that this answers your questions and please let us know if you need anything else here.

3

Thank you very much for the clarification and giving examples.
I actually started experimenting and I got answers for the 1st and 2nd questions and I was going to post what I found here, but this reply answers the 3rd question as well, I couldn’t explains what was run_cap for! Maybe the documentation should be updated that it is per agent if count is not provided in a random search.

a couple of questions out of curiosity:

  1. if both count and run_cap are given which one would the agent follow?
  2. is there a way to change the order grid search follows?
4

Hey @osama2o1ooo ,

Thanks for messaging back –

Regarding which takes precedence – the ‘run_cap’ is global across all agents and a maximum number of runs in a sweep is defined here in our documentation.

It is possible to have multiple agents for the same sweep (each will try to run the number of runs in the count argument) unless the total number of runs equals the ‘run_cap’ in which case if an agent reaches the maximum number of runs set in ‘run_cap’ it will exit.

Regarding the order of parameters at present it is not possible to configure the order of parameters in grid search. would you mind sharing your use case I’d be happy to log this as feedback for our product team.

Hope this makes sense and look forward to hearing back

5

Hey @osama2o1ooo , please let us know if you need anything else here and if not would you mind if close this off for you?

6

1st. Ah! So it can be said that the smaller between the 2 will be used to stop the agents.
for example all agents will run once and then exit in the following 2 cases:

  • agent count = 20, sweep run_cap = 1
  • agent count = 1, sweep run_cap = 20

2nd. from the docs:

run_cap Specify a maximum number of runs in a sweep.

that doesn’t really clarify that it is per agent! I honestly understood that it is the max all agents combined can run, and not for each agent!

3rd. regarding the order of parameters, I just thought of it like the magic wand button that shows the most useful columns in runs table or hyperparameters in a sweep, we don’t know which a hyper-parameter has the biggest effect, and in gird search one usually wants to try every combination but it’s also good to see early on how different combinations can impact the outcome of the experiment.

Thanks once more :smiley: