Run best model off sweep?

[cyrilw]

Hi,
I am using Sweeps to run through different configuration models and I was told by the wandb chat support that to run the best model configuration off sweeps is to create a new sweep with the best performing parameter set and running off it.

But this is lot of tedious work, is there any other elegant way of quering wandb project for the best model configuration and running off it?

tldr: I run a sweep with different configuration, would like to run predictions off a specific set of parameters (or best performing set of parameters). How to do it with the sweep API?

[_scott]

Hi @cyrilw

Thanks for persisting with this and posting it here, here is how you do it with the Api.

import wandb

api = wandb.Api()
sweep = api.sweep(f"_scott/project-name/sweeps/qwbwbwbz")

# Get best run parameters
best_run = sweep.best_run(order='validation/accuracy')
best_parameters = best_run.config
print(best_parameters)

Hope this helps

[cyrilw]

Thank you for the reply.

in this,
best_run = sweep.best_run(order='validation/accuracy')

do I need to change the validation/accuracy based on my sweep columns?

[_scott]

Yes, sorry I should have said that. order is the metric you want it to order by.

If you have set a goal in your Sweep config, it’ll use that if order is not given.

Here’s the source if you’re curious: https://github.com/wandb/client/blob/a339333b3ee93864daf416f04c1501186dffac5c/wandb/apis/public.py#L2137

[cyrilw]

I see. Just to confirm, my current sweep configuration is;

method: random
metric:
  goal: minimize
  name: KL
parameters:
  K:
    distribution: int_uniform
    max: 15
    min: 3

since I do have the goal/metric set I don’t need to worry about order, right? but if not I do collect KL (column name is KL) score (kl-divergence) so I’d modify the code snipet as;

best_run = sweep.best_run(order='KL')

But in case of KL, the lower is better any recommendation on how to select best_run based on it?

[_scott]

Because your configuration has a goal, you can leave order out.
So you can just do:

best_run = sweep.best_run()