W&B Sweeps w/ Self-Supervised Learning

I’m reaching out to get some thoughts on integrating W&B Sweeps with some of the code we’re interested in using. An example of the code we’re running is linked here. Note the 2 key sections ‘Training Self-Supervised Learning Model’ and ‘Fine Tuning Model’ which contain the !python commands we’re interested in tuning (model, technique, learning rate, etc.)

Based on this documentation, I’ve set up sweep_config but I’m unsure how to incorporate the 2 !python commands in train() when running an agent. Do you have any input on how to integrate a wandb sweep with these 2 commands?

An additional point I wanted to discuss was the strategy for a Sweep. The SSL code we’re running requires training 2 sequential models (the SSL and the final classification Model) where the output SSL model is the input to the final classification model (see the linked code above). We’re interested in doing hyperparameter tuning for both of the models - should we set up 2 independent sweeps for each? Or should we run a sweep on the first SSL model, pick the best performing model and use that as the input into the second classification model where we run a second sweep?

Hi @anmolseth, thank you for writing in!

Have you looked into using a custom command in your sweep config? Here is some information on this. You can pass any arguments for the sweep in via the command line so that your train.py can pick them up there instead of through wandb.config

As for the second part of your question, I can’t speak to which method would give better accuracy but my intuition would tell me that optimizing your input model first, and then using that model in the sweep for the second model would give a better result. Sweeping over both independently may lead the second model to expect input data to look differently than it will look from an optimized version of the first model.

Let me know if you have any further questions on either of these and I’d be happy help out.

Thank you,
Nate

1 Like

Thanks @nathank! The custom command functionality seems to be well-suited for my use case. I wanted to follow up on how to set up SSL’s train.py to be compatible with command line Sweeps. The SSL train.py cli_main() method includes ArgumentParser() to pull out the arguments from command line but this parser doesn’t seem to be present in any of the W&B examples online.

Do you know if this parser is needed to be compatible with W&B command line sweeps? Additionally, are there are any other changes I should be making to the SSL code to get prepped for my first Sweep?

Thank you,
Anmol

Hi @anmolseth,
What parameters would you like to sweep over? It looks like the ArgumentParser() is already setup to take in a number of arguments. If you are sweeping over parameters that ArgumentParser() already handles then it will simply grab them automatically from the sweep if you use the same name for them.

If not you will need to add arguments to the parser for any additions parameters you would like to sweep over and make sure your model is using the arguments passed in via the ArgumentParser()

Other than that, you should be ready to start your sweep. Let me know if you have any issues and I’d be happy to help!

Thank you,
Nate

Hi @anmolseth,
I wanted to follow up and see if you would like any additional help with this or if you were able to get your sweep running?

Thank you,
Nate