Track train script version along with hyperaparams (ideally automated)

[darek]

I’m just getting started with experiments, my goal is to ablation-study some potentially new methods for finetuning NLP models. I use HuggingFace trainer and the W&B integration works flawlessly, so the hyperparams are super easy to track. I’m going to be doing some experimentation within the training code though - is there an easy way to track the training script version/content along with the experiments, ideally integrated with Github? Thanks! Darek

[_scott]

Hi Darek,
They sound like interesting experiments, we’re happy you’ve chosen W&B to track them!

You can track your code by turning on code saving in your settings: https://docs.wandb.ai/ref/app/features/panels/code

You can also call wandb.run.log_code(".") after calling wandb.init() to capture all python source code files in the current directory and all subdirectories as an artifact.

Soon you should be able to use wandb Sweeps as a backend of the hyperparameter_search call in HuggingFace if this PR gets merged: Add W&B backend for hyperparameter sweep by AyushExel · Pull Request #14582 · huggingface/transformers · GitHub

Code Saving

[darek]

Thanks Scott, this is exactly what I needed! Also looking forward to the HF sweep PR