When is one supposed to run wandb.watch so that weights and biases tracks params and gradients?

[brando]

I run wandb.watch before my training script starts but that doesn’t seem to track the histograms of weights and gradients. The script I have is nothing too complicated - just generating random data and fitting it after applying a quadratic:

code in github: ultimate-utils/my_wandb_basic1.py at master · brando90/ultimate-utils · GitHub

sample run: Weights & Biases

[bhutanisanyam1]

Thanks for sharing!
Let me dig into the code and get back with an answer

[charlesfrye]

There are two things you might be running into here – can’t confirm because your code relies on the ultimate-utils package.

1. wandb.watch will only start working once you call wandb.log after a backwards pass that touches the watched Module (docs).
2. The frequency with which gradients/params are logged is controlled by the log_freq argument. If the number of logging calls is less than the value of log_freq, then no information will be logged. Here’s a short colab reproducing this behavior.

Also, if you want params and gradients, you need to set the log kwarg to "all". By default, we log only gradients.