When is one supposed to run wandb.watch so that weights and biases tracks params and gradients?

I run wandb.watch before my training script starts but that doesn’t seem to track the histograms of weights and gradients. The script I have is nothing too complicated - just generating random data and fitting it after applying a quadratic:

code in github: ultimate-utils/my_wandb_basic1.py at master · brando90/ultimate-utils · GitHub

sample run: Weights & Biases

1 Like

Thanks for sharing!
Let me dig into the code and get back with an answer :slight_smile:

There are two things you might be running into here – can’t confirm because your code relies on the ultimate-utils package.

  1. wandb.watch will only start working once you call wandb.log after a backwards pass that touches the watched Module (docs).
  2. The frequency with which gradients/params are logged is controlled by the log_freq argument. If the number of logging calls is less than the value of log_freq, then no information will be logged. Here’s a short colab reproducing this behavior.

Also, if you want params and gradients, you need to set the log kwarg to "all". By default, we log only gradients.

1 Like