Log stats with different global steps

[parisi]

wandb.log(train_stats, step=train_step)
wandb.log(test_stats, step=test_step)

In my code, training and testing happen in parallel. I get training stats at every step, but testing stats are available only once in a while (I don’t pause training to wait for testing results). For instance, I may get testing stats at steps [10, 12, 20, 33].
The problem with this is that above commands doesn’t work if test_step and train_step are not the same. Only the first log succeeds, and the second is not logged at all (no error, the program keeps running). If I pass the same step, e.g., wandb.log(test_stats, step=train_step) then it works.

Is it possible to log stats with different steps?

[mohammadbakir]

Hi @parisi , happy to help. An important note about wandb step is it must always monotonically increase and it increases with each wandb.log call. There are several way to specify which step to log at, including custom logging axes, or you could through conditional logic

wandb.log({"train_acc": acc}) #Logs every training step
if epoch%5 ==0:
          wandb.log({"test_acc": acc}) #Logs  every 5 training steps

See this workspace for a few toy examples of difference approach, code is included for each run.

[parisi]

Thanks, I am using define_metric and it works!
Just one more question. What I am doing is

wandb.define_metric('Steps')
wandb.define_metric("*", step_metric="Steps")
wandb.log({**train_stats, 'Steps': train_step})
wandb.log({**test_stats, 'Steps': test_step})

However, wandb creates one plot for ‘Steps’ as well. Can I avoid that? I tried with define_metric('Steps', hidden=True, step_sync=False) but the plot is still there. It is just an useless plot and I would like to get rid of it.

Thanks!

[mohammadbakir]

Hi @parisi ,

You’re welcome! I’m glad to hear that define_metric is working well for your use case. Unfortunately, there isn’t a direct method to hide the steps charts in the SDK. However, you can drag the steps chart to the hidden panel section of your main/run workspace. By doing so, any subsequent runs logged will automatically have the steps chart hidden from view. Marking initial inquiry resolved on our end, but please do reach out again anytime we could be of help.