wandb.log(train_stats, step=train_step)
wandb.log(test_stats, step=test_step)
In my code, training and testing happen in parallel. I get training stats at every step, but testing stats are available only once in a while (I don’t pause training to wait for testing results). For instance, I may get testing stats at steps [10, 12, 20, 33].
The problem with this is that above commands doesn’t work if test_step
and train_step
are not the same. Only the first log
succeeds, and the second is not logged at all (no error, the program keeps running). If I pass the same step, e.g., wandb.log(test_stats, step=train_step)
then it works.
Is it possible to log stats with different steps?
Hi @parisi , happy to help. An important note about wandb step is it must always monotonically increase and it increases with each wandb.log call. There are several way to specify which step to log at, including custom logging axes, or you could through conditional logic
wandb.log({"train_acc": acc}) #Logs every training step
if epoch%5 ==0:
wandb.log({"test_acc": acc}) #Logs every 5 training steps
See this workspace for a few toy examples of difference approach, code is included for each run.
Thanks, I am using define_metric
and it works!
Just one more question. What I am doing is
wandb.define_metric('Steps')
wandb.define_metric("*", step_metric="Steps")
wandb.log({**train_stats, 'Steps': train_step})
wandb.log({**test_stats, 'Steps': test_step})
However, wandb creates one plot for ‘Steps’ as well. Can I avoid that? I tried with define_metric('Steps', hidden=True, step_sync=False)
but the plot is still there. It is just an useless plot and I would like to get rid of it.
Thanks!
Hi @parisi ,
You’re welcome! I’m glad to hear that define_metric
is working well for your use case. Unfortunately, there isn’t a direct method to hide the steps charts
in the SDK. However, you can drag the steps chart to the hidden panel section of your main/run workspace. By doing so, any subsequent runs logged will automatically have the steps chart hidden from view. Marking initial inquiry resolved on our end, but please do reach out again anytime we could be of help.