I have been using wandb for a while now, but since last week the training runs on W&B only show the validation losses and not the training losses. If I open tensorboard from the W&B run page, I also can only see the validation stats, but if I open tensoboard from terminal independently from W&B the both training and validation stats are shown correctly.
I am not sure of what’s going on, as I haven’t changed my workflow at all, and since I don’t get any error message I am not sure of how to debug the problem.
If it helps, the folder structure of my project looks like this:
-main_folder -training_notebook -output (folder) -run_name_folder -events.out.tfevents.1673259721.ip-..... (tensorboard file for training) -validation (folder) -events.out.tfevents.1673259726.ip-.... (tensorboard file for validation)
and I run the training with the following code:
with wandb.init(name=run_name, project=project_name, id=run_id, job_type="training_job", sync_tensorboard=True) as run:
Any idea of why only the validation losses are logged on W&B?