Tensorboard not logged correctly on wandb run

[fgraffitti]

Hello,
I have been using wandb for a while now, but since last week the training runs on W&B only show the validation losses and not the training losses. If I open tensorboard from the W&B run page, I also can only see the validation stats, but if I open tensoboard from terminal independently from W&B the both training and validation stats are shown correctly.
I am not sure of what’s going on, as I haven’t changed my workflow at all, and since I don’t get any error message I am not sure of how to debug the problem.
If it helps, the folder structure of my project looks like this:

-main_folder
    -training_notebook
    -output     (folder)
        -run_name_folder
        -events.out.tfevents.1673259721.ip-.....   (tensorboard file for training)
        -validation     (folder)
            -events.out.tfevents.1673259726.ip-.... (tensorboard file for validation)

and I run the training with the following code:

with wandb.init(name=run_name,
                project=project_name, 
                id=run_id,
                job_type="training_job",
                sync_tensorboard=True) as run:

Any idea of why only the validation losses are logged on W&B?

[lesliewandb]

Hi Francesco, thank you for writing in! Can you tell me if you had done anything different between when it was showing the training losses where not appearing. For example, have you upgraded or changed your environment where you run your script?

[lesliewandb]

Hi Francesco, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!