Hi - I have the same problem as first filed here. Training plots (but not the eval and system plots) disappeared when the run is finished. I have the same issue with all my recent runs. I had no issue with the plots during the runs. I’d like to share these plots publicly so I appreciate someone can take a look at this and resolve it. Thanks.
Hi @onuralp! Apologies you are seeing this behavior on your side. Could you please send me a link to your workspace so I can take a peek at it? What type of graphs are you expecting to see there, that are not showing up in the workspace for you?
Thank you @artsiom for looking into this. Here is one example run. I can see the eval plots but training plots (e.g., train loss) show only a single data point instead of the entire run.
Thank you for sending over your workspace, when you say the train plots have dissapeared, do you mean you used to see them before, and now there is only a single point you see of the full data?
I have went and looked into your run and it does look like train data never got captured through the experiment in the first place.
Yes, that’s what I mean.
I used axolotl to register my training run and monitored the train and eval loss on wandb dashboard during each experiment without any issue. Eval plots look fine but you think that train loss data was captured only during the run and not captured somewhere for post-run inspection?
That’s too bad. Any suggestion what might be the problem and how I can make sure that training progress is also captured?
Thank you.
Could you please provide your training script? This is a strange behavior that where one metric gets recorded and the other one doesn’t. I’ll take a peek at it and see if there is anything obvious that could have caused this behavior.
Hi, wanted to follow up with you regarding this script and see if you could provide a training script for debugging.
Hi, since we have not heard back from you, we are going to close this request. If you would like to reopen the conversation, please let us know! Unfortunately, at the moment, we do not receive notifications if a thread reopens on Discourse. So, please feel free to create a new ticket regarding your concern if you’d like to continue the conversation.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.