Resumed run plotting broken

W&B Help
1

Hi, I’ve resumed a training run that I closed, it is running fine but the plotting seems to be completely broken. Before resuming it was plotting like expected, however afterwards it only shows a random selection of points from the plot, with this changing every 30 seconds or so. Wondering if there is an easy fix?

The screenshot shows a correct run (dark brown) next to a one of the random configurations of the broken one (light brown)

image
image3256×1720 190 KB

2

Hi @reeceomahoney Good day and thank you for reaching out to us. Happy to help you on this!

To further investigate this issue with your plots after resuming your runs, I would like to request for the following information:

  • You mentioned that you resumed a training run that is closed, do you recall or know what is the status of your run before you tried to resume it?
  • Would you be able to provide the workspace URL so we can see your plots?
  • SDK version, you can get this by running (wandb --version)
  • A code snippet/script of how you are trying to resume your run. This is for us to see if there something on the execution that would result to your issue.

Thank you!

3

I learnt some more salient information after making this post. Firstly, this error was fixed when I zoomed into any selected area of the plot, no matter how large. Secondly, it sorted itself out when I checked back the next day. Given this do you still need the above information?

4

Hi @reeceomahoney our team identified an issue that was causing this chart behavior and it was fixed. We can now mark this as solved, but please reach out to us if you encounter any issues. Thank you!