I’m running a training program on a server and using wandb to upload logs. My lab’s network was down for about half a day, during which my program continued running on the server without any errors. After the network was restored, I found that the status of my run on the wandb website showed ‘crashed’, even though my training was still continuing, Wandb didn’t automatically upload the log data when network is fixed.
After the training completed, I tried to synchronize the data using commands like ‘wandb sync run-’ / 'wandb sync run-.wandb’ / ‘wandb sync --clean run-*’, but they all resulted in errors:
Either ‘AssertionError: invalid padding’ or ‘AssertionError: invalid checksum’
I also tried synchronizing using the Python API, which ran without any error output, but nothing appeared on the website.
I tried opening the .wandb file and it does contain content, so why can’t it be uploaded? If there’s an issue with my local files, it’s strange because nothing was done except for the unexpected network disconnection.
My request:
I’d like to either upload the training data to the website or be able to plot the graphs locally. Thanks!