I am running wandb in offline mode since I don’t have an internet connection on the compute nodes that I use for my experiments.
This works fine when I’m logging training loss and other things.
When I try to log images, however, I get the following warning wandb: Network error (ConnectTimeout), entering retry loop. and the run waits forever.
The logging happens through:
images = []
for i in range(10):
images.append(wandb.Image(image[i], caption=f"{caption}.{i}"))
wandb.log({category: images})
I can even see that the offline mode is active since I get the following output when I stop the run:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /scratch_emmy/outputs/wandb/offline-run-20230208_142336-9685bcf5ea8d5d35ccc9d93b2d035832
Hi @arnenix , happy to look into this for you. I ran a test on my end and did not run into the same issue you face. I was successful in logging images offline then syncing the run to wandb using the same code example you provided.
Which version of wandb are you using? If not our latest, upgrade and try again. If the problem persists, provide me a copy of your debug.log and debug-internal.log files of the runs that are failing. These are located in the run/logs folder under the wandb folder of your working directory. Thanks
Hi @arnenix , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!