Getting ConnectTimeout in offline mode when trying to log an image

I am running wandb in offline mode since I don’t have an internet connection on the compute nodes that I use for my experiments.
This works fine when I’m logging training loss and other things.
When I try to log images, however, I get the following warning wandb: Network error (ConnectTimeout), entering retry loop. and the run waits forever.

The logging happens through:

images = []
for i in range(10):
    images.append(wandb.Image(image[i], caption=f"{caption}.{i}"))
wandb.log({category: images})

I can even see that the offline mode is active since I get the following output when I stop the run:

wandb: You can sync this run to the cloud by running:
wandb: wandb sync /scratch_emmy/outputs/wandb/offline-run-20230208_142336-9685bcf5ea8d5d35ccc9d93b2d035832

Hi @arnenix , happy to look into this for you. I ran a test on my end and did not run into the same issue you face. I was successful in logging images offline then syncing the run to wandb using the same code example you provided.

Which version of wandb are you using? If not our latest, upgrade and try again. If the problem persists, provide me a copy of your debug.log and debug-internal.log files of the runs that are failing. These are located in the run/logs folder under the wandb folder of your working directory. Thanks

Hi @arnenix , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

Hi @mohammadbakir ,

sorry for not responding earlier.
I also opened an issue on github and they were able to reproduce my issue and are working on a fix.
For further info see: [CLI]: Getting ConnectTimeout in offline mode when trying to log an image · Issue #4946 · wandb/wandb · GitHub

I think this thread can be closed.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.