I am running wandb in offline mode since I don’t have an internet connection on the compute nodes that I use for my experiments.
This works fine when I’m logging training loss and other things.
When I try to log images, however, I get the following warning wandb: Network error (ConnectTimeout), entering retry loop. and the run waits forever.
The logging happens through:
images = 
for i in range(10):
I can even see that the offline mode is active since I get the following output when I stop the run:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /scratch_emmy/outputs/wandb/offline-run-20230208_142336-9685bcf5ea8d5d35ccc9d93b2d035832
Which version of wandb are you using? If not our latest, upgrade and try again. If the problem persists, provide me a copy of your debug.log and debug-internal.log files of the runs that are failing. These are located in the run/logs folder under the wandb folder of your working directory. Thanks