Hi,
Is there a frequency limit for wandb.log? I was using wandb to track my code status. I run my python code (in which wandb.log is called maybe 2 times per Sec ) via a .sh file. My code was supposed to run 10times, however I saw only 3 or 4 logs, both on wandb dashboard and my local log. The last update was 1:00 this morning, but it’s jammed till 10:00 this morning, so I manually stopped all my jammed tasks.
In my case, I started 12 master shell scripts at the same time, and each master shell scripts is supposed to run a python script 20 times one by one, each run of each python file is associated to one wandb run (although in the same wandb project). As is mentioned above, each run of each python file calls wandb.log() 2 times per seconds, so in total, the frequency should be 12*2=24 times per Sec.
I could see plenty of python -m wandb -service ..... commands running, but the process of my own code has already stpped. I 'm wondering that, is the 24 times per Sec logging frequency jammed wandb process thus jammed my code? Or is there a limitation on logging frequency?
Best regards, thanks alot 