Hi,
I’m not sure whether this is related to a W&B connection issue, but since yesterday my runs have stopped updating metrics correctly. For example in this run Weights & Biases, global_step stops at around 1.5M steps, even though the training actually runs until 75M steps, as confirmed by the logs.
The code has not changed. A few days ago, the same setup worked as expected, for example: Weights & Biases . Since yesterday, it consistently stops updating after around 1.5M steps. The only thing I did was updating wandb from 0.22.x to 0.24.0
I’ve looked through related issues but haven’t found a solution that applies to this case. My wandb version is 0.24.0.
Thanks in advance for any help.
Update: This is fixed after downgrading to wandb 0.22.1 (and use python 3.11)