# Strange global\_step restarts affecting learning rates and performance? **URL:** \ **Category:** W&B Help\ **Tags:** wandb\ **Created:** [November 6, 2023, 4:57pm UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349 "2023-11-06T16:57:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![versae](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/versae/32/1518_2.png) [@versae](https://community.wandb.ai/u/versae)\ **Post date:** [November 6, 2023, 4:57pm UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/1 "2023-11-06T16:57:28Z")
I’m seeing the weirdest `global_step` restarts during my training runs. And it does affect the optimizer as it restarts the warmup every time global\_step goes to zero. ![image](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/2X/3/3a47aeaa9c6fb435667083102ac3ae09f5cdd669.png) This is in Jax using a joint scheduler from Optax to create the learning rate function. I log separately the step number when doing evals and it the counter seems fine, but for some reason the logged values in wandb are pretty strange. ---
**Author:** ![carlo-catimbang](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlo-catimbang](https://community.wandb.ai/u/carlo-catimbang)\ **Post date:** [November 8, 2023, 10:23am UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/2 "2023-11-08T10:23:39Z")
hi @versae Thank you for reaching out for support. I’ll check this on our end and we’ll get back to you for updates. ---
**Author:** ![carlo-catimbang](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlo-catimbang](https://community.wandb.ai/u/carlo-catimbang)\ **Post date:** [November 13, 2023, 10:01am UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/3 "2023-11-13T10:01:22Z")
Hi @versae, The issue you’re experiencing might be due to multiple calls to `wandb.log` for the same training step. The wandb SDK has its own internal step counter that is incremented every time a `wandb.log` call is made. This means that there is a possibility that the wandb log counter is not aligned with the training step in your training loop. To avoid this, you can specifically define your x-axis step using `wandb.define_metric`. You only need to do this once, after `wandb.init` is called. Here is an example: ```auto wandb.init(...) wandb.define_metric("*", step_metric="global_step") ``` The glob pattern, “\*”, means that every metric will use “global\_step” as the x-axis in your charts. If you only want certain metrics to be logged against “global\_step”, you can specify them instead: ```auto wandb.define_metric("train/loss", step_metric="global_step") ``` This should help align your `global_step` with the internal wandb step counter and prevent it from restarting to zero. ---
**Author:** ![carlo-catimbang](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlo-catimbang](https://community.wandb.ai/u/carlo-catimbang)\ **Post date:** [November 16, 2023, 4:09am UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/4 "2023-11-16T04:09:26Z")
Hi @versae , I just want to follow up if this helps and you still need assistance. Regards, Carlo Argel ---
**Author:** ![versae](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/versae/32/1518_2.png) [@versae](https://community.wandb.ai/u/versae)\ **Post date:** [November 17, 2023, 3:22pm UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/5 "2023-11-17T15:22:21Z")
Hi! We’re still investigating. Thanks for your help! Cheers. ---
**Author:** ![carlo-catimbang](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@carlo-catimbang](https://community.wandb.ai/u/carlo-catimbang)\ **Post date:** [November 21, 2023, 12:34am UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/6 "2023-11-21T00:34:23Z")
Hi @versae , Thank you for informing us. We are closing this out due to internal tracking purposes. You can write back anytime you are ready to proceed with this. Regards, Carlo Argel ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [January 20, 2024, 12:35am UTC](https://community.wandb.ai/t/strange-global-step-restarts-affecting-learning-rates-and-performance/5349/7 "2024-01-20T00:35:13Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.