# Training hangs with GPU Utilization 100% and wandb trying to sync **URL:** \ **Category:** W&B Help\ **Created:** [November 3, 2022, 11:12pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376 "2022-11-03T23:12:28Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ndrwnaguib](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndrwnaguib](https://community.wandb.ai/u/ndrwnaguib)\ **Post date:** [November 3, 2022, 11:12pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/1 "2022-11-03T23:12:28Z")
I’ve been trying to get wandb to work with pytorch lightning on multiple gpus, it works fine, in the sense that the model is being trained, and metrics are being reported properly to the dashboard; however, only after a couple of hours and sometimes 20 mins, the system is maxed to use all the resources, causing the whole training process to freeze without any progress. I used `py-spy` to generate the following dumps ![image](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/2X/b/b7f9aeb6b00a64f0949160291dd29702c3f2a805.png) Hopefully, they’d be helpful to figure out where is the issue. Thanks. ---
**Author:** ![ndrwnaguib](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndrwnaguib](https://community.wandb.ai/u/ndrwnaguib)\ **Post date:** [November 3, 2022, 11:13pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/2 "2022-11-03T23:13:03Z")
Here is also the flamegraph, ![image](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/2X/a/a1ca2be3989e2908d9be36a70421b54199727142.png) ---
**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\ **Post date:** [November 9, 2022, 11:45am UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/3 "2022-11-09T11:45:25Z")
Hi @ndrwnaguib thank you for reporting this. Can you please provide a bit more context on the training environment and resources, eg how many gpus, model size? Would it be possible to share your `debug.log` and `debug-internal.log` files of the run that’s hanging? I also checked your W&B account, and it seems you’re part of a team where the rate limits are higher. Could you log your experiments in the team so as to rule out the possibility this being caused by rate limits? Thanks! ---
**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\ **Post date:** [November 14, 2022, 2:01pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/5 "2022-11-14T14:01:26Z")
Hi @ndrwnaguib I wanted to follow up with you regarding this issue, can you please provide some more information to help us debug this? thanks! ---
**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\ **Post date:** [November 17, 2022, 4:22pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/6 "2022-11-17T16:22:55Z")
Gi @ndrwnaguib since we haven’t heard back from you in a while, I will go ahead and close this ticket for now. If the issue still persists, please provide us with some further information requested above, and we will be happy to reopen this and keep investigating! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [January 16, 2023, 4:23pm UTC](https://community.wandb.ai/t/training-hangs-with-gpu-utilization-100-and-wandb-trying-to-sync/3376/7 "2023-01-16T16:23:15Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.