# Lighting: Checkpoints silently fail to save

**URL:** <https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048>\
**Category:** W&B Help\
**Created:** [September 1, 2022, 11:10pm UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048 "2022-09-01T23:10:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zaptrem](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/zaptrem/32/958_2.png) [@zaptrem](https://community.wandb.ai/u/zaptrem)\
**Post date:** [September 1, 2022, 11:10pm UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048/1 "2022-09-01T23:10:02Z")

</div>

Every couple of hours I silently get this error during training then checkpoints silently fail to save from then on. I’ve now lost \>30 hrs of training because of this weird issue. Does anyone know what’s causing this and how to fix it?

for future searchers:  
`NVMLError_OperatingSystem: The operating system has blocked the request.`

 ![IMG_0367](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/40594953988cd414c66052042a88418ba4eda5a4.jpeg)

---

<div class="post-metadata">

**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\
**Post date:** [September 7, 2022, 3:25pm UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048/2 "2022-09-07T15:25:10Z")

</div>

Hi @zaptrem thank you for reporting this. We will need a bit more context here on your setup both hardware and environments. Is this a WSL terminal? there might be some issues with NVML and WSL.  
Also wanted to clarify, did it not log anything from epoch 1876 to 2392? would you be interested in running the experiments in offline mode and write the data to your disk and sync to W&B afterwards?

---

<div class="post-metadata">

**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\
**Post date:** [September 12, 2022, 5:36pm UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048/3 "2022-09-12T17:36:47Z")

</div>

Hey @zaptrem I wanted to follow up with you regarding this issue, could you provide some more information asked above to help debug this? Please let me know if I can be of further assistance or if your issue has been resolved.

---

<div class="post-metadata">

**Author:** ![thanos-wandb](https://avatars.discourse-cdn.com/v4/letter/t/4af34b/32.png) [@thanos-wandb](https://community.wandb.ai/u/thanos-wandb)\
**Post date:** [September 16, 2022, 11:39am UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048/4 "2022-09-16T11:39:46Z")

</div>

Hi @zaptrem as we haven’t heard back from you, we are going to close the ticket for now. Please feel free to message us here if the issue hasn’t been resolved for you, and we will be happy to keep investigating!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [November 15, 2022, 11:40am UTC](https://community.wandb.ai/t/lighting-checkpoints-silently-fail-to-save/3048/5 "2022-11-15T11:40:13Z")

</div>

Closed on November 15, 2022, 11:40am UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
