# WARNING .wandb file is incomplete (invalid padding)

**URL:** <https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153>\
**Category:** W&B Help\
**Tags:** wandb\
**Created:** [March 30, 2023, 12:37pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153 "2023-03-30T12:37:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![evanv](https://avatars.discourse-cdn.com/v4/letter/e/73ab20/32.png) [@evanv](https://community.wandb.ai/u/evanv)\
**Post date:** [March 30, 2023, 12:37pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153/1 "2023-03-30T12:37:07Z")

</div>

Hello all!

I am running a lot of runs per day (~2K sometimes) and have been encountering some strange errors in a handful of my runs. I am doing this on a large computation cluster, so to avoid putting too much strain on the network for every run I

1. set wandb to run offline (`export WANDB_MODE="offline"`)
2. set the `WANDB_DIR` to be a tmp directory (`WANDB_DIR=$(mktemp -d)`)
3. Run my run as normal (runs are relatively short often taking ~2-20 minutes)
4. Sync my wandb runs (`wandb sync $WANDB_DIR/wandb/offline*`)
5. Clean up my tmpdir (`rm -rf $WANDB_DIR `)

The full script is below:

```auto
my_config= # some config unique to this run
export WANDB_MODE="offline"
export WANDB_DIR=$(mktemp -d)
python train.py --config $my_config 
wandb sync $WANDB_DIR/wandb/offline*
rm -rf $WANDB_DIR 

```

In 99% of runs this works totally fine, however in a handful I get messages like:

```auto
Syncing: https://wandb.ai/some_run ... wandb: WARNING .wandb file is incomplete (invalid padding), be sure to sync this run again once it's finished
done.

```

If I actually _look_ at `some_run`, it seems totally normal and I don’t see any missing data. Furthermore the `wandb sync` command returns 0 exit code so I would assume all is well despite the error message. But the existence of the error is concerning and I am not sure the best way to deal with it or if it needs to be delt with at all. I am grateful for any advice people have!

---

<div class="post-metadata">

**Author:** ![mohammadbakir](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@mohammadbakir](https://community.wandb.ai/u/mohammadbakir)\
**Post date:** [April 6, 2023, 8:31pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153/2 "2023-04-06T20:31:11Z")

</div>

Hi @evanv , that warning is safe to ignore, the run will eventually sync. For the runs that throw this warning, are you seeing there are discrepancies in the data logged to the workspace?

---

<div class="post-metadata">

**Author:** ![evanv](https://avatars.discourse-cdn.com/v4/letter/e/73ab20/32.png) [@evanv](https://community.wandb.ai/u/evanv)\
**Post date:** [April 7, 2023, 10:31pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153/3 "2023-04-07T22:31:49Z")

</div>

Hi there @mohammadbakir ! Nothing appears to be wrong wtih the runs. Out of curiosity, why is the wanring thrown?

---

<div class="post-metadata">

**Author:** ![mohammadbakir](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@mohammadbakir](https://community.wandb.ai/u/mohammadbakir)\
**Post date:** [April 17, 2023, 8:01pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153/4 "2023-04-17T20:01:28Z")

</div>

Hi @evanv , apologies for the delay.

The `invalid padding` error occurs when wandb tries to read data from your run file, but the file may not be in an expected format. This could happen for a variety of reasons including file corruption, or issues when the file is read. When wandb scans your file and finds a discrepency with the format, it raises a warning informing you to `sync this run again once it's finished` as precautionary measure. If it successfully synced the first time around, then great, if not try again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [June 16, 2023, 8:01pm UTC](https://community.wandb.ai/t/warning-wandb-file-is-incomplete-invalid-padding/4153/5 "2023-06-16T20:01:43Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.