# Checkpoint path error using wandb.agent

**URL:** <https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966>\
**Category:** W&B Help\
**Tags:** sweeps\
**Created:** [June 19, 2024, 7:10pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966 "2024-06-19T19:10:33Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![pdomingues](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@pdomingues](https://community.wandb.ai/u/pdomingues)\
**Post date:** [June 19, 2024, 7:10pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/1 "2024-06-19T19:10:33Z")

</div>

Hello, I’m running my jupyter notebook using sweeps and wandb.agent but most of my sweep runs are failing with the following error message:

```auto
wandb: Adding directory to artifact (/home/phdomingues/masters/results/ViT/UNIFESP/masked/SN-UNIFESP/checkpoint-77)... Done. 2.4s
wandb: Adding directory to artifact (/home/phdomingues/masters/results/ViT/UNIFESP/masked/SN-UNIFESP/checkpoint-154)... Done. 3.2s
Traceback (most recent call last):
  File "/tmp/ipykernel_4543/1672163409.py", line 32, in train
    trainer.train()
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 1885, in train
    return inner_training_loop(
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 2311, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 2733, in _maybe_log_save_evaluate
    self.control = self.callback_handler.on_save(self.args, self.state, self.control)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer_callback.py", line 487, in on_save
    return self.call_event("on_save", args, state, control)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer_callback.py", line 498, in call_event
    result = getattr(callback, event)(
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/integrations/integration_utils.py", line 847, in on_save
    artifact.add_dir(artifact_path)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/wandb/sdk/artifacts/artifact.py", line 1226, in add_dir
    raise ValueError("Path is not a directory: {}".format(local_path))
ValueError: Path is not a directory: /home/phdomingues/masters/results/ViT/UNIFESP/masked/SN-UNIFESP/checkpoint-231

```

A snippet of my code:

```auto
def train(config=None):
    with wandb.init(config=config):
        config = wandb.config

        training_args = TrainingArguments(
            output_dir=output_dir,
            report_to='wandb',
            save_strategy='epoch',
            evaluation_strategy='epoch',
            logging_strategy='epoch',
            learning_rate=config.learning_rate,
            weight_decay=config.weight_decay,
            num_train_epochs=config.epochs,
            per_device_train_batch_size=config.batch_size,
            per_device_eval_batch_size=2,
            save_total_limit=2,
            remove_unused_columns=False,
            push_to_hub=False,
            fp16=True,
            load_best_model_at_end=True,
        )

        trainer = Trainer(
            model=model,
            args=training_args,
            data_collator=collate_fn,
            compute_metrics=partial(compute_metrics, metrics=[load_metric(m, trust_remote_code=True) for m in METRICS]),
            train_dataset=ds['train'],
            eval_dataset=ds['test'],
            tokenizer=processor,
        )
        trainer.train()

wandb.agent(sweep_id, train, count=20)
wandb.finish()

```

It seems to me that the library fails to create a checkpoint directory and breaks when trying to access it, or maybe it removes it too soon…

I would appreciate it if someone could help me figure out what is happening and how to solve it.

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [June 25, 2024, 7:57am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/2 "2024-06-25T07:57:25Z")

</div>

Hi @pdomingues Good day and thank you for reaching out to us! Happy to help you on this.

It seems that the error is triggered by an artifact directory not being recognized.

`ValueError: Path is not a directory: /home/phdomingues/masters/results/ViT/UNIFESP/masked/SN-UNIFESP/checkpoint-231`

As a first step of troubleshooting, have you checked if this directory exist?

---

<div class="post-metadata">

**Author:** ![pdomingues](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@pdomingues](https://community.wandb.ai/u/pdomingues)\
**Post date:** [June 25, 2024, 4:47pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/3 "2024-06-25T16:47:45Z")

</div>

Hello @paulo-sabile and thanks for the response.  
I can confirm that the directory does not exist when the error is thrown.

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [June 26, 2024, 5:54am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/4 "2024-06-26T05:54:57Z")

</div>

Thank you for clarifying this @pdomingues

To further troubleshoot this error, could you please provide us a copy of your `debug-internal.log` and `debug.log` for the affected run. These files are under your local folder `wandb/run-_-/logs` in the same directory where you’re running your code. These files will help us with more details about this error.

Can you also please share your current SDK version, you can get this by running `wandb --version`

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![pdomingues](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@pdomingues](https://community.wandb.ai/u/pdomingues)\
**Post date:** [June 26, 2024, 9:05pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/5 "2024-06-26T21:05:43Z")

</div>

Hello @paulo-sabile , sdk version is 0.17.2 and I tried downgrading to 0.15, but the error persists.

Unfortunately, I don’t have the exact logs of the error I posted anymore, so I reproduced the experiment again.

Here you can see the error is still the same:

```auto
Run fahuw2h2 errored:
Traceback (most recent call last):
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 307, in _run_job
    self._function()
  File "/tmp/ipykernel_476/1672163409.py", line 32, in train
    trainer.train()
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 1885, in train
    return inner_training_loop(
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 2311, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer.py", line 2733, in _maybe_log_save_evaluate
    self.control = self.callback_handler.on_save(self.args, self.state, self.control)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer_callback.py", line 487, in on_save
    return self.call_event("on_save", args, state, control)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/trainer_callback.py", line 498, in call_event
    result = getattr(callback, event)(
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/transformers/integrations/integration_utils.py", line 847, in on_save
    artifact.add_dir(artifact_path)
  File "/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/wandb/sdk/artifacts/artifact.py", line 1226, in add_dir
    raise ValueError("Path is not a directory: {}".format(local_path))
ValueError: Path is not a directory: /home/phdomingues/masters/results/ViT/UNIFESP/masked/FP-UNIFESP/checkpoint-60

```

The forum won’t let me upload the files so [here](https://drive.google.com/drive/folders/160Z5IHcS6HyGdMYGV0Gce-10rCkIqQIt?usp=sharing) is a link to both uploaded to my Google Drive.

I couldn’t find any errors on the logs for some reason, but I made sure they are the correct ones and you can even check the run id (fahuw2h2) in debug-internal.log.

Thanks again for the response.

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [July 2, 2024, 6:41am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/6 "2024-07-02T06:41:27Z")

</div>

Hi @pdomingues Good day and thank you for patiently waiting for our update.

We reviewed this and we think that the local directory doesn’t exist yet when adding the artifact. this might be a concurrency issue.

To further investigate this, we would like to ask the following:

- What is the size of that directory from your checkpoints?
- Could you please try to add a call like `time.sleep(100)` just to check if that would resolve the issue?

Thank you!

---

<div class="post-metadata">

**Author:** ![pdomingues](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@pdomingues](https://community.wandb.ai/u/pdomingues)\
**Post date:** [July 4, 2024, 7:55pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/7 "2024-07-04T19:55:07Z")

</div>

Hello @paulo-sabile,

I also suspect concurrency problems, but since it’s not coming directly from my code, I’m not sure how to proceed.

As for what you asked:

- I’m constantly recreating the directory so it doesn’t get too large from models being saved after every test I run, but the parent directory is 13GB.
- I’ve tried adding multiple sleep calls to my code. Still, I can only control execution before and after training (since I’m using the wandb-huggingface interface), and the problem occurs mid-training, so this didn’t solve anything. I’ve also tried tracking the file and the exact line where the error occurs (/home/phdomingues/.miniconda3/envs/wandb/lib/python3.8/site-packages/wandb/sdk/artifacts/artifact.py) and adding sleep and retry logic to that, together with some print statements to ensure everything was executed, but no success either.

Unrelated to what you asked, but another test I made recently was switching to PyTorch CPU, but that yielded nothing as well.

Again, thanks for the responses, and no worries about the waiting time. I’m already glad anyone is giving me a hand with this.

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [July 9, 2024, 1:28am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/8 "2024-07-09T01:28:11Z")

</div>

Thank you for confirming this @pdomingues . We’ll further review this and will get back to you for an update.

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [July 17, 2024, 7:22am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/9 "2024-07-17T07:22:55Z")

</div>

Hi @pdomingues Good day and thank you for patiently waiting for our update on this. What we can suggest here to move forward is you can add some logic on your end, to make sure that the` dir_name` exists. If it doesn’t exist, then try to add a code that should either have to wait with `time.sleep` or create the directory. This check needs to happen before the `add_dir` method.

Let me know if you can try to setup something like this and if this can help.

Best Regards,  
Paulo

---

<div class="post-metadata">

**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\
**Post date:** [July 23, 2024, 9:39am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/10 "2024-07-23T09:39:39Z")

</div>

Hi @pdomingues , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

---

<div class="post-metadata">

**Author:** ![pdomingues](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@pdomingues](https://community.wandb.ai/u/pdomingues)\
**Post date:** [July 23, 2024, 11:54am UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/11 "2024-07-23T11:54:38Z")

</div>

Hello @paulo-sabile and sorry for the delay in my response I had some personal matters to resolve and missed the notification about your message. I’ve tried all I could so I think I’ll move on and try other approaches to solve my problem. I appreciate the help and agree to close this request.

---

<div class="post-metadata">

**Author:** ![geminiproduo1](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/geminiproduo1/32/2935_2.png) [@geminiproduo1](https://community.wandb.ai/u/geminiproduo1)\
**Post date:** [March 31, 2026, 3:42pm UTC](https://community.wandb.ai/t/checkpoint-path-error-using-wandb-agent/6966/12 "2026-03-31T15:42:52Z")

</div>

LongPathTool manages issues caused by excessive directory length.