Cannot download files from a crashed run

Hi, could anyone tell me what’s going on with the API call?

I’m downloading a checkpoint through API from a run that crashed for some reason in Kaggle notebook, I promise the permission is the point of why the errors appeared(I own this project), and then the internet reason, I have tried in the other computer, and different notebook platform.

The below is my code:

wandb.login(key=config["wandb"]["api_key"])
api = wandb.Api()
run = api.run("keahi/finalProjectCV&algorithm/ujp50jup")
file_name = "expt_6_prototype_vit_weightedSampling_dataset_ps16_tl12_numhead12_state_dict_epoch_60.pth"
file = run.file(file_name)
file.download()

and then output as shown in the below:

wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin
wandb: WARNING If you're specifying your api key in code, ensure this code is not shared publicly.
wandb: WARNING Consider setting the WANDB_API_KEY environment variable, or running `wandb login` from the command line.
wandb: Appending key for api.wandb.ai to your netrc file: /root/.netrc
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py:131, in Retry.__call__(self, *args, **kwargs)
    130 try:
--> 131     result = self._call_fn(*args, **kwargs)
    132     # Only print resolved attempts once every minute

File /opt/conda/lib/python3.10/site-packages/wandb/apis/public/files.py:172, in File.download(self, root, replace, exist_ok, api)
    168         raise ValueError(
    169             "File already exists, pass replace=True to overwrite or exist_ok=True to leave it as is and don't error."
    170         )
--> 172 util.download_file_from_url(path, self.url, api.api_key)
    173 return open(path)

File /opt/conda/lib/python3.10/site-packages/wandb/util.py:1318, in download_file_from_url(dest_path, source_url, api_key)
   1310 response = requests.get(
   1311     source_url,
   1312     auth=auth,
   (...)
   1316     timeout=5,
   1317 )
-> 1318 response.raise_for_status()
   1320 if os.sep in dest_path:

File /opt/conda/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Not Found for url: https://api.wandb.ai/files/keahi/finalProjectCV&algorithm/ujp50jup/expt_6_prototype_vit_weightedSampling_dataset_ps16_tl12_numhead12_state_dict_epoch_60.pth

During handling of the above exception, another exception occurred:

CommError                                 Traceback (most recent call last)
Cell In[5], line 6
      4 file_name = "expt_6_prototype_vit_weightedSampling_dataset_ps16_tl12_numhead12_state_dict_epoch_60.pth"
      5 file = run.file(file_name)
----> 6 file.download()

File /opt/conda/lib/python3.10/site-packages/wandb/apis/normalize.py:73, in normalize_exceptions.<locals>.wrapper(*args, **kwargs)
     69         raise CommError(message, err.last_exception).with_traceback(
     70             sys.exc_info()[2]
     71         )
     72 except Error as err:
---> 73     raise err
     74 except Exception as err:
     75     # gql raises server errors with dict's as strings...
     76     if len(err.args) > 0:

File /opt/conda/lib/python3.10/site-packages/wandb/apis/normalize.py:41, in normalize_exceptions.<locals>.wrapper(*args, **kwargs)
     39 message = "Whoa, you found a bug."
     40 try:
---> 41     return func(*args, **kwargs)
     42 except requests.HTTPError as error:
     43     errors = parse_backend_error_messages(error.response)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py:212, in retriable.<locals>.decorator.<locals>.wrapped_fn(*args, **kargs)
    210 @functools.wraps(fn)
    211 def wrapped_fn(*args: Any, **kargs: Any) -> Any:
--> 212     return retrier(*args, **kargs)

File /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/retry.py:147, in Retry.__call__(self, *args, **kwargs)
    144     return result
    145 except self._retryable_exceptions as e:
    146     # if the secondary check fails, re-raise
--> 147     retry_timedelta_triggered = check_retry_fn(e)
    148     if not retry_timedelta_triggered:
    149         raise

File /opt/conda/lib/python3.10/site-packages/wandb/util.py:907, in no_retry_auth(e)
    905     raise CommError(f"Permission denied to access {wandb.run.path}")
    906 else:
--> 907     raise CommError(
    908         "It appears that you do not have permission to access the requested resource. "
    909         "Please reach out to the project owner to grant you access. "
    910         "If you have the correct permissions, verify that there are no issues with your networking setup."
    911         f"(Error {e.response.status_code}: {e.response.reason})"
    912     )

CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 404: Not Found)

The process of troubleshooting I’ve tried:

  • change platform → Not working
  • try to download a file from a different run → succeeded and very smoothly
  • try to download the same file from the dashboard directly → succeeded.

Looking forward to your help!!!
Thank you in advance.

Sorry, something that I missed is:

I’ve double-checked that the file name or run_id is correct.

Hi @keahixie thank you for writing in. Can you please try to print all file names as follows to see if there’s a matched output with the specified one?

wandb.login(key=config["wandb"]["api_key"])
api = wandb.Api()
run = api.run("keahi/finalProjectCV&algorithm/ujp50jup")
for f in run.files():
  print(f.name)

Hi @keahixie we wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Sorry, sir!
I’m quite busy these days because I’m in final exam week! I’ll send the feedback back in a week from now. Thanks for your understanding !!

Keahi

Thank you for the update, @keahixie, and best of luck with your exams! Sounds good, let me know once you had time to look into this next week and we can take it from there.