Failure to upload data after run

[tomjur]

Hi,

I have several runs that upon termination are getting stuck on the following error (unable to copy as text due to the rotating “|” sign):

Any ideas on how to fix without stopping my run?

Edit: I checked the debug-internal.log and it seems the below error message pops up.
Additional details -
(1) The server uploading the data is in a private network (.technion.ac.il)
(2) Unlike other runs that do succeed in the meantime, here I upload a video using:

wandb.log({"video": wandb.Video(all_images, fps=4, format="mp4")})

Error:

2022-12-28 13:30:28,582 ERROR Thread-17 :3928098 [internal_api.py:upload_file():1875] upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/tomjur/[my proj]/[current run]/wandb-metadata.json?Expires=1672313428&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=UKkwGiiPdbchVt3D9Np7HXU91ioIwnWZa5EqapeDR0UFZbjElm2TU5XOF93P2z7%2BnvCTcAEeSkV0tVg3ln0aMTUw%2BYTVSiyBBHncYCZkJcHWS2MDgcA5v9LzjnQ3c5kJoUA5dJwMWNVLebZ8LWkUDKdBSeafEfogP3xE%2FLIDdPtHolpISNJIvEY3JDPOrTJUh9Ge%2Bw3%2BgLgvep9LbG2qEtmj%2FIeeW%2FXqi9wIpfWdDjZ6ZlkGkohYjJPdSGA6GM9RLrglaDVnwmJQXTgyLEIpD9%2FsuKJeBN3U0v6qz1aE2OBoDCK%2Fs4lDEspXy%2FgOqVS6FEHrwQ813rkabFR6dvC1tA%3D%3D: HTTPSConnectionPool(host=‘storage.googleapis.com’, port=443): Max retries exceeded with url: /wandb-production.appspot.com/tomjur/[my proj]/[current run]/wandb-metadata.json?Expires=1672313428&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=UKkwGiiPdbchVt3D9Np7HXU91ioIwnWZa5EqapeDR0UFZbjElm2TU5XOF93P2z7%2BnvCTcAEeSkV0tVg3ln0aMTUw%2BYTVSiyBBHncYCZkJcHWS2MDgcA5v9LzjnQ3c5kJoUA5dJwMWNVLebZ8LWkUDKdBSeafEfogP3xE%2FLIDdPtHolpISNJIvEY3JDPOrTJUh9Ge%2Bw3%2BgLgvep9LbG2qEtmj%2FIeeW%2FXqi9wIpfWdDjZ6ZlkGkohYjJPdSGA6GM9RLrglaDVnwmJQXTgyLEIpD9%2FsuKJeBN3U0v6qz1aE2OBoDCK%2Fs4lDEspXy%2FgOqVS6FEHrwQ813rkabFR6dvC1tA%3D%3D (Caused by SSLError(CertificateError(“hostname ‘storage.googleapis.com’ doesn’t match either of ‘*.technion.ac.il’, ‘technion.ac.il’”)))

Thanks,
Tom

[mohammadbakir]

Hi @tomjur , happy to help. From the error message it appears you are encountering and SSL certification error, SSLError(CertificateError(“...."), most likely attributed to your private networks firewall. Please refer to our guide here on addressing network related issues. Do let me know if you still encounter issues after addressing the SSL error.

[mohammadbakir]

Hi @tomjur , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!