Wandb logged my images then stopped and I cant find them in the media folder

W&B Help
1

wandb suddenly stopped logging my images during a run after it logged successfully. On the first iteration it logged the images then subsequent iterations (of the same run) it stopped logging. I had left the run continue overnight on a paid server - so now I’m not sure what to do - do I need to rerun everything again? its a paid server which is quite expensive so I used the first iteration to check that everything was ok then left it to run overnight. Is there any other place I can look for to recover the images? they don’t exist in the wand media folder - really upset because I’ll have to pay again and still not sure I’ll get the images. Please can I get help! I checked the internal debug file and I can see lots of this:

2024-02-19 10:33:07,312 ERROR   wandb-upload_11:158274 [internal_api.py:upload_file():2767] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'SUmL3Hri3gQbE8KfLSWhzg==', 'Content-Type': 'image/png', 'Content-Length': '395913'}
2024-02-19 10:33:07,312 ERROR   wandb-upload_11:158274 [internal_api.py:upload_file():2769] upload_file response body: 
2024-02-19 10:33:07,336 INFO    wandb-upload_13:158274 [upload_job.py:push():89] Uploaded file /root/.local/share/wandb/artifacts/staging/tmppxhigo5u
2024-02-19 10:33:07,341 ERROR   wandb-upload_58:158274 [internal_api.py:upload_file():2765] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/141059310/725738762/33e8bd484e9d49a616df1ad312fdb4db?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=KypByNideEpNx6Ix6jGAg%2BmGPlnWLHOTdfEBsYci4hA9twOtHddKSjQGsPm61zLf1o%2BWDbk9whE8vulznr5jTzWBPFPgmpKczOGn8fK270dQKjDeZqqKp%2B7pjV8T9yADYU9mFGKj0RfeuFyugJyyyO9LRavBYWmBZpzATzuT2pNPAkNiJs2%2FMWin1Tg%2FYKvvruMZp5CAcyLdNSPpKIFtVMq7sp%2Fb%2FOT6s1YiYi7Nz1JUm%2Fec1dgn3Ls3UIMAi%2FQ2joHpOMrJr1w%2BASLsrmyk2n96xga%2F%2F%2FDkqwjd%2FHAB3ZebwpYGQ6aV62MELV9iB0VabEXhwqVYpM94CVqJh%2BEUWw%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/141059310/725738762/33e8bd484e9d49a616df1ad312fdb4db?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=KypByNideEpNx6Ix6jGAg%2BmGPlnWLHOTdfEBsYci4hA9twOtHddKSjQGsPm61zLf1o%2BWDbk9whE8vulznr5jTzWBPFPgmpKczOGn8fK270dQKjDeZqqKp%2B7pjV8T9yADYU9mFGKj0RfeuFyugJyyyO9LRavBYWmBZpzATzuT2pNPAkNiJs2%2FMWin1Tg%2FYKvvruMZp5CAcyLdNSPpKIFtVMq7sp%2Fb%2FOT6s1YiYi7Nz1JUm%2Fec1dgn3Ls3UIMAi%2FQ2joHpOMrJr1w%2BASLsrmyk2n96xga%2F%2F%2FDkqwjd%2FHAB3ZebwpYGQ6aV62MELV9iB0VabEXhwqVYpM94CVqJh%2BEUWw%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-02-19 10:33:07,341 ERROR   wandb-upload_58:158274 [internal_api.py:upload_file():2767] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'M+i9SE6dSaYW3xrTEv202w==', 'Content-Type': 'image/png', 'Content-Length': '429304'}
2024-02-19 10:33:07,341 ERROR   wandb-upload_58:158274 [internal_api.py:upload_file():2769] upload_file response body: 
2024-02-19 10:33:07,345 ERROR   wandb-upload_17:158274 [internal_api.py:upload_file():2765] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/141059310/725738762/025b1042109a46f7018fc717b970fab6?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Zf7x1FJtU9eO0Kaop8Nf0EbOcj%2BQj5G4KiNGgWPhtQpeYTDzp4oi789ArtKKKMR37Dx8ZLrYGngW1wFfOxmgwgpV%2BHNTYtczVhCRZ80S5A%2Brn9yggm2Q9TeEsky7Bu25LAud1qFvZjr4gGnuR6CsXhi1ywRQ%2FSyRDjqc24zAeaCX%2Fc1Xo2QJ7Wj9wOvTMQABo4sMsJF7zWrQs6e2UcOqnpsdnTT6FbNAGhgVU93xoOaBlJ8OW%2FRWa9SfWS7cDrsf7UntdX4IWZMC4AZEcRanfQgXsiIGIaE0lN13Xnjfx5Xeh9ZraUETc6hFz2XQ3CHbYSWYlDDygRe628EjkTuiRg%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/141059310/725738762/025b1042109a46f7018fc717b970fab6?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Zf7x1FJtU9eO0Kaop8Nf0EbOcj%2BQj5G4KiNGgWPhtQpeYTDzp4oi789ArtKKKMR37Dx8ZLrYGngW1wFfOxmgwgpV%2BHNTYtczVhCRZ80S5A%2Brn9yggm2Q9TeEsky7Bu25LAud1qFvZjr4gGnuR6CsXhi1ywRQ%2FSyRDjqc24zAeaCX%2Fc1Xo2QJ7Wj9wOvTMQABo4sMsJF7zWrQs6e2UcOqnpsdnTT6FbNAGhgVU93xoOaBlJ8OW%2FRWa9SfWS7cDrsf7UntdX4IWZMC4AZEcRanfQgXsiIGIaE0lN13Xnjfx5Xeh9ZraUETc6hFz2XQ3CHbYSWYlDDygRe628EjkTuiRg%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-02-19 10:33:07,345 ERROR   wandb-upload_17:158274 [internal_api.py:upload_file():2767] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'AlsQQhCaRvcBj8cXuXD6tg==', 'Content-Type': 'image/png', 'Content-Length': '390353'}
2024-02-19 10:33:07,345 ERROR   wandb-upload_17:158274 [internal_api.py:upload_file():2769] upload_file response body: 
2024-02-19 10:33:07,346 INFO    wandb-upload_57:158274 [upload_job.py:push():89] Uploaded file /root/.local/share/wandb/artifacts/staging/tmp6n80elz0
2024-02-19 10:33:07,353 ERROR   wandb-upload_25:158274 [internal_api.py:upload_file():2765] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/141059310/725738762/4e35ea83336eb0602cef1341dd109b99?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=mu3fzHlQ8NiZiIxwJ%2FRoqK9CdqDOP4W5vcOsUcC6ad6t34vXeCyezrMdKLDIZpa%2FaxyerHxhq27tJ0ey3HbSrwGVsB3TCdQ%2BZ7p6G0X4yh4zJlKV%2F94mCQgEK6b0BgOueBshNPBerB8CKIORxa7GchV6h3XKZv6kOtRzqU2kSGSuULUDWrJZyIUE0gqy%2F3oQc5rMWw7cwXutqAIAN%2FEYQY38NqZRSqDozznUkOFC8AYwQ%2BL9DUqISzizC6Fk%2BZqnXdyfE%2FiFqqsqYShqqiU3aJI0JzqD98QWJzyBTdSvY6IDHVBAPo1WQJlmp8CrUWKukM05H9Ds7DBbHev9%2B%2BMFoA%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/141059310/725738762/4e35ea83336eb0602cef1341dd109b99?Expires=1708425164&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=mu3fzHlQ8NiZiIxwJ%2FRoqK9CdqDOP4W5vcOsUcC6ad6t34vXeCyezrMdKLDIZpa%2FaxyerHxhq27tJ0ey3HbSrwGVsB3TCdQ%2BZ7p6G0X4yh4zJlKV%2F94mCQgEK6b0BgOueBshNPBerB8CKIORxa7GchV6h3XKZv6kOtRzqU2kSGSuULUDWrJZyIUE0gqy%2F3oQc5rMWw7cwXutqAIAN%2FEYQY38NqZRSqDozznUkOFC8AYwQ%2BL9DUqISzizC6Fk%2BZqnXdyfE%2FiFqqsqYShqqiU3aJI0JzqD98QWJzyBTdSvY6IDHVBAPo1WQJlmp8CrUWKukM05H9Ds7DBbHev9%2B%2BMFoA%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024
2

hey @suemayah-eldursi - few questions to help me dig into this:

  • what wandb SDK version are you using?
  • could you also provide a link to the run where you’re seeing this?
  • It seems you’re encountering SSL-related errors when wandb attempts to upload images to Google Cloud Storage, which is where wandb stores artifacts and logs. This problem can occur due to network issues or SSL certificate verification problems. Could you ensure that the REQUESTS_CA_BUNDLEenvironment variable is pointing to a file containing a set of trusted CA certificates?
3

Hi Suemayah, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!