Wandb.sync crashed

[larinaleo12]

2023-08-01T03:42:16Z
when I use wandb in my own projects, I set a offline mode. So I get a friendly reminder as follows

wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/zcy/pycode/graphtrans-main/wandb/offline-run-20230729_205424-77nmb8jc

I activate my conda env and use the wandb sync to upload my offline run. But I get the errors like this

File “/home/zcy/anaconda3/envs/graph-aug/lib/python3.8/site-packages/requests/models.py”, line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.wandb.ai/files/larinaleo12/ucca4gnntransformer-cor-sc/77nmb8jc/file_stream
upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/larinaleo12/ucca4gnntransformer-cor-sc/77nmb8jc/output.log?Expires=1690946538&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=veI0HqYb44O3n2wmu4G7qLm6rt7JvDAklVzKPyu%2B0iG6e2GBnP1GCNJ2zEBaQPwQRncGcZ1%2B1te8AFnT%2FYvjzg1Ke9iR19qbRozGVGoRrtnrtSUbC4qlzWo%2FK90lvUWyNQb1uhCk4O8OnftREfEa4Sw3XbPNJlX%2BcU3Qc3ncMiFenPGpqMYXsKRMxy6gpPo%2F23TN20MTLI419dYmbfN16GLovNDgoyB4tdYC7QkOYsa6Gcr%2FrFbGlucgaAHR3lu6AAqcJkAdgRBad8g0XNaxU%2B9hIbzbUFkuGJj%2Ba4nrxs82ZMFrYFsBIoVyY07kpXMqMA%2B%2BPb34S%2FKigCn8KU%2Fv0Q%3D%3D (‘Connection aborted.’, TimeoutError(110, ‘Connection timed out’))

what can I do to fix this error and sync my offline files?

[nathank]

Hi @larinaleo12, do you get this error for any run that you try to sync or just this one? For example, could you try making a very basic offline run without much data and sync it?

Also, what version of wandb are you using in your condo env?

[larinaleo12]

I think the problem is just for this run,because I can sync another run a short ago. And this error are for tow runs with large amount of data which are more than 12GB. So I suspect this error may be caused by the file size. But I don’t know how to solve this problem and it’s just my guess.

[nathank]

@larinaleo12, I agree that it is likely caused by the size of the run. It seems like we may need to break up file uploads for this sort of large files. Would it maybe be possible to share the actual run folder with us so we can reproduce it from our side? If so, could you try uploading it do a file sharing site and send the link to me at Nathan.kuneman@wandb.com?

Or if you have a minimal script that could reproduce the run, that would work as well.

[mohammadbakir]

Hi @larinaleo12, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!