Wandb cannot upload files

As far as I understand wandb cannot finish uploading a file after the training process ends.
Logs are like the following:

2023-03-17 10:58:41,519 INFO    SenderThread:38526 [sender.py:transition_state():587] send defer: 14
2023-03-17 10:58:41,519 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: defer
2023-03-17 10:58:41,520 DEBUG   SenderThread:38526 [sender.py:send():336] send: final
2023-03-17 10:58:41,520 INFO    HandlerThread:38526 [handler.py:handle_request_defer():170] handle defer: 14
2023-03-17 10:58:41,520 DEBUG   SenderThread:38526 [sender.py:send():336] send: footer
2023-03-17 10:58:41,521 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: defer
2023-03-17 10:58:41,521 INFO    SenderThread:38526 [sender.py:send_request_defer():583] handle sender defer: 14
2023-03-17 10:58:41,522 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: poll_exit
2023-03-17 10:58:41,523 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: server_info
2023-03-17 10:58:41,523 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: poll_exit
2023-03-17 10:58:41,524 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: get_summary
2023-03-17 10:58:41,525 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: server_info
2023-03-17 10:58:41,525 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: sampled_history
2023-03-17 10:58:42,143 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: shutdown
2023-03-17 10:58:42,144 INFO    HandlerThread:38526 [handler.py:finish():842] shutting down handler
2023-03-17 10:58:42,525 INFO    WriterThread:38526 [datastore.py:close():298] close: /home/batu/wandb/run-20230317_105832-lvhco0to/run-lvhco0to.wandb
2023-03-17 10:58:43,142 INFO    SenderThread:38526 [sender.py:finish():1504] shutting down sender
2023-03-17 10:58:43,142 INFO    SenderThread:38526 [file_pusher.py:finish():164] shutting down file pusher
2023-03-17 10:58:43,143 INFO    SenderThread:38526 [file_pusher.py:join():169] waiting for file pusher

Hi Batuhan,

Could you send me your debug logs for the run that is not uploading files?

Also the workspace to the run that this is happening to?

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.