Wandb cannot upload files

W&B Help
1

As far as I understand wandb cannot finish uploading a file after the training process ends.
Logs are like the following:

2023-03-17 10:58:41,519 INFO    SenderThread:38526 [sender.py:transition_state():587] send defer: 14
2023-03-17 10:58:41,519 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: defer
2023-03-17 10:58:41,520 DEBUG   SenderThread:38526 [sender.py:send():336] send: final
2023-03-17 10:58:41,520 INFO    HandlerThread:38526 [handler.py:handle_request_defer():170] handle defer: 14
2023-03-17 10:58:41,520 DEBUG   SenderThread:38526 [sender.py:send():336] send: footer
2023-03-17 10:58:41,521 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: defer
2023-03-17 10:58:41,521 INFO    SenderThread:38526 [sender.py:send_request_defer():583] handle sender defer: 14
2023-03-17 10:58:41,522 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: poll_exit
2023-03-17 10:58:41,523 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: server_info
2023-03-17 10:58:41,523 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: poll_exit
2023-03-17 10:58:41,524 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: get_summary
2023-03-17 10:58:41,525 DEBUG   SenderThread:38526 [sender.py:send_request():363] send_request: server_info
2023-03-17 10:58:41,525 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: sampled_history
2023-03-17 10:58:42,143 DEBUG   HandlerThread:38526 [handler.py:handle_request():144] handle_request: shutdown
2023-03-17 10:58:42,144 INFO    HandlerThread:38526 [handler.py:finish():842] shutting down handler
2023-03-17 10:58:42,525 INFO    WriterThread:38526 [datastore.py:close():298] close: /home/batu/wandb/run-20230317_105832-lvhco0to/run-lvhco0to.wandb
2023-03-17 10:58:43,142 INFO    SenderThread:38526 [sender.py:finish():1504] shutting down sender
2023-03-17 10:58:43,142 INFO    SenderThread:38526 [file_pusher.py:finish():164] shutting down file pusher
2023-03-17 10:58:43,143 INFO    SenderThread:38526 [file_pusher.py:join():169] waiting for file pusher
2

Hi Batuhan,

Could you send me your debug logs for the run that is not uploading files?

Also the workspace to the run that this is happening to?

3

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.