Run.finish() hangs

[arthur-howard]

I’m using wandb version 0.14.0 in an ipynb file using vscode as part of assignment 1 of the ‘Effective MLOps’ course (logging dataset as artifact and visualising data with a table)

When I execute run.finish() at the end of my file the cell hangs indefinitely with the message

Waiting for W&B process to finish... (success).

[morgan]

How much data are you logging? It might still be uploading in the background. You can check one of the debug.log or debug-internal.log files in the wandb folder to see if there is any upload activity happening

[luis_bergua]

Hi @arthur-howard, thanks for reporting this! In addition to Morgan’s suggestion, could you please share those files for a specific run affected? They are under your local folder wandb/run-<date-time>-<run-id>/logs in the same directory where you’re running your code. Also, if you restart runtime on your notebook is the issue still raising or it only appears sometimes?

[arthur-howard]

Hi @luis_bergua as Morgan suggested there is a rather large artifact uploading the debug log is 70,000 lines long and growing so I think everything is working as expected thank you!

[silver-way]

Hi there, I have the same issue. From time to time wand.finish() hangs and will not finish unless terminated. wandb==0.13.10

app.log

2023-05-04 03:20:08,768 - INFO - utils.process_results - Logging wandb results
2023-05-04 03:20:11,232 - INFO - utils.process_results - Finishing wandb run (after the logging there is wandb.finish() line)

debug.log

2023-05-04 03:14:40,270 INFO    MainThread:1501 [wandb_init.py:init():775] starting run threads in backend
2023-05-04 03:14:40,328 INFO    MainThread:1501 [wandb_run.py:_console_start():2114] atexit reg
2023-05-04 03:14:40,328 INFO    MainThread:1501 [wandb_run.py:_redirect():1969] redirect: SettingsConsole.WRAP_RAW
2023-05-04 03:14:40,328 INFO    MainThread:1501 [wandb_run.py:_redirect():2034] Wrapping output streams.
2023-05-04 03:14:40,328 INFO    MainThread:1501 [wandb_run.py:_redirect():2059] Redirects installed.
2023-05-04 03:14:40,329 INFO    MainThread:1501 [wandb_init.py:init():817] run started, returning control to user process
2023-05-04 03:20:11,232 INFO    MainThread:1501 [wandb_run.py:_finish():1854] finishing run silver-way/FR BT 3/jz5k5gh9
2023-05-04 03:20:11,233 INFO    MainThread:1501 [wandb_run.py:_atexit_cleanup():2083] got exitcode: 0
2023-05-04 03:20:11,233 INFO    MainThread:1501 [wandb_run.py:_restore():2066] restore
2023-05-04 03:20:11,233 INFO    MainThread:1501 [wandb_run.py:_restore():2072] restore done

debug-internal.log

2023-05-04 03:20:13,343 INFO    SenderThread:1680 [sender.py:transition_state():587] send defer: 10
2023-05-04 03:20:13,343 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: poll_exit
2023-05-04 03:20:13,344 DEBUG   HandlerThread:1680 [handler.py:handle_request():144] handle_request: defer
2023-05-04 03:20:13,344 INFO    HandlerThread:1680 [handler.py:handle_request_defer():170] handle defer: 10
2023-05-04 03:20:13,345 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: defer
2023-05-04 03:20:13,345 INFO    SenderThread:1680 [sender.py:send_request_defer():583] handle sender defer: 10
2023-05-04 03:20:13,345 INFO    SenderThread:1680 [file_pusher.py:finish():162] shutting down file pusher
2023-05-04 03:20:13,784 INFO    wandb-upload_37:1680 [upload_job.py:push():138] Uploaded file /optimalasset/wandb/run-20230504_031439-jz5k5gh9/files/output.log
2023-05-04 03:20:13,798 INFO    wandb-upload_29:1680 [upload_job.py:push():138] Uploaded file /optimalasset/wandb/run-20230504_031439-jz5k5gh9/files/config.yaml
2023-05-04 03:20:13,810 INFO    wandb-upload_33:1680 [upload_job.py:push():138] Uploaded file /optimalasset/wandb/run-20230504_031439-jz5k5gh9/files/wandb-summary.json
2023-05-04 03:20:13,810 INFO    wandb-upload_0:1680 [upload_job.py:push():138] Uploaded file /optimalasset/wandb/run-20230504_031439-jz5k5gh9/files/requirements.txt
2023-05-04 03:20:14,235 DEBUG   HandlerThread:1680 [handler.py:handle_request():144] handle_request: poll_exit
2023-05-04 03:20:14,235 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: poll_exit
2023-05-04 03:20:15,235 DEBUG   HandlerThread:1680 [handler.py:handle_request():144] handle_request: poll_exit
2023-05-04 03:20:15,235 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: poll_exit
2023-05-04 03:20:16,236 DEBUG   HandlerThread:1680 [handler.py:handle_request():144] handle_request: poll_exit
2023-05-04 03:20:16,236 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: poll_exit
2023-05-04 03:20:17,236 DEBUG   HandlerThread:1680 [handler.py:handle_request():144] handle_request: poll_exit
2023-05-04 03:20:17,236 DEBUG   SenderThread:1680 [sender.py:send_request():363] send_request: poll_exit

The poll_exit loop is keep logging forever until stopped.