Waiting for W&B process to finish... (success)

susbrock · December 7, 2022, 8:22pm

I am using wandb to log Tensorflow model training. Unfortunately the runs keep on running forever even though the training has finished minutes before.

I am receiving a lot of the following debug messages in debug-internals.log:

2022-12-07 21:19:47,595 DEBUG   HandlerThread:796 [handler.py:handle_request():139] handle_request: keepalive
```

debug.log  is already finished with the following last messages:
```
2022-12-07 21:09:05,301 INFO    MainThread:12168 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'session_history') code\_session_history.ipynb None
2022-12-07 21:09:05,323 INFO    MainThread:12168 [jupyter.py:_save_ipynb():389] looking for notebook: None
2022-12-07 21:09:05,323 INFO    MainThread:12168 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
2022-12-07 21:09:05,323 INFO    MainThread:12168 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
2022-12-07 21:09:05,324 INFO    MainThread:12168 [wandb_run.py:_restore():1938] restore
2022-12-07 21:09:05,325 INFO    MainThread:12168 [wandb_run.py:_restore():1944] restore done
```

I have already updated to wandb-013.6 but this did not solve the issues.

liuyee · December 8, 2022, 7:53am

I also encountered this problem, but it has not been resolved.

liuyee · December 8, 2022, 8:00am

object at 0x000001B3207410A0>>, args: ('https://api.wandb.ai/files/liuye/Image_Enhancement/1nykox3a/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}}
2022-12-08 15:56:47,968 WARNING FileStreamThread:7688 [file_stream.py:request_with_retry():667] requests_with_retry encountered retryable exception: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /files/liuye/Image_Enhancement/1nykox3a/file_stream (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))). func: <bound method Session.post of <requests.sessions.Session object at 0x000001B3207410A0>>, args: ('https://api.wandb.ai/files/liuye/Image_Enhancement/1nykox3a/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}}

Blockquote

luis_bergua · December 12, 2022, 8:45am

Hi @susbrock @liuyee, thanks for reporting this! I’ll try to reproduce this issue and see what’s happening here but, in the meantime, there are some points that will be helpful:

Are you using Tensorboard in your experiment?
Could you confirm me if your code is something like:

wandb.init()
##Create and train tf model
wandb.log({metrics})

If you aren’t doing it yet, could you try to assign the run to a variable like run = wandb.init() and call run.finish() ate the end of your experiment?

Thanks!

sydholl · December 12, 2022, 8:45am

Hi @susbrock @liuyee, thanks for reporting this! I’ll try to reproduce this issue and see what’s happening here but, in the meantime, there are some points that will be helpful:

Are you using Tensorboard in your experiment?
Could you confirm me if your code is something like:

wandb.init()
##Create and train tf model
wandb.log({metrics})
If you aren’t doing it yet, could you try to assign the run to a variable like run = wandb.init() and call run.finish() ate the end of your experiment?

Thanks!

[Discourse post]

liuyee · December 12, 2022, 10:06am

Are you using Tensorboard in your experiment? A: Yes.
Could you confirm me if your code is something like that? A: i don’t use run.finish().

luis_bergua · December 12, 2022, 12:19pm

Thanks for confirming this liuyee, could you try using run.finish() and see if the same issue appears?

liuyee · December 13, 2022, 8:29am

It still does not work.

susbrock · December 13, 2022, 8:07pm

@luis_bergua I am using Tensorboard. I am using

wandb.init(....)
...

wandb.finish()

because I am running my code in jupyter notebook for VS code.

Last week I was struggling with wandb but currently it works fine. I have add “run = wandb.init(…)” sometime during the last days so that I cannot say if this is the important part. I am still finishing my code with “wandb.finish()”. I had picked up the “run = wand.init(…)” implementation while I was looking for a solution to resume runs…

luis_bergua · December 15, 2022, 5:02pm

Hi @susbrock @liuyee, thanks for your answers! I would highly recommend you to always assign wandb.init() to a variable since it can avoid errors when running multiple experiments. Have you experienced this issue again? If so, would you mind sharing with me the debug.log and debug-internal.log files under the local wandb folder? Thanks!

susbrock · December 25, 2022, 10:12am

@luis_bergua The problem is reappearing right now. Therefore I am posting the debug files.

debug.log

2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Configure stats pid to 1764
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from C:\Users\Susanne\.config\wandb\settings
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from i:\tinyml\tiny_cnn\wandb\settings
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from environment variables: {'_require_service': 'True', 'mode': 'online'}
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {'program': '<python with no main file>'}
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_log_setup():476] Logging user logs to i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\logs\debug.log
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_log_setup():477] Logging internal logs to i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\logs\debug-internal.log
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_jupyter_setup():426] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x000001FC77A5D900>
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():516] calling init triggers
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {}
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():569] starting backend
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():573] setting up manager
2022-12-25 10:24:34,060 INFO    MainThread:1764 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=spawn, using: spawn
2022-12-25 10:24:34,065 INFO    MainThread:1764 [wandb_init.py:init():580] backend started and connected
2022-12-25 10:24:34,080 INFO    MainThread:1764 [wandb_run.py:_label_probe_notebook():1116] probe notebook
2022-12-25 10:24:34,083 INFO    MainThread:1764 [wandb_run.py:_label_probe_notebook():1126] Unable to probe notebook: 'NoneType' object has no attribute 'get'
2022-12-25 10:24:34,083 INFO    MainThread:1764 [wandb_init.py:init():658] updated telemetry
2022-12-25 10:24:34,183 INFO    MainThread:1764 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2022-12-25 10:24:35,152 INFO    MainThread:1764 [wandb_run.py:_on_init():2006] communicating current version
2022-12-25 10:24:35,417 INFO    MainThread:1764 [wandb_run.py:_on_init():2010] got version response 
2022-12-25 10:24:35,418 INFO    MainThread:1764 [wandb_init.py:init():728] starting run threads in backend
2022-12-25 10:24:40,442 INFO    MainThread:1764 [wandb_run.py:_console_start():1986] atexit reg
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1844] redirect: SettingsConsole.WRAP_RAW
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1909] Wrapping output streams.
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1931] Redirects installed.
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_init.py:init():765] run started, returning control to user process
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set batch_size = 32 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb batch_size 32 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set learn_rate = 0.001 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb learn_rate 0.001 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set momentum = 0.9 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb momentum 0.9 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set epochs = 5 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb epochs 5 None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set classes = 3 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb classes 3 None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set id = 2dyabu0c - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb id 2dyabu0c None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set architecture = efficientNetB0_0.1_96_c3_o3_keras - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb architecture efficientNetB0_0.1_96_c3_o3_keras None
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set optimizer = SGD - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb optimizer SGD None
2022-12-25 10:24:40,536 INFO    MainThread:1764 [wandb_run.py:_tensorboard_callback():1299] tensorboard callback: i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\files\train, True
2022-12-25 10:25:31,797 INFO    MainThread:1764 [wandb_run.py:_tensorboard_callback():1299] tensorboard callback: i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\files\validation, True
2022-12-25 10:28:02,476 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'visualize', 'conf_mat') {'panel_type': 'Vega2', 'panel_config': {'panelDefId': 'wandb/confusion_matrix/v1', 'fieldSettings': {'Actual': 'Actual', 'Predicted': 'Predicted', 'nPredictions': 'nPredictions'}, 'stringSettings': {'title': ''}, 'transform': {'name': 'tableWithLeafColNames'}, 'userQuery': {'queryFields': [{'name': 'runSets', 'args': [{'name': 'runSets', 'value': '${runSets}'}], 'fields': [{'name': 'id', 'fields': []}, {'name': 'name', 'fields': []}, {'name': '_defaultColorIndex', 'fields': []}, {'name': 'summaryTable', 'args': [{'name': 'tableKey', 'value': 'conf_mat_table'}], 'fields': []}]}]}}} None
2022-12-25 10:28:03,896 INFO    MainThread:1764 [wandb_run.py:_finish():1753] finishing run susbrock/efficientNetB0/2dyabu0c
2022-12-25 10:28:03,980 INFO    MainThread:1764 [jupyter.py:save_history():477] saving 27 cells to _session_history.ipynb
2022-12-25 10:28:03,980 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'session_history') code\_session_history.ipynb None
2022-12-25 10:28:04,018 INFO    MainThread:1764 [jupyter.py:_save_ipynb():389] looking for notebook: None
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_restore():1938] restore
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_restore():1944] restore done

debug-internal.log will follow in separate message.
Thanks for your support.

vincpc · January 2, 2023, 5:39pm

I do have a similar issue, Running the code from de MLOps course 01_EDA.ipynb

The notebook basicaly does:
start a run with run = wandb.init()
create an antifact and store a Table.
but when I run run.finish(), it keeps going for ever and never finish running the cell
(I’m using VS Code Jupyter notebook), running the same code as it is in the course Effective MLOps: Model Development, processing data for exploratory analysis.

UPDATE:

Running the code using Colab works just fine.
Still would like to work in my local PC
My internet speed last test is 200/111 Mb download/upload so it’s not a connection problem.

this is the log I’m having (debug-internal.log):

2023-01-02 17:18:30,630 INFO Thread-3998:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\f4\d1e18a269ee5ea2dfe1f31f62ace7e
2023-01-02 17:18:30,635 INFO Thread-3975:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ef\805451c61c3548e0b64d7cd43fb495
2023-01-02 17:18:30,637 INFO Thread-3983:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\f1\159ed57aec7385894be1f50ff3db6b
2023-01-02 17:18:30,641 INFO Thread-3974:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ef\40545a3365286db74b9eec276aafe5
2023-01-02 17:18:30,662 INFO Thread-3962:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ec\5bb019ae113d7a08c9bb6eb9ac6fe8
2023-01-02 17:18:30,773 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:31,775 INFO Thread-4012:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fa\336027db262331cf3de1c2bc489161
2023-01-02 17:18:31,779 INFO Thread-4023:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fd\72fdd8e9b81196f34a27936872ec85
2023-01-02 17:18:31,802 INFO Thread-4026:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fe\23e9669b121a89dd294fbcd063240e
2023-01-02 17:18:35,832 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:40,870 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:45,886 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:50,958 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:55,987 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:01,020 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:06,043 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:11,062 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:16,108 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:21,124 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:26,138 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:31,175 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:36,203 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:41,221 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:46,246 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:51,287 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:56,303 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:01,336 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:06,359 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:11,360 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:16,378 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive

This last line is still being printed…

Log (debug.log):

2023-01-02 17:08:08,443 INFO 2023-01-02 17:08:08,443 INFO 2023-01-02 17:08:08,444 INFO 2023-01-02 17:08:08,444 INFO 2023-01-02 17:08:08,444 INFO 2023-01-02 17:08:08,444 INFO 2023-01-02 17:08:08,445 INFO 2023-01-02 17:08:08,445 INFO 2023-01-02 17:08:08,445 INFO 2023-01-02 17:08:08,445 INFO config: {}
2023-01-02 17:08:08,445 INFO 2023-01-02 17:08:08,446 INFO 2023-01-02 17:08:08,449 INFO 2023-01-02 17:08:08,454 INFO 2023-01-02 17:08:08,468 INFO 2023-01-02 17:08:08,470 INFO 2023-01-02 17:08:08,470 INFO 2023-01-02 17:08:08,513 INFO 2023-01-02 17:08:09,065 INFO 2023-01-02 17:08:09,380 INFO 2023-01-02 17:08:09,380 INFO 2023-01-02 17:08:09,638 INFO 2023-01-02 17:08:09,638 INFO 2023-01-02 17:08:09,638 INFO 2023-01-02 17:08:09,638 INFO 2023-01-02 17:08:09,640 INFO 2023-01-02 17:08:09,640 INFO 2023-01-02 17:08:09,640 INFO 2023-01-02 17:08:20,386 INFO 2023-01-02 17:08:20,389 INFO 2023-01-02 17:08:20,389 INFO 2023-01-02 17:08:21,422 INFO 2023-01-02 17:08:21,446 INFO 2023-01-02 17:08:21,446 INFO 2023-01-02 17:08:22,609 INFO 2023-01-02 17:08:22,612 INFO 2023-01-02 17:08:22,612 INFO 2023-01-02 17:08:23,625 INFO 2023-01-02 17:08:28,241 INFO 2023-01-02 17:08:28,241 INFO 2023-01-02 17:08:28,349 INFO 2023-01-02 17:08:28,360 INFO 2023-01-02 17:08:28,360 INFO 2023-01-02 17:08:28,410 INFO 2023-01-02 17:08:28,413 INFO 2023-01-02 17:08:28,413 INFO 2023-01-02 17:08:29,158 INFO 2023-01-02 17:12:21,179 INFO 2023-01-02 17:12:21,179 INFO 2023-01-02 17:12:21,272 INFO 2023-01-02 17:12:31,453 INFO 2023-01-02 17:12:31,453 INFO 2023-01-02 17:12:31,531 INFO 2023-01-02 17:12:31,899 INFO 2023-01-02 17:12:31,899 INFO 2023-01-02 17:12:31,993 INFO 2023-01-02 17:12:31,993 INFO 2023-01-02 17:12:31,993 INFO 2023-01-02 17:12:32,009 INFO 2023-01-02 17:12:32,009 INFO 2023-01-02 17:12:32,009 INFO 2023-01-02 17:12:32,009 INFO 2023-01-02 17:12:32,009 INFO MainThread:15880 [wandb_setup.py:_flush():68] Configure stats pid to 15880
MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from C:\Users\Vinc.config\wandb\settings
MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\settings
MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from environment variables: {‘_require_service’: ‘True’}
MainThread:15880 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {‘program’: ‘’}
MainThread:15880 [wandb_init.py:_log_setup():476] Logging user logs to d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\run-20230102_170808-2gezouc7\logs\debug.log
MainThread:15880 [wandb_init.py:_log_setup():477] Logging internal logs to d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\run-20230102_170808-2gezouc7\logs\debug-internal.log
MainThread:15880 [wandb_init.py:_jupyter_setup():426] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x0000023EB863CAF0>
MainThread:15880 [wandb_init.py:init():516] calling init triggers
MainThread:15880 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
MainThread:15880 [wandb_init.py:init():569] starting backend
MainThread:15880 [wandb_init.py:init():573] setting up manager
MainThread:15880 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=spawn, using: spawn
MainThread:15880 [wandb_init.py:init():580] backend started and connected
MainThread:15880 [wandb_run.py:_label_probe_notebook():1116] probe notebook
MainThread:15880 [wandb_run.py:_label_probe_notebook():1126] Unable to probe notebook: ‘NoneType’ object has no attribute ‘get’
MainThread:15880 [wandb_init.py:init():658] updated telemetry
MainThread:15880 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
MainThread:15880 [wandb_run.py:_on_init():2006] communicating current version
MainThread:15880 [wandb_run.py:_on_init():2010] got version response
MainThread:15880 [wandb_init.py:init():728] starting run threads in backend
MainThread:15880 [wandb_run.py:_console_start():1986] atexit reg
MainThread:15880 [wandb_run.py:_redirect():1844] redirect: SettingsConsole.WRAP_RAW
MainThread:15880 [wandb_run.py:_redirect():1909] Wrapping output streams.
MainThread:15880 [wandb_run.py:_redirect():1931] Redirects installed.
MainThread:15880 [wandb_init.py:init():765] run started, returning control to user process
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
MainThread:15880 [wandb_run.py:_finish():1753] finishing run vincpc/mlops-course-001/2gezouc7
MainThread:15880 [jupyter.py:save_history():448] not saving jupyter history
MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
MainThread:15880 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
MainThread:15880 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
MainThread:15880 [wandb_run.py:_restore():1938] restore
MainThread:15880 [wandb_run.py:_restore():1944] restore done

VS Code is not using any Network at all, so it’s not Uploading anymore.

system · March 3, 2023, 5:39pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Taking forever to finish after Waiting for W&B process to finish... (success) W&B Help wandb	8	3594	September 8, 2023
Waiting for W&B process to finish… (success) W&B Help sweeps , wandb	6	876	January 12, 2024
Waiting for W&B process to finish (success) W&B Help dashboard , projects , questions , wandb , beginner-friendly	4	1416	September 26, 2022
Wandb takes too much time after each run ends W&B Help sweeps	6	1171	October 25, 2022
Wandb.finish() takes too long to finish W&B Help wandb	2	761	July 16, 2023

Waiting for W&B process to finish... (success)

Related topics