Waiting for W&B process to finish... (success)

I am using wandb to log Tensorflow model training. Unfortunately the runs keep on running forever even though the training has finished minutes before.

I am receiving a lot of the following debug messages in debug-internals.log:

2022-12-07 21:19:47,595 DEBUG   HandlerThread:796 [handler.py:handle_request():139] handle_request: keepalive
```

debug.log  is already finished with the following last messages:
```
2022-12-07 21:09:05,301 INFO    MainThread:12168 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'session_history') code\_session_history.ipynb None
2022-12-07 21:09:05,323 INFO    MainThread:12168 [jupyter.py:_save_ipynb():389] looking for notebook: None
2022-12-07 21:09:05,323 INFO    MainThread:12168 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
2022-12-07 21:09:05,323 INFO    MainThread:12168 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
2022-12-07 21:09:05,324 INFO    MainThread:12168 [wandb_run.py:_restore():1938] restore
2022-12-07 21:09:05,325 INFO    MainThread:12168 [wandb_run.py:_restore():1944] restore done
```

I have already updated to wandb-013.6 but this did not solve the issues.

I also encountered this problem, but it has not been resolved.

1 Like
object at 0x000001B3207410A0>>, args: ('https://api.wandb.ai/files/liuye/Image_Enhancement/1nykox3a/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}}
2022-12-08 15:56:47,968 WARNING FileStreamThread:7688 [file_stream.py:request_with_retry():667] requests_with_retry encountered retryable exception: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /files/liuye/Image_Enhancement/1nykox3a/file_stream (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)')))). func: <bound method Session.post of <requests.sessions.Session object at 0x000001B3207410A0>>, args: ('https://api.wandb.ai/files/liuye/Image_Enhancement/1nykox3a/file_stream',), kwargs: {'json': {'complete': False, 'failed': False, 'dropped': 0, 'uploaded': []}}

Blockquote

Hi @susbrock @liuyee, thanks for reporting this! I’ll try to reproduce this issue and see what’s happening here but, in the meantime, there are some points that will be helpful:

  • Are you using Tensorboard in your experiment?
  • Could you confirm me if your code is something like:
wandb.init()
##Create and train tf model
wandb.log({metrics})
  • If you aren’t doing it yet, could you try to assign the run to a variable like run = wandb.init() and call run.finish() ate the end of your experiment?

Thanks!

Hi @susbrock @liuyee, thanks for reporting this! I’ll try to reproduce this issue and see what’s happening here but, in the meantime, there are some points that will be helpful:

  • Are you using Tensorboard in your experiment?

  • Could you confirm me if your code is something like:

    wandb.init()
    ##Create and train tf model
    wandb.log({metrics})

  • If you aren’t doing it yet, could you try to assign the run to a variable like run = wandb.init() and call run.finish() ate the end of your experiment?

Thanks!

[Discourse post]

  • Are you using Tensorboard in your experiment? A: Yes.
  • Could you confirm me if your code is something like that? A: i don’t use run.finish().

Thanks for confirming this liuyee, could you try using run.finish() and see if the same issue appears?


It still does not work.

@luis_bergua1 I am using Tensorboard. I am using

wandb.init(....)
...

wandb.finish()

because I am running my code in jupyter notebook for VS code.

Last week I was struggling with wandb but currently it works fine. I have add “run = wandb.init(…)” sometime during the last days so that I cannot say if this is the important part. I am still finishing my code with “wandb.finish()”. I had picked up the “run = wand.init(…)” implementation while I was looking for a solution to resume runs…

Hi @susbrock @liuyee, thanks for your answers! I would highly recommend you to always assign wandb.init() to a variable since it can avoid errors when running multiple experiments. Have you experienced this issue again? If so, would you mind sharing with me the debug.log and debug-internal.log files under the local wandb folder? Thanks!

@luis_bergua1 The problem is reappearing right now. Therefore I am posting the debug files.

debug.log

2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Configure stats pid to 1764
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from C:\Users\Susanne\.config\wandb\settings
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from i:\tinyml\tiny_cnn\wandb\settings
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Loading settings from environment variables: {'_require_service': 'True', 'mode': 'online'}
2022-12-25 10:24:34,055 INFO    MainThread:1764 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {'program': '<python with no main file>'}
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_log_setup():476] Logging user logs to i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\logs\debug.log
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_log_setup():477] Logging internal logs to i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\logs\debug-internal.log
2022-12-25 10:24:34,056 INFO    MainThread:1764 [wandb_init.py:_jupyter_setup():426] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x000001FC77A5D900>
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():516] calling init triggers
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {}
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():569] starting backend
2022-12-25 10:24:34,057 INFO    MainThread:1764 [wandb_init.py:init():573] setting up manager
2022-12-25 10:24:34,060 INFO    MainThread:1764 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=spawn, using: spawn
2022-12-25 10:24:34,065 INFO    MainThread:1764 [wandb_init.py:init():580] backend started and connected
2022-12-25 10:24:34,080 INFO    MainThread:1764 [wandb_run.py:_label_probe_notebook():1116] probe notebook
2022-12-25 10:24:34,083 INFO    MainThread:1764 [wandb_run.py:_label_probe_notebook():1126] Unable to probe notebook: 'NoneType' object has no attribute 'get'
2022-12-25 10:24:34,083 INFO    MainThread:1764 [wandb_init.py:init():658] updated telemetry
2022-12-25 10:24:34,183 INFO    MainThread:1764 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2022-12-25 10:24:35,152 INFO    MainThread:1764 [wandb_run.py:_on_init():2006] communicating current version
2022-12-25 10:24:35,417 INFO    MainThread:1764 [wandb_run.py:_on_init():2010] got version response 
2022-12-25 10:24:35,418 INFO    MainThread:1764 [wandb_init.py:init():728] starting run threads in backend
2022-12-25 10:24:40,442 INFO    MainThread:1764 [wandb_run.py:_console_start():1986] atexit reg
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1844] redirect: SettingsConsole.WRAP_RAW
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1909] Wrapping output streams.
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_run.py:_redirect():1931] Redirects installed.
2022-12-25 10:24:40,443 INFO    MainThread:1764 [wandb_init.py:init():765] run started, returning control to user process
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set batch_size = 32 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb batch_size 32 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set learn_rate = 0.001 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb learn_rate 0.001 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set momentum = 0.9 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb momentum 0.9 None
2022-12-25 10:24:40,444 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set epochs = 5 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb epochs 5 None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set classes = 3 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb classes 3 None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set id = 2dyabu0c - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb id 2dyabu0c None
2022-12-25 10:24:40,445 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set architecture = efficientNetB0_0.1_96_c3_o3_keras - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb architecture efficientNetB0_0.1_96_c3_o3_keras None
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_config.py:__setitem__():155] config set optimizer = SGD - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x000001FC452EB040>>
2022-12-25 10:24:40,446 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb optimizer SGD None
2022-12-25 10:24:40,536 INFO    MainThread:1764 [wandb_run.py:_tensorboard_callback():1299] tensorboard callback: i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\files\train, True
2022-12-25 10:25:31,797 INFO    MainThread:1764 [wandb_run.py:_tensorboard_callback():1299] tensorboard callback: i:\tinyml\tiny_cnn\wandb\run-20221225_102434-2dyabu0c\files\validation, True
2022-12-25 10:28:02,476 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'visualize', 'conf_mat') {'panel_type': 'Vega2', 'panel_config': {'panelDefId': 'wandb/confusion_matrix/v1', 'fieldSettings': {'Actual': 'Actual', 'Predicted': 'Predicted', 'nPredictions': 'nPredictions'}, 'stringSettings': {'title': ''}, 'transform': {'name': 'tableWithLeafColNames'}, 'userQuery': {'queryFields': [{'name': 'runSets', 'args': [{'name': 'runSets', 'value': '${runSets}'}], 'fields': [{'name': 'id', 'fields': []}, {'name': 'name', 'fields': []}, {'name': '_defaultColorIndex', 'fields': []}, {'name': 'summaryTable', 'args': [{'name': 'tableKey', 'value': 'conf_mat_table'}], 'fields': []}]}]}}} None
2022-12-25 10:28:03,896 INFO    MainThread:1764 [wandb_run.py:_finish():1753] finishing run susbrock/efficientNetB0/2dyabu0c
2022-12-25 10:28:03,980 INFO    MainThread:1764 [jupyter.py:save_history():477] saving 27 cells to _session_history.ipynb
2022-12-25 10:28:03,980 INFO    MainThread:1764 [wandb_run.py:_config_callback():1163] config_cb ('_wandb', 'session_history') code\_session_history.ipynb None
2022-12-25 10:28:04,018 INFO    MainThread:1764 [jupyter.py:_save_ipynb():389] looking for notebook: None
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_restore():1938] restore
2022-12-25 10:28:04,018 INFO    MainThread:1764 [wandb_run.py:_restore():1944] restore done

debug-internal.log will follow in separate message.
Thanks for your support.

I do have a similar issue, Running the code from de MLOps course 01_EDA.ipynb

The notebook basicaly does:
start a run with run = wandb.init()
create an antifact and store a Table.
but when I run run.finish(), it keeps going for ever and never finish running the cell
(I’m using VS Code Jupyter notebook), running the same code as it is in the course Effective MLOps: Model Development, processing data for exploratory analysis.


UPDATE:

Running the code using Colab works just fine.
Still would like to work in my local PC
My internet speed last test is 200/111 Mb download/upload so it’s not a connection problem.


this is the log I’m having (debug-internal.log):

2023-01-02 17:18:30,630 INFO Thread-3998:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\f4\d1e18a269ee5ea2dfe1f31f62ace7e
2023-01-02 17:18:30,635 INFO Thread-3975:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ef\805451c61c3548e0b64d7cd43fb495
2023-01-02 17:18:30,637 INFO Thread-3983:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\f1\159ed57aec7385894be1f50ff3db6b
2023-01-02 17:18:30,641 INFO Thread-3974:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ef\40545a3365286db74b9eec276aafe5
2023-01-02 17:18:30,662 INFO Thread-3962:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\ec\5bb019ae113d7a08c9bb6eb9ac6fe8
2023-01-02 17:18:30,773 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:31,775 INFO Thread-4012:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fa\336027db262331cf3de1c2bc489161
2023-01-02 17:18:31,779 INFO Thread-4023:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fd\72fdd8e9b81196f34a27936872ec85
2023-01-02 17:18:31,802 INFO Thread-4026:6476 [upload_job.py:push():101] Uploaded file C:\Users\Vinc.cache\wandb\artifacts\obj\md5\fe\23e9669b121a89dd294fbcd063240e
2023-01-02 17:18:35,832 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:40,870 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:45,886 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:50,958 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:18:55,987 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:01,020 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:06,043 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:11,062 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:16,108 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:21,124 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:26,138 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:31,175 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:36,203 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:41,221 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:46,246 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:51,287 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:19:56,303 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:01,336 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:06,359 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:11,360 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive
2023-01-02 17:20:16,378 DEBUG HandlerThread:6476 [handler.py:handle_request():139] handle_request: keepalive

This last line is still being printed…



Log (debug.log):

2023-01-02 17:08:08,443 INFO MainThread:15880 [wandb_setup.py:_flush():68] Configure stats pid to 15880
2023-01-02 17:08:08,443 INFO MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from C:\Users\Vinc.config\wandb\settings
2023-01-02 17:08:08,444 INFO MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\settings
2023-01-02 17:08:08,444 INFO MainThread:15880 [wandb_setup.py:_flush():68] Loading settings from environment variables: {‘_require_service’: ‘True’}
2023-01-02 17:08:08,444 INFO MainThread:15880 [wandb_setup.py:_flush():68] Inferring run settings from compute environment: {‘program’: ‘’}
2023-01-02 17:08:08,444 INFO MainThread:15880 [wandb_init.py:_log_setup():476] Logging user logs to d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\run-20230102_170808-2gezouc7\logs\debug.log
2023-01-02 17:08:08,445 INFO MainThread:15880 [wandb_init.py:_log_setup():477] Logging internal logs to d:\GitHub\Repositories\MLOps\mlops-001\lesson1\wandb\run-20230102_170808-2gezouc7\logs\debug-internal.log
2023-01-02 17:08:08,445 INFO MainThread:15880 [wandb_init.py:_jupyter_setup():426] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x0000023EB863CAF0>
2023-01-02 17:08:08,445 INFO MainThread:15880 [wandb_init.py:init():516] calling init triggers
2023-01-02 17:08:08,445 INFO MainThread:15880 [wandb_init.py:init():519] wandb.init called with sweep_config: {}
config: {}
2023-01-02 17:08:08,445 INFO MainThread:15880 [wandb_init.py:init():569] starting backend
2023-01-02 17:08:08,446 INFO MainThread:15880 [wandb_init.py:init():573] setting up manager
2023-01-02 17:08:08,449 INFO MainThread:15880 [backend.py:_multiprocessing_setup():102] multiprocessing start_methods=spawn, using: spawn
2023-01-02 17:08:08,454 INFO MainThread:15880 [wandb_init.py:init():580] backend started and connected
2023-01-02 17:08:08,468 INFO MainThread:15880 [wandb_run.py:_label_probe_notebook():1116] probe notebook
2023-01-02 17:08:08,470 INFO MainThread:15880 [wandb_run.py:_label_probe_notebook():1126] Unable to probe notebook: ‘NoneType’ object has no attribute ‘get’
2023-01-02 17:08:08,470 INFO MainThread:15880 [wandb_init.py:init():658] updated telemetry
2023-01-02 17:08:08,513 INFO MainThread:15880 [wandb_init.py:init():693] communicating run to backend with 60 second timeout
2023-01-02 17:08:09,065 INFO MainThread:15880 [wandb_run.py:_on_init():2006] communicating current version
2023-01-02 17:08:09,380 INFO MainThread:15880 [wandb_run.py:_on_init():2010] got version response
2023-01-02 17:08:09,380 INFO MainThread:15880 [wandb_init.py:init():728] starting run threads in backend
2023-01-02 17:08:09,638 INFO MainThread:15880 [wandb_run.py:_console_start():1986] atexit reg
2023-01-02 17:08:09,638 INFO MainThread:15880 [wandb_run.py:_redirect():1844] redirect: SettingsConsole.WRAP_RAW
2023-01-02 17:08:09,638 INFO MainThread:15880 [wandb_run.py:_redirect():1909] Wrapping output streams.
2023-01-02 17:08:09,638 INFO MainThread:15880 [wandb_run.py:_redirect():1931] Redirects installed.
2023-01-02 17:08:09,640 INFO MainThread:15880 [wandb_init.py:init():765] run started, returning control to user process
2023-01-02 17:08:09,640 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:09,640 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:20,386 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:20,389 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:20,389 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:21,422 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:21,446 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:21,446 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:22,609 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:22,612 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:22,612 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:23,625 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:28,241 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:28,241 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:28,349 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:28,360 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:28,360 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:28,410 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:08:28,413 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:08:28,413 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:08:29,158 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:12:21,179 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:12:21,179 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:12:21,272 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:12:31,453 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:12:31,453 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:12:31,531 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:12:31,899 INFO MainThread:15880 [wandb_init.py:_pause_backend():389] pausing backend
2023-01-02 17:12:31,899 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:12:31,993 INFO MainThread:15880 [wandb_init.py:_resume_backend():398] resuming backend
2023-01-02 17:12:31,993 INFO MainThread:15880 [wandb_run.py:_finish():1753] finishing run vincpc/mlops-course-001/2gezouc7
2023-01-02 17:12:31,993 INFO MainThread:15880 [jupyter.py:save_history():448] not saving jupyter history
2023-01-02 17:12:32,009 INFO MainThread:15880 [jupyter.py:save_ipynb():378] not saving jupyter notebook
2023-01-02 17:12:32,009 INFO MainThread:15880 [wandb_init.py:_jupyter_teardown():408] cleaning up jupyter logic
2023-01-02 17:12:32,009 INFO MainThread:15880 [wandb_run.py:_atexit_cleanup():1955] got exitcode: 0
2023-01-02 17:12:32,009 INFO MainThread:15880 [wandb_run.py:_restore():1938] restore
2023-01-02 17:12:32,009 INFO MainThread:15880 [wandb_run.py:_restore():1944] restore done

VS Code is not using any Network at all, so it’s not Uploading anymore.