ProxyError with wandb in VSCode Jupyter-notebook when using a US proxy via v2rayn

740840403 · October 17, 2023, 12:39pm

Hello,

I’m encountering an issue when using the wandb library within a Jupyter-notebook on VSCode. I’m based in China, and everything works fine when I’m not using a proxy. However, I need to use a US proxy via v2rayn to access certain services, such as ChatGPT. When the proxy is active, wandb throws an error stating “wandb: Network error (ProxyError), entering retry loop.”, and the wandb.init() process never completes.

Any assistance or insights into this issue would be greatly appreciated.

Thanks.

carlo-catimbang · October 19, 2023, 6:07am

Hi there,

Thank you for reaching out for support. I’ll be glad to assist you with your concern. There seems to be a Proxy that is blocking the connection between your client and the wandb server. so I’ll take a deeper look into this. Can you provide the following please:

A sample of your code snippet to fully understand your workflow
Link to your workspace where you are encountering the error if available
the debug.log and debug.internal.log for the affected run. These are located in your wandb working directory under wandb//logs

Regards,
Carlo Argel

740840403 · October 20, 2023, 8:22am

this is debug-internal.log

2023-10-20 16:16:30,955 INFO    StreamThr :37552 [internal.py:wandb_internal():86] W&B internal server running at pid: 37552, started at: 2023-10-20 16:16:30.955004
2023-10-20 16:16:30,956 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status
2023-10-20 16:16:30,962 INFO    WriterThread:37552 [datastore.py:open_for_write():85] open: c:\code\myaudio\augement_train\wandb\run-20231020_161630-29u4crzy\run-29u4crzy.wandb
2023-10-20 16:16:30,962 DEBUG   SenderThread:37552 [sender.py:send():380] send: header
2023-10-20 16:16:30,963 DEBUG   SenderThread:37552 [sender.py:send():380] send: run
2023-10-20 16:16:34,207 INFO    SenderThread:37552 [retry.py:__call__():172] Retry attempt failed:
Traceback (most recent call last):
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\connectionpool.py", line 996, in _prepare_proxy
    conn.connect()
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\connection.py", line 364, in connect
    self.sock = conn = self._connect_tls_proxy(hostname, conn)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\connection.py", line 499, in _connect_tls_proxy
    socket = ssl_wrap_socket(
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\util\ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\util\ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "d:\yangshu_tem\anaconda\envs\audio\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "d:\yangshu_tem\anaconda\envs\audio\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\lib\retry.py", line 131, in __call__
    result = self._call_fn(*args, **kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\internal\internal_api.py", line 366, in execute
    return self.client.execute(*args, **kwargs)  # type: ignore
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\vendor\gql-0.2.0\wandb_gql\client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\vendor\gql-0.2.0\wandb_gql\client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\lib\gql_request.py", line 58, in execute
    request = self.session.post(self.url, **post_args)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\requests\sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\requests\adapters.py", line 513, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))
2023-10-20 16:16:36,016 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:16:41,051 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:16:46,093 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:16:51,120 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:16:56,145 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:01,176 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:06,222 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:11,267 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:16,299 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:21,344 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:26,388 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:31,426 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:36,452 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:41,481 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:46,526 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:51,553 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:17:56,606 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: keepalive
2023-10-20 16:18:01,668 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: cancel
2023-10-20 16:18:01,668 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: cancel
2023-10-20 16:18:01,668 DEBUG   SenderThread:37552 [sender.py:send():389] Record cancelled: run
2023-10-20 16:18:01,669 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:06,694 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:11,723 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:16,751 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:21,780 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:26,826 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:31,864 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:36,893 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:41,928 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:46,953 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:52,008 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:18:57,051 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:02,106 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:07,155 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:12,179 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:17,234 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:22,277 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:27,299 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:32,360 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:37,425 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:42,449 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report
2023-10-20 16:19:47,466 DEBUG   HandlerThread:37552 [handler.py:handle_request():144] handle_request: status_report

this is debug-log

2023-10-20 16:16:30,951 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Current SDK version is 0.15.11
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Configure stats pid to 28292
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Loading settings from C:\Users\aaa\.config\wandb\settings
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Loading settings from c:\code\myaudio\augement_train\wandb\settings
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Loading settings from environment variables: {'api_key': '***REDACTED***'}
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program': '<python with no main file>'}
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:_log_setup():528] Logging user logs to c:\code\myaudio\augement_train\wandb\run-20231020_161630-29u4crzy\logs\debug.log
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:_log_setup():529] Logging internal logs to c:\code\myaudio\augement_train\wandb\run-20231020_161630-29u4crzy\logs\debug-internal.log
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:_jupyter_setup():474] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x00000138B336D520>
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:init():568] calling init triggers
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:init():575] wandb.init called with sweep_config: {}
config: {'learning_rate': 0.001, 'dataset_id': 'us8k', 'seed': 52, 'batch_size': 128}
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:init():618] starting backend
2023-10-20 16:16:30,952 INFO    MainThread:28292 [wandb_init.py:init():622] setting up manager
2023-10-20 16:16:30,954 INFO    MainThread:28292 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=spawn, using: spawn
2023-10-20 16:16:30,956 INFO    MainThread:28292 [wandb_init.py:init():628] backend started and connected
2023-10-20 16:16:30,961 INFO    MainThread:28292 [wandb_run.py:_label_probe_notebook():1276] probe notebook
2023-10-20 16:16:30,961 INFO    MainThread:28292 [wandb_run.py:_label_probe_notebook():1286] Unable to probe notebook: 'NoneType' object has no attribute 'get'
2023-10-20 16:16:30,961 INFO    MainThread:28292 [wandb_init.py:init():720] updated telemetry
2023-10-20 16:16:30,962 INFO    MainThread:28292 [wandb_init.py:init():753] communicating run to backend with 90.0 second timeout
2023-10-20 16:18:01,668 ERROR   MainThread:28292 [wandb_init.py:init():779] encountered error: Run initialization has timed out after 90.0 sec. 
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
2023-10-20 16:18:01,746 ERROR   MainThread:28292 [wandb_init.py:init():1188] Run initialization has timed out after 90.0 sec. 
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
Traceback (most recent call last):
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\wandb_init.py", line 1170, in init
    run = wi.init()
  File "d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\wandb_init.py", line 785, in init
    raise error
wandb.errors.CommError: Run initialization has timed out after 90.0 sec. 
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-

740840403 · October 20, 2023, 8:25am

“This is where I call the wandb library; the rest of the code involves some neural network configurations.”

        os.environ["WANDB_API_KEY"] = '5dc260112e52b18f65c4c032b9285f48f4a35b2e'  
        config = dict (
        learning_rate = 0.001,
        dataset_id = "us8k",
        seed = 52,
        batch_size = 128
        )
        wandb.init(
            project='us8k_origin',
            notes=name,
            tags=["baseline", model_name],
            config=config,
            name = name
        )

740840403 · October 20, 2023, 8:27am

this is the error information

Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Network error (ProxyError), entering retry loop.
wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin
Waiting for wandb.init()...
Problem at: C:\Users\aaa\AppData\Local\Temp\ipykernel_28292\1251992745.py 32 <module>
---------------------------------------------------------------------------
CommError                                 Traceback (most recent call last)
Cell In[19], line 32
     25 os.environ["WANDB_API_KEY"] = '5dc260112e52b18f65c4c032b9285f48f4a35b2e'  
     26 config = dict (
     27 learning_rate = 0.001,
     28 dataset_id = "us8k",
     29 seed = 52,
     30 batch_size = 128
     31 )
---> 32 wandb.init(
     33     project='us8k_origin',
     34     notes=name,
     35     tags=["baseline", model_name],
     36     config=config,
     37     name = name
     38 )
     39 # if CONTINUE:
     40 #     #start = START
     41 #     model.load_state_dict(torch.load("./models/cutonly_se_resnext10_32x4d_cutmix_236_29epochs_saved_weights.pth"))
     44 train_input, test_input, train_label, test_label = esc_split(fold)

File d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\wandb_init.py:1189, in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1187     if logger is not None:
   1188         logger.exception(str(e))
-> 1189     raise e
   1190 except KeyboardInterrupt as e:
   1191     assert logger

File d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\wandb_init.py:1170, in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1168 except_exit = wi.settings._except_exit
   1169 try:
-> 1170     run = wi.init()
   1171     except_exit = wi.settings._except_exit
   1172 except (KeyboardInterrupt, Exception) as e:

File d:\yangshu_tem\anaconda\envs\audio\lib\site-packages\wandb\sdk\wandb_init.py:785, in _WandbInit.init(self)
    783         backend.cleanup()
    784         self.teardown()
--> 785     raise error
    787 assert run_result is not None  # for mypy
    789 if not run_result.HasField("run"):

CommError: Run initialization has timed out after 90.0 sec. 
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-

carlo-catimbang · October 24, 2023, 10:21pm

Hi there,

Can you try running on offline mode then sync your run to wandb? Troubleshooting | Weights & Biases Documentation
Regards,
Carlo Argel

carlo-catimbang · October 31, 2023, 10:09am

Hi @740840403 , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

system · December 30, 2023, 10:09am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Wandb: Network error (ProxyError), entering retry loop W&B Help wandb	3	1964	February 26, 2023
WanDB connection error W&B Help	10	1933	August 26, 2023
Run initialization has timed out W&B Help	11	164	July 10, 2024
Using WandB in Visual Studio Code W&B Help	3	2037	March 31, 2023
ConnectionRefusedError W&B Help wandb	3	594	November 28, 2022

ProxyError with wandb in VSCode Jupyter-notebook when using a US proxy via v2rayn

Related topics