Wandb Comm Error

W&B Help
,
1

Hi I am facing below problem while trying to run sweeps using wandb.

wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: ERROR Error uploading “code/wandb_hyperparameter.py”: CommError, <Response [400]>
wandb: ERROR Error uploading “wandb-metadata.json”: CommError, <Response [400]>
wandb: ERROR Error uploading “wandb-summary.json”: CommError, <Response [400]>
wandb: ERROR Error uploading “config.yaml”: CommError, <Response [400]>
wandb: ERROR Error uploading “requirements.txt”: CommError, <Response [400]>

Please help.

2

Hi @mithileshia,

I’m sorry you are facing this issue. Could you share the debug.log and debug-internal.log files associated with one of the runs which displays this error? It would be very helpful in order to gain more visibility into this error and understand why you are seeing CommErrors here.

Thanks,
Ramit

3

Hi Ramit sharing the debug.log and debug-internal.log files with you.

debug.log

2022-03-16 10:23:25,658 INFO Thread-11 :28804 [wandb_init.py:_log_setup():293] Logging user logs to /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/logs/debug.log
2022-03-16 10:23:25,658 INFO Thread-11 :28804 [wandb_init.py:_log_setup():294] Logging internal logs to /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/logs/debug-internal.log
2022-03-16 10:23:25,658 INFO Thread-11 :28804 [wandb_setup.py:_flush():69] setting env: {‘api_key’: ‘xxxxxxxxxxxxxxxxx’, ‘project’: ‘blstm’, ‘entity’: ‘mithileshia’, ‘root_dir’: ‘/home/demo_user2/Mithilesh/BayesianLSTM_W&B’, ‘sweep_id’: ‘lndyf4e1’, ‘run_id’: ‘g3s746w2’, ‘config_paths’: ‘/home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/sweep-lndyf4e1/config-g3s746w2.yaml’}
2022-03-16 10:23:25,658 INFO Thread-11 :28804 [wandb_setup.py:_flush():69] setting user settings: {‘save_code’: True, ‘email’: ‘xxxxxxxxxx@gmail.co’}
2022-03-16 10:23:25,658 INFO Thread-11 :28804 [wandb_setup.py:_flush():69] multiprocessing start_methods=fork,spawn,forkserver
2022-03-16 10:23:26,598 INFO Thread-11 :28804 [wandb_run.py:_console_start():1290] atexit reg
2022-03-16 10:23:26,599 INFO Thread-11 :28804 [wandb_run.py:_redirect():1160] redirect: SettingsConsole.REDIRECT
2022-03-16 10:23:26,600 INFO Thread-11 :28804 [wandb_run.py:_redirect():1163] Redirecting console.
2022-03-16 10:23:26,600 INFO Thread-11 :28804 [redirect.py:install():196] install start
2022-03-16 10:23:26,600 INFO Thread-11 :28804 [redirect.py:install():211] install stop
2022-03-16 10:23:26,600 INFO Thread-11 :28804 [redirect.py:install():196] install start
2022-03-16 10:23:26,602 INFO Thread-11 :28804 [redirect.py:install():211] install stop
2022-03-16 10:23:26,602 INFO Thread-11 :28804 [wandb_run.py:_redirect():1207] Redirects installed.
2022-03-16 10:23:26,614 INFO Thread-11 :28804 [wandb_run.py:finish():988] finishing run mithileshia/blstm/g3s746w2

debug-internal.log

2022-03-16 10:23:26,477 INFO MainThread:28972 [internal.py:wandb_internal():62] W&B internal server running at pid: 28972
2022-03-16 10:23:26,480 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: check_version
2022-03-16 10:23:26,480 INFO WriterThread:28972 [datastore.py:open_for_write():76] open: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/run-g3s746w2.wandb
2022-03-16 10:23:26,481 DEBUG SenderThread:28972 [sender.py:send():89] send: header
2022-03-16 10:23:26,481 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,481 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: check_version
2022-03-16 10:23:26,483 DEBUG Thread-4 :28972 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1):2022-03-16 10:23:26,529 DEBUG Thread-4 :28972 [connectionpool.py:_make_request():433] “GET /pypi/wandb/json HTTP/1.1” 200 62723
2022-03-16 10:23:26,550 DEBUG SenderThread:28972 [sender.py:send():89] send: run
2022-03-16 10:23:26,554 DEBUG SenderThread:28972 [git.py:repo():30] git repository is invalid

2022-03-16 10:23:26,611 DEBUG Thread-11 :28972 [connectionpool.py:_new_conn():226] Starting new HTTP connection (1): localhost: 8080
2022-03-16 10:23:26,613 DEBUG Thread-11 :28972 [connectionpool.py:_make_request():433] http: // localhost : 8080 “PUT //local-files/mithileshia/blstm/g3s746w2/wandb-metadata.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=11545c50288dbb6a0b2ec1f09c833776f78f73ec3a4b5742d55d589d6c41713e HTTP/1.1” 400 406
2022-03-16 10:23:26,614 ERROR Thread-11 :28972 [internal_api.py:upload_file():1210] upload_file exception http: // localhost :8080 //local-files/mithileshia/blstm/g3s746w2/wandb-metadata.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=11545c50288dbb6a0b2ec1f09c833776f78f73ec3a4b5742d55d589d6c41713e 400 Client Error: Bad Request for url: http:// localhost:8080//local-files/mithileshia/blstm/g3s746w2/wandb-metadata.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=11545c50288dbb6a0b2ec1f09c833776f78f73ec3a4b5742d55d589d6c41713e
2022-03-16 10:23:26,615 DEBUG SenderThread:28972 [connectionpool.py:_make_request():433] http : //l ocalhost:8080 “POST /graphql HTTP/1.1” 200 65
2022-03-16 10:23:26,615 DEBUG Thread-12 :28972 [connectionpool.py:_make_request():433] localhost:8080 “POST /graphql HTTP/1.1” 200 431
2022-03-16 10:23:26,617 DEBUG SenderThread:28972 [sender.py:send():89] send: run
2022-03-16 10:23:26,623 DEBUG Thread-12 :28972 [connectionpool.py:_new_conn():226] Starting new HTTP connection (1): localhost:8080
2022-03-16 10:23:26,626 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: poll_exit
2022-03-16 10:23:26,626 DEBUG Thread-12 :28972 [connectionpool.py:_make_request():433] // localhost:8080 “PUT //local-files/mithileshia/blstm/g3s746w2/code/wandb_hyperparameter.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=ba1e408c529f3e8f79ad2150ee5468f05c21064c63b7e20e1a5e3a97b0bf9bd6 HTTP/1.1” 400 424
2022-03-16 10:23:26,627 ERROR Thread-12 :28972 [internal_api.py:upload_file():1210] upload_file exception http:// localhost:8080//local-files/mithileshia/blstm/g3s746w2/code/wandb_hyperparameter.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=ba1e408c529f3e8f79ad2150ee5468f05c21064c63b7e20e1a5e3a97b0bf9bd6 400 Client Error: Bad Request for url: http: // localhost: 8080//local-files/mithileshia/blstm/g3s746w2/code/wandb_hyperparameter.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=ba1e408c529f3e8f79ad2150ee5468f05c21064c63b7e20e1a5e3a97b0bf9bd6
2022-03-16 10:23:26,627 ERROR Thread-11 :28972 [upload_job.py:push():115] Failed to upload file: /tmp/tmpc_2asjh8wandb/gc64ldxd-wandb-metadata.json
Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 24, in wrapper
return func(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 132, in wrapped_fn
return retrier(*args, **kargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 96, in call
result = self._call_fn(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1220, in upload_file
util.sentry_reraise(e)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/util.py”, line 95, in sentry_reraise
six.reraise(type(exc), exc, sys.exc_info()[2])
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/six.py”, line 703, in reraise
raise value
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1208, in upload_file
response.raise_for_status()
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/requests/models.py”, line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http: // localhost:8080//local-files/mithileshia/blstm/g3s746w2/wandb-metadata.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=11545c50288dbb6a0b2ec1f09c833776f78f73ec3a4b5742d55d589d6c41713e

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/filesync/upload_job.py”, line 107, in push
self._api.upload_file_retry(
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 26, in wrapper
raise CommError(err.response, err)
wandb.errors.error.CommError: <Response [400]>
2022-03-16 10:23:26,629 DEBUG raven-sentry.BackgroundWorker:28972 [connectionpool.py:_new_conn():939] Starting new HTTPS connection (1): o151352.ingest.sentry.io:443
2022-03-16 10:23:26,631 ERROR Thread-12 :28972 [upload_job.py:push():115] Failed to upload file: /tmp/tmpc_2asjh8wandb/brxvnwm4-code/wandb_hyperparameter.py
Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 24, in wrapper
return func(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 132, in wrapped_fn
return retrier(*args, **kargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 96, in call
result = self._call_fn(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1220, in upload_file
util.sentry_reraise(e)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/util.py”, line 95, in sentry_reraise
six.reraise(type(exc), exc, sys.exc_info()[2])
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/six.py”, line 703, in reraise
raise value
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1208, in upload_file
response.raise_for_status()
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/requests/models.py”, line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http: // localhost: 8080//local-files/mithileshia/blstm/g3s746w2/code/wandb_hyperparameter.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102326Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=ba1e408c529f3e8f79ad2150ee5468f05c21064c63b7e20e1a5e3a97b0bf9bd6

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/filesync/upload_job.py”, line 107, in push
self._api.upload_file_retry(
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 26, in wrapper
raise CommError(err.response, err)
wandb.errors.error.CommError: <Response [400]>
2022-03-16 10:23:26,631 DEBUG SenderThread:28972 [git.py:repo():30] git repository is invalid
2022-03-16 10:23:26,638 DEBUG SenderThread:28972 [connectionpool.py:_new_conn():226] Starting new HTTP connection (1): localhost:8080
2022-03-16 10:23:26,650 DEBUG SenderThread:28972 [connectionpool.py:_make_request():433] localhost:8080 “POST /graphql HTTP/1.1” 200 591
2022-03-16 10:23:26,652 INFO SenderThread:28972 [sender.py:send_run():392] updated run: g3s746w2
2022-03-16 10:23:26,652 DEBUG SenderThread:28972 [sender.py:send():89] send: exit
2022-03-16 10:23:26,652 INFO SenderThread:28972 [sender.py:send_exit():167] handling exit code: 1
2022-03-16 10:23:26,652 INFO SenderThread:28972 [sender.py:send_exit():175] send defer
2022-03-16 10:23:26,652 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,652 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: poll_exit
2022-03-16 10:23:26,652 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: defer
2022-03-16 10:23:26,653 INFO HandlerThread:28972 [handler.py:handle_request_defer():68] handle defer: 0
2022-03-16 10:23:26,653 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,653 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: defer
2022-03-16 10:23:26,653 INFO SenderThread:28972 [sender.py:send_request_defer():184] handle sender defer: 0
2022-03-16 10:23:26,653 INFO SenderThread:28972 [sender.py:send_request_defer():220] send defer: 1
2022-03-16 10:23:26,653 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: defer
2022-03-16 10:23:26,654 INFO HandlerThread:28972 [handler.py:handle_request_defer():68] handle defer: 1
2022-03-16 10:23:26,657 DEBUG raven-sentry.BackgroundWorker:28972 [connectionpool.py:_make_request():433] "https:// o151352.ingest. sentry. io:443 “POST /api/5288891/store/ HTTP/1.1” 200 41
2022-03-16 10:23:26,661 DEBUG raven-sentry.BackgroundWorker:28972 [connectionpool.py:_make_request():433] https:// o151352.i ngest. sentry. io :443 “POST /api/5288891/store/ HTTP/1.1” 200 41
2022-03-16 10:23:26,665 DEBUG raven-sentry.BackgroundWorker:28972 [connectionpool.py:_make_request():433] https:// o151352. ingest.sentry. io:443 “POST /api/5288891/store/ HTTP/1.1” 200 41
2022-03-16 10:23:26,670 DEBUG raven-sentry.BackgroundWorker:28972 [connectionpool.py:_make_request():433] https:// o151352. ingest. sentry. io:443 “POST /api/5288891/store/ HTTP/1.1” 200 41
2022-03-16 10:23:26,691 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,691 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: defer
2022-03-16 10:23:26,691 INFO SenderThread:28972 [sender.py:send_request_defer():184] handle sender defer: 1
2022-03-16 10:23:26,691 INFO SenderThread:28972 [sender.py:send_request_defer():220] send defer: 2
2022-03-16 10:23:26,692 DEBUG SenderThread:28972 [sender.py:send():89] send: stats
2022-03-16 10:23:26,692 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: defer
2022-03-16 10:23:26,692 INFO HandlerThread:28972 [handler.py:handle_request_defer():68] handle defer: 2
2022-03-16 10:23:26,692 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,692 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: defer
2022-03-16 10:23:26,692 INFO SenderThread:28972 [sender.py:send_request_defer():184] handle sender defer: 2
2022-03-16 10:23:26,692 INFO SenderThread:28972 [sender.py:send_request_defer():220] send defer: 3
2022-03-16 10:23:26,693 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: defer
2022-03-16 10:23:26,693 INFO HandlerThread:28972 [handler.py:handle_request_defer():68] handle defer: 3
2022-03-16 10:23:26,693 DEBUG SenderThread:28972 [sender.py:send():89] send: summary
2022-03-16 10:23:26,693 INFO SenderThread:28972 [sender.py:_save_file():567] saving file wandb-summary.json with policy end
2022-03-16 10:23:26,693 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,693 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: defer
2022-03-16 10:23:26,694 INFO SenderThread:28972 [sender.py:send_request_defer():184] handle sender defer: 3
2022-03-16 10:23:26,694 INFO SenderThread:28972 [sender.py:send_request_defer():220] send defer: 4
2022-03-16 10:23:26,694 DEBUG HandlerThread:28972 [handler.py:handle_request():54] handle_request: defer
2022-03-16 10:23:26,694 INFO HandlerThread:28972 [handler.py:handle_request_defer():68] handle defer: 4
2022-03-16 10:23:26,694 DEBUG SenderThread:28972 [sender.py:send():89] send: request
2022-03-16 10:23:26,694 DEBUG SenderThread:28972 [sender.py:send_request():98] send_request: defer
2022-03-16 10:23:26,694 INFO SenderThread:28972 [sender.py:send_request_defer():184] handle sender defer: 4
2022-03-16 10:23:26,694 INFO SenderThread:28972 [dir_watcher.py:finish():280] shutting down directory watcher
2022-03-16 10:23:27,577 INFO Thread-8 :28972 [dir_watcher.py:_on_file_modified():228] file/dir modified: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/config.yaml
2022-03-16 10:23:27,578 INFO SenderThread:28972 [dir_watcher.py:_on_file_created():215] file/dir created: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/wandb-metadata.json
2022-03-16 10:23:27,578 INFO SenderThread:28972 [dir_watcher.py:_on_file_created():215] file/dir created: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/output.log
2022-03-16 10:23:27,578 INFO SenderThread:28972 [dir_watcher.py:_on_file_created():215] file/dir created: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/requirements.txt
2022-03-16 10:23:27,579 INFO SenderThread:28972 [dir_watcher.py:_on_file_created():215] file/dir created: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/code
2022-03-16 10:23:27,579 INFO SenderThread:28972 [dir_watcher.py:finish():309] scan:

2022-03-16 10:23:27,602 DEBUG Thread-13 :28972 [connectionpool.py:_new_conn():226] Starting new HTTP connection (1): localhost:8080
2022-03-16 10:23:27,605 DEBUG Thread-15 :28972 [connectionpool.py:_new_conn():226] Starting new HTTP connection (1): localhost:8080
2022-03-16 10:23:27,605 DEBUG Thread-14 :28972 [connectionpool.py:_make_request():433] http: // localhost: 8080 “POST /graphql HTTP/1.1” 200 417
2022-03-16 10:23:27,606 DEBUG Thread-13 :28972 [connectionpool.py:_make_request():433] http: // localhost: 8080 “PUT //local-files/mithileshia/blstm/g3s746w2/wandb-summary.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=71b63953009d547f05162851b5762e9f28fe65117cc7a293ba71138648f76525 HTTP/1.1” 400 404
2022-03-16 10:23:27,608 ERROR Thread-13 :28972 [internal_api.py:upload_file():1210] upload_file exception http: // localhost: 8080//local-files/mithileshia/blstm/g3s746w2/wandb-summary.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=71b63953009d547f05162851b5762e9f28fe65117cc7a293ba71138648f76525 400 Client Error: Bad Request for url: http: // localhost:8080// local-files/mithileshia/blstm/g3s746w2/wandb-summary.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-
Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 24, in wrapper
return func(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 132, in wrapped_fn
return retrier(*args, **kargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 96, in call
result = self._call_fn(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1220, in upload_file
util.sentry_reraise(e)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/util.py”, line 95, in sentry_reraise
six.reraise(type(exc), exc, sys.exc_info()[2])
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/six.py”, line 703, in reraise
raise value
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1208, in upload_file
response.raise_for_status()
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/requests/models.py”, line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http: // localhost:8080// local-files/mithileshia/blstm/g3s746w2/wandb-summary.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=71b63953009d547f05162851b5762e9f28fe65117cc7a293ba71138648f76525

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/filesync/upload_job.py”, line 107, in push
self._api.upload_file_retry(
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 26, in wrapper
raise CommError(err.response, err)
wandb.errors.error.CommError: <Response [400]>
2022-03-16 10:23:27,618 DEBUG Thread-14 :28972 [connectionpool.py:_make_request():433] http: // localhost:8080 “PUT //local-files/mithileshia/blstm/g3s746w2/config.yaml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a22fd49e13ede15a3b6ea19c938b6e0c79b4219d3534c0bd1f6bc246770bd085 HTTP/1.1” 400 390
2022-03-16 10:23:27,621 ERROR Thread-15 :28972 [upload_job.py:push():115] Failed to upload file: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/requirements.txt
Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 24, in wrapper
return func(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 132, in wrapped_fn
return retrier(*args, **kargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 96, in call
result = self._call_fn(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1220, in upload_file
util.sentry_reraise(e)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/util.py”, line 95, in sentry_reraise
six.reraise(type(exc), exc, sys.exc_info()[2])
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/six.py”, line 703, in reraise
raise value
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1208, in upload_file
response.raise_for_status()
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/requests/models.py”, line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http:// localhost:8080//local-files/mithileshia/blstm/g3s746w2/requirements.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=00bd693017e41aa2739f274a3e398cb45c70d4f075eb44662473dfd17b821a7a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/filesync/upload_job.py”, line 107, in push
self._api.upload_file_retry(
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 26, in wrapper
raise CommError(err.response, err)
wandb.errors.error.CommError: <Response [400]>
2022-03-16 10:23:27,622 ERROR Thread-14 :28972 [internal_api.py:upload_file():1210] upload_file exception http: // localhost: 8080//local-files/mithileshia/blstm/g3s746w2/config.yaml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a22fd49e13ede15a3b6ea19c938b6e0c79b4219d3534c0bd1f6bc246770bd085 400 Client Error: Bad Request for url: http: // localhost:8080//local-files/mithileshia/blstm/g3s746w2/config.yaml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-
2022-03-16 10:23:27,631 DEBUG SenderThread:28972 [sender.py:send():89] send: final
2022-03-16 10:23:27,631 ERROR Thread-14 :28972 [upload_job.py:push():115] Failed to upload file: /home/demo_user2/Mithilesh/BayesianLSTM_W&B/wandb/run-20220316_102325-g3s746w2/files/config.yaml
Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 24, in wrapper
return func(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 132, in wrapped_fn
return retrier(*args, **kargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/old/retry.py”, line 96, in call
result = self._call_fn(*args, **kwargs)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1220, in upload_file
util.sentry_reraise(e)
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/util.py”, line 95, in sentry_reraise
six.reraise(type(exc), exc, sys.exc_info()[2])
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/six.py”, line 703, in reraise
raise value
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/internal/internal_api.py”, line 1208, in upload_file
response.raise_for_status()
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/requests/models.py”, line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http: // localhost: 8080//local-files/mithileshia/blstm/g3s746w2/config.yaml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0IGHhIFOZrr9%2BXoTgbQ%2Blg%3D%3D%2F20220316%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220316T102327Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a22fd49e13ede15a3b6ea19c938b6e0c79b4219d3534c0bd1f6bc246770bd085

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/filesync/upload_job.py”, line 107, in push
self._api.upload_file_retry(
File “/home/demo_user2/miniconda3/lib/python3.8/site-packages/wandb/apis/normalize.py”, line 26, in wrapper
raise CommError(err.response, err)
wandb.errors.error.CommError: <Response [400]>

2022-03-16 10:23:29,658 INFO SenderThread:28972 [file_pusher.py:finish():169] shutting down file pusher
2022-03-16 10:23:29,658 INFO SenderThread:28972 [file_pusher.py:join():174] waiting for file pusher
2022-03-16 10:23:29,659 INFO MainThread:28972 [internal.py:handle_exit():137] Internal process exited

4

Hi @mithileshia,

I apologize about the delay here. From the looks of it it seems like the URL http: // localhost:8080// local-files/mithileshia/blstm/g3s746w2/wandb-summary.json is malformed, though I can not see the line creating from your source code in the debug logs. Could you share the line from the code causing this error?

Thanks,
Ramit

5

​Hi @mithileshia,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

6

​Hi @mithileshia, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

​

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.