Sweep - Broken Pipe

ahs63 · August 4, 2023, 6:34am

Hi there,

I saw similar problems were asked a few times but I didn’t see any solution. I am facing the following problem. I am using native Ubuntu 22.04, a personal laptop at home with “NO” VPN/firewall/proxy/load balancer.

I am using Python 3.8.13 and wandb client 0.15.8.

Any idea?

wandb: WARNING Calling wandb.login() after wandb.init() has no effect.
wandb: Waiting for W&B process to finish… (success).
wandb: View run neat-brook-42 at: Weights & Biases
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230803_232558-8nul30m9/logs
wandb: Agent Starting Run: 5cojxnxb with config:
wandb: var_smoothing: 0.0001
Exception in thread Thread-4:
Traceback (most recent call last):
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/agents/pyagent.py”, line 298, in _run_job
self._function()
File “test.py”, line 35, in main
wandb.log({
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 390, in wrapper
return func(self, *args, **kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 341, in wrapper_fn
return func(self, *args, **kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 331, in wrapper
return func(self, *args, **kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 1752, in log
self._log(data=data, step=step, commit=commit)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 1527, in _log
self._partial_history_callback(data, step, commit)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 1397, in _partial_history_callback
self._backend.interface.publish_partial_history(
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface.py”, line 653, in publish_partial_history
self._publish_partial_history(partial_history)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py”, line 89, in _publish_partial_history
self._publish(rec)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py”, line 51, in _publish
self._sock_client.send_record_publish(record)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 221, in send_record_publish
self.send_server_request(server_req)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 155, in send_server_request
self._send_message(msg)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 152, in _send_message
self._sendall_with_error_handle(header + data)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
self.run()
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/agents/pyagent.py”, line 303, in _run_job
wandb.finish(exit_code=1)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 3717, in finish
wandb.run.finish(exit_code=exit_code, quiet=quiet)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 390, in wrapper
return func(self, *args, **kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 331, in wrapper
return func(self, *args, **kwargs)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 1885, in finish
return self._finish(exit_code, quiet)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 1893, in _finish
tel.feature.finish = True
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/telemetry.py”, line 42, in exit
self._run._telemetry_callback(self._obj)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 713, in _telemetry_callback
self._telemetry_flush()
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_run.py”, line 724, in _telemetry_flush
self._backend.interface._publish_telemetry(self._telemetry_obj)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py”, line 101, in _publish_telemetry
self._publish(rec)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py”, line 51, in _publish
self._sock_client.send_record_publish(record)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 221, in send_record_publish
self.send_server_request(server_req)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 155, in send_server_request
self._send_message(msg)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 152, in _send_message
self._sendall_with_error_handle(header + data)
File “/home/cps/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py”, line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

ahs63 · August 4, 2023, 6:43am

Here are additional logs from running this simple example - Sweeps Walkthrough

Run history:

score ▁

Run summary:

score 3.00017

View run breezy-sweep-9 at: Weights & Biases
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)

Find logs at: ./wandb/run-20230803_233830-4kik86d1/logs

wandb: Agent Starting Run: hxihjp2o with config: wandb: x: 0.016518566399011086 wandb: y: 3 wandb: WARNING Ignored wandb.init() arg project when running a sweep.

Tracking run with wandb version 0.15.8

Run data is saved locally in /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_233841-hxihjp2o

Syncing run clear-sweep-10 to Weights & Biases (docs)
Sweep page: Weights & Biases

View project at Weights & Biases

View sweep at Weights & Biases

View run at Weights & Biases

Waiting for W&B process to finish… (success).

0.001 MB of 0.019 MB uploaded (0.000 MB deduped)

Run history:

score ▁

Run summary:

score 3.0

View run clear-sweep-10 at: Weights & Biases
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)

Find logs at: ./wandb/run-20230803_233841-hxihjp2o/logs

Error in callback <function _WandbInit._pause_backend at 0x7feafdf8d940> (for post_run_cell):

--------------------------------------------------------------------------- BrokenPipeError Traceback (most recent call last) File ~/anaconda3/envs/dissertation/lib/python3.8/site-packages/backcall/backcall.py:104, in callback_prototype…adapt…adapted(*args, **kwargs) 102 kwargs.pop(name) 103 # print(args, kwargs, unmatched_pos, cut_positional, unmatched_kw) → 104 return callback(*args, **kwargs) File ~/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/wandb_init.py:419, in _WandbInit._pause_backend(self) 417 if self.backend.interface is not None: 418 logger.info(“pausing backend”) # type: ignore → 419 self.backend.interface.publish_pause() File ~/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface.py:733, in InterfaceBase.publish_pause(self) 731 def publish_pause(self) → None: 732 pause = pb.PauseRequest() → 733 self._publish_pause(pause) File ~/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py:349, in InterfaceShared._publish_pause(self, pause) 347 def _publish_pause(self, pause: pb.PauseRequest) → None: 348 rec = self._make_request(pause=pause) → 349 self._publish(rec) File ~/anaconda3/envs/dissertation/lib/python3.8/site-packages/wandb/sdk/interface/interface_sock.py:51, in InterfaceSock._publish(self, record, local) 49 def _publish(self, record: “pb.Record”, local: Optional[bool] = None) → None: 50 self._assign(record)

…

→ 130 sent = self._sock.send(data) 131 # sent equal to 0 indicates a closed socket 132 if sent == 0: BrokenPipeError: [Errno 32] Broken pipe

Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings…

artsiom · August 8, 2023, 9:21pm

Hi @ahs63 ! Thank you for writing in! Could you please provide the debug.log and debug-internal.log files associated with the run where you are running into this issue? These files should be located in the wandb folder relative to your working directory.

ahs63 · August 9, 2023, 6:18am

Hi @artsiom , here are the logs. Thanks.

debug.log

2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO config: {}
2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,726 INFO 2023-08-03 23:42:43,728 INFO 2023-08-03 23:42:43,729 INFO 2023-08-03 23:42:43,732 INFO 2023-08-03 23:42:43,733 INFO 2023-08-03 23:42:44,080 INFO 2023-08-03 23:42:44,188 INFO 2023-08-03 23:42:44,188 INFO 2023-08-03 23:42:48,363 INFO 2023-08-03 23:42:48,363 INFO 2023-08-03 23:42:48,363 INFO 2023-08-03 23:42:48,363 INFO 2023-08-03 23:42:48,364 INFO 2023-08-03 23:42:50,216 INFO 2023-08-03 23:42:55,077 MainThread:525208 [wandb_setup.py:_flush():76] Current SDK version is 0.15.8
MainThread:525208 [wandb_setup.py:_flush():76] Configure stats pid to 525208
MainThread:525208 [wandb_setup.py:_flush():76] Loading settings from /home/cps/.config/wandb/settings
MainThread:525208 [wandb_setup.py:_flush():76] Loading settings from /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/settings
MainThread:525208 [wandb_setup.py:_flush():76] Loading settings from environment variables: {‘notebook_name’: ‘./naive-bayes.ipynb’, ‘api_key’: ‘REDACTED’}
MainThread:525208 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {‘program_relpath’: ‘test.py’, ‘program’: ‘test.py’}
MainThread:525208 [wandb_init.py:_log_setup():507] Logging user logs to /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/logs/debug.log
MainThread:525208 [wandb_init.py:_log_setup():508] Logging internal logs to /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/logs/debug-internal.log
MainThread:525208 [wandb_init.py:init():547] calling init triggers
MainThread:525208 [wandb_init.py:init():554] wandb.init called with sweep_config: {}
MainThread:525208 [wandb_init.py:init():596] starting backend
MainThread:525208 [wandb_init.py:init():600] setting up manager
MainThread:525208 [backend.py:_multiprocessing_setup():106] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
MainThread:525208 [wandb_init.py:init():606] backend started and connected
MainThread:525208 [wandb_init.py:init():697] updated telemetry
MainThread:525208 [wandb_init.py:init():730] communicating run to backend with 60.0 second timeout
MainThread:525208 [wandb_run.py:_on_init():2180] communicating current version
MainThread:525208 [wandb_run.py:_on_init():2189] got version response
MainThread:525208 [wandb_init.py:init():781] starting run threads in backend
MainThread:525208 [wandb_run.py:_console_start():2159] atexit reg
MainThread:525208 [wandb_run.py:_redirect():2014] redirect: wrap_raw
MainThread:525208 [wandb_run.py:_redirect():2079] Wrapping output streams.
MainThread:525208 [wandb_run.py:_redirect():2104] Redirects installed.
MainThread:525208 [wandb_init.py:init():822] run started, returning control to user process
MainThread:525208 [pyagent.py:run():314] Starting sweep agent: entity=None, project=None, count=10
WARNING MsgRouterThr:525208 [router.py:message_loop():77] message_loop has been closed

debug-internal.log

2023-08-03 23:42:43,729 INFO StreamThr :525240 [internal.py:wandb_internal():86] W&B internal server running at pid: 525240, started at: 2023-08-03 23:42:43.729344
2023-08-03 23:42:43,730 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: status
2023-08-03 23:42:43,733 INFO WriterThread:525240 [datastore.py:open_for_write():85] open: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/run-18agjlcf.wandb
2023-08-03 23:42:43,734 DEBUG SenderThread:525240 [sender.py:send():379] send: header
2023-08-03 23:42:43,734 DEBUG SenderThread:525240 [sender.py:send():379] send: run
2023-08-03 23:42:44,075 INFO SenderThread:525240 [dir_watcher.py:init():211] watching files in: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files
2023-08-03 23:42:44,076 INFO SenderThread:525240 [sender.py:_start_run_threads():1121] run started: 18agjlcf with start time 1691131363.729363
2023-08-03 23:42:44,076 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: summary_record
2023-08-03 23:42:44,076 INFO SenderThread:525240 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end
2023-08-03 23:42:44,081 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: check_version
2023-08-03 23:42:44,081 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: check_version
2023-08-03 23:42:44,195 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: run_start
2023-08-03 23:42:44,197 DEBUG HandlerThread:525240 [system_info.py:init():31] System info init
2023-08-03 23:42:44,197 DEBUG HandlerThread:525240 [system_info.py:init():46] System info init done
2023-08-03 23:42:44,197 INFO HandlerThread:525240 [system_monitor.py:start():181] Starting system monitor
2023-08-03 23:42:44,197 INFO SystemMonitor:525240 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-08-03 23:42:44,197 INFO HandlerThread:525240 [system_monitor.py:probe():201] Collecting system info
2023-08-03 23:42:44,198 INFO SystemMonitor:525240 [interfaces.py:start():190] Started cpu monitoring
2023-08-03 23:42:44,200 INFO SystemMonitor:525240 [interfaces.py:start():190] Started disk monitoring
2023-08-03 23:42:44,201 INFO SystemMonitor:525240 [interfaces.py:start():190] Started memory monitoring
2023-08-03 23:42:44,202 INFO SystemMonitor:525240 [interfaces.py:start():190] Started network monitoring
2023-08-03 23:42:44,207 DEBUG HandlerThread:525240 [system_info.py:probe():195] Probing system
2023-08-03 23:42:44,211 DEBUG HandlerThread:525240 [gitlib.py:_init_repo():53] git repository is invalid
2023-08-03 23:42:44,211 DEBUG HandlerThread:525240 [system_info.py:probe():240] Probing system done
2023-08-03 23:42:44,211 DEBUG HandlerThread:525240 [system_monitor.py:probe():210] {‘os’: ‘Linux-5.19.0-50-generic-x86_64-with-glibc2.17’, ‘python’: ‘3.8.13’, ‘heartbeatAt’: ‘2023-08-04T06:42:44.207128’, ‘startedAt’: ‘2023-08-04T06:42:43.724125’, ‘docker’: None, ‘cuda’: None, ‘args’: (), ‘state’: ‘running’, ‘program’: ‘test.py’, ‘codePath’: ‘test.py’, ‘host’: ‘cps’, ‘username’: ‘cps’, ‘executable’: ‘/home/cps/anaconda3/envs/dissertation/bin/python’, ‘cpu_count’: 4, ‘cpu_count_logical’: 8, ‘cpu_freq’: {‘current’: 1.2143749999999998, ‘min’: 400.0, ‘max’: 4400.0}, ‘cpu_freq_per_core’: [{‘current’: 1.203, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.201, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.123, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.256, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.341, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.253, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.204, ‘min’: 400.0, ‘max’: 4400.0}, {‘current’: 1.775, ‘min’: 400.0, ‘max’: 4400.0}], ‘disk’: {‘total’: 467.8895797729492, ‘used’: 248.06146621704102}, ‘memory’: {‘total’: 15.325130462646484}}
2023-08-03 23:42:44,211 INFO HandlerThread:525240 [system_monitor.py:probe():211] Finished collecting system info
2023-08-03 23:42:44,211 INFO HandlerThread:525240 [system_monitor.py:probe():214] Publishing system info
2023-08-03 23:42:44,211 DEBUG HandlerThread:525240 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-08-03 23:42:44,212 DEBUG HandlerThread:525240 [system_info.py:_save_pip():67] Saving pip packages done
2023-08-03 23:42:44,212 DEBUG HandlerThread:525240 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment
2023-08-03 23:42:45,077 INFO Thread-12 :525240 [dir_watcher.py:_on_file_created():271] file/dir created: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-summary.json
2023-08-03 23:42:45,078 INFO Thread-12 :525240 [dir_watcher.py:_on_file_created():271] file/dir created: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/conda-environment.yaml
2023-08-03 23:42:45,078 INFO Thread-12 :525240 [dir_watcher.py:_on_file_created():271] file/dir created: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/requirements.txt
2023-08-03 23:42:48,356 DEBUG HandlerThread:525240 [system_info.py:_save_conda():86] Saving conda packages done
2023-08-03 23:42:48,356 INFO HandlerThread:525240 [system_monitor.py:probe():216] Finished publishing system info
2023-08-03 23:42:48,360 DEBUG SenderThread:525240 [sender.py:send():379] send: files
2023-08-03 23:42:48,360 INFO SenderThread:525240 [sender.py:_save_file():1375] saving file wandb-metadata.json with policy now
2023-08-03 23:42:48,363 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: stop_status
2023-08-03 23:42:48,363 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: stop_status
2023-08-03 23:42:48,551 DEBUG SenderThread:525240 [sender.py:send():379] send: telemetry
2023-08-03 23:42:48,849 INFO wandb-upload_0:525240 [upload_job.py:push():131] Uploaded file /tmp/tmp0ab21fgawandb/i4ghvnzx-wandb-metadata.json
2023-08-03 23:42:49,077 INFO Thread-12 :525240 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/conda-environment.yaml
2023-08-03 23:42:49,077 INFO Thread-12 :525240 [dir_watcher.py:_on_file_created():271] file/dir created: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-metadata.json
2023-08-03 23:42:49,552 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: status_report
2023-08-03 23:42:50,922 DEBUG SenderThread:525240 [sender.py:send():379] send: exit
2023-08-03 23:42:50,922 INFO SenderThread:525240 [sender.py:send_exit():584] handling exit code: 0
2023-08-03 23:42:50,922 INFO SenderThread:525240 [sender.py:send_exit():586] handling runtime: 6
2023-08-03 23:42:50,924 INFO SenderThread:525240 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end
2023-08-03 23:42:50,925 INFO SenderThread:525240 [sender.py:send_exit():592] send defer
2023-08-03 23:42:50,925 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,925 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 0
2023-08-03 23:42:50,925 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,925 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 0
2023-08-03 23:42:50,925 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 1
2023-08-03 23:42:50,926 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,926 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 1
2023-08-03 23:42:50,926 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,926 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 1
2023-08-03 23:42:50,926 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 2
2023-08-03 23:42:50,926 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,926 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 2
2023-08-03 23:42:50,926 INFO HandlerThread:525240 [system_monitor.py:finish():190] Stopping system monitor
2023-08-03 23:42:50,927 DEBUG SystemMonitor:525240 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-08-03 23:42:50,927 INFO HandlerThread:525240 [interfaces.py:finish():202] Joined cpu monitor
2023-08-03 23:42:50,927 DEBUG SystemMonitor:525240 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-08-03 23:42:50,927 INFO HandlerThread:525240 [interfaces.py:finish():202] Joined disk monitor
2023-08-03 23:42:50,927 DEBUG SystemMonitor:525240 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-08-03 23:42:50,927 INFO HandlerThread:525240 [interfaces.py:finish():202] Joined memory monitor
2023-08-03 23:42:50,928 INFO HandlerThread:525240 [interfaces.py:finish():202] Joined network monitor
2023-08-03 23:42:50,928 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,928 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 2
2023-08-03 23:42:50,928 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 3
2023-08-03 23:42:50,928 DEBUG SenderThread:525240 [sender.py:send():379] send: stats
2023-08-03 23:42:50,929 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,929 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 3
2023-08-03 23:42:50,929 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,929 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 3
2023-08-03 23:42:50,929 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 4
2023-08-03 23:42:50,929 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,929 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 4
2023-08-03 23:42:50,929 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,929 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 4
2023-08-03 23:42:50,929 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 5
2023-08-03 23:42:50,929 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,929 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 5
2023-08-03 23:42:50,930 DEBUG SenderThread:525240 [sender.py:send():379] send: summary
2023-08-03 23:42:50,930 INFO SenderThread:525240 [sender.py:_save_file():1375] saving file wandb-summary.json with policy end
2023-08-03 23:42:50,930 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,930 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 5
2023-08-03 23:42:50,930 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 6
2023-08-03 23:42:50,931 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:50,931 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 6
2023-08-03 23:42:50,931 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:50,931 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 6
2023-08-03 23:42:50,936 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: status_report
2023-08-03 23:42:51,071 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 7
2023-08-03 23:42:51,071 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:51,071 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 7
2023-08-03 23:42:51,071 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:51,071 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 7
2023-08-03 23:42:51,078 INFO Thread-12 :525240 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/config.yaml
2023-08-03 23:42:51,078 INFO Thread-12 :525240 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-summary.json
2023-08-03 23:42:51,078 INFO Thread-12 :525240 [dir_watcher.py:_on_file_created():271] file/dir created: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/output.log
2023-08-03 23:42:51,922 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: poll_exit
2023-08-03 23:42:52,222 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 8
2023-08-03 23:42:52,222 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: poll_exit
2023-08-03 23:42:52,222 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:52,223 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 8
2023-08-03 23:42:52,223 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:52,223 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 8
2023-08-03 23:42:52,223 INFO SenderThread:525240 [job_builder.py:build():281] Attempting to build job artifact
2023-08-03 23:42:52,224 INFO SenderThread:525240 [job_builder.py:_get_source_type():411] no source found
2023-08-03 23:42:52,224 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 9
2023-08-03 23:42:52,224 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:52,224 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 9
2023-08-03 23:42:52,225 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:52,225 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 9
2023-08-03 23:42:52,225 INFO SenderThread:525240 [dir_watcher.py:finish():358] shutting down directory watcher
2023-08-03 23:42:52,923 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: poll_exit
2023-08-03 23:42:53,079 INFO Thread-12 :525240 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/output.log
2023-08-03 23:42:53,080 INFO SenderThread:525240 [dir_watcher.py:finish():388] scan: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files
2023-08-03 23:42:53,080 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/output.log output.log
2023-08-03 23:42:53,080 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/config.yaml config.yaml
2023-08-03 23:42:53,081 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/conda-environment.yaml conda-environment.yaml
2023-08-03 23:42:53,088 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/requirements.txt requirements.txt
2023-08-03 23:42:53,092 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-summary.json wandb-summary.json
2023-08-03 23:42:53,094 INFO SenderThread:525240 [dir_watcher.py:finish():402] scan save: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-metadata.json wandb-metadata.json
2023-08-03 23:42:53,094 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 10
2023-08-03 23:42:53,094 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: poll_exit
2023-08-03 23:42:53,095 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:53,095 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 10
2023-08-03 23:42:53,096 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:53,096 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 10
2023-08-03 23:42:53,100 INFO SenderThread:525240 [file_pusher.py:finish():159] shutting down file pusher
2023-08-03 23:42:53,408 INFO wandb-upload_0:525240 [upload_job.py:push():131] Uploaded file /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/output.log
2023-08-03 23:42:53,441 INFO wandb-upload_1:525240 [upload_job.py:push():131] Uploaded file /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/config.yaml
2023-08-03 23:42:53,494 INFO wandb-upload_3:525240 [upload_job.py:push():131] Uploaded file /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/requirements.txt
2023-08-03 23:42:53,496 INFO wandb-upload_4:525240 [upload_job.py:push():131] Uploaded file /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/wandb-summary.json
2023-08-03 23:42:53,519 INFO wandb-upload_2:525240 [upload_job.py:push():131] Uploaded file /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/files/conda-environment.yaml
2023-08-03 23:42:53,720 INFO Thread-11 :525240 [sender.py:transition_state():612] send defer: 11
2023-08-03 23:42:53,721 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:53,721 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 11
2023-08-03 23:42:53,721 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:53,721 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 11
2023-08-03 23:42:53,722 INFO SenderThread:525240 [file_pusher.py:join():164] waiting for file pusher
2023-08-03 23:42:53,722 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 12
2023-08-03 23:42:53,722 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:53,722 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 12
2023-08-03 23:42:53,723 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:53,723 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 12
2023-08-03 23:42:53,723 INFO SenderThread:525240 [file_stream.py:finish():593] file stream finish called
2023-08-03 23:42:53,827 INFO SenderThread:525240 [file_stream.py:finish():597] file stream finish is done
2023-08-03 23:42:53,827 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 13
2023-08-03 23:42:53,828 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:53,828 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 13
2023-08-03 23:42:53,829 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:53,829 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 13
2023-08-03 23:42:53,829 INFO SenderThread:525240 [sender.py:transition_state():612] send defer: 14
2023-08-03 23:42:53,829 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: defer
2023-08-03 23:42:53,830 INFO HandlerThread:525240 [handler.py:handle_request_defer():170] handle defer: 14
2023-08-03 23:42:53,830 DEBUG SenderThread:525240 [sender.py:send():379] send: final
2023-08-03 23:42:53,830 DEBUG SenderThread:525240 [sender.py:send():379] send: footer
2023-08-03 23:42:53,830 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: defer
2023-08-03 23:42:53,830 INFO SenderThread:525240 [sender.py:send_request_defer():608] handle sender defer: 14
2023-08-03 23:42:53,831 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: poll_exit
2023-08-03 23:42:53,832 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: poll_exit
2023-08-03 23:42:53,833 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: poll_exit
2023-08-03 23:42:53,834 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: poll_exit
2023-08-03 23:42:53,835 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: server_info
2023-08-03 23:42:53,835 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: server_info
2023-08-03 23:42:53,839 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: get_summary
2023-08-03 23:42:53,840 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: sampled_history
2023-08-03 23:42:53,841 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: job_info
2023-08-03 23:42:53,902 DEBUG SenderThread:525240 [sender.py:send_request():406] send_request: job_info
2023-08-03 23:42:53,902 INFO MainThread:525240 [wandb_run.py:_footer_history_summary_info():3481] rendering history
2023-08-03 23:42:53,902 INFO MainThread:525240 [wandb_run.py:_footer_history_summary_info():3513] rendering summary
2023-08-03 23:42:53,902 INFO MainThread:525240 [wandb_run.py:_footer_sync_info():3440] logging synced files
2023-08-03 23:42:53,903 DEBUG HandlerThread:525240 [handler.py:handle_request():144] handle_request: shutdown
2023-08-03 23:42:53,903 INFO HandlerThread:525240 [handler.py:finish():857] shutting down handler
2023-08-03 23:42:54,842 INFO WriterThread:525240 [datastore.py:close():294] close: /home/cps/tech-market-fit/my_notebooks/experiments/hyper-tuning/wandb/run-20230803_234243-18agjlcf/run-18agjlcf.wandb
2023-08-03 23:42:54,902 INFO SenderThread:525240 [sender.py:finish():1552] shutting down sender
2023-08-03 23:42:54,902 INFO SenderThread:525240 [file_pusher.py:finish():159] shutting down file pusher
2023-08-03 23:42:54,902 INFO SenderThread:525240 [file_pusher.py:join():164] waiting for file pusher

artsiom · August 11, 2023, 9:42pm

Thank you for sending over your logs!

They do look pretty clean, is there a possibility you could send me the minimal reproduction code? Will try reproing this on my side.

artsiom · August 16, 2023, 2:48pm

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

artsiom · August 18, 2023, 3:18pm

Hi Aik,

Since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know! You can reopen the conversation by opening a new thread and referring to this thread in it.

Warmly,
Artsiom

system · October 10, 2023, 9:42pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Server socket closed W&B Help sweeps , wandb	4	1386	May 19, 2023
BrokenPipeError on Ubuntu machine W&B Help sweeps	5	5898	June 6, 2023
Broken Pipe error W&B Help sweeps , wandb	2	1141	February 9, 2024
BrokenPipeError when doing sweeps W&B Help sweeps , wandb	5	584	January 22, 2024
BUG: Sweep in Jupyter makes new runs impossible to start W&B Help	3	290	August 22, 2022

Sweep - Broken Pipe

Run history:

Run summary:

Run history:

Run summary:

debug.log

debug-internal.log

Related topics