Wandb server connection? How to fix?

brando · August 15, 2023, 11:22pm

Why is this error happening/

Traceback (most recent call last):
  File "/lfs/hyperturing1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 578, in <module>
    experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
  File "/lfs/hyperturing1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 518, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
    print(f'{next(iter(batch))=}')
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1353, in __iter__
    for key, example in ex_iterable:
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1013, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 398, in __iter__
    for i in indices_iterator:
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 587, in _iter_random_indices
    yield from (int(i) for i in rng.choice(num_sources, size=random_batch_size, p=p))
  File "_generator.pyx", line 821, in numpy.random._generator.Generator.choice
ValueError: a and p must have same size
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: 🚀 View run ['c4', 'wikitext'] div_coeff_num_batches=600 (today='2023-m08-d15-t16h_16m_01s' (name=['en', 'wikitext-103-v1']) data_mixture_name='uniform' probabilities=[0.5, 0.5]) at: https://wandb.ai/brando/beyond-scale/runs/q51mv95t
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)
wandb: Find logs at: ./wandb/run-20230815_161602-q51mv95t/logs
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 256, in check_network_status
    self._loop_check_status(
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 212, in _loop_check_status
    local_handle = request()
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface.py", line 864, in deliver_network_status
    return self._deliver_network_status(status)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 610, in _deliver_network_status
    return self._deliver_record(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 569, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

How to fix?

brando · August 15, 2023, 11:23pm

[CLI]: Multiprocessing brokenpipe error

opened 01:28PM - 08 Feb 23 UTC

closed 07:31AM - 02 Mar 23 UTC

srulikbd

cli

### Describe the bug I use sweeps with tf.keras multiprocessing option. Aft…er few runs (about 7-15) the run crash. I get broken pipe error. It seems that I get OOM on the CPU and each run add more memory usage to the CPU, so after some runs it floods. ```python import wandb wandb.login() os.environ['CUDA_VISIBLE_DEVICES'] = "0" def train_standard(): tqdm_callback = tfa.callbacks.TQDMProgressBar() callbacks = [keras.callbacks.ModelCheckpoint(filepath=model_path + "/save_at_{epoch}.h5"), tqdm_callback, WandbMetricsLogger()] model.fit( x=data_generator.train_generator(batch_size), steps_per_epoch=math.ceil(data_generator.get_train_size() / batch_size), epochs=epochs, shuffle=True, initial_epoch=0, verbose=2, validation_data=data_generator.valid_generator(batch_size), validation_steps=math.ceil(data_generator.get_valid_size() / batch_size), callbacks=callbacks, use_multiprocessing=True, workers=6 def main(): wandb.init(name=DATASET) wandb.require("service") config = wandb.config preprocessing(smooth_factor, sample_factor) train_standard(batch_size, epochs, learning_rate, dropout, dense_layer, activation, hyperparameter_search=True) wandb.finish() sweep_config = { 'method': 'grid', 'metric': { 'goal': 'maximize', 'name': 'weighted avg.f1-score' }, 'parameters': { 'batch_size': {'values': [8,16,32]}, 'epochs': {'values': [3,6,9]}, 'lr': {'values': [0.001, 0.0001]}, 'dropout': {'values': [0.1, 0.2, 0.5]}, 'dense_layer' : {'values': [16, 32, 64]}, 'activation': {'values': ['relu', 'tanh', 'sigmoid']}, 'smooth_factor' : {'values': [10]}, 'sample_factor' : {'values': [3, 5, 10]}, } } if __name__ == '__main__': sweep_id = wandb.sweep(sweep_config, project="shaoxing-hyperparameter-tuning1") wandb.agent(sweep_id, function=main) ``` ```shell Training: 0%| 0/6 ETA: ?s, ?epochs/sEpoch 1/6 Epoch 1/6 ETA: ?s - 6805/6805█ ETA: 00:00s - loss: 0.4434 - categorical_accuracy: 0.8402 - val_loss: 0.5995 - val_categorical_accuracy: 0 Training: 17%|███████████▌ 1/6 ETA: 20:51s, 250.28s/epochs6805/6805 - 250s - loss: 0.4434 - categorical_accuracy: 0.8402 - val_loss: 0.5995 - val_categorical_accuracy: 0.7694 - 250s/epoch - 37ms/step Epoch 2/6 Epoch 2/6 ETA: ?s - 6805/6805█ ETA: 00:00s - loss: 0.2890 - categorical_accuracy: 0.9054 - val_loss: 0.5200 - val_categorical_accuracy: 0 Training: 33%|███████████████████████ 2/6 ETA: 16:36s, 249.10s/epochs6805/6805 - 248s - loss: 0.2890 - categorical_accuracy: 0.9054 - val_loss: 0.5200 - val_categorical_accuracy: 0.7900 - 248s/epoch - 36ms/step Epoch 3/6 Epoch 3/6 ETA: ?s - 6805/6805█ ETA: 00:00s - loss: 0.2802 - categorical_accuracy: 0.9076 - val_loss: 0.5400 - val_categorical_accuracy: 0 Training: 50%|██████████████████████████████████▌ 3/6 ETA: 12:28s, 249.48s/epochs6805/6805 - 250s - loss: 0.2802 - categorical_accuracy: 0.9076 - val_loss: 0.5400 - val_categorical_accuracy: 0.8057 - 250s/epoch - 37ms/step Epoch 4/6 Epoch 4/6 ETA: ?s - 6805/6805█ ETA: 00:00s - loss: 0.2605 - categorical_accuracy: 0.9134 - val_loss: 0.4429 - val_categorical_accuracy: 0 Training: 67%|██████████████████████████████████████████████ 4/6 ETA: 08:20s, 250.47s/epochs6805/6805 - 252s - loss: 0.2605 - categorical_accuracy: 0.9134 - val_loss: 0.4429 - val_categorical_accuracy: 0.8262 - 252s/epoch - 37ms/step Epoch 5/6 Epoch 5/6 ETA: ?s - 6805/6805█ ETA: 00:00s - loss: 0.2627 - categorical_accuracy: 0.9078 - val_loss: 0.5205 - val_categorical_accuracy: 0 Training: 83%|█████████████████████████████████████████████████████████▌ 5/6 ETA: 04:10s, 250.99s/epochs6805/6805 - 252s - loss: 0.2627 - categorical_accuracy: 0.9078 - val_loss: 0.5205 - val_categorical_accuracy: 0.7949 - 252s/epoch - 37ms/step Epoch 6/6 Epoch 6/6 ETA: ?s - Process Keras_worker_ForkPoolWorker-149:███████████████████▉ ETA: 00:00s - loss: 0.2400 - categorical_accuracy: 0.9175 Process Keras_worker_ForkPoolWorker-152: Process Keras_worker_ForkPoolWorker-151: [3]+ Killed python train.py Killed Process Keras_worker_ForkPoolWorker-150: (tensorflow) ubuntu@ip-172-31-92-126:~/walkky-ML/classifier/ml/heart_beat_cnn$ git pull && python train.pyTraceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 409, in _send_bytes self._send(header) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 410, in _send_bytes self._send(buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 409, in _send_bytes self._send(header) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 409, in _send_bytes self._send(header) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() Traceback (most recent call last): File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/local/lib/python3.10/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe ``` ### Additional Files [https://wandb.ai/walkky/shaoxing-hyperparameter-tuning1/sweeps/u1jduer9/overview?workspace=user-srulikbd](url) ### Environment WandB version: 0.13.9 OS: Ubuntu 20.04.5 LTS (GNU/Linux 5.15.0-1026-aws x86_64) with Deep Learning AMI GPU TensorFlow 2.11.0 Python version: 3.10.7 Versions of relevant libraries: NVIDIA driver version: 515.65.01 CUDA version: 11.2 TensorFlow 2.11.0 ### Additional Context it seems that there is a lot of python processes created during running: ![image](https://user-images.githubusercontent.com/35503583/217542844-faa69a1f-0ce4-4d1b-9b1d-bf401a25947e.png) maybe it's related to the error? ![image](https://user-images.githubusercontent.com/35503583/217543017-f74f8456-ce17-4d5a-ac3e-cdb33901edfd.png)

nathank · August 18, 2023, 4:52pm

Hi @brando, would it be possible to share any errors that show up in the debug-internal.log from the local run folder? Also, are you only seeing this when you get the Value error crash that you saw?

nathank · August 22, 2023, 9:39pm

Hi @brando, I just wanted to follow up on this and see if we could still help with this?