Wandb server connection? How to fix?

Why is this error happening/

Traceback (most recent call last):
  File "/lfs/hyperturing1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 578, in <module>
    experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
  File "/lfs/hyperturing1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 518, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
    print(f'{next(iter(batch))=}')
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1353, in __iter__
    for key, example in ex_iterable:
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1013, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 398, in __iter__
    for i in indices_iterator:
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 587, in _iter_random_indices
    yield from (int(i) for i in rng.choice(num_sources, size=random_batch_size, p=p))
  File "_generator.pyx", line 821, in numpy.random._generator.Generator.choice
ValueError: a and p must have same size
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: πŸš€ View run ['c4', 'wikitext'] div_coeff_num_batches=600 (today='2023-m08-d15-t16h_16m_01s' (name=['en', 'wikitext-103-v1']) data_mixture_name='uniform' probabilities=[0.5, 0.5]) at: https://wandb.ai/brando/beyond-scale/runs/q51mv95t
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)
wandb: Find logs at: ./wandb/run-20230815_161602-q51mv95t/logs
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 256, in check_network_status
    self._loop_check_status(
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 212, in _loop_check_status
    local_handle = request()
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface.py", line 864, in deliver_network_status
    return self._deliver_network_status(status)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 610, in _deliver_network_status
    return self._deliver_record(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 569, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

How to fix?

related:

Hi @brando, would it be possible to share any errors that show up in the debug-internal.log from the local run folder? Also, are you only seeing this when you get the Value error crash that you saw?

Hi @brando, I just wanted to follow up on this and see if we could still help with this?