Why do I have an army of wandb files everywhere? It's overwhelming my disk quota

[brando]

I think this was caused by wandb

(mds_env_gpu) brando9~ $ ls
data						   pymp-5u3y_5ds  pymp-bkacd1x6  pymp-hvosikw7	pymp-o4ubsmvp  pymp-u2hxw56_
debug-cli.brando9.log				   pymp-5ufiwnum  pymp-bkeosgoh  pymp-hwp6mlgl	pymp-o5709m1c  pymp-u3pu2_8j
diversity-for-predictive-success-of-meta-learning  pymp-5v6yw2ny  pymp-bli8hdga  pymp-hxkt2k6m	pymp-o5pz2fw4  pymp-u4atcmq1
iit-term-synthesis				   pymp-5vfjyiy8  pymp-blu04xre  pymp-hy_6a7yv	pymp-o5qihl07  pymp-u4hs1l_h
miniconda					   pymp-5vkol09u  pymp-bm04j6u_  pymp-hybr3wao	pymp-o5v4yyjh  pymp-u5hwvurq
miniconda.sh					   pymp-5w0ycex5  pymp-bm1af1s4  pymp-hyd9kp9k	pymp-o78e5rv_  pymp-_u6saect
proverbot9001					   pymp-5wk26dbe  pymp-bm_hml09  pymp-hyg2wzja	pymp-o7u0brbq  pymp-u7341z_u
pycoq						   pymp-5wv1264k  pymp-bmwsk0ui  pymp-hyujmqqx	pymp-o8dzl_uf  pymp-u75jwgya
pymp-00y0lbly					   pymp-5x3lfjus  pymp-bnduqemj  pymp-hyvxx7e3	pymp-o8rez_1_  pymp-u7ca8y7o
...
				   pymp-be9ogs4j  pymp-hptd30xx  pymp-ny7y25f5	pymp-tudnkgfd  tmpft4wdrhxwandb
pymp-5no2jlmt					   pymp-bedy_tkt  pymp-hpybolff  pymp-nyra71s5	pymp-t_uf7u5j  tmpfvg5and3wandb
pymp-5ntrk5up					   pymp-beha_4zu  pymp-hq7fmd8n  pymp-nzpgew4t	pymp-tvqwp5ey  tmpj6j0zfbj
pymp-5omzfs4r					   pymp-bei9ikn0  pymp-hqydweky  pymp-o08sw9t7	pymp-tvwdr69z  tmpj8tzqwx4
pymp-5_pfdtfd					   pymp-berkyhno  pymp-hr4yovs8  pymp-o0c0lcxr	pymp-twvu_nlb  tmplgskh1xrwandb-artifacts
pymp-5p_xds_i					   pymp-bgxwfpek  pymp-hr5p_4ss  pymp-o0hmcwva	pymp-tx00ffcm  tmpmfxif8o4wandb-media
pymp-5q_5m8lf					   pymp-_bgy7q56  pymp-hra_cgtz  pymp-o0jepdlw	pymp-txu77bbs  tmpol_hgq43wandb-artifacts
pymp-5qxdpjjs					   pymp-bhelzwte  pymp-hrc22cf7  pymp-_o0s21so	pymp-ty8mdfqa  tmppybf10yp
pymp-5rlt16x0					   pymp-bhfw5927  pymp-hrtr23se  pymp-o1hh7338	pymp-tykpaaaa  tmpq3i3awq2wandb-media
pymp-5rlwn6_h					   pymp-bhfz8nzu  pymp-hs_0tj_l  pymp-o1mmetf7	pymp-tz38gxnk  tmpvbj3c1glwandb-artifacts
pymp-5sf007t_					   pymp-bhzzq3ji  pymp-hs60ap_i  pymp-o2pg14r2	pymp-tz5zx4w5  tmpx795b348wandb-media
pymp-5smqiuo8					   pymp-_bi71ltm  pymp-ht16ywuf  pymp-o3sgbsuh	pymp-u03tywik  tmpz6nn0yezwandb-media
pymp-5svxeand					   pymp-bia6j600  pymp-htb33wdv  pymp-o41jeuwi	pymp-u19gfzmg  tmpzkt1j4lt
pymp-5tpni86a					   pymp-bip9y835  pymp-htj63od5  pymp-o48ihbt6	pymp-u1a3qbtw  tmpzov5_ez7wandb-artifacts
pymp-5turfdfv					   pymp-b_j5cszm  pymp-hu621te7  pymp-o4hse0ac	pymp-u215nj9s  ultimate-utils
pymp-5_u0d4ov					   pymp-bk6h_1kx  pymp-huvexmsv  pymp-o4tpqmbs	pymp-u21uakvb  wandb

I have this:

# - use local machine as home, can't start with cd because like in .bashrc.user since we need to figure out local path in lfs
#export LOCAL_MACHINE_PWD=$(python3 -c "import socket;hostname=socket.gethostname().split('.')[0];print(f'/lfs/{hostname}/0/brando9');")
export LOCAL_MACHINE_PWD=$(python3 -c "import socket;hostname=socket.gethostname().split('.')[0];print('/lfs/'+str(hostname)+'/0/brando9');")
mkdir -p $LOCAL_MACHINE_PWD
export WANDB_DIR=$LOCAL_MACHINE_PWD
export HOME=$LOCAL_MACHINE_PWD

is it wrong?

Also, why would it take 11T of info?

(mds_env_gpu) brando9~ $ df -h /lfs/hyperturing2/0/brando9/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md127       11T   11T   48M 100% /lfs/hyperturing2/0

[thanos-wandb]

Hi @brando thanks for reporting this issue. Could you please clarify in which files/directories are you referring to?

The pymp-* folders seem to be coming from the multiprocessing python package, and it’s an issue that has been addressed in certain Python versions, what’s your current one? However, the tmp<run-id>wandb are definitely created by wandb, could you please output disk usage per directory to see what’s actually related with wandb ones?

[thanos-wandb]

Hi @brando I wanted to follow up with you and see if you could provide us any further information requested in the previous message, so that we could understand what caused this issue for you. Thanks!

[thanos-wandb]

Hi @brando since we haven’t heard back from you, I will go ahead and close this ticket for now. Please let us now though if the issue persists for you, and we will be happy to keep investigating!