How to stop weights & biases (wandb) from creating random tmp files?

W&B Help
1

I have a million tmp files due to wandb on my home folder. I don’t know why. Why are they being created & how do I stop it?

anaconda					   tmpa0wf77f_wandb-artifacts  tmpmv2ewoi1wandb-media
anaconda.sh					   tmpa3t655lewandb-media      tmpmvhp3i6ewandb-media
data						   tmpa6roiz80wandb	       tmpmwszwea4
debug-cli.brando9.log				   tmpal7kticfwandb-artifacts  tmpn0p49w1hwandb-media
diversity-for-predictive-success-of-meta-learning  tmpalsnd4g1wandb-media      tmpnbxtnojbwandb-artifacts
div_install_miniconda				   tmpambb3rm9wandb-artifacts  tmpngpy96dawandb-artifacts
div_install.out					   tmpapco0xetwandb-media      tmpnqezeggnwandb-media
iit-term-synthesis				   tmpaqf80v_hwandb-media      tmpo3cug5nzwandb-media
main.sh.e449240					   tmpaqigpze6wandb-artifacts  tmpoc4x6l22wandb
main.sh.e457075					   tmpaw1kvgtgwandb-media      tmpofql583uwandb-media
main.sh.e760266					   tmpay63rbxgwandb-media      tmponqiggzswandb-artifacts
main.sh.err748250				   tmpb2clycf5		       tmpoqem6uclwandb-media
main.sh.err849450				   tmpbbrfn_kmwandb-artifacts  tmpoqth0mgpwandb-media
main.sh.err923818				   tmpbcxatqdiwandb-artifacts  tmppbd5bfm_wandb
main.sh.err962904				   tmpbgewkz10wandb	       tmppbnpm41gwandb-media
main.sh.o449240					   tmpbsd96o99wandb-media      tmppwxmebn1
main.sh.o457075					   tmpbtlp8zomwandb	       tmpq396kfo1wandb-artifacts
main.sh.o748250					   tmpby3a9u8ywandb	       tmpq8jryat0wandb-media
main.sh.o760266					   tmpc45e2nlxwandb-media      tmpqc65bfs0wandb
main.sh.o849450					   tmpc4m5b21_		       tmpqexdhp6gwandb-artifacts
main.sh.o923818					   tmpcap20jmdwandb-media      tmpqh3uu7v2wandb-media
main.sh.o950686					   tmpcl2sb6j_wandb	       tmpqh99a72vwandb-media
main.sh.o962904					   tmpcsncx8x4wandb-media      tmpqmim4sxywandb
miniconda					   tmpd7dhluxmwandb	       tmpqpfcq9uwwandb
miniconda.sh					   tmpdbbb3hw_wandb-artifacts  tmpqtds4jdiwandb-artifacts
nohup.out					   tmpdfrjyk90wandb-media      tmp_qz8pu0xwandb-artifacts
nohup.out449240					   tmpdhqwaxygwandb	       tmpr98qj7auwandb
nohup.out457075					   tmpdpj3bfz0wandb-artifacts  tmprfwooa22wandb-artifacts
nohup.out760266					   tmpdqzzy7v3		       tmpri9xu8i_wandb-media
pycoq						   tmpdr6fbpctwandb	       tmprj4g0kkhwandb
test.py						   tmpejwo7axlwandb	       tmp_rla0cb9wandb-media
tmp						   tmpekqp7b2dwandb-media      tmprmrasn0fwandb-media
tmp03kmjan0wandb				   tmpf3pk0_3t		       tmpr_yrhzj_wandb
tmp07zhon11wandb-media				   tmpf4w8yhsswandb-media      tmprzxltg0lwandb
tmp0pkwjwg8wandb				   tmpf_6vd6hkwandb-media      tmps0beul64wandb-media
tmp0ypuhnktwandb-media				   tmpf7vuwlipwandb	       tmps5qf0_w0wandb
tmp0zk3_ok1wandb				   tmpfc8ltujrwandb-media      tmpsp2djjg6wandb-artifacts
tmp14xa24j_wandb				   tmpfmcmwgb8		       tmpsqe0vylnwandb
tmp1f3gqdq1wandb-media				   tmpfqhl6c9vwandb	       tmpstniop3twandb-media
tmp1hmrx3xnwandb				   tmpfvkvyklpwandb-media      tmpsv3n4fi7wandb-media
tmp1nxq8dmowandb				   tmpfxuc2zwjwandb-artifacts  tmp_t3mkuy4
tmp1r2xah97wandb-media				   tmpg051c49z		       tmptb0urf26wandb
tmp1sdb3vnqwandb-media				   tmpg16e6zpxwandb-media      tmptgq1h308wandb-media
tmp1wq9i7tmwandb-media				   tmpg2qfjo5pwandb-artifacts  tmpthtghn1wwandb-media
tmp27k3evykwandb-artifacts			   tmpg34wt2g1wandb-media      tmptkp9qpgxwandb-media
tmp2ncmg9jmwandb-media				   tmpggaltim9wandb-media      tmptqn9w7rawandb-artifacts
tmp2qxmugpjwandb-media				   tmpgj6gyqw6wandb-media      tmptsqb0lwrwandb
tmp2w92xlzowandb				   tmpgpv_1hxk		       tmptub9i1zzwandb-media
tmp39lds7tywandb-media				   tmpgswv7jpn		       tmpu0k6cuycwandb-media
tmp3ncj9tdewandb-artifacts			   tmpgvz0_o1h		       tmpu6uv_y0pwandb
tmp3qlpfrylwandb-media				   tmpgyarr2jxwandb	       tmpumz7hmaiwandb-artifacts
tmp3snbanfnwandb				   tmph6m9dpa_wandb	       tmpun08cdmwwandb-artifacts
tmp3xrxd920wandb-artifacts			   tmph8n3b36swandb-media      tmp_uqnbz5n
tmp3zmnx6jxwandb-artifacts			   tmphddkq3_3wandb	       tmpurv7_fe2wandb
tmp4103eum2wandb				   tmphmva83y4wandb	       tmpuwoxzzfvwandb-media
tmp421qmhu3wandb				   tmphs6erdxrwandb-media      tmpvb5bk2js
tmp48khxd0nwandb-artifacts			   tmphshrf9juwandb-artifacts  tmpvd_wklrtwandb
tmp49fv73y2wandb-media				   tmpi31q87a0wandb-artifacts  tmpvg_71vtdwandb-media
tmp49sad_g1wandb-artifacts			   tmpiu05wr2_wandb	       tmpvlxyr3eawandb-media
tmp4c4800_xwandb-media				   tmpivnhmojfwandb	       tmpvqmyjo4pwandb-media
tmp4clbe6xvwandb-media				   tmpj16iv0rbwandb-media      tmpw10pvrxxwandb-media
tmp4nuizjduwandb-media				   tmpj4nmef2_wandb-media      tmpw8eaus7xwandb-media
tmp5aiik94rwandb-media				   tmpj6k4pajlwandb-artifacts  tmpw97zp6pqwandb-media
tmp5jusc1czwandb-media				   tmpjetcrm92wandb-media      tmpwkzzglljwandb-media
tmp5ks7vxpqwandb				   tmp_jfnbfwcwandb-artifacts  tmpwlpoppuwwandb-media
tmp5ss5gfoqwandb-media				   tmpjhcfo3sjwandb-media      tmpwok9yxtqwandb-media
tmp61l257guwandb-media				   tmpjhkja0n4wandb-media      tmpwqbb7793wandb
tmp66a_30crwandb				   tmpjq3bc0iywandb-media      tmpwu7oid1swandb-media
tmp6_95ss09					   tmpjseq6pjrwandb	       tmpwwmlqm3gwandb-artifacts
tmp6eb3e1v_wandb-artifacts			   tmpjywyihxswandb	       tmpwys0txyz
tmp6ev3bw0kwandb-media				   tmpk7eb9cxxwandb-artifacts  tmpx0i8_uxdwandb-media
tmp6j_pagmjwandb-media				   tmpki8mvo7pwandb	       tmpxby6g44swandb-media
tmp6uz84wzpwandb				   tmpkiqc2rxywandb-media      tmpxdsg3tk8wandb-artifacts
tmp7dmpqrecwandb				   tmpklsmildcwandb-media      tmpxm2j1915wandb
tmp7fzpg3pjwandb-artifacts			   tmpkvhsusnzwandb-artifacts  tmp_xmydpcnwandb-media
tmp7iafm3cywandb-media				   tmpkvt13pjiwandb	       tmpxpj1qkhnwandb-media
tmp7m0tkcx7wandb				   tmpkxhoutmnwandb	       tmpxqnwoio_wandb-media
tmp7p7ko5c1					   tmpl32i_q8cwandb-artifacts  tmpy3sbukw0wandb-artifacts
tmp7xmnpnjxwandb-media				   tmpldwit_dswandb-media      tmpy4tqgd9q
tmp80lef2dvwandb				   tmplf9oolt5wandb	       tmpy5mlqvf2
tmp89e0j4bjwandb-artifacts			   tmplgmiofgnwandb-artifacts  tmpy5y0mxbrwandb
tmp8h7rchd9wandb-artifacts			   tmplw1n5b69wandb-media      tmpydoskv75wandb
tmp8l4njuz2					   tmplx9285iywandb	       tmpyx791iakwandb-media
tmp8lxb4u_0wandb				   tmp_lzx3b9dwandb-media      tmpyy2hv95pwandb-artifacts
tmp8lyo8smzwandb				   tmpm1c4zy4twandb-artifacts  tmpz0gx4ikiwandb-media
tmp8q4h8lu7wandb-artifacts			   tmpm2755ginwandb-artifacts  tmpz26cajmh
tmp_8uvnuf2wandb				   tmpm56u1aa5wandb-media      tmpz5s198hnwandb-artifacts
tmp96o0qfii					   tmpm9gk_r6swandb-media      tmpz6oxqu4vwandb-media
tmp974f9ciawandb-media				   tmpm_9gv20owandb-media      tmpzgz2lbnmwandb
tmp98ec7tz8wandb-media				   tmpmdak3eqkwandb-media      tmpzyqel_hcwandb-artifacts
tmp9i4bx28vwandb				   tmpmekovp_5wandb-artifacts  tmpzzjlqqh8wandb-media
tmp9l8xrnlqwandb-media				   tmp_mpucxdiwandb-artifacts  ultimate-utils
tmp9y_56adfwandb-media				   tmpmsn3sy8mwandb-artifacts  wandb

Additional Files

No response

Environment

WandB version:
(metalearning_gpu) brando9~ $ python
Python 3.9.13 (main, Oct 13 2022, 21:15:33)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import wandb
wandb.version
‘0.13.5’

OS: ubuntu/linux

(metalearning_gpu) brando9~ $ cat /etc/os-release
NAME=“Ubuntu”
VERSION=“16.04.7 LTS (Xenial Xerus)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 16.04.7 LTS”
VERSION_ID=“16.04”
HOME_URL=“http://www.ubuntu.com/”
SUPPORT_URL=“http://help.ubuntu.com/”
BUG_REPORT_URL=“Bugs : Ubuntu”
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Python version: 3.9.13

Versions of relevant libraries:

related:

2

Hi @brando thank you for reporting this. Besides these directories which are controlled by the environment variables WANDB_DIR, WANDB_CACHE_DIR and WANDB_CONFIG_DIR, the wandb SDK in some cases such as logging wandb.Image objects or Artifacts it will additionally write some temporary files in the tmp directory for code’s efficiency reasons.

It seems that in your system, these were created in your home directory. Could you please post here the output of ls -lah /t* to check the write permissions, and of pwd from within the directory where these files were generated? I have filed a feature request to our engineering team to expose an environment variable for this directory so that it can be controlled where to write the temp files. Please let me know if that would help. Also, in terms of logged data, has everything uploaded fine in W&B, and in that case would you need any help to remove these temp files?

3

HI!

I’m encountering this logging thing in my /tmp folder when downloading artifacts. From your explanation, I guess it’s the intended behaviour, but is there any way to just avoid wandb to produce these files?

Thanks!

4

Hi @jceamoran indeed that’s intended and it isn’t currently possible to switch this off. You can however remove these files once your training has completed and everything has synced with W&B App. May I please ask what’s the reason that you wouldn’t want to have some temporary files written in your disk? I can add this context to an existing feature request to allow users control the location of this directory, would this work for you?

5

Hi @thanos-wandb !

I was just wondering how to make my deployments lighter. Nothing I cannot live with. However, being able to choose location of these files would be an interesting feature tho!

Thanks for your answer :slight_smile:

6

Hi @jceamoran thanks for the context, good to hear it’s not currently blocking you. You could in the meantime call a command as follows, to cleanup the /tmp folder after your training has finished and synced everything to W&B App:
os.system('find /tmp -type d -name "*wandb*" -exec rm -rf {} \;')

However, I understand that this might not be ideal to be done by you, so I have also increased the requests for this feature, and we will reach out to you here on any related updates!

7

Hi, I just noticed this behavior too. We have literally millions of wandb directories in /tmp, so just listing the contents of the directory is slow, even though the total space they take up is only a few GB. I guess it would be nice if wandb did not produce so much trash, and also if it does, maybe they could be stored in per-user subdirectories of /tmp instead of directly in /tmp

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.