Recently I got a banner saying “Your[sic] have exceeded your 100 GB of free storage.” when visiting the wandb website. This already seems a bit strange to me, as I have been using wandb consistently for the last year – but my storage is shown as 700 GB / 100GB so I feel like I should have gotten this message earlier. Therefore I wanted to go through the files and see if I maybe logged something really big by accident.
However, when using the file browser wandb provides to delete big files, the size of the folders seems inconsistent.
(Sorry for sending a screenshot of a screenshot but I was not allowed to show two images in my post)
Screenshot (1)
→ Run c4e8le0a is shown as the runs with the biggest size of ca 150 GB data logged.
Screenshot (2)
→ However, when I browse into this folders, the files do not add up to this amount of data at all – It seems off by a factor 100
So I really don’t know what is going on here and how to fix this problem. Only thing I can imagine is that either some invisible files are being logged or that there is some sort of bug regarding the storage calculation.
Hi @carla_s , happy to help. Appreciate the screenshot, it helps us more investigate the issue as this might be part of a known one which is currently being worked on. To check more, is it possible if you can provide your user profile, to further investigate the inconsistency.
We would like to update you with our internal discussion that this is a known issue where the sub-directory breakdown currently only fetches up to 1000 items, even if the user’s storage may contain significantly more items.
As for your storage the larger high-level summary value is correct, and that the /media and lower level folders are currently only showing the size of the displayed subset of files.
From there you can trust the high level summary value to check what to delete on your files. Hope this helps.
Rest assured that we do have some tentative design updates for this page coming in the future to address this issue.
I have a very similar issue with the displayed and calculated used storage numbers.
I arrived at this stage after naively storing every single checkpoint for some of the earlier model runs, after which I used the API (according to Delete W&B Artifacts) to delete all artifact versions without aliases (in a loop for all existing runs).
In the “manage usage overview” for the model artifacts the folders now still display multiple GB of stored data, but if I enter the folder I see one model checkpoint of 48.5MB in size.
I tried deleting the whole model via the web interface as seen in the screenshot below, which actually freed up used storage appropriately.
To me this seems to be some kind of inconsistency in the tracked folder size, or there do exist some cached versions of deleted artifact versions that are not accessible neither via the API nor via the the web interface. Interestingly, the artifact tracking summary on the top of the sizes shows 3.5 GB if I enable “View cached data” in the settings, but the sizes in the table are still the large numbers for each model. The table also still contains some models that I had previously deleted (from the runs overview).
Is there any way to update the cached folder sizes for models, or to somehow free up cached artifacts that do not exist any more?
I am assisting my colleague Joana furthur on this request. Thank you for your patience regarding this matter. I do recognize how being unable to verify your own usage numbers is problematic.
There are two main reasons why there is a mismatch. Our backend has not gotten around to marking the deleted files, once it does, it should update on your usage page, but I will test it out and flag if I notice incorrect behavior.
In terms of sometimes seeing mismatch in storage numbers in the individual folders on the usage page and their parent folder, this is due to our current design of that page. Currently the sub-directory breakdown only fetches up to 1000 items, even if the user’s storage may contain significantly more items. So for example, a parent folder might read 1.8 GB, but the subfolders only reflect a few hundred megabytes, if there are more than 1000 items in the subfolder, only those file sizes are summed up. Most often we see this in /media folders when users log many images. We do indicate similar message on the main project page for more than 10K logged files
File view is truncated to the first 10000 files
This is purposeful on the workspace to prevent performance degradation
We do have some tentative design updates for this page coming in the future to address these issues. These changes will address the inconsistency you’ve been seeing.
thanks a lot for looking into it and your reply. As a matter of fact, the backend now seems to have updated the deleted files and the total storage use has gone down significantly!
The explanation regarding the sub-folder breakdowns also make a lot of sense, I had, indeed, generated a lot of media / image files for every epoch, resulting in way more than 1000 files. Looking forward to the updates there. It didn’t impact me in a severe way yet, I was trying to maybe help track down the problem
Thanks for doing an awesome job with WandB overall, I really enjoy using it.
Hi @mohammadbakir ,
is there any news on the ability to remove one entire folder, regardless of the number of files inside?
I have multiple runs in which the numbers of files started to pile up and occuping a lot of space, and now I have to keep deleting them at 1000 files at a time even if I select the entire folder, which is too few files to clean the runs in a reasonable time.
