In an effort to comply with the 100GB storage limits, I have been looking closely at the “usage” page (https://wandb.ai/usage/your-username-here), and indexing, then deleting files using the export API. There is sometimes a significant discrepancy between the runs size reported by https://wandb.ai/usage/your-username-here/your-project-here/runs VS https://wandb.ai/usage/your-username-here/your-project-here/runs/your-run-here.
There is a set of files that are returned when using Run.files() call, but they are invisible when looking at the run on the web dashboard (I suspect they are the cause of that size report discrepancy). In the extreme case of some of my runs, they are json files, with paths starting with artifact/, and they number in the tens of thousands: e.g.:
These files obviously relate to artifacts, but the artifacts themselves are queried through Run. logged_artifacts(). I have been deleting artifacts in my past runs, keeping only less than 400 per run (these are very small artifacts), and I have been expecting the count of these “manifest” files to go down, But it instead seemed to have created additional .deadlist files…
My questions are:
What are these files?
Are they important?
What can I expect to happen if I manually delete them through the export API?
Firstly, could you describe the significant discrepancy between, preferably with screenshots? And to confirm some things so that I can reproduce this myself:
You are deleting the artifacts via run.logged_artifacts() then using .delete() these artifacts
These artifacts are of old runs and so all runs are finished when deleting these artifacts.
As for the wandb_manifest.json, this is our file that we use to store data about the artifact. For example, if the same file is uploaded twice under different artifacts, then we will not have you upload it twice and have it reference the other file. This is done via the wandb_manifest.json. As for the .deadlist that is showing up, that should be a temporary file that is generated while the manifests are being deleted.
As for the affect of deleting this file, it is important to keep these since we rely on this manifest when using Weave functions and when referencing the Artifact in other parts of the code. Since these files are generated per artifact, they are not normally a significant amount of data that is stored. However, if you have many runs that each create its own artifact, this manifest will be created within each one. I will have to check if the manifest gets included in this calculation.
Here are the screenshots that show the discrepancy. As you can see, The overall run storage size is listed as 9.4GB, but when clicking into the run and seeing the breakdown, the overall size becomes 3.3GB. Note that those artifacts/ files are not visible in the breakdown.
Thanks for the insight in the role of the files. If the .deadlist files are meant to be temporary, can I expect them to disappear on their own over time? Note that in the run I show in the screenshots, there are almost as many .deadlist files as there are wandb_manifest.json, at the count of around 30,000 each (working from memory for the count).
