Using the (relatively new) functionality wandb.init(..., resume_from=f"id?_step=N), runs can be resumed at a given step and previously logged metrics for steps N+1, … will be overwritten. The data for steps 0-N should stay. This seems to work for scalars but does not work for images, based on my observation.
If a run logs images and crashes at iteration >N, the images are initially logged correctly (they can be displayed in the web interface). However, when I resume the run at step=N using the method above, all the images for steps 0-N are suddenly not available while the newly logged images for steps >N are again ok. In the web interface, the previous images for steps 0-N are “seemingly there” - the step slider goes to 0 - but they take forever to load and effectively cannot be displayed.
I can provide a minimal example code if interested.