Hi I’ve ran an offline run only to realise resume is disabled for offline runs.
Now I’ve got several runs which all start from epoch 0
I was wondering if it is easily possible to add up the epochs?
For example if run0 = 0-140 epochs, run1 = 0-200 epochs:
Can I modify run1 and increase all the numbers by 140 so it goes 140-340 instead?
Hi @masoudr, thank you for reaching out with your query.
Currently is not possible to override previously logged metrics and steps in offline mode.
Going forward, for online runs, we are introducing the [Rewind](https://docs.wandb.ai/guides/runs/rewind) feature which would allow to do this.
What is your use case for running the runs offline ? I’d be happy to raise a feature request to have Rewind available for offline Runs as well on your behalf
Hi fmamberti,
My use case for running offline is that I am running on a remote (and national) multi-GPU slurm based system which operates in an offline mode during job executions.
Meaning that during runtime, the nodes don’t have internet connectivity.
I believe this is common practice for security and resource isolation in HPC environments, so it would definitely be a good idea to extend any new features to offline runs as well.
Thank you
Hi @masoudr , thank you for sharing your use case. I have raise a feature request for this for our product team to review.
Thanks, please let me know if something comes of it.
No problem, I will do indeed.