Hi all,
I have two experiments logged in WandB with nearly identical configurations, but one crashed, and the other completed successfully. Each experiment logs a Git state with a branch name and commit hash, like this:
git checkout -b "branch_name" <commit_hash>
However:
- I can’t retrieve one of the commit hashes locally—
git checkout
gives me a “bad object” error, and it’s not found withgit fetch --all
. - In the WandB code directory, I only see the main module that launches the experiment. In my case, it’s just:
from trainer import train
train()
I suspect the branch and commit were only created in the environment where the training job ran and were not pushed to the remote repository.
Question:
- Can I retrieve the full code or Git state directly from WandB or the environment?
- Does the branch/hash information in WandB mean the code is accessible, and if so, how do I retrieve it?
- Are there best practices to ensure such states are reproducible for debugging?
Thanks in advance!