The runs charts changed after run finished

W&B Help
1

I am using wandb with openai finetuning according to this guide fine-tuning-integrations, after run finished, i have charts in the above part of the screenshot, but after several mins ago (It already run finished, so for sure the charts should be the same as before), but charts changed (in the bottom part of the screenshot), so what happened?

aaa
aaa1406×1158 179 KB

2

Hi @kevinzhang19870314 Good day and thank you for reporting this to us. Are you using Full fidelity for the charts? If yes, do you know if the charts will return the previous state when you switch it to Random sampling?

image

3
  1. Yes, I am using Full fidelity for the charts by default.
  2. Charts will not return the previous state when switch it to random sampling.
4

Thanks for confirming this @kevinzhang19870314 Do you know if this same behavior can happen on any new experiments that you create? Can I also ask for the workspace URL where we can view these charts so we can review? Thank you!

5
  1. It not happen on my recently runs;
  2. This is the workspace url that I copied from here: Weights & Biases;
    image
6

Thank you @kevinzhang19870314 for sharing more details about this. Are you able to log your experiment on a new project and confirm if the same issue appears?

7

The same thing happened on my latest runs. See below screenshot. Here is the workspace url workspace url.

Untitled
Untitled1831×1901 364 KB

8

Thanks for flagging this @kevinzhang19870314 . We’ll take further review into this behavior and investigate.

It will help if you could please share a code snippet so we can review how the runs are logged to wandb? If you can share more details about your setup, we can take a look at it for review.

Also, could you please share your current SDK version? You can get this by running wandb --version. Thank you!

9

Unfortunately, I can’t provide more info, as I am using OpenAI fine-tune API. So the code and more details you can find in this docs.

10

Hi @kevinzhang19870314 Thanks for this! Does this mean the code you are using is directly from the docs? If not, will it be okay just to give us some details on what experiment are you trying to setup just for our review on what could be happening on the charts? Thank you