Run summary statistics do not match run history logs


I recently ran a sweep, and all the runs had summary statistics incongruent with the respective run history logs. For instance, a run may show a summary statistic for metric X to be 3.3705 but the last history log for the same metric shows 3.380950. Normally, if the last history log is 3.3705, then the last history log should be 3.371.

The sweep’s runs were run in offline mode and then synced afterwards with wandb sync. I’ve done this before and never experienced any problems, so I dont know what went wrong this time.

I can share the run/sweep but not publicly in this forum. I can also share my training code.

Also, the summary statistic does not appear at all in the metric’s entire log history, so it seems to be a totally fabricated value.