# Chart and artifact logging is very non-deterministic when the process is running in the cloud **URL:** \ **Category:** W&B Help\ **Tags:** artifacts, wandb\ **Created:** [November 20, 2023, 11:28am UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428 "2023-11-20T11:28:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ptrochim](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ptrochim/32/1534_2.png) [@ptrochim](https://community.wandb.ai/u/ptrochim)\ **Post date:** [November 20, 2023, 11:28am UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428/1 "2023-11-20T11:28:37Z")
I’m running finetuning jobs in the cloud using the [runpod.io](http://runpod.io) service and I use wandb to log training and evaluation metrics as well as to save the model checklpoints at the end of the finetuning process. The logging is notoriously non-deterministic - all charts and events might be logged fine in one run, and then be completely missing in the next. In most runs though a random selection of charts is being logged. For example, if I have 3 metrics A, B and C, in run one only A will be logged, in run 2 A and C might be logged, in run 3 only B might be logged - and so on and so forth. The same phenomenon applies to the artifacts - sometimes they are logged, sometimes they aren’t. It’s a real guesswork. This situation makes relying on wandb in our projects really difficult. I would like to ask for your help in understanding what might be causing this and if it’s a configuration issue, what should I do to configure the job properly. I’m happy to share how my stack is configured and grant you access to the code, but TL;DR: - I’m using Huggingface transformers.Trainer for finetuning. It logs all training metrics for me. The logging frequency for those has been set. - I’m using proprietary Huggingface transformers.TrainerCallbacks to log evaluation metrics and the artifacts - I’m running the training job out of a docker container from a pod running in [runpod.io](http://runpod.io) Please let me know what details you would require to assist me with this problem. Kind regards, Piotr Trochim ---
**Author:** ![nathank](https://avatars.discourse-cdn.com/v4/letter/n/7ea924/32.png) [@nathank](https://community.wandb.ai/u/nathank)\ **Post date:** [November 22, 2023, 4:21pm UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428/2 "2023-11-22T16:21:12Z")
Hi @ptrochim, Could you share a link to a few a runs where you are seeing different logging behavior and I can take a look? Also, if you are able to share a minimal reproduction of your code that we can use to test with I can try to reproduce this on my side. Thank you, Nate ---
**Author:** ![nathank](https://avatars.discourse-cdn.com/v4/letter/n/7ea924/32.png) [@nathank](https://community.wandb.ai/u/nathank)\ **Post date:** [December 4, 2023, 11:00pm UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428/3 "2023-12-04T23:00:33Z")
Hi @ptrochim, I wanted to follow up and see if this was still an issue? ---
**Author:** ![ptrochim](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/ptrochim/32/1534_2.png) [@ptrochim](https://community.wandb.ai/u/ptrochim)\ **Post date:** [December 4, 2023, 11:16pm UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428/4 "2023-12-04T23:16:36Z")
Hi Nathan, Turns out this was caused by a bug in my code. Thank you for following up, let’s close the thread. Piotr ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [February 2, 2024, 11:16pm UTC](https://community.wandb.ai/t/chart-and-artifact-logging-is-very-non-deterministic-when-the-process-is-running-in-the-cloud/5428/5 "2024-02-02T23:16:44Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.