Hi everyone!
I’ve been using Weights & Biases to track some reinforcement learning experiments I’m running on iOS devices, but I’ve hit a weird synchronization issue with my “Step” counts and loss curves.
I’m currently using a mobile environment where I’ve deployed some custom delta scripts to handle the environment interaction and data collection. I’ve been following the logic for these delta scripts to ensure that the mobile executor is efficiently capturing state changes, which I then try to pipe into my W&B project via a simple API call. The problem is that whenever the delta scripts fire off updates at a high frequency (more than 5–10 times per second), the W&B dashboard seems to drop about 30% of the logged metrics, or I see “Network Timeout” errors in the background log.
Has anyone else here integrated delta scripts or similar mobile-based executors with the W&B Python or REST API? I’m trying to figure out if I should be implementing an “on-device” buffer to batch my logs before sending them to the W&B servers, or if there’s a specific rate-limit on the WandB log() function that I’m hitting because of the executor’s speed. I’m also seeing some strange timestamp drift where the metrics from the delta scripts don’t align with my system’s UTC time, making the “System/CPU” charts in my workspace look completely disjointed. If any MLOps experts here have tips on stabilizing high-speed data streams from mobile executors into W&B, I’d really appreciate the help!