The default interval of monitoring points for GPU utilization is the 30s. How can I set this time?
In my model, the latency of one batch is shorter than the 30s. So I don’t think this interval is suitable.
Can I modify it by setting any variable?
I modify the two function sample_rate_seconds(self) and sample_rate_average(self) in stats.py. The path of stats.py is <Python_Path>\Lib\site-packages\wandb\sdk\internal\stats.py.
And the default value is 2 and 15. I don’t know how it works, so I set it as small as possible.
There are some notes in code,
“”“Sample system stats every this many seconds, defaults to 2, min is 0.5"”"
“”“The number of samples to average before pushing, defaults to 15 valid range (2:30)”“”
Finally, the interval is 1s. Have any other way to make the interval shorter? Like 0.5s?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
Hi Boqian, according to lines 85-87 in this code (https://github.com/wandb/client-ng/blob/fa51b2fbaf12a22a3e48f8c9c2c157642986946b/wandb/internal/stats.py#L85) we can’t return anything faster than one second. However, if you’re willing to edit the client code on your instance to where you make the interval shorter on your machine you can.
Hi Boqian,
We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
Hi Boqian, I’m going to close this request because we haven’t heard back from you but let me know if you need any further help!
Thanks a lot!
I see, but I am still confused about why the min value is 0.5. Why not set it shorter? like 0.1?
Do you have any idea about how to show the GPU utilization per 0.1 seconds?
I think it is hard to continuously modify the client code to achieve that.
Thanks!