I’m trying to compute some runtime statistics about runs in the finished state. I want to retrieve runtime, start-time, and end-time timestamps shown in the runs table app but through the SDK. I searched in the doc and also tried dir(run)
to check if some Run object attribute is not documented. Still, I didn’t find any useful attribute except for an undocumented attribute called run.createdAt
.
Any hint?
After digging a little bit I discovered:
- there is a “private” attribute in
run.summary['_runtime']
expressed in number of seconds. I can add this duration to therun.createdAt
to get the end datetime. - there is also another (undocumented)
heartbeatAt
attribute that should signal the last heartbeat sent from the run process to the server, with some error, this should be more or less coincident with the end of a run). I can subtractheartbeatAt - createdAt
to get the runtime.
In this way, everything matches what is shown in the app UI. However, I found something a little bit odd in the UI app:
- The “create” attribute (column of the table) shows the date-time keeping the original timezone (I run the script on an AWS EC2 instance that by default has the UTC timezone)
- the “end time” is shown in local time
Can you confirm that my reasoning and findings are correct?
Hi @pio-fina , thank you for reaching out with your questions and sharing your findings regarding the timestamps.
Regarding the UI - Yes your findings are correct. Currently, the Created column is displayed in UTC while the End Time is displayed in Local time. This is an issue we are aware of and we are working to make sure the timezones displayed across the UI are consistent. I will add your report to the open ticket and will keep you posted on the progress.
Regarding the timestamps you can use via the API:
run.createdAt
is the best options for the Created time and runtime- To get the Runtime for a run, you may want to use
run.summary['_wandb']['runtime']
rather thanrun.summary['_runtime']
.summary['_runtime']
uses the timestamp from thesummary
which correspond to the timestamp for the latest update for the Run summary and may not be the same as when the Run is finished (i.e. when the scripts complete orwandb.finish()
is called). run.heartbeatAt
is indeed the most reliable option to get the End Time of a run.
Please let me know if you have any further questions.
thanks @fmamberti-wandb, even a simple timezone designator in the string could help in this case
Hi @pio-fina , We have now merged a fix and when covering on the Created column the data will be shown in local time, as all the other times for other fields.
I will mark this as complete, please don’t hesitate to reach out if you have any further questions.