How to get runtime, start time and end time of finished run

I’m trying to compute some runtime statistics about runs in the finished state. I want to retrieve runtime, start-time, and end-time timestamps shown in the runs table app but through the SDK. I searched in the doc and also tried dir(run) to check if some Run object attribute is not documented. Still, I didn’t find any useful attribute except for an undocumented attribute called run.createdAt.
Any hint?

After digging a little bit I discovered:

  • there is a “private” attribute in run.summary['_runtime'] expressed in number of seconds. I can add this duration to the run.createdAt to get the end datetime.
  • there is also another (undocumented) heartbeatAt attribute that should signal the last heartbeat sent from the run process to the server, with some error, this should be more or less coincident with the end of a run). I can subtract heartbeatAt - createdAt to get the runtime.

In this way, everything matches what is shown in the app UI. However, I found something a little bit odd in the UI app:

  • The “create” attribute (column of the table) shows the date-time keeping the original timezone (I run the script on an AWS EC2 instance that by default has the UTC timezone)
  • the “end time” is shown in local time

Can you confirm that my reasoning and findings are correct?

Hi @pio-fina , thank you for reaching out with your questions and sharing your findings regarding the timestamps.

Regarding the UI - Yes your findings are correct. Currently, the Created column is displayed in UTC while the End Time is displayed in Local time. This is an issue we are aware of and we are working to make sure the timezones displayed across the UI are consistent. I will add your report to the open ticket and will keep you posted on the progress.

Regarding the timestamps you can use via the API:

  • run.createdAt is the best options for the Created time and runtime
  • To get the Runtime for a run, you may want to use run.summary['_wandb']['runtime'] rather than run.summary['_runtime']. summary['_runtime'] uses the timestamp from the summary which correspond to the timestamp for the latest update for the Run summary and may not be the same as when the Run is finished (i.e. when the scripts complete or wandb.finish() is called).
  • run.heartbeatAt is indeed the most reliable option to get the End Time of a run.

Please let me know if you have any further questions.

thanks @fmamberti-wandb, even a simple timezone designator in the string could help in this case :slight_smile: