I guess this use-case is common but I cannot figure it out…
I would like to log one model per run, and in the end, be able to load the best overall model for production.
So like :
Run 1,2,3,4,5…
run.log_artifact(my_model_artifact)
Production:
artifact = api.artifact.get_best_of_all_my_runs()
For now my solution is :
Runs :
artifact.save() # with the same name so only one artifact for all runs
Production
artifact = api.artifact("entity/project/artifact:alias") # Get the only model (which also should be the best)
Hi there! You can do this in a couple of ways but you don’t have to limit yourself to logging just one artifact. A common flow would be to log a model checkpoint every but then to also log a “best model” artifact. Since artifacts are versioned you don’t have to worry about renaming the new “best model” artifact. Then at the end of your run you not only have an artifact history of your model at each of the checkpoints but also a versioned history of all the best models.
I indeed do this and have my best model for each run and it works great.
Then I can see all my runs in the Table Tab, with their hyperparameters, and a column for the score/accuracy.
However my questions is : Could I get the best model among all my runs so I can use it in production ?
Let say I have one parameter P :
I run Run1 with P1, get Accuracy A1 // Run 2 with P2 get Accuracy A2 // Run3 with P3 get Accurary A3
Then in the table I can see Run1, Run2 and Run3 each with logs / metrics / models / Parameters.
I would like to be able to do in my production/inference code :
“Select best Accuracy model from Run1, Run2, Run3 so I can use the best model in inference mode in production”
I hope my explanation was clear…
Thanks a lot for your time.,
I think I understand your question a little better. We are currently working on a feature that will make exactly what you’re asking for super clean and easy
In the meantime the most straightforward way of going about this would be to save the performance of all of the metrics you care about to the run.summary of the run that produced that model. Then from the api you can query all the runs in the project and select the best run using the metrics you set in the summary. Then you can get the download the best model and use it as an input to your production code.
Here is a related example querying the best model from a sweep but the process will be slightly different for you use case.
Thanks for the reply, this is indeed what I was looking for !
I will be glad to test the new feature (even in beta) but I’m sure you will communicate when it will be available.
Else I was thinking about using the API and you just confirmed it so I will be headed for that.
