How do I create a custom metric for bayesian sweeps?

I am trying to figure out how to integrate a custom metric for sweeps. It should be a composite of the number of clusters created as well as the number of outliers. I’m just getting started and the answer doesn’t jump out from the documentation. Thanks in advance.

Hi @drob707 ,

Thank you for writing in. If you are intending to track a single metric that is derived from others, then declare that variable in your script, example, count_clusters_outliers = num_clusters_created + num_outliers. You would then set the metric name in your sweep config to the variable declared above, more on sweep configuration here.

metric:
  name: count_cluster_outliers
  goal: minimize

If you want to optimize multiple metrics, consider using a weighted optimization metric. Lastly, we do support custom controllers allowing for you to create your algorithms with sweeps if you are more interested in this route.

Please let me know if you have any questions.

Regards,

Mohammad

That is super Mohammad, thanks for the link. I knew I had seen it somewhere but I just couldn’t find the reference. In my case I’m optimizing for two factors. One is the resultant number of clusters generated using HDBSCAN parameters and the other is the number of outliers identified. So in this case a particular sweep may produce 14 clusters with 1200 outliers. Two other runs might create outputs with 13/400 and 15/500. In this scenario 14/1200 is better than 13/400 or 15/500. The raw loss function values would be 0/1200, 1/400 and 1/500. The question is how should can I weight these outputs so that the bayesian algorithm doesn’t over/under compensate on for a given result?

Hi @drob707 ,

Apologies for my delayed response, I missed the notification of your reply Thank-you for following up though. This would be tricky and unfortunately I can’t come up with an equation that would work. I did come across this article online on how to balance metrics using Bayesian optimization, see here. Hope this helps.

Regards,

Mohammad