How to plot performance metrics against non-summary stats?

How to plot performance metrics against non-summary stats?

I am running sweeps over a few different datasets within the same job (each for a different location, it’s a multi-task learning problem) and I am making wandb logging calls such as:

  wandb.log({
    "false_negative_rate" : 0.043,
    "false_positive_rate" : 0.261,
    "location": "Boston",
    "number_training_instances": 50
  })

  ...

  wandb.log({
    "false_negative_rate" : 0.017,
    "false_positive_rate" : 0.145,
    "location": "Boston",
    "number_training_instances": 100
  })

  ...

  wandb.log({
    "false_negative_rate" : 0.076,
    "false_positive_rate" : 0.334,
    "location": "Miami",
    "number_training_instances": 50
  })

  ...

  wandb.log({
    "false_negative_rate" : 0.048,
    "false_positive_rate" : 0.172,
    "location": "Miami",
    "number_training_instances": 100
  })

After logging, I’d like to be able to use the dashboard to create various scatter plots such as

Plot false_negative_rate versus false_positive_rate
  where location == "Boston"
    and number_training_instances == 50

Plot false_negative_rate versus number_training_instances
  where location == "Washington"

Plot false_negative_positve_rate versus location
  where number_training_instances == 100

However, the filters in the dashboard only provide me the ability to filter/plot the last values logged (as opposed to any of the values logged). E.g., I can only filter on

 - number_training_instances == 100
 - location == "Miami"

because these were the last values logged for these attributes in each sweep.

Is there a way to get the plotting flexibility I want using wandb’s existing features?
Cheers

Hey @ted-positronix,

wandb offers flexible plotting capabilities via the usage of the Custom Chart panel. You can access all logs of your metrics via the history or historyTable tab. This can then be passed to something like the custom scatter plot chart we offer.. At this point your filtering should adjust the custom chart as you’ve seen with the summary plots you tried.

Let me know if I’m misinterpreting your use case!

Regards,
Anish Shah

Many thanks Ashish! I will need to do some reading to see how to use this functionality. I don’t know Vega, etc.

I will post further questions here if I run into trouble.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.