Creating Artifact Manually While Using Metaflow <> WandB Integration

W&B Help
1

Hi everyone,

I’m using the MetaFlow Integration with WandB. I understand the decorator will help me log instance variables (e.g., pandas data frames) as artifacts to WandB automatically.

In one of Metaflow steps, I’m trying to write a set of arrow files to disk and then upload these to an artifact with WandB. To do such I must run:

run.log_artifact(artifact)

Is there an easy way to extract / get the run object created by the wandb metaflow decorator so that it can be used to log this artifact explicitly?

Thanks!
Sahil

1 Like
2

Hi Sahil!

We don’t expose the run object directly, but you can log to that run implicitly by calling this inside your step.

artifact = wandb.Artifact(name, type="dataset")
artifact.add_file("file.arrow")
wandb.log_artifact(artifact)

Are you trying to save dataframes as .arrow? We save this as .parquet today, but can add an arrow option.

1 Like
3

That’s awesome! Thanks so much :slight_smile:

1 Like
4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.