HuggingFace Trainer Doesn't Log Validation Loss to WandB

W&B Help
1

I’m trying to use HuggingFace’s Trainer API to fine-tune BERT:

    # Skip trainer till we can figure out how to specify MSE as loss
    trainer = Trainer(
        model=model,  # the instantiated 🤗 Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,
        eval_dataset=eval_dataset)

    trainer.train()

However, the validation loss is not logged to WandB:

image
image1015×195 6.61 KB

This problem was observed by someone else on the HuggingFace forum: No loss being logged, when running MLM script (Colab) - 🤗Transformers - Hugging Face Forums

Can someone please tell me how to ensure the validation loss is logged to WandB?

2

Hi Rylan, what do you have stored in the training_args variable? There are several ways to integrate W&B for HuggingFace. One is to make sure that training_args is a transformers.TrainingArguments object with TrainingArguments(report_to='wandb').

Hope this helps! Full W&B documentation for HuggingFace can be found here:

3

Yep! report_to='wandb' is successfully logging the training loss to W&B, but the validation loss isn’t being logged…

4

It seems that adding the following arguments to the TrainingArguments solved the problem :slight_smile:

        do_eval=True,
        evaluation_strategy="steps",
        eval_steps=10,
5

:+1: Nice to hear that you solved it!