Can I implement a custom stopping condition in an Evaluation in Weave?

W&B Help
1

I’m using Weave to evaluate a model on sentence pairs generated from a long text, consisting of a reference sentence and a changing potentially connected sentence. However, I don’t want to evaluate all pairs (too expensive) I want to stop further evaluation for a reference sentence after detecting three consecutive sentences that didn’t show a connection (so I move away from the reference sentence on a sentence-by-sentence basis, evaluating each sentence on the way and ideally stopping after three turned out to not show a connection).

Is is possible to add a condition to stop the evaluation loop when that condition is met? And if not, how would you set up an Evaluation for such a use case?

2

Hello Mirek!
You can achieve this by wrapping the “model” function in another wrap that has the behavior you want. For example:

dataset = [{'reference_sentence': ...}, ...]@weave.opdef eval_driver(sentence): missed_connection_count = 0 while missed_connect_count < 3: result = my_model_fn(...) ...eval = weave.Evaluation(dataset=dataset)eval.evaluate(eval_driver)

Best, W&B

3

Thank you, will try it out!

4

Sure! Please let me know if you have any more questions!

5

Hi Mirek,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

6

Hi Mirek, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!