Evaluation · L3 Code

Implement a reference-based scorer, an LLM-as-judge pairwise comparator, and a simple Elo update.

Reference-based scorer (exact match / unit tests)

LLM-as-judge pairwise comparator

Elo rating aggregator

Three evaluation building blocks you can implement directly.

python

import re

def extract_final_number(text):
    # crude but typical: grab the last number in the response
    matches = re.findall(r"-?\d+\.?\d*", text)
    return float(matches[-1]) if matches else None

def score_math_benchmark(model_responses, gold_answers):
    correct = 0
    for response, gold in zip(model_responses, gold_answers):
        pred = extract_final_number(response)
        if pred is not None and abs(pred - gold) < 1e-4:
            correct += 1
    return correct / len(gold_answers)

A simple reference-based scorer for a math benchmark like GSM8K.