Evaluation · L2 Mechanics

How benchmarks, LLM-as-judge, and preference-based leaderboards are actually constructed and scored.

Reference-based scoring

Compare to a known correct answer
Exact match / multiple choice / unit tests
Cheap, objective, but rigid

Reference-free scoring

No single correct answer exists
LLM-judge or human rates quality directly
Flexible, but subjective and judge-dependent

The fundamental scoring split underneath every evaluation method.

Benchmarks split into two families by how they're scored. Reference-based benchmarks (MMLU's multiple choice, GSM8K's numeric answer, HumanEval's unit tests) have an objectively checkable correct answer, so scoring is fast, cheap, and deterministic. Reference-free evaluation (most real chat quality, creative writing, following nuanced instructions) has no single correct answer — you need a judge, human or model, to assess quality directly.