A benchmark is to artificial intelligence what a standardized exam is to students: an identical test for everyone, designed to objectively compare the performance of different models on a well-defined task. Without a benchmark, claiming one model is "better" than another is mere opinion; with one, you get a reproducible number.
What a benchmark is made of
A benchmark combines three ingredients:
- an evaluation dataset (questions, images, problems);
- a precise task (classify, translate, answer, reason);
- a metric that turns answers into a score.
The metric depends on the task: accuracy for classification, F1 score, BLEU for translation, or pass rate. Accuracy is computed simply:
$$\text{Accuracy} = \frac{\text{correct predictions}}{\text{total number of examples}}$$
Famous examples
| Benchmark | Domain evaluated |
|---|---|
| ImageNet | Image recognition |
| GLUE / SuperGLUE | Language understanding |
| MMLU | General knowledge and reasoning |
| HumanEval | Code generation |
Strengths and limits
Benchmarks structure progress: they give research a shared target. But they have a downside. Data contamination (a model having already seen the test during training) artificially inflates scores. And optimizing for a benchmark can drift away from real-world use — this is Goodhart's law: "when a measure becomes a target, it ceases to be a good measure."
A benchmark measures what it knows how to measure, never intelligence as a whole: it is a compass, not a map of the territory.