Picture two automatic graders that score an essay without truly reading it: they simply count how many words and phrases from the AI's output also appear in a human-written "model" answer. BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are these two tape-measures — the historical benchmarks for evaluating machine translation and text summarization respectively.
The principle: counting n-grams
Both metrics rely on n-grams (sequences of 1, 2, 3… consecutive words) shared between the candidate text and one or more references. The difference lies in the angle of measurement:
- BLEU measures precision: of the words the AI produced, how many are "correct"? It adds a brevity penalty to discourage overly short outputs.
- ROUGE mainly measures recall: of the words in the reference, how many did the AI manage to reproduce? Hence its summary-oriented bias, where the main fear is omitting essential content.
$$\text{BLEU} = BP \cdot \exp!\left(\sum_{n=1}^{N} w_n \log p_n\right)$$
where $p_n$ is the precision of n-grams of order $n$ and $BP$ the brevity penalty.
BLEU vs ROUGE
| Criterion | BLEU | ROUGE |
|---|---|---|
| Typical task | Translation | Summarization |
| Main axis | Precision | Recall |
| Variants | — | ROUGE-N, ROUGE-L (longest common subsequence) |
Limitations
These metrics do not "understand" meaning: a perfect synonym is penalized, while a grammatically absurd sentence can still score well. Yet they remain fast, reproducible, and free of human cost.
Fast and objective, BLEU and ROUGE measure surface overlap — never genuine understanding.