Picture a language model as a gambler betting on the next word in a sentence. Perplexity measures its degree of hesitation: the more confident and correct the model, the lower the perplexity. It is one of the oldest and most widely used metrics for evaluating language models.
The intuition
Perplexity quantifies a model's "surprise factor" when facing text. A perplexity of 10 means that, at each word, the model hesitates on average as if choosing among 10 equally likely options. The smaller this number, the better the model predicts real language.
It derives directly from cross-entropy: it is simply its exponential.
The formula
For a word sequence $w_1, \dots, w_N$, perplexity is written:
$$\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log p(w_i \mid w_1, \dots, w_{i-1})\right)$$
where $p(w_i \mid \dots)$ is the probability the model assigns to word $w_i$ given the preceding context.
Strengths and limits
| Strength | Limit |
|---|---|
| Automatic, no human annotation needed | Depends on the tokenizer: biased cross-model comparisons |
| Good measure of pure language modeling | Measures neither truthfulness nor usefulness |
| Fast and reproducible | Weak correlation with perceived quality on complex tasks |
- Low perplexity = a model more confident in its predictions.
- It is only validly compared across identical vocabulary and corpus.
Perplexity tells you whether a model speaks the language well — not whether it tells the truth.