Picture a student who memorizes the answers to past exam papers without understanding the material: dazzling on questions they've already seen, lost the moment one word changes. That is exactly overfitting: an AI model that fits its training data so perfectly that it memorizes the noise and accidental details, at the expense of its ability to generalize to new data.
Spotting overfitting
The classic symptom is a wide gap between two scores. The model reaches a very low training error, yet its validation error (on data never seen before) starts climbing back up.
| Signal | Underfitting | Good fit | Overfitting |
|---|---|---|---|
| Training error | high | low | very low |
| Validation error | high | low | high |
| Model complexity | too simple | right-sized | too high |
The real goal is to minimize the expected error on new data, not merely on the known sample:
$$\mathbb{E}_{(x,y)}\big[\,L!\left(f(x),\,y\right)\big]$$
How to prevent it
- Regularization (L1/L2 penalties, dropout) to curb complexity.
- Cross-validation and early stopping.
- Data augmentation to diversify the examples.
- Prefer a simpler model when performance is comparable.
Overfitting is the opposite extreme of underfitting: the whole craft lies in balancing the bias-variance trade-off.
A good model does not recite the past: it anticipates the unknown.