Picture a student cramming: at first, every hour of study lifts their grades. Then comes a point where, by memorizing the same textbook word-for-word, they absorb useless details and stop actually understanding. Early stopping tells a machine learning model: "stop at exactly the right moment, before you start learning noise."
The mechanism
During training, we track two error curves: one on the training data and one on a held-out validation set. Training error almost always keeps falling. But validation error eventually rises again — the hallmark of overfitting.
Early stopping monitors that validation error and halts training once it stops improving for a fixed number of epochs — a grace period called patience.
When to stop?
| Phase | Training error | Validation error | Decision |
|---|---|---|---|
| Underfitting | high | high | keep going |
| Optimal | low | minimal | stop here |
| Overfitting | very low | rising | too late |
Formally, we keep the weights $\theta^*$ at the validation-error minimum:
$$\theta^* = \arg\min_{t} \; E_{\text{val}}(\theta_t)$$
In practice
- Patience: number of epochs tolerated without improvement before stopping.
- Restoration: reload the best weights, not the last ones.
- It acts as a nearly free, widely used form of regularization.
A model that stops in time beats one that memorizes its own mistakes.