Picture an archer: some arrows miss the target consistently to one side (systematic error), others scatter everywhere (instability). The bias-variance trade-off captures this fundamental tension: a machine-learning model cannot be both too simple and too sensitive to its data at once. It is one of the central concerns of model evaluation.
Two opposing sources of error
- High bias: the model is too rigid and misses the real structure of the data. This is underfitting. Example: a straight line for a curved relationship.
- High variance: the model hugs the training data too closely, including its noise. This is overfitting. It shines on training data but fails on new examples.
| Symptom | High bias | High variance |
|---|---|---|
| Training error | high | low |
| Test error | high | high |
| Model complexity | too low | too high |
| Remedy | richer model | regularize, more data |
Decomposing the error
For squared loss, the expected error decomposes as:
$$\mathbb{E}\left[(y - \hat{f}(x))^2\right] = \underbrace{\text{Bias}^2}{} + \underbrace{\text{Variance}}{} + \underbrace{\sigma^2}_{\text{irreducible noise}}$$
The $\sigma^2$ term is irreducible: even a perfect model cannot remove it.
Finding the balance
Raising complexity lowers bias but inflates variance. The practical levers are cross-validation, regularization (L1/L2), and gathering more data.
The best model is neither the simplest nor the most complex: it is the one that generalizes best to the unknown.