Picture a student who memorizes the answer keys of past exams: brilliant on those exact questions, lost the moment something new appears. Regularization is the family of techniques that keep a machine learning model from falling into this trap. It deliberately limits the model's ability to fit the training data so that it captures the general trend rather than the noise, and therefore generalizes better to data it has never seen.
The problem: overfitting
A model that is too flexible ends up memorizing every detail of the training set, including its random errors. This is overfitting: near-zero training error, but poor performance on new data. Regularization introduces a controlled bias to reduce the model's variance — the heart of the bias-variance tradeoff.
L1 and L2 penalties
Classic methods add a penalty term to the loss function that discourages overly large weights:
$$ J(\theta) = \text{Loss}(\theta) + \lambda \sum_{i} |\theta_i|^p $$
The coefficient $\lambda$ sets the strength of the brake. With $p=2$ we get L2 (ridge), which shrinks weights; with $p=1$, L1 (lasso), which drives some to zero and thus performs feature selection.
| Method | Effect on weights | Typical use |
|---|---|---|
| L1 (Lasso) | Sets weights to zero | Feature selection |
| L2 (Ridge) | Shrinks all weights | General smoothing |
| Dropout | Randomly disables neurons | Deep networks |
| Early stopping | Stops training in time | Any architecture |
Beyond penalties
In deep learning we also rely on dropout (randomly switched-off neurons), data augmentation, and early stopping.
To regularize well is to teach a model to forget just enough to understand more.