Picture a recipe where every ingredient arrives in a different unit — grams, liters, pinches: impossible to measure properly. Normalization does the opposite in deep learning: it brings the values flowing through a neural network to a common scale, so training is stable, fast, and reproducible.
The core idea
Inside a network, activations (the intermediate outputs of layers) can take wildly different magnitudes. When their distribution keeps shifting during training, later layers must constantly readapt — slowing convergence.
Normalization recenters these values around zero and rescales them to unit variance. For an activation vector $x$:
$$\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}$$
where $\mu$ is the mean, $\sigma^2$ the variance, and $\epsilon$ a small term avoiding division by zero. Two learnable parameters, gamma ($\gamma$) and beta ($\beta$), are then applied so the network can restore a useful scale if needed: $y_i = \gamma \hat{x}_i + \beta$.
The main variants
| Method | Normalizes over | Typical use |
|---|---|---|
| Batch Norm | the batch of examples | vision, CNNs |
| Layer Norm | the features of one example | Transformers, NLP |
| Group Norm | groups of channels | small batches, segmentation |
Why it became essential
Normalization enables higher learning rates, reduces sensitivity to initialization, and adds a mild regularizing effect. Layer Normalization sits at the heart of every block in Transformers, the architecture behind today's large language models.
To normalize is to give the network a level playing field: less turbulence, faster travel toward the solution.