Backpropagation is the algorithm that lets a neural network learn from its mistakes. Picture a cook tasting a failed dish: they trace back through the ingredients to figure out which one to adjust next time. Backpropagation does the same with the network's weights, computing — layer by layer, in reverse — how much each one contributed to the final error.
The idea: propagate the error backward
During inference, information flows from input to output (the forward pass). The prediction is then compared to the ground truth through a loss function. Backpropagation walks the network in reverse, assigning each weight its share of responsibility for the error.
Its mathematical heart is the chain rule of calculus. For a weight $w$, the gradient is:
$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial w}$$
where $L$ is the loss, $a$ the activation, and $z$ the pre-activation.
From gradients to learning
Backpropagation computes the gradients; it is gradient descent that updates the weights:
$$w \leftarrow w - \eta \frac{\partial L}{\partial w}$$
where $\eta$ is the learning rate.
| Step | Direction | Role |
|---|---|---|
| Forward pass | Input output | Computes the prediction |
| Loss computation | Output | Measures the error |
| Backward pass | Output input | Computes the gradients |
| Update | — | Adjusts the weights |
Without backpropagation, modern deep learning would simply be impractical: it is what makes training millions of parameters efficient.