Picture a staircase where, at every landing, you can either climb the steps or take a slide that drops you straight onto the next landing. A residual connection (or skip connection) is that slide: it adds a layer's input directly to its output, letting information bypass the intermediate transformations. Introduced in 2015 with the ResNet architecture (He et al.), it made it possible to train networks with hundreds of layers.
The mathematical idea
Instead of asking a block of layers to learn a target function $H(x)$ directly, we ask it to learn only the residual $F(x) = H(x) - x$. The output becomes:
$$ y = F(x) + x $$
The $+x$ term is the identity: if the optimal transformation is to "do nothing," the network just needs to drive $F(x)$ toward zero — far easier than learning the identity from nonlinear layers.
Why it matters
Without a shortcut, the gradient weakens as it propagates through many layers — the vanishing gradient problem. The residual connection builds a "highway" through which the gradient flows without attenuation during backpropagation.
| Without residual connection | With residual connection |
|---|---|
| Gradient vanishes with depth | Gradient flow preserved |
| Accuracy degrades beyond ~20 layers | 100+ layer networks become trainable |
| Identity hard to learn | Identity is "free" by default |
Today this mechanism is everywhere: it sits at the heart of Transformers (every attention and feed-forward sub-layer is wrapped in a residual connection), and therefore of models like GPT and modern LLMs.
Learn the difference rather than the whole — that simple idea unlocked depth.