Picture a team where, at every meeting, a few members are randomly told to stay silent: the rest must learn to cope without relying on one indispensable colleague. That is exactly the idea behind Dropout, a regularization technique introduced in 2014 by Srivastava, Hinton and colleagues. During training, a fraction of neurons is randomly deactivated on each forward pass, forcing the network not to lean too heavily on a few connections.
Why it works
A deep neural network is prone to overfitting: it memorizes the noise in the training data instead of learning general patterns. By cutting neurons at random, Dropout prevents co-adaptation — neurons can no longer rely on each other through fragile combinations. The effect resembles implicitly training an ensemble of smaller networks whose predictions are averaged.
How it is computed
Each neuron is assigned a probability $p$ of being kept. We sample a binary mask $r_j \sim \text{Bernoulli}(p)$ and compute:
$$\tilde{y}_j = \frac{r_j}{p} \cdot y_j$$
Dividing by $p$ (inverted dropout) keeps the expected activation stable, so no adjustment is needed at inference time.
Training vs. Inference
| Phase | Behavior |
|---|---|
| Training | Neurons randomly switched off (rate 0.2–0.5) |
| Inference | All neurons active, no cutting |
Dropout remains common in dense layers and Transformers today, even though batch normalization sometimes competes with it in convolutional networks.
Switching off neurons at random teaches a network humility: never depend on a single one.