Think of an artificial neuron as a smart valve: it receives a stream of signals, sums them up, then decides how much to let through. That is precisely the role of an activation function — the small non-linear transformation applied to each neuron's output.
Why non-linearity changes everything
A neural network stacks layers of multiplications and additions. Yet without an activation function, stacking linear layers stays… linear. No matter how deep, the network could only model a straight line.
The activation function injects curvature. It is what lets a network approximate complex boundaries — telling a cat from a dog, translating a sentence, generating an image.
$$ a = f\left(\sum_i w_i x_i + b\right) $$
where $w_i$ are the weights, $b$ the bias, and $f$ the activation function.
The main families
| Function | Formula | Typical use |
|---|---|---|
| Sigmoid | $\sigma(x) = \frac{1}{1+e^{-x}}$ | Probabilistic output (0–1) |
| Tanh | $\tanh(x)$ | Centered outputs (−1 to 1) |
| ReLU | $\max(0, x)$ | Hidden layers (standard) |
| Softmax | $\frac{e^{x_i}}{\sum_j e^{x_j}}$ | Multi-class classification |
The reign of ReLU
ReLU (Rectified Linear Unit) dominates hidden layers today: cheap to compute, it eases the vanishing gradient problem that crippled older sigmoids in deep networks. Its variants (Leaky ReLU, GELU, SiLU) fix its "dead neuron" weakness.
Without an activation function, a deep network would be nothing but a disguised linear regression. That non-linear spark is what makes deep learning truly deep.