Picture a jury that must split 100% of its confidence among several candidates: none is ignored, but the favorite gets the largest share. That is exactly what the Softmax function does. It takes a list of raw scores (called logits) — which may be negative, positive, large or small — and converts them into a probability distribution: values between 0 and 1 that sum to exactly 1.
The formula
For a logit vector $z = (z_1, \dots, z_n)$, the probability of class $i$ is:
$$\text{softmax}(z)i = \frac{e^{z_i}}{\sum$$}^{n} e^{z_j}
The exponential plays two roles: it makes every value positive and it amplifies the gaps. A logit slightly larger than the others becomes a clearly dominant probability — hence "soft-max," a smoothed version of the strict maximum.
Why not a plain maximum?
| Approach | Output | Differentiable? |
|---|---|---|
| argmax (hard max) | A single winner (1 / 0) | No |
| Softmax | Graded probabilities | Yes |
This differentiability is key: it enables gradient backpropagation during training. Softmax is almost always paired with the cross-entropy loss.
Where do we meet it?
- As the output layer of multi-class classifiers.
- At the heart of the attention mechanism in Transformers, where it weighs the relative importance of each token.
- With a temperature $T$: dividing the logits by $T$ makes the distribution sharper (small T) or more uniform (large T), controlling the creativity of a generative model.
Softmax is the universal translator between the internal language of networks — arbitrary scores — and that of probabilities, readable and usable.