Temperature is the dial that decides how much risk a language model is willing to take. Picture a slider: all the way left, the model plays it safe and always repeats the most likely word; all the way right, it grows bold, surprising, sometimes incoherent. It is a sampling parameter applied at inference time, leaving the model's weights untouched.
How it works
A model doesn't predict a single word but a probability distribution over its entire vocabulary. Temperature $T$ divides the raw scores (logits) before the softmax step:
$$P(x_i) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$
- $T \to 0$: the distribution collapses onto the most likely word (nearly deterministic).
- $T = 1$: the original distribution is preserved.
- $T > 1$: the distribution flattens, and rarer words gain a greater chance of being picked.
Choosing the right value
| Temperature | Behaviour | Typical use |
|---|---|---|
| 0 – 0.3 | Precise, repeatable | Code, extraction, math, facts |
| 0.5 – 0.8 | Balanced | Writing, dialogue |
| 1.0 – 1.5 | Creative, varied | Brainstorming, poetry, fiction |
The core trade-off
Temperature embodies the trade-off between reliability and diversity. Too low, and answers turn monotonous and rigid; too high, and they drift toward incoherence and hallucination. It is often combined with other controls such as top-p (nucleus sampling) to better govern the window of possible choices.
Tuning the temperature doesn't make the model "smarter" — it lets you choose between caution and imagination.