When a language model generates a word, it computes a probability for every possible token in its vocabulary. Always picking the most likely one makes text flat and repetitive; picking purely at random makes it incoherent. Top-k and top-p are two sampling filters that draw the line between what is allowed and what is discarded at each step.
Two ways to trim the list
Top-k keeps only the k most probable tokens, then renormalizes and samples among them. Simple but rigid: if one word clearly dominates, you still keep k candidates; if the distribution is flat, you may exclude good ones.
Top-p (or nucleus sampling) is adaptive: it keeps the smallest set of tokens whose cumulative probability reaches p. We retain the minimal set $V^{(p)}$ such that:
$$\sum_{x \in V^{(p)}} P(x) \geq p$$
The size of the nucleus therefore varies with context: wide when the model is uncertain, narrow when it is confident.
Comparison
| Criterion | Top-k | Top-p |
|---|---|---|
| Threshold | Fixed count (k) | Probability mass (p) |
| Pool size | Constant | Variable, adaptive |
| Typical values | k = 40 | p = 0.9 |
| Weakness | Ignores distribution shape | Sensitive to long tails |
In practice
These filters are often combined with temperature, which flattens or sharpens the probabilities before sampling. They are frequently applied together: top-k first, then top-p on the remaining subset. A low setting favors reliability (factual answers); a high setting favors creativity (writing, brainstorming).
Top-k cuts by count, top-p cuts by confidence: these are the two dials that tune how bold a model dares to be.