Picture a storyteller writing word after word, never knowing the ending in advance: each word depends strictly on everything that came before. That is precisely how an autoregressive model works. It generates data (text, audio, images) one element at a time, feeding what it just produced back in at every step to predict the next element.
The core idea: predict the next
An autoregressive model factorizes the probability of an entire sequence into a product of conditional probabilities. For a sequence of tokens $x_1, x_2, \dots, x_T$:
$$P(x_1, \dots, x_T) = \prod_{t=1}^{T} P(x_t \mid x_1, \dots, x_{t-1})$$
In other words, the model only ever learns to answer one question: "given everything so far, what is the most likely element now?". Generation is therefore sequential: produce a token, append it to the context, and repeat.
Why it powers LLMs
This is exactly how large language models such as the GPT family operate. During generation they never "see" the whole sentence at once — they build it token by token. This simple mechanism, trained on massive corpora, gives rise to abilities in reasoning, translation, and writing.
| Aspect | Autoregressive model | Diffusion model |
|---|---|---|
| Generation | Sequential (one element at a time) | Parallel (global denoising) |
| Flagship domain | Text (GPT, LLaMA) | Images (Stable Diffusion) |
| Weakness | Slow, errors accumulate | Many costly steps |
The trade-off: generation is slow (one pass per token), and an early mistake can propagate through the rest of the output.
To generate is to predict the immediate future from the past — one token at a time.