Picture an insatiable reader who has consumed a vast slice of human text and, at every moment, plays a game of guessing the next word. That is the essence of a GPT — Generative Pre-trained Transformer. Introduced by OpenAI starting in 2018, this family of models now powers conversational assistants such as ChatGPT.
Unpacking the acronym
Each word tells part of the story:
- Generative: the model produces new text, token by token, rather than merely classifying.
- Pre-trained: it is first trained on massive corpora in a self-supervised way, before any fine-tuning.
- Transformer: the underlying architecture, introduced in 2017, built on the attention mechanism.
How it learns
The training objective is deceptively simple: predict the next token given the previous ones. The model learns to maximise the probability of the observed sequence:
$$P(x_1, x_2, \dots, x_n) = \prod_{t=1}^{n} P(x_t \mid x_1, \dots, x_{t-1})$$
This auto-regressive task forces the network to capture grammar, facts and implicit reasoning. Attention lets every word "look at" all the others to weigh their contextual importance.
From pre-training to assistant
A raw GPT completes text; it becomes useful after several stages:
| Stage | Purpose |
|---|---|
| Pre-training | Learn language and the world |
| Supervised fine-tuning | Follow instructions |
| RLHF | Align with human preferences |
A GPT does not "understand" the way a human does: it models the probability of language so finely that the illusion of understanding becomes remarkably useful.