Imagine reading a book knowing only isolated letters: slow and costly. Memorizing every possible word would be unrealistic. Byte Pair Encoding (BPE) strikes the balance: it splits text into frequent subwords, the vocabulary in which large language models truly think.
From compression to tokenization
BPE began as a data compression technique: iteratively replace the most frequent pair of adjacent symbols with a new symbol. Applied to text, this becomes vocabulary learning:
- Start from elementary units (characters or bytes).
- Count every adjacent pair in the corpus.
- Merge the most frequent pair into a new token.
- Repeat until reaching a target vocabulary size.
Formally, at each step we pick the pair $(a, b)$ that maximizes its frequency:
$$ (a, b) = \arg\max_{(x, y)} \; \text{count}(x, y) $$
Why LLMs love it
BPE solves the out-of-vocabulary problem: even an unseen word decomposes into known fragments. The byte-level variant (used by GPT-2 and its successors) starts from raw bytes, guaranteeing that no character is ever out of vocabulary, whatever the language or emoji.
| Approach | Vocabulary | Unknown words |
|---|---|---|
| Word-level | Huge | Frequent |
| Character-level | Tiny | None, but very long sequences |
| BPE (subwords) | Moderate | Almost none |
BPE is the trade-off that lets models cover every language with a finite vocabulary: an alphabet reinvented for machines.