A language model does not "read" words or letters: it manipulates tokens, small units of text turned into numbers. Tokenization is the step that splits a sentence into these fragments — much like cutting text into reusable LEGO bricks the machine knows how to stack.
What is a token?
A token is the atomic unit processed by the model. Depending on the method, it may be a whole word, a word fragment (subword), a character, or a punctuation mark. Modern models (GPT, Llama, Gemini) mostly rely on subword tokenization, using algorithms such as Byte Pair Encoding (BPE) or WordPiece. The benefit: representing a huge vocabulary with a finite set of tokens while still handling rare or invented words.
For instance, the word "tokenization" might split into token + ization. Each token is then assigned an integer ID and a vector (embedding).
Why it matters
- Cost and context limit: API billing and the context window are measured in tokens, not words.
- Languages: Arabic or Chinese often consume more tokens per word than English.
- Performance: good splitting reduces the sequence length to process.
| Text | Approximate tokens |
|---|---|
| 1 common English word | ~1.3 tokens |
| 1 French word | ~1.5–2 tokens |
| 100 words | ~130–150 tokens |
A common rule of thumb for English: 1 token ≈ 4 characters, roughly:
$$ N_{tokens} \approx \frac{N_{characters}}{4} $$
Understanding tokenization means understanding an LLM's true native language: not words, but tokens.