Imagine reading a sentence whose words have all been tossed into a bag with no order. That is exactly what a Transformer "sees" without positional encoding: its attention mechanism treats the input as an unordered set of tokens. Positional encoding is the trick that reinjects information about order and distance between words.
Why do we need it?
Unlike recurrent networks (RNNs) that read words one by one, a Transformer's attention is permutation-invariant: "the cat eats the mouse" and "the mouse eats the cat" would yield identical representations. To distinguish them, a vector that depends on each word's position in the sequence is added to its embedding.
The original sinusoidal encoding
In the foundational paper Attention Is All You Need (2017), the authors propose sine and cosine functions of varying frequencies:
$$PE_{(pos,\,2i)} = \sin!\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{(pos,\,2i+1)} = \cos!\left(\frac{pos}{10000^{2i/d}}\right)$$
where $pos$ is the position, $i$ the dimension index and $d$ the embedding size. Each position thus receives a unique signature, and relative positions can be expressed through simple linear combinations.
The main families
| Type | Principle | Examples |
|---|---|---|
| Fixed absolute | Predefined sinusoids | Original Transformer |
| Learned absolute | Trained vectors | BERT, GPT-2 |
| Relative / rotary | Encodes token-to-token distance | RoPE (LLaMA), ALiBi |
Modern approaches like RoPE have the advantage of generalizing better to sequences longer than those seen during training.
Without positional encoding, a Transformer would hear the words but ignore their order — positional encoding gives the sentence back its meaning.