Picture a meeting room where, to understand each word of a sentence, you could instantly consult every other word and decide which ones truly matter. That is precisely what the attention mechanism (or self-attention) does: it lets a model dynamically weigh the importance of each element in a sequence relative to the others. Introduced at the heart of the Transformer architecture in 2017, it replaced recurrent networks and became the engine of today's large language models.
The intuition: who looks at whom?
In the sentence "the cat sleeps because it is tired," to interpret "it" the model must link it back to "cat." Attention learns these links. Each word generates three vectors:
- a Query: "what am I looking for?"
- a Key: "what can I offer?"
- a Value: "what information I carry"
The formula
Attention compares queries and keys, normalizes the scores, then blends the values:
$$\text{Attention}(Q, K, V) = \text{softmax}!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$
The factor $\sqrt{d_k}$ stabilizes the gradients. The softmax turns scores into weights that sum to 1.
Why it is revolutionary
| Criterion | Recurrence (RNN) | Self-Attention |
|---|---|---|
| Processing | sequential | parallel |
| Long-range dependencies | hard | direct |
| Context | limited | global |
The multi-head variant runs the operation in parallel to capture several kinds of relationships (syntax, semantics…).
No attention, no ChatGPT: it is the building block that ushered AI into the Transformer era.