Picture a reader who, instead of scanning a text from left to right, takes in the whole sentence at a glance to guess a missing word from everything around it. That is the idea behind BERT (Bidirectional Encoder Representations from Transformers), a language model introduced by Google in 2018. Its breakthrough: understanding each word using its left and right context simultaneously, whereas earlier models read in a single direction.
A bidirectional encoder architecture
BERT uses only the encoder part of the Transformer architecture, built on the attention mechanism. Each word (or token) becomes a vector enriched by the whole sentence's context. The core computation is attention:
$$\text{Attention}(Q,K,V) = \text{softmax}!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$
where $Q$, $K$ and $V$ are the queries, keys and values derived from the words, and $d_k$ is the dimension of the keys.
Two pre-training tasks
BERT learns from huge unlabeled corpora through two objectives:
- Masked Language Modeling (MLM): about 15% of words are masked and the model must recover them.
- Next Sentence Prediction (NSP): predict whether two sentences truly follow one another.
Pre-train, then fine-tune
| Phase | Data | Goal |
|---|---|---|
| Pre-training | Massive unlabeled corpus | General linguistic representations |
| Fine-tuning | Small labeled set | Specific task (sentiment, Q&A, NER) |
BERT popularized the "pre-train once, fine-tune everywhere" paradigm, paving the way for today's large language models.