Picture a student handed a text with words erased: to fill the blanks, they must grasp grammar, meaning and context. That is the core of self-supervised learning (SSL): a model learns from raw, unlabeled data by generating its own supervision signal from the data's internal structure.
The principle: free labels
Instead of relying on costly human annotation, we design a pretext task where one part of the data predicts another. Two major families exist:
- Predictive / masked: hide part of the input and reconstruct it (masked words in BERT, next tokens in GPT).
- Contrastive: pull together two views of the same example and push apart different ones (SimCLR, vision).
For an autoregressive language model, the objective is to maximize the likelihood of the next token:
$$\mathcal{L} = -\sum_{t=1}^{T} \log P(x_t \mid x_1, \dots, x_{t-1})$$
Where it sits
| Paradigm | Labels | Signal source |
|---|---|---|
| Supervised | Required (human) | External annotator |
| Unsupervised | None | Structure (clusters) |
| Self-supervised | Derived from data | The data itself |
Why it matters now
It is the engine behind foundation models: we pre-train on massive unlabeled corpora, then fine-tune on a small targeted task. This taps the abundant data of the web exactly where manual labeling is scarce and expensive.
Self-supervision freed AI from its dependence on human labels: raw data becomes its own teacher.