Imagine a recipe where, each time you double the ingredients, the oven, and the baking time, the cake gets measurably better — and predictably so. Scaling laws describe exactly this for AI: a model's performance improves smoothly and predictably as you increase three levers — model size (number of parameters), the amount of training data, and the compute invested.
A power-law relationship
The central finding, popularized by OpenAI's work (2020) and later refined by DeepMind, is that a model's error (its loss) decreases as a power law with respect to each lever. Schematically:
$$L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha}$$
where $L$ is the loss, $N$ the number of parameters, and $\alpha$ a positive exponent. The practical consequence is striking: on a log-log plot, performance follows a near-perfect straight line across several orders of magnitude.
The Chinchilla balance
For a long time, the dominant strategy was simply making models bigger. In 2022, DeepMind's Chinchilla model showed that many models were undertrained: for a fixed compute budget, parameters and data must be balanced.
| Lever | Effect of increasing it | Limit |
|---|---|---|
| Parameters ($N$) | More capacity | Memory/compute cost |
| Data ($D$) | Better generalization | Finite high-quality data |
| Compute ($C$) | Enables larger $N$ and $D$ | Energy and financial cost |
Why it's fundamental
Scaling laws turned AI into a discipline of predictable engineering: you can estimate a future model's performance before training it, justifying enormous investments.
More isn't always better — but thanks to scaling laws, we know how much more, and where to invest it.