Picture a hiker lost in the fog on a mountain, trying to reach the valley. Unable to see the horizon, they feel the ground underfoot and take a step in the steepest downward direction. Gradient descent works exactly like this: it is the optimization algorithm that lets an AI model learn by gradually reducing its error.
The principle
A model is defined by parameters (the network's weights). A cost function $J(\theta)$ measures how wrong its predictions are. The goal is to find the parameters that minimize this cost. The gradient $\nabla J(\theta)$ points in the direction of steepest ascent, so we move in the opposite direction:
$$\theta \leftarrow \theta - \eta \, \nabla J(\theta)$$
The learning rate $\eta$ controls the step size: too large and we bounce around without converging; too small and learning drags on forever.
Three variants
| Variant | Data per step | Trait |
|---|---|---|
| Batch | Whole dataset | Stable but slow |
| Stochastic (SGD) | 1 example | Fast, noisy |
| Mini-batch | Small batch | Dominant trade-off |
In practice
Gradient descent is the engine of deep learning. Combined with backpropagation (which efficiently computes the gradient layer by layer), it trains today's largest models. Optimizers such as Adam and RMSProp improve it by automatically adapting the step size for each parameter.
For a machine, learning is simply walking down the slope of its own errors until it reaches the bottom of the valley.