Picture a hiker lost in fog, trying to reach the bottom of a valley as fast as possible: at every step they feel the slope underfoot and move in the steepest downhill direction. The optimizer is exactly that hiker — the algorithm that adjusts a neural network's weights, step by step, to minimize the loss function (the model's error).
The core idea: gradient descent
The optimizer relies on the gradient, the derivative of the loss with respect to each weight — the mathematical "slope". The simplest update rule, stochastic gradient descent (SGD), reads:
$$\theta_{t+1} = \theta_t - \eta \nabla_\theta L(\theta_t)$$
where $\theta$ are the weights, $\eta$ is the learning rate (step size), and $\nabla_\theta L$ is the gradient. Too large a step makes training diverge; too small a step makes it endless.
The main families
Modern optimizers add memory and adapt the step size per parameter.
| Optimizer | Key idea | Strength |
|---|---|---|
| SGD | Fixed step | Simple, robust |
| Momentum | Accumulates velocity | Crosses plateaus |
| RMSProp | Per-weight adaptive step | Handles scales |
| Adam | Momentum + adaptive | Default standard |
Adam (Kingma & Ba, 2014) blends the inertia of momentum with RMSProp's adaptation; it is today the default choice for training most large models.
Without an optimizer, a network would be just a box of frozen numbers: the optimizer is what turns error into learning.