Picture walking down a mountain in fog to reach the lowest valley. The learning rate (often denoted $\eta$) is the length of each step you take. Too large, and you leap over the valley; too small, and you take forever to descend. It is the single most influential hyperparameter in training a neural network.
The mechanism
At each iteration, gradient descent adjusts the model's weights $w$ in the direction that reduces error. The learning rate controls how big that correction is:
$$w_{t+1} = w_t - \eta \, \nabla L(w_t)$$
where $\nabla L$ is the gradient of the loss function. The gradient gives the direction; $\eta$ decides the distance travelled.
Striking the balance
| Learning rate | Consequence |
|---|---|
| Too high | Divergence, oscillation, exploding loss |
| Too low | Very slow convergence, risk of getting stuck |
| Well tuned | Fast and stable convergence |
Typical values range from $10^{-1}$ to $10^{-5}$ depending on the architecture and optimizer.
Making it dynamic
Rather than fixing $\eta$, it is often varied during training:
- Decay: large steps early, small steps later to fine-tune.
- Warmup: a gradual start to stabilize the first iterations.
- Adaptive optimizers (Adam, RMSprop): tune an effective rate per parameter.
Choosing the right learning rate means finding the rhythm between the boldness that drives progress and the caution that ensures convergence.