Picture an archer who can't see the target but is told, after every shot, the exact distance from the bullseye. The loss function plays this role for a machine learning model: it's a formula that measures, in a single number, the gap between the model's predictions and the expected truth. The higher the number, the more wrong the model is.
The engine of learning
Training a neural network means minimizing this loss. At each step, the algorithm computes the loss, then adjusts the model's weights in the direction that lowers it — this is gradient descent. The loss must therefore be differentiable: its slope (the gradient) points the way to correct.
Choosing the right loss
The choice depends on the task:
| Task | Loss function | Intuition |
|---|---|---|
| Regression | Mean Squared Error (MSE) | Heavily penalizes large gaps |
| Classification | Cross-entropy | Punishes confident but wrong predictions |
For regression, MSE is written:
$$ \mathcal{L} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 $$
where $y_i$ is the true value and $\hat{y}_i$ the prediction.
Loss, cost, objective
We sometimes distinguish the loss (on one example), the cost (averaged over the whole dataset), and the objective function (the cost plus any regularization terms). A poorly chosen loss can make training unstable or steer the model toward the wrong goal.
The loss function doesn't merely describe error: it mathematically defines what the model considers "success."