Picture a student who only revises the exact questions on the final exam: they would score well, but did they truly learn? Cross-validation solves this same problem in machine learning. It is an evaluation method that trains and tests a model across several different splits of the same data, in order to estimate its performance in a robust way rather than by luck.
The k-fold principle
The most common form is k-fold cross-validation. The dataset is divided into $k$ equal parts called folds. In each round, one fold serves as the test set and the remaining $k-1$ serve as the training set. The process is repeated $k$ times, and the scores are averaged:
$$\text{Score}{CV} = \frac{1}{k}\sum_i$$}^{k} \text{Score
With $k=5$ or $k=10$ as typical values, every data point is used in turn for both training and testing.
Why it matters
A single train/test split can be misleading: a lucky partition artificially inflates the score. Cross-validation reduces this variance and helps detect overfitting.
| Method | Data tested | Reliability |
|---|---|---|
| Single train/test | 1 portion | Low |
| k-fold | All, $k$ times | High |
| Leave-one-out | 1 point at a time | Very high, costly |
Useful variants
- Stratified: preserves class proportions (useful for imbalanced data).
- Leave-one-out: extreme case where $k$ equals the number of samples.
- Time-series: respects the chronological order of sequences.
Cross-validation trades a little compute time for a lot of confidence in the result.