Picture studying a textbook for an exam. Reading the whole book once is an epoch. But you don't memorize it all at once: you study chapter by chapter — each chapter is a batch. And every time you finish a chapter and adjust your understanding, that's an iteration. These three notions structure the rhythm of learning for a neural network.
The three units, precisely
- Epoch : one complete pass over the entire training dataset. After one epoch, the model has "seen" every example once.
- Batch : a subset of the dataset processed together before the weights are updated. Its size is the batch size.
- Iteration : a single weight update, corresponding to the processing of one batch.
The numerical relationship
Letting $N$ be the total number of examples and $B$ the batch size, the number of iterations per epoch is:
$$\text{iterations per epoch} = \left\lceil \frac{N}{B} \right\rceil$$
| Term | Unit | Triggers a weight update? |
|---|---|---|
| Iteration | 1 batch | Yes (once) |
| Epoch | whole dataset | Yes (multiple times) |
| Batch size | nb of examples | sets the granularity |
Example: with $N = 10\,000$ examples and $B = 100$, one epoch contains $100$ iterations. Over $20$ epochs, the model performs $2\,000$ updates in total.
Why it matters
A batch that is too large smooths the gradient but costs memory; too small, it makes learning noisy yet sometimes more generalizing. The number of epochs arbitrates between underfitting (too few) and overfitting (too many).
Epoch, batch and iteration don't describe what the model learns, but at what pace it learns.