Picture someone reading a book: to understand a word, they recall the previous ones. A Recurrent Neural Network (RNN) works the same way — an architecture built to process sequences (text, speech, time series) by keeping a memory of what it has already seen.
The recurrence principle
Unlike a standard network that treats each input in isolation, an RNN carries a hidden state ($h_t$) that flows from one time step to the next. At each step it blends the new input $x_t$ with the previous memory:
$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b)$$
The state $h_{t-1}$ acts as a summary of the whole history. The same weights ($W_x$, $W_h$) are reused at every step: the network "unfolds" one cell through time.
Strengths and limits
RNNs excel on ordered data but suffer from the vanishing gradient: they forget distant dependencies. Variants fix this:
| Variant | Key contribution |
|---|---|
| Vanilla RNN | Short memory |
| LSTM | Gates for long memory |
| GRU | Streamlined LSTM |
Uses
- Translation and text generation
- Speech recognition
- Time-series forecasting (prices, weather)
RNNs laid the foundations of sequential processing before Transformers largely replaced them on text — yet their idea of memory remains foundational.