Picture a reader who, while moving through a novel, keeps the hero's name from chapter one in mind while letting trivial details fade away. That is precisely what an LSTM (Long Short-Term Memory) does: a type of recurrent neural network (RNN) designed to retain information across long sequences without letting it vanish.
The problem it solves
Classic RNNs suffer from the vanishing gradient: during training, the error signal weakens as it propagates back through time, making it impossible to remember distant events. The LSTM, introduced by Hochreiter and Schmidhuber in 1997, adds a memory cell ($C_t$) that flows almost unchanged from one time step to the next, like a conveyor belt of information.
Gates: the heart of the mechanism
The LSTM regulates its memory through three gates, small layers whose output, between 0 and 1, decides what passes through.
| Gate | Role |
|---|---|
| Forget | What should be erased from memory? |
| Input | What new information to store? |
| Output | What to reveal at the present moment? |
The forget gate, for instance, is computed as:
$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$
where $\sigma$ is the sigmoid function. The cell is then updated: part of the old state is erased, and the new information is added.
Uses and limits
LSTMs long dominated machine translation, speech recognition, and time-series forecasting. Today, Transformers have largely overtaken them on text, yet LSTMs remain relevant for lightweight sequential data and embedded systems.
The LSTM taught machines an essentially human skill: knowing what to keep, and what to forget.