An autoencoder is a neural network that learns to reconstruct its own inputs by passing them through a bottleneck. Picture a translator who summarizes a long novel into a few sentences, then tries to rewrite the entire novel from that summary: the more faithful the reconstruction, the better the summary captured what truly mattered. This is the essence of representation learning.
Encoder, latent space, decoder
The architecture has three parts:
- The encoder $f$ compresses the input $x$ into a compact code $z$ (the latent space).
- The decoder $g$ reconstructs an approximation $\hat{x}$ from $z$.
- The network trains to minimize the reconstruction error:
$$\mathcal{L}(x) = \lVert x - g(f(x)) \rVert^2$$
The bottleneck forces the model to keep the useful information and discard the noise: it simply cannot memorize everything.
Main variants
| Variant | Core idea | Typical use |
|---|---|---|
| Denoising | Reconstructs a clean input from a corrupted one | Image cleanup |
| Sparse | Forces few active neurons | Feature detection |
| Variational (VAE) | Learns a latent distribution $\mathcal{N}(\mu, \sigma)$ | Generating images, sound |
Why it is generative
The VAE makes the latent space continuous and structured. By sampling a random point $z$ and feeding it to the decoder, you create brand-new, never-seen data. This mechanism places the autoencoder among the foundational building blocks of generative AI, alongside GANs and diffusion models.
Compress to understand, then understand to create: that is the twofold calling of the autoencoder.