Picture a consulting firm: when a question arrives, you don't summon all 100 specialists — only the 2 or 3 whose expertise fits the problem. That is exactly the idea behind the Mixture of Experts (MoE): instead of activating one giant neural network for every input, you activate only a small fraction of specialized sub-networks, called experts.
How it works
An MoE block replaces a standard dense layer with two components:
- a set of $N$ experts (small independent networks);
- a router (gating network) that, for each token, picks the $k$ most relevant experts (typically $k=1$ or $k=2$).
The output is a weighted combination of the chosen experts:
$$ y = \sum_{i \in \text{Top-}k} g_i(x)\, E_i(x) $$
where $g_i(x)$ is the weight the router assigns to expert $E_i$. Unselected experts compute nothing — this is sparse activation.
Why it is powerful
| Dense network | MoE network |
|---|---|
| All parameters active | Only $k$ experts active |
| Compute ∝ total size | Compute nearly constant |
| Expensive to scale | Huge capacity, controlled cost |
This lets us train models with hundreds of billions of parameters while keeping the per-token compute close to that of a much smaller model. It underpins several recent large models (Mixtral and leading architectures from Google and other labs).
The balancing challenge
The risk: the router learns to always call the same few experts. So a load-balancing loss is added to spread the work across all experts.
MoE decouples a model's capacity from its inference cost: grow large without paying the full price every time.