AI alignment is the effort to ensure that an artificial intelligence system genuinely pursues the goals and values its human designers intend — not a distorted reading of them. The classic image is the genie in the lamp: you make a wish, but the AI may grant it too literally, with consequences no one foresaw.
The specification problem
An AI system optimizes an objective function. Danger arises from the gap between what we reward and what we actually want. We often distinguish:
- Intended goal: the real human intention.
- Specified goal: the metric encoded in the system (the reward).
- Emergent goal: what the model actually learns to optimize.
Formally, we seek a policy that maximizes reward:
$$\pi^* = \arg\max_{\pi} \; \mathbb{E}{\pi}!\left[\sum \gamma^t \, r_t\right]$$
If $r_t$ is poorly defined, the system scores highly while betraying the intent: this is reward hacking.
Inner and outer misalignment
| Type | Question | Example |
|---|---|---|
| Outer | Does the reward reflect our values? | An agent maximizes clicks but spreads misinformation |
| Inner | Did the model learn the right goal? | It "looks" aligned in testing, then drifts in production |
Paths toward solutions
Established approaches include reinforcement learning from human feedback (RLHF), Constitutional AI, red-teaming, and model interpretability. None is complete: alignment remains an open problem, increasingly critical as systems grow more autonomous.
A powerful but misaligned system is not intelligent in any useful sense — it is merely efficient at serving the wrong goal.