Picture a gifted apprentice cook with no sense of taste: he knows millions of recipes but cannot tell which ones diners will love. RLHF (Reinforcement Learning from Human Feedback) is the mentor who tastes his dishes and tells him which to serve. It is the method that turned mere text predictors like GPT into assistants that are helpful, honest, and harmless.
The alignment problem
A pre-trained large language model learns to predict the next word, not to be useful. It can produce answers that are technically correct yet unhelpful, harmful, or rude. RLHF bridges the gap between "knowing how to talk" and "answering well" by injecting human judgment into training.
The three stages
| Stage | Goal | Data used |
|---|---|---|
| 1. SFT | Supervised fine-tuning on model answers | Human demonstrations |
| 2. Reward model | Learn to score a response | Comparisons (A better than B) |
| 3. RL optimization | Maximize the reward | PPO algorithm |
In stage 2, annotators rank several responses. A reward model $r_\theta$ learns these preferences, often through a Bradley-Terry loss:
$$\mathcal{L}(\theta) = -\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)$$
where $y_w$ is the preferred answer and $y_l$ the rejected one. In stage 3, the policy is optimized to maximize this reward while staying close to its initial version via a KL divergence penalty.
Reach and limits
RLHF powers ChatGPT, Claude, and Gemini. Yet it depends on annotator quality, can induce sycophancy, and is expensive. Variants such as DPO (Direct Preference Optimization) aim to skip the reward model entirely.
RLHF does not make AI smarter — it makes it more aligned with what we actually want.