RLHF pipeline

RLHF, reinforcement learning from human feedback, is the recipe that turned raw language models into helpful assistants. The intuition is teaching by comparison: instead of writing down the perfect answer, you repeatedly show people two model responses and let them say which is better, then push the model toward the kind of answer people keep preferring.

The classic pipeline has three stages. First, supervised fine-tuning trains the model on human-written demonstrations of good behavior. Second, humans compare pairs of model outputs, and those comparisons train a reward model that scores any response by how much a person would likely prefer it. Third, the model is optimized with reinforcement learning, usually PPO, to produce responses the reward model rates highly, while a KL penalty keeps it from drifting too far from the fine-tuned starting point.

Each stage can fail in its own way: demonstrations can be narrow, the reward model is only a proxy for real human values, and the RL step can learn to exploit that proxy. RLHF is powerful but fragile, which is exactly why so much alignment research focuses on making each link in this chain more robust.