RL from human feedback (RLHF)

Some goals are easy to recognise but nearly impossible to write down as a reward function — 'be helpful', 'sound natural', 'do not be rude'. RLHF sidesteps the problem of writing the reward by asking people instead. You let the agent produce behaviours, have humans judge which are better, and turn those judgments into the reward the agent optimises. In effect the human becomes the specification you could never quite put into code.

The classic pipeline has three stages. Start from a model trained by ordinary supervised learning. Collect human comparisons of its outputs and fit a reward model that predicts which output a person would prefer. Then improve the policy with reinforcement learning — usually PPO — against that learned reward, while a penalty keeps it from drifting too far from where it began. The reward model is the multiplier that lets a few thousand human labels generalise to millions of fresh situations.

RLHF is how modern chat assistants are shaped to be helpful and harmless, and it works remarkably well in practice. But it inherits every weakness of its parts: the reward model is only an imperfect proxy for human values, and optimising it too hard surfaces reward hacking and Goodhart effects. It is a powerful, practical recipe — not a finished solution to value alignment.