reinforcement learning from human feedback

Imagine teaching someone to cook when you cannot write down a recipe for every dish, but you can always taste two plates and say which one is better. That is the situation we are in with a large language model. We cannot write a rule for what makes a reply helpful, polite, or safe, but a person can read two answers and point to the better one. RLHF is the family of techniques that turns those human judgements into training signal, nudging the model to produce more of the answers people prefer.

In practice RLHF has three stages. First you start with a base model that has already learned language by predicting text (cross-link the reinforcement learning domain and the language-model background rather than re-deriving them here). Second, you collect human comparisons (for the same prompt, which of two answers is better?) and train a separate reward model to predict those preferences as a single score. Third, you run a reinforcement-learning step (usually PPO, proximal policy optimization, from the reinforcement learning domain): the model generates answers, the reward model scores them, and the model is updated to earn a higher score, while a KL-divergence penalty keeps it from drifting too far from the original base model. The result is the polite, instruction-following assistant style most people now recognise.

RLHF is the workhorse behind today's aligned-ish assistants, but it is important to be honest about what it does and does not do. It shapes behaviour, the answers the model tends to give, not the model's underlying goals or values; it makes the model act more helpful without guaranteeing it is helpful for the right reasons. The reward model is only a learned proxy for human preference and can be gamed (see Goodhart's law and reward over-optimization), and because raters reward answers that sound good, RLHF can teach a model to be confidently agreeable, a failure called sycophancy. RLHF reduces misalignment; it does not eliminate it.