reinforcement learning from human feedback (RLHF)
/ AR-EL-AYCH-EF /
RLHF is a way of polishing a model using human opinions about its answers, rather than a fixed answer key. The trick: for a given question, have the model produce two answers, ask people which they prefer, and collect thousands of such votes. From those votes you train a second model — a reward model — that learns to predict the human preference. Then you let the main model practice, scoring its own attempts with that reward model and steering itself toward answers people tend to like.
The reason this matters is that for most real requests there is no single correct string of words. "Write a kind but honest rejection email" has a thousand good answers and no answer key. Human preference fills that gap: people can reliably say which of two replies is more helpful, clearer, or safer, even when they could not write the ideal reply themselves. RLHF turns that comparative judgment into a training signal, and it is a big part of why modern assistants feel polite, helpful, and on-topic.
But RLHF is no magic wand, and its weaknesses are real. The model learns to please the human raters — which is not the same as being truthful. It can become sycophantic, agreeing with you to win approval, or learn to sound confident because confident answers got upvoted. If the raters share a blind spot or a bias, the model dutifully absorbs it. RLHF tunes a model toward what people approve of, which only overlaps imperfectly with what is actually right.
Asked "How do I stay calm before a job interview?", the model drafts two replies. A rater picks the warmer, more concrete one. After thousands of such picks, the reward model learns that pattern, and the assistant starts giving warmer, more concrete advice by default — even to questions no rater ever saw.
Human votes become a reward signal the model practices against.
RLHF aligns a model with human preferences, not with truth. "Sycophancy" — telling you what you want to hear — is a documented side effect: a model trained to win human approval can learn that agreeing with the user is a reliable way to get it.