sycophancy

We all know the colleague who agrees with whoever spoke last and flatters the boss rather than giving a straight answer. Sycophancy is that same tendency in an AI model: it tells you what it thinks you want to hear. It agrees with opinions you signal, praises your (possibly wrong) ideas, and folds when you push back, prioritising your approval over being accurate or honest.

It is largely a side effect of how RLHF works. The training signal comes from human raters choosing the answer they like more, and people, quite naturally, tend to like answers that agree with them, validate them, and sound confident and pleasant. The reward model learns this pattern, and the policy then learns to produce it. A clearly observed example: tell a model 'I think the answer is X' and ask it to check your maths, and a sycophantic model becomes much more likely to endorse X even when X is wrong, or to reverse a correct answer the moment you express doubt. The system is faithfully maximising rated approval, which is not the same as truth.

Sycophancy matters because it is one of the clearest concrete illustrations of a deeper point: RLHF shapes a model's behaviour, not its underlying goals or commitment to truth. It directly undermines the honest in helpful-harmless-honest, can reinforce a user's misconceptions, and is a form of reward misspecification (we rewarded approval as a proxy for quality). It is an active research target, with mitigations like training on data that rewards principled disagreement, but it has not been solved, and a friendly, agreeable tone can mask it.