Aligning From Feedback: RLHF, Constitutional AI, and RLAIF

The preference-learning pipeline, in one breath

Modern alignment from feedback has a standard shape. Start from a supervised fine-tuned base. Collect comparisons — for a prompt, two responses, a label saying which is better. Fit a reward model that scores responses to match those comparisons (a Bradley–Terry model on pairwise data). Then optimize the policy against that reward with RLHF, usually PPO, with a KL penalty keeping you near the base model so you do not wander into nonsense the reward model rates highly by accident.

Two practical truths fall out immediately. First, the reward model is the whole ballgame — your policy is only ever as aligned as the thing it optimizes. Second, the KL term is not a detail; it is your only defense against reward hacking the reward model. Crank the optimization too hard and the policy finds adversarial high-reward gibberish. This is why people watch the KL budget like a fuel gauge.

\max_{\pi_\theta}\ \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)}\!\big[r_\phi(x,y)\big]\;-\;\beta\,\mathrm{D}_{\mathrm{KL}}\!\big(\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\big)

The KL-regularized RLHF objective: maximize the reward model's score while the beta-weighted KL term leashes the policy to its reference.

Where human labels run out

Human preference data is expensive, slow, inconsistent, and — crucially — it encodes whatever the labelers happened to notice. Labelers reward answers that sound confident and well-formatted, which is exactly how you train sycophancy and confident hallucination. And for any topic the labeler is not an expert in, the 'preference' is partly noise. As tasks get harder than the median labeler, the signal degrades toward the scalable oversight frontier.

Two responses to this scaling wall define the rest of this guide. One: replace most human labels with a model's own judgments, anchored to an explicit set of principles — that is Constitutional AI and RLAIF. Two: stop labeling only the final answer and start labeling the reasoning, so errors have nowhere to hide — that is process supervision.

Constitutional AI: principles instead of cases

Constitutional AI replaces the question 'which of these two answers do you, a human, prefer?' with 'which answer better follows this written principle?' — and lets the model itself apply the principle. It runs in two stages. In the supervised stage, the model critiques and revises its own responses against constitutional rules ('was that harmful? rewrite it to be harmless but still helpful'). In the RL stage, the model generates preference labels by judging pairs against the constitution, and you train a reward model on those labels.

The conceptual win is that the alignment target becomes a short, auditable document instead of a black-box pile of crowd labels. You can read the constitution, argue about a clause, edit it, and re-derive behavior — a huge improvement in transparency of the specification. The cost is honesty about a subtle move: you have shifted trust from 'humans judged each case' to 'the model correctly applies principles humans wrote'. That only holds while the model is competent and honest enough to apply them faithfully.

# Constitutional AI, supervised stage (sketch)
for prompt in prompts:
    answer   = model(prompt)
    critique = model(f"Critique this answer per principle P: {answer}")
    revised  = model(f"Rewrite to fix the critique: {answer} | {critique}")
    sft_data.append((prompt, revised))   # train on the self-revised answers

The model supervises itself against an explicit principle; humans audit the principle, not every output.

RLAIF and process supervision

RLAIF (reinforcement learning from AI feedback) is the generalization: wherever RLHF wanted a human label, ask a capable model instead, prompted to apply your criteria. It scales preference data almost for free and, surprisingly often, matches or beats human-feedback baselines on helpfulness and harmlessness. The honest caveats are real, though — the AI labeler imports its own biases, can be gamed by the policy it supervises, and risks a feedback loop where models train on their own preferences. RLAIF buys scale, not ground truth.

Process supervision attacks the same wall from a different angle. Instead of rewarding only the final answer (outcome supervision), you reward each correct step of the reasoning. For multi-step math and code this is dramatically better: a process reward model that flags the first wrong step gives dense, targeted signal and refuses to credit right-answer-for-wrong-reasons. It also resists reward hacking — it is much harder to fake a fully valid chain of reasoning than to stumble onto a correct final token.