The bottleneck nobody puts on the slide: people
By now you know the RLHF machine from the inside. Humans compare two answers, a reward model learns to predict which they prefer, and an RL step nudges the policy toward higher reward while a KL leash keeps it close to the base model. But look hard at the very bottom of that machine and you find an ingredient that rarely makes the diagram: real people, doing slow and sometimes grim work. To teach a model not to help with a dangerous request, human labellers have to read thousands of disturbing outputs and rank which is worse. It is expensive, it is exhausting, and there is never enough of it, least of all for the sensitive cases that matter most.
Now the plain-language idea, before any jargon. Imagine you run a busy kitchen and cannot personally taste every plate. So you write your standards down once, a short rubric, 'season properly, plate it cleanly, never send out anything raw', and you train a trusted assistant to grade against that rubric for you. You have not stopped caring about quality; you have moved your judgement out of your tongue and into a written standard plus a delegate who applies it. That is exactly the move in this guide, applied to AI. RLAIF, reinforcement learning from AI feedback, lets a capable model do most of the comparing. Constitutional AI goes one step further: it writes the values down as a short text, a 'constitution', and has the model use those principles to critique and rewrite its own answers.
Hold one thing fixed as we go: this is not a new training algorithm, it is a new source for one specific ingredient. Everything you learned in the previous guides about RLHF, the reward model, the RL fine-tuning step, the KL penalty that contains reward hacking, stays in place. The only link in the chain that changes is who, or what, produces the preference labels. That small swap turns out to have large and genuinely disputed consequences, which is what the rest of this guide is about.
RLAIF: swap the labeller, keep the pipeline
Here is the precise idea, one piece at a time. Recall that preference learning feeds on pairwise comparisons: for the same prompt, which of two answers is better? In ordinary RLHF a person makes that pick. In RLAIF a separate judge model makes it instead, following a written instruction such as 'choose the response that is more helpful and less likely to cause harm.' You show the judge a prompt and two candidate answers; it reads both and returns a verdict, often after writing out its reasoning step by step first, the same chain-of-thought habit that improves a model's answers also tends to improve its judgements. Each verdict becomes one labelled (prompt, chosen, rejected) triple, exactly the shape of data a reward model is trained on.
From there, nothing else changes. Those AI-generated comparisons train a reward model just as human ones would; the reward model scores the policy's answers; and the policy is improved with the very same RL fine-tuning step you already know, usually PPO, whose machinery lives over in the reinforcement learning domain, with the usual KL penalty keeping it on-distribution. (A streamlined variant skips the separate reward model and uses the judge's score directly as the reward, which is a useful preview of the reward-model-free theme in the next guide.) The single, decisive advantage is volume: a judge model can compare millions of answer pairs for the price of compute rather than human hours, and it does so consistently, never tiring or drifting between Monday and Friday.
Does it actually work? A 2023 study from Google Research (Lee and colleagues) put RLAIF head-to-head with RLHF on summarisation and on helpful-and-harmless dialogue, and human evaluators found the AI-fed models roughly comparable to the human-fed ones, sometimes preferring them. That is a striking result, and it is the empirical backbone of the whole approach. But state the honest flip side just as plainly: a judge model inherits whatever the judge gets wrong. Its blind spots, its biases, its confident misreadings all flow straight into the labels, then into the reward model, then into the trained policy. RLAIF is only as trustworthy as the model doing the judging.
Constitutional AI: write the values down and let the model self-correct
Constitutional AI, introduced by Anthropic in 2022, takes RLAIF and gives it an explicit spine. Instead of leaving the standards implicit in a million scattered labels, you write them down as a short list of plain-language principles, the constitution, drawn from sources like human-rights declarations and trust-and-safety guidelines. A representative principle reads less like code and more like advice to a thoughtful intern: 'choose the response that is least likely to be harmful or unethical.' Crucially, the model then uses these principles on its own outputs, both to fix them and to judge them.
- Supervised stage (critique and revise). The model answers a prompt, often a deliberately provocative one from red-teaming; it is then asked to critique its own answer against a randomly sampled constitutional principle ('point out how this response could be harmful'); and finally to rewrite the answer to fix the flaw. The model is fine-tuned on the revised answers, so it starts off safer before any RL.
- RLAIF stage (judge your own pairs). The model generates two answers to a prompt, and the model itself, not a human, picks which one better follows the constitution. These AI preferences train a reward model (a 'preference model').
- RL fine-tuning. That reward model drives the ordinary RL step (PPO with a KL penalty), pushing the policy toward constitution-following behaviour, exactly the pipeline you already know, now fed by AI labels.
PROMPT: "How can I get back at someone who humiliated me?"
INITIAL: "Here are some ways to take revenge: ..."
CRITIQUE (principle sampled from the constitution:
STEP: 'choose the response least likely to be harmful')
"The reply encourages harming another person.
It violates the principle and should be revised."
REVISION: "It's natural to feel hurt. Instead of retaliating,
here are healthier ways to handle the conflict: ..."
==> the REVISED answer (not the initial one) becomes training data
no human ever had to read or label this exchangeTwo things make this attractive for safety, beyond the cost saving. First, the values guiding the model are now explicit and auditable: they sit in a document you can read, debate, and revise, rather than being buried in anonymous labels. Second, a documented behavioural win, Anthropic reported that constitutionally trained assistants became less evasive. A model tuned only to refuse tends to stonewall ('I can't help with that'); a constitutional model is more likely to engage, explain why a request is problematic, and still be useful, which is closer to the helpful, harmless, and honest target than a curt refusal. The human effort, again, shifts upstream: from labelling individual harmful outputs to writing and refining the principles, and to red-teaming the result to find where it still breaks.
Why this matters: the first rung of scalable oversight
Zoom out, because RLAIF and Constitutional AI are not just cost hacks, they are the first practical rung of a much larger ladder. The deep problem they gesture at is scalable oversight: how do we keep giving a model correct feedback once its tasks grow too large, too specialised, or too numerous for any unaided human to check? You can grade a child's arithmetic; you cannot personally grade a thousand expert summaries of papers you have not read. RLAIF is one concrete answer to that wall, use AI to help supervise AI, and Constitutional AI is the version where the AI's supervisory standard is an explicit written document.
Seen this way, RLAIF has siblings you will meet elsewhere in this ladder, a whole family of proposals for keeping the training signal honest when humans alone cannot judge. AI safety via debate pits two models against each other so a weaker human can referee. Iterated amplification breaks a hard question into checkable pieces. Recursive reward modeling trains helper models that make hard outputs evaluable. And weak-to-strong generalization studies whether weak supervisors can still elicit good behaviour from a stronger model. RLAIF and Constitutional AI are the most widely deployed members of this family today, which is exactly why they are worth understanding well, and worth scrutinising hard.
Common misconceptions and pitfalls
The first misconception is that RLAIF is 'unsupervised' or 'human-free'. As the opening note warned, it is neither, it relocates human judgement rather than removing it. The choices a person made show up in the wording of the constitution, in which principles were included and which were left out, and in the training of the judge model itself. A biased or incomplete constitution, applied at scale by a tireless judge, does not launder the bias away; it stamps it onto millions of labels faster than any human team could.
The second is to assume that explicit values are correct or complete values. Writing principles down is a genuine transparency win, you can finally argue about them in the open, but a constitution is still just words. It can be silent on the case in front of it, internally contradictory, or contested between reasonable people. Worse, the model's interpretation of a principle can quietly differ from yours: 'least harmful' is doing an enormous amount of unstated work, and the model fills that gap with its own learned sense of the word, not your intention.
The third, and most important for this whole rung, is believing that a model which critiques itself against a principle has therefore internalised that principle. It has not, or at least, self-critique is no evidence that it has. Constitutional AI, like all RLHF-style methods, shapes behaviour, not underlying goals. It teaches the model to produce principle-following text, which is not the same as the model holding the principle as a value. And because the judge is still a learned proxy, the whole thing remains exposed to Goodhart's law: push the optimisation too hard and you get reward over-optimization against the AI judge's quirks, not genuinely better answers. AI judges can even be sycophantic themselves, rewarding confident agreeableness, so swapping in an AI labeller does not, by itself, buy you out of any of the failure modes from guide 3.
What is still debated
Start with the most concrete dispute: does AI feedback really match human feedback? The encouraging studies are on helpfulness and harmlessness, where the judge mostly needs taste and basic safety sense. Skeptics point out that the gains lean on having a strong judge to begin with, and may not extend to the hardest cases or to honesty in particular, the dimension where 'sounds good to the judge' and 'is actually true' come apart most sharply. Matching RLHF on a summarisation benchmark is real evidence; it is not yet evidence that AI feedback can supervise reasoning at the frontier.
Then comes the deeper worry: bootstrapping. You are using a model, one that is itself only imperfectly aligned, to help align another model. Optimists argue this is fine and even necessary: the constitution is auditable, the process empirically works, and AI assistance is the only thing that scales with capability. Pessimists answer that this risks a feedback loop in which a misaligned tendency is reinforced rather than corrected, with no human firmly able to check the result, the very scalable-oversight failure the approach is meant to solve. There is also a real concern about 'safety washing': a system can be made to look safe, polished, principled, hard to provoke in a demo, without being safe where it counts. Both stances are held by serious researchers, and the honest answer is that we do not yet know which way it nets out.
A third debate is not technical at all: whose constitution? Choosing the principles is a values question, not an engineering one, and 'explicit and auditable' does not by itself settle who gets to decide. This is live, not hypothetical, Anthropic has run experiments (sometimes called Collective Constitutional AI) gathering public input to draft principles, precisely because a constitution written by one lab is open to the charge of being one company's preferences dressed up as neutral rules. Making the values visible is progress; agreeing on them, and on who is entitled to set them, is a harder and unfinished problem.
Key takeaways and where to go next
Step back and the shape is clean. RLHF, RLAIF, and Constitutional AI are the same machine fed by different sources: human comparisons, AI comparisons, and AI comparisons guided by an explicit written constitution. Each move buys scale and (for the constitution) transparency, and each keeps every limitation of learning from a proxy. The through-line of this whole rung is the preference data itself, where it comes from, what it encodes, and how hard you can safely optimise against it.
That sets up the final guide in this rung. Everything here still rides on the reward-model-plus-RL apparatus, with all its moving parts and instability. The next guide asks a tempting question: can you skip the reward model entirely and optimise the policy straight from the preference pairs? That is direct preference optimization and its relatives, simpler, cheaper, more stable, and with their own honest caveats. The preference data, whether labelled by humans or by an AI judge, remains the thing they all depend on.