constitutional feedback (RLAIF)

Human comparisons are slow, expensive, and inconsistent, and they expose labellers to unpleasant content. What if the feedback came from a written set of principles instead — a short 'constitution' of rules like 'prefer the more harmless response' — applied by an AI model rather than a person? Constitutional feedback does this: the comparisons that train the reward model are produced by a model judging outputs against explicit principles, not by humans labelling each pair.

A capable model is prompted with a principle from the constitution and asked which of two responses better satisfies it; its choices become the preference dataset, which trains a reward model and then a policy through the usual RLHF machinery — a setup often called RL from AI feedback (RLAIF). The human effort moves up a level: people write and revise the principles rather than labelling thousands of individual comparisons, which makes the values explicit, auditable, and cheap to scale.

This trades human labour for transparency and scale, but the feedback is only as good as the principles and the judging model. Biases and gaps in either propagate straight into the policy, and the model can misread its own rules or apply them unevenly. Constitutional feedback does not remove humans from alignment — it relocates them, from labelling outputs to authoring and stress-testing the constitution itself.