Constitutional AI

Most early RLHF for safety relied on people reading lots of harmful or disturbing outputs to label which were worse, slow, costly, and unpleasant work. Constitutional AI, introduced by Anthropic, asks a different question: what if we write down the principles we care about as a short text (a 'constitution') and have the AI use those principles to critique and improve its own answers? The values are stated openly in words, and the model does much of the labour of applying them.

It runs in two phases. First, a supervised phase: the model produces an answer, then is prompted to critique that answer against a constitutional principle (for example 'is this response harmful or evasive?') and rewrite it to be better; training on these self-revised answers makes the model safer to begin with. Second, an RLAIF phase: the model generates pairs of answers, and instead of humans, the model itself picks the one that better follows the constitution; those AI preferences train a reward model that then drives the usual reinforcement-learning step. Human effort shifts from labelling individual outputs to writing and refining the principles.

For safety this has real attractions: the values guiding the model are explicit and auditable rather than buried in millions of anonymous labels, it reduces how much harmful content humans must read, and it scales as a form of AI-assisted oversight. But it is not a solution to alignment. The constitution is only words and may be incomplete or contested; the model's interpretation of a principle can differ from ours; and like all RLHF-style methods it shapes behaviour rather than installing values, and remains vulnerable to Goodhart-style gaming. It makes the target clearer and cheaper to pursue, not automatically correct.