Constitutional AI

Constitutional AI aligns a model to a short written set of principles — a constitution — instead of relying only on case-by-case human labels. The idea is to make the values explicit and steerable: rather than thousands of raters implicitly deciding what harmless means, you write down the principles and have the model help apply them.

It runs in two phases. In the supervised phase, the model critiques and revises its own responses against the constitution, generating safer answers to learn from. In the reinforcement phase, the model compares pairs of its own responses and labels which better follows the principles, producing AI-generated preference data that trains a reward model in the usual RLHF way. Humans still write and refine the constitution; the labor of applying it at scale is shifted to the model.

The appeal is transparency and scale: the values live in editable text you can read and debate, and you need far fewer humans labelling potentially distressing content. The risk is that the model's reading of a principle, not your intended reading, is what actually gets optimized.