value learning

Try to write down everything you care about, and you will quickly give up. Human values are vast, context-dependent, full of unspoken exceptions, and sometimes contradictory. We know not to win a board game by flipping the table, not to clean a room by throwing the contents out the window, without anyone ever stating those rules. Value learning is the project of getting an AI to learn what we value from data, behavior, and feedback, instead of being handed an explicit, complete rulebook we cannot actually write.

Precisely, value learning covers a family of approaches to inferring the objective rather than specifying it. Inverse reinforcement learning tries to infer the underlying reward from observed human behavior. Learning from preferences and feedback, as in RLHF, infers what we want from comparisons and ratings. Cooperative inverse reinforcement learning frames the agent as genuinely uncertain about the true objective and motivated to learn it from the human, an uncertainty that, helpfully, also makes the agent more willing to be corrected. The micro-picture: rather than coding do not break the vase, you show examples and let the system infer the preference behind them.

Value learning matters because it targets the hardest part of outer alignment, actually specifying human values. The honesty here is unavoidable, because this is deeply contested terrain. Human values conflict between people and shift over time; we are often irrational and act against our own stated values, so behavior does not cleanly reveal them; and there are open questions about whose values an AI should learn and the old is-ought gap between what people do and what they should do. Learning to predict human behavior is not the same as learning human values.