process supervision

Process supervision trains and rewards a model on the quality of each reasoning step, not just on whether the final answer is right. The contrast is outcome supervision, which gives one reward at the end and is indifferent to how the answer was reached. The intuition is that a sound final answer reached by flawed or lucky reasoning is fragile and hard to trust, so checking the work, step by step, yields both better accuracy and more transparent, verifiable reasoning.

In practice each step of a chain of thought is labeled correct, neutral, or wrong, and a process reward model is trained to score steps; that model then supervises generation, by reranking sampled solutions, by reward at each step, or by guiding search. On hard mathematical benchmarks, process reward models trained on step-level human labels outperform outcome reward models at selecting correct solutions, and they localize exactly where reasoning first goes wrong.

For safety the appeal is alignment of the reasoning process, not only the answer: rewarding faithful intermediate steps discourages reward hacking that games the outcome and produces more legible traces a supervisor can audit, a concrete form of scalable oversight. The costs are heavy step-level annotation, the risk that the process model is itself gamed, and an open question of whether the visible chain of thought faithfully reflects the model's actual computation.