process reward models

When an agent solves a long problem — a multi-step proof, a chain of reasoning — you can reward it two ways. Outcome supervision checks only the final answer: right or wrong. Process supervision instead grades each step of the work along the way. A process reward model is trained to score those intermediate steps, telling the agent not just whether it succeeded but where, specifically, its reasoning went right or wrong.

A process reward model is trained on labels (human or model-generated) of individual reasoning steps as correct, helpful, or flawed, and it emits a reward at each step rather than only at the end. This gives a dense, well-localised learning signal: credit lands on the specific step that earned it, instead of one coarse reward smeared across an entire solution. It also discourages a notorious failure — reaching the right final answer through wrong reasoning, which an outcome-only reward would happily reinforce.

Process supervision tends to produce more reliable and more interpretable reasoning than outcome supervision, and it has become a leading direction for training reasoning models. The honest cost is that step-level labels are far more expensive to collect, and deciding what makes an intermediate step 'good' is itself subtle and contestable. It sits squarely at the intersection of reward design and the old problem of credit assignment.