value-equivalent models

There are two ways a model can be "good." One is to reproduce the world faithfully, pixel for pixel. The other — the value-equivalent view — is to be good only where it counts: a model is value-equivalent if planning with it yields the same values and the same decisions as planning with the true environment, even if it gets everything else wrong. For control, the second kind is what you actually need.

This reframes model learning. Instead of training a model to minimise prediction error on observations, you train it so that the planning computations done on top of it — the value backups, the policy it implies — come out correct. A model can throw away vast amounts of irrelevant visual detail and still be perfect for choosing actions, which is exactly what makes systems like MuZero work: their learned dynamics are optimised for consequences, not appearances.

The appeal is focus: a value-equivalent model spends its limited capacity on what matters for the task. The trade-off is generality — a model tuned to one reward and one planner may be useless for a different objective, since it was never trying to capture the world, only the decisions.