JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Planning Where It Counts: Value-Equivalent Models & MuZero

The frontier idea: don't model the whole world — model just enough to plan correctly, then search like a champion.

The trouble with predicting everything

Every model so far has tried to reproduce the environment — next state, or next observation, as faithfully as possible. But faithful reconstruction is the wrong objective: a model can be excellent at predicting irrelevant detail and still useless for choosing actions, or sloppy about appearances yet perfect for planning. What we actually need is a model that lets us compute the right values and decisions, nothing more.

Value-equivalent models

A value-equivalent model flips the objective. Instead of asking the model to predict the next state accurately, we ask it to make the same predictions about future value and reward that the real environment would, when used for planning. The model's internal states need not resemble the real world at all; they only have to support correct Bellman-style backups. This frees the model to ignore everything that does not affect the answer — a powerful antidote to the wasted-capacity problem of reconstruction.

V^{\pi}(s) = \mathbb{E}\!\left[\, r_{t+1} + \gamma\, V^{\pi}(s_{t+1}) \;\middle|\; s_t = s \right]

A value-equivalent model only has to get this right — the future value the Bellman equation defines, not the literal next state.

MuZero: planning with a value-equivalent model

MuZero is the landmark system built on this principle. It learns three networks together — a representation function that maps observations to an abstract latent state, a dynamics function that predicts the next latent state and reward, and a prediction function that outputs a policy and value from a latent state. None of them is trained to reconstruct observations; they are trained only so that planning through them reproduces the right policies, values, and rewards. The same algorithm masters Go, chess, shogi and Atari without ever being told the rules.

s^{0}=h_\theta(o_{\le t}),\quad (s^{k+1},\hat{r}^{k+1})=g_\theta(s^{k},a^{k}),\quad (\boldsymbol{p}^{k},v^{k})=f_\theta(s^{k})

MuZero's three learned networks: a representation h that encodes observations into a latent state, a dynamics g that predicts the next latent state and reward, and a prediction f that outputs a policy and value.

Decision-time search with MCTS

MuZero plans at decision time by running Monte Carlo tree search inside its learned latent model. The search builds a tree of imagined latent futures, and its outcome does two jobs: it picks the move to play now, and the improved policy it discovers becomes a training target that makes the networks better — a self-improving loop reminiscent of AlphaZero, but with no given simulator.

Monte Carlo tree search in action — MuZero runs exactly this search inside its learned latent model.

An interactive Monte Carlo tree search expanding nodes, selecting promising branches, and backing up values.

  1. Encode the current observation into a latent root state with the representation network.
  2. Search forward with MCTS, expanding the tree using the dynamics and prediction networks only.
  3. Act by playing the most-visited action at the root, then re-search from the next observation.
  4. Train the networks so their predictions match the search's improved policy, the observed reward, and the bootstrapped value.

The frontier and what you have learned

Value-equivalent planning is one of the most active frontiers of model-based RL: later work pushes MuZero toward extreme sample efficiency, offline data, and continuous control. The throughline of this whole track is simple — *a model is a tool for thinking ahead, and the best model is the one that makes your planning correct, not the one that paints the prettiest picture of the world.*