MuZero planning

MuZero asked a radical question: do you really need to predict what the world looks like in order to plan well? Its answer was no. MuZero learns a model that never reconstructs observations at all — it only predicts the quantities planning actually consumes, namely rewards, values, and policies — and then runs powerful tree search inside that abstract model to choose moves, mastering Go, chess, shogi, and Atari without ever being told the rules.

It has three learned pieces working in a latent space: a representation function that maps the real observation to an initial hidden state, a dynamics function that maps a hidden state and action to the next hidden state plus a predicted reward, and a prediction function that reads a hidden state into a policy and value. Monte Carlo tree search unrolls the dynamics function to build a search tree, and the whole model is trained end-to-end so that its predicted rewards, values, and policies match what really happens and what the search recommends.

MuZero matters because it unified planning and learning without a given simulator, learning its own model where AlphaZero needed the true rules. The catch is that its hidden states are value-equivalent, not interpretable — the model is right about consequences, but it cannot show you a picture of the future it imagines.