Monte Carlo prediction

Imagine wanting to know how good a chess opening really is. You cannot read the answer off a rulebook, so you just play many full games from that position and average how they turned out. Monte Carlo prediction does exactly this for a fixed policy: to estimate the value of a state, it runs complete episodes, then averages the actual returns — the total discounted reward earned afterward — observed each time the agent passed through that state.

The method is pure experience: it needs no model of the environment, only episodes that eventually end. Because it learns from real, completed returns rather than from other guesses, its estimates are unbiased and settle onto the true values as more episodes accumulate. The price is patience and noise. You must wait for each episode to finish before you can update, and one lucky or unlucky run can swing a return a long way, so the estimates carry high variance.