every-visit Monte Carlo

Every-visit Monte Carlo is the relaxed sibling of the first-visit method. Instead of counting only the first time a state appears in an episode, it counts the return following every occurrence of that state, then averages all of them. If a state shows up three times in one episode, all three of its subsequent returns feed the estimate.

This uses more data per episode, which can help when visits are scarce, but it gives up the clean independence of first-visit. Returns from repeated visits inside one episode overlap and are correlated, so the estimator is biased for any finite sample. Reassuringly, that bias washes out as episodes accumulate: every-visit MC is still consistent and converges to the same true values, and in practice it often performs about as well while being simpler to code.