first-visit Monte Carlo

Within a single episode an agent may pass through the same state several times. First-visit Monte Carlo handles this by being strict: for each episode it records only the return that followed the very first time the state appeared, and ignores any later visits in that same episode. Averaging those first-visit returns across many episodes gives the state's value estimate.

The reason for the rule is statistical cleanliness. Each episode contributes at most one return per state, and those returns are independent samples of the same quantity, so the average is an unbiased estimator that converges to the true value with the usual one-over-square-root-N error shrinkage. It is the textbook default precisely because its behavior is easy to reason about.