reward-to-go

Naive REINFORCE weights every action in an episode by the same total return — including rewards that arrived before the action was even taken. That is plainly unfair: an action cannot have caused a reward in the past. Reward-to-go fixes this by crediting each action only with the rewards that came after it, the return from that step onward. It is a small change with an outsized effect on noise.

The justification is causality: in expectation, multiplying a score by rewards earned before the action contributes zero to the gradient, because past rewards are independent of the current action's score. So dropping those terms leaves the gradient unbiased while removing a chunk of pure noise. Each action's weight becomes G_t = Σ_{t'≥t} γ^{t'−t} r_{t'} instead of the full-trajectory return.

Reward-to-go is one of the cheapest and most reliable variance reducers, and it composes freely with baselines and advantages — in fact advantage estimates are built on reward-to-go returns. It is usually the first refinement you make after writing the textbook policy gradient, and almost every serious implementation includes it by default.