return decomposition

Return decomposition is the small algebraic trick that powers nearly all of value-based RL. The return from now is just the very next reward plus the discounted return from the next step onward. You peel off one reward at a time, and what remains has the exact same shape as what you started with — a return, only one step later and discounted by gamma.

Written out, the return at time t equals the next reward plus gamma times the return at time t plus one. Take expectations and this recursion becomes the Bellman equation; that is the bridge from a definition to a usable update. It also explains why a learned value function is so useful: it lets you replace the unknown future tail with an estimate of the next state's value, so you can learn from a single transition without waiting for an entire episode to finish.