Bellman expectation equation

The Bellman expectation equation captures a beautifully simple idea: the value of where you are equals the reward you get next plus the discounted value of where you land. Instead of summing an endless stream of future rewards, you only look one step ahead and trust the value function to summarize everything beyond. It turns an infinite lookahead into a single recursive relationship that the value of a fixed policy must satisfy.

For a policy, it says a state's value equals the average — over the actions the policy takes and the next states the environment produces — of the immediate reward plus the discounted value of the next state. This self-consistency is the engine behind policy evaluation: if your value estimates do not satisfy the equation, the mismatch tells you exactly how to correct them. Temporal-difference methods are, at heart, just nudging estimates to make this equation hold.