Bellman backup operator

A "backup" takes the value of where you might go next and pushes it back onto the state you are in. The Bellman operator packages that into a single mathematical machine: feed it a whole table of value estimates and it hands back a new, improved table. Treating one full sweep as applying an operator turns "run the algorithm" into "apply this function again and again".

There are two versions. The expectation operator averages over the actions a fixed policy takes; the optimality operator takes a max over actions. Both map a value function to a new value function by adding immediate reward to the discounted value of successors. Iterative policy evaluation is just repeated application of the expectation operator; value iteration is repeated application of the optimality operator. Each has a unique fixed point.