value of a policy

When we ask is this policy any good, we need a single number to compare against another policy. The value of a policy provides it: a summary score of how much expected return that whole way of behaving earns. Sometimes it means the value at a particular starting state, sometimes the average over wherever episodes tend to begin, but either way it collapses an entire strategy into one figure of merit.

It is the same object as the objective J of pi — the expected return under the policy — viewed as a yardstick for ranking policies rather than as something to optimize. This is what lets us say one policy is better than another: one policy beats another if its value is at least as high from every state. Policy improvement and policy iteration are precisely procedures for producing a new policy whose value is provably higher.