state-value function

Standing in some situation, you might ask: if I keep behaving the way I usually do, how well will things turn out from here? The state-value function answers exactly that. For a given policy, it assigns to each state a single number: the expected return you will collect starting from that state and following the policy thereafter. Good states — close to the goal, full of opportunity — get high values; bad states — near a cliff, low on options — get low values.

It is written V-pi of s, read as the value of state s under policy pi. Note the value depends on the policy: a state is only as good as what you will do from it. The same chessboard is winning for a strong player and losing for a weak one. The state-value function turns the vague feeling of this looks promising into a precise expected number, which is what lets algorithms compare situations and learn.