value function
/ VAL-yoo FUNK-shun /
A value function answers a forward-looking question: starting from this situation and following my usual strategy, how much total reward should I expect from here on out? It is not about the reward I get right now — it's about the whole future stretching ahead. A square one step from the goal has high value; a square next to a cliff has low value, even if standing on it costs nothing today.
This is the key trick that lets agents plan beyond the immediate moment. A chess position with no captures available might give zero reward, yet a strong value function recognizes it as winning because of what it leads to. By learning a value for each state, the agent gets a map of which situations are promising and which are traps — and then it can simply walk toward higher value, like water finding the downhill path.
Two cautions. First, value is always tied to a particular policy: 'how good is this state' only means something once you say 'assuming I keep playing this way.' Change the strategy and the values change. Second, the value function is an estimate, learned from limited experience, so it can be confidently wrong about parts of the world the agent rarely visits — which is one reason agents keep exploring rather than trusting their map completely.
On a grid where the exit gives +10, the square right next to the exit has value near +10 (one step away). A square in a far corner has lower value — many steps, and any could go wrong. The agent reaches the exit just by always stepping toward the higher-value neighbor.
Value = expected future reward from here. It turns 'win or lose at the end' into guidance at every step.
Don't confuse value with reward. Reward is the immediate payout for one step; value is the predicted sum of all future rewards. A move with small immediate reward can have huge value if it sets up a big payoff later — and that gap is exactly what the value function exists to capture.