Bellman equation
/ BEL-mun ee-KWAY-zhun /
The Bellman equation is one of those rare ideas that is both profound and obvious once you see it. It says: the value of where I stand right now equals the reward I get for my next step, plus the value of wherever that step lands me. In other words, a big future reward can always be sliced into 'what I get now' plus 'what's left to come.' The whole long-term value of a situation is defined in terms of itself one step later.
This self-referential shape is what makes reinforcement learning computable. Instead of imagining every possible future to the end of time, the agent only needs to relate each state to its immediate neighbors. Knowledge then ripples outward: the value near the goal becomes accurate first, then the squares one step away pick it up, then two steps away, and so on, until good estimates have flooded backward across the whole problem.
Named after Richard Bellman, who developed dynamic programming in the 1950s, this equation is the silent engine inside almost every RL method. Q-learning, SARSA, and value iteration are all, at heart, just different ways of nudging estimates until they satisfy the Bellman equation. The honest caveat: solving it exactly requires knowing the environment's rules, which agents usually don't — so in practice they approximate it from sampled experience, and the approximation can drift, especially once neural networks enter the picture.
On a path to a +10 goal with no step cost, the goal's neighbor has value 10. Its neighbor must then satisfy: value = 0 (this step's reward) + 10 (the neighbor's value) = 10. The next one back: 0 + 10 = 10. The '10' propagates backward, one Bellman update at a time.
Value here = reward now + value of where I land next. Estimates flood backward from the goal.
The Bellman equation defines what a correct value function looks like; it doesn't, by itself, tell you the numbers. Algorithms reach those numbers by repeatedly applying the equation until the estimates stop changing — a process that is guaranteed to converge with exact tables, but only hoped to converge once neural networks approximate the values.