distributional Bellman equation

Ordinary value learning asks one question: on average, how much return will follow from here? But two situations can share the same average and feel completely different — a steady trickle of points versus a coin flip between jackpot and ruin. The distributional view keeps the whole spread, not just the mean. Instead of learning the expected return, the agent learns the distribution of returns: the full range of outcomes and how likely each is.

The distributional Bellman equation is the recursive rule that distribution obeys. The return from a state equals the immediate reward plus a discounted copy of the next state's return distribution — so you shift the next distribution by the reward and scale it by gamma, rather than just scaling and adding numbers. Training matches the predicted distribution to this bootstrapped target distribution. Even when you ultimately act on the mean, learning the full distribution gives a richer, more stable training signal, which is why distributional agents like C51 and QR-DQN often outperform their expectation-only cousins.