distributional reinforcement learning
Distributional reinforcement learning learns the whole distribution of returns from a state-action pair, not just its average. Two actions can share the same expected return yet differ wildly in spread, with one occasionally disastrous and the other steady; modeling only the mean throws away exactly that information.
It replaces the scalar Bellman equation with a distributional Bellman operator acting on the return random variable: the return distribution equals the immediate reward plus a discounted, shifted copy of the next state-action's distribution. Representations include a categorical fixed support trained by a projected cross-entropy loss (C51), and quantile regression that learns the locations of a set of quantiles (QR-DQN, IQN). The operator is a contraction in a maximal-form Wasserstein metric rather than in KL.
Even when you ultimately act on the mean, carrying the full distribution often improves representation learning and stability, and it opens the door to risk-sensitive policies that prefer or avoid variance.
The distributional Bellman equation: the return random variable equals reward plus a discounted next-step return, in distribution.
The distributional Bellman operator contracts in Wasserstein, not KL; using a KL-based loss is a practical approximation, not the underlying theory.