Predict the distribution, not just the average
An action-value Q(s, a) is the expected return — a single number that averages away all the risk. But two actions can share the same mean while one is a safe trickle and the other a coin-flip between jackpot and disaster. Distributional RL keeps the whole distribution of returns Z(s, a) instead of collapsing it to its mean.
An interactive Markov chain whose probabilistic transitions between states produce a spread of possible outcomes.
Why bother, if you act by the mean anyway? Because predicting a richer target is a better learning signal: the network forms sharper internal representations, gradients are more informative, and empirically the agent learns faster and scores higher even when the final policy is still greedy on the mean.
The distributional Bellman equation
The ordinary Bellman update says expected value equals reward plus discounted next value. The distributional Bellman equation says the same thing about random variables: the return distribution equals the (shifted and scaled) next-state return distribution, Z(s, a) =ᴰ r + γ·Z(s′, a′). Reward shifts the distribution; the discount γ squeezes it toward zero.
The distributional Bellman equation: the return is a random variable equal, in distribution, to the reward plus the discounted next return.
C51: a categorical distribution
Categorical DQN (C51) fixes a grid of 51 return 'atoms' spanning a range like [−10, +10] and has the network output a probability for each atom — a histogram over outcomes. Applying the distributional Bellman update shifts and scales this histogram, which usually lands between atoms, so C51 projects the result back onto the fixed grid and trains with a cross-entropy loss.
The name: 51 atoms. Its weakness is that you must guess the value range and bin count up front — too narrow and returns saturate at the edges; too coarse and the histogram is blurry.
QR-DQN: flip the axes
Quantile Regression DQN (QR-DQN) turns the picture sideways. Instead of fixing the value positions and learning probabilities, it fixes the probabilities (e.g. the 1st, 3rd, … 99th percentiles) and learns the return value at each quantile. Now there is no preset range to mis-guess — the support adapts to whatever the game pays out.
Training uses the quantile (pinball) loss, which is just an asymmetric absolute error — and that asymmetry is what teaches each output to settle at its target percentile. It removes the projection step C51 needs and generally trains more robustly.
The quantile (pinball) loss: an asymmetric absolute error whose tilt τ pins each QR-DQN output to its target percentile.
Noisy nets: exploration the network learns
Classic ε-greedy explores by acting randomly a fixed fraction of the time — undirected and uniform across the whole state space, which is hopeless when only a few states need bold exploration. Noisy networks instead inject learnable Gaussian noise into the network's weights.
Because the noise magnitude is a parameter trained by gradient descent, the agent can learn to be noisy (exploratory) in unfamiliar states and quiet (decisive) where it already knows the right move — a state-dependent, self-annealing answer to the exploration–exploitation tradeoff that needs no ε schedule.