categorical DQN (C51)

C51 was the first deep agent to make distributional RL work in practice. Its trick for representing a return distribution is deliberately simple: pin down a fixed grid of possible return values — fifty-one evenly spaced points, hence the name — and have the network output a probability for each. The distribution becomes a little histogram over those fixed bins, and the agent learns to shape that histogram for every state-action pair.

The subtlety is the update. The distributional Bellman target shifts and scales the next distribution, which generally lands between the fixed grid points, so C51 projects it back onto the grid before measuring the loss as a cross-entropy between predicted and target histograms. This projection is the fiddly heart of the method. C51 markedly outperformed plain DQN on Atari and proved distributional RL was more than a curiosity, but its weakness is the grid: you must choose its range and resolution in advance, and returns outside that range get clipped.