quantile regression DQN (QR-DQN)

QR-DQN flips C51's representation inside out. C51 fixes the return values and learns their probabilities; QR-DQN fixes the probabilities and learns the return values. It asks: what is the return at the tenth percentile, the thirtieth, the ninetieth? The network outputs a set of quantile estimates, and together they trace out the distribution without anyone having to guess its range in advance. The support adapts to wherever the returns actually live.

It learns these quantiles by quantile regression, using a tilted absolute loss — the pinball loss — that penalises under- and over-estimates asymmetrically so each output settles at its target quantile. The Huber-smoothed version of that loss gives the practical algorithm. Freeing the method from a fixed grid removed C51's clipping problem and generally improved results, and the quantile idea led on to even more flexible distributional agents like IQN. It is the natural companion to C51 in the distributional-RL toolkit.