prioritized experience replay

Uniform replay treats every stored memory as equally worth revisiting, but that is wasteful — most transitions are unsurprising and teach the agent little, while a few rare, shocking ones carry most of the lesson. Prioritized experience replay samples the surprising ones more often. Surprise is measured by the temporal-difference error: how far the agent's prediction was from what actually happened. A big error means there is something here worth learning, so replay it sooner and more frequently.

Each transition's sampling probability is set proportional to its TD error raised to a power, so high-error events bubble to the top while everything still has a nonzero chance. But sampling non-uniformly biases the gradient — you are now learning from a skewed distribution — so importance-sampling weights are applied to correct it. The result learns markedly faster than uniform replay on many Atari games. The cost is bookkeeping: priorities must be updated as errors change, usually via a sum-tree data structure for efficient sampling.