experience replay

Imagine a student who, instead of reading each page once and forgetting it, keeps every page in a folder and re-studies random pages each night. Experience replay does this for an agent: every transition it lives through — state, action, reward, next state — is dropped into a memory, and training samples random batches from that memory rather than only the latest moment. Old lessons get revisited many times, long after the situation that produced them has passed.

This solves two problems at once. First, consecutive transitions are highly correlated — a stream of nearly identical frames — and gradient methods assume roughly independent samples, so random sampling decorrelates them. Second, it improves data efficiency, because each costly interaction with the environment is reused dozens of times instead of once. Replay is what made off-policy deep RL like DQN practical; it works precisely because Q-learning is off-policy and can learn from data the current policy would not have generated.