replay buffer

The replay buffer is the actual container that experience replay draws from — think of it as a rolling logbook of the agent's recent life. Each entry is one transition: where it was, what it did, what reward it got, where it ended up, and whether the episode ended. New experiences are written to the end, and when the buffer fills, the oldest entries are overwritten, so it always holds a sliding window of the most recent few hundred thousand or million steps.

Its size is a real design knob. Too small, and the buffer forgets useful old experience and stays dominated by whatever the agent is doing right now, reintroducing the correlations replay was meant to remove. Too large, and it clings to transitions from a long-dead, much worse policy that no longer reflect how the agent behaves, slowing learning. Typical Atari agents keep around a million transitions. The buffer is plain storage; the intelligence lives in how you sample from it.