recurrent replay RL

Frame stacking handles short bursts of motion, but some environments hide so much — what is behind a wall, a number you saw and must remember — that no fixed window of recent frames is enough. The cure is to give the agent a memory, usually a recurrent network like an LSTM that carries a hidden state through time. Recurrent replay RL is how you train such an agent off-policy: you store and replay whole sequences of transitions rather than single ones, so the recurrence has a timeline to unroll over.

The hard part is the hidden state. A transition pulled from the buffer was experienced under a recurrent state the agent had built up, but that state was produced by an old version of the network and is not stored. Replaying the sequence from a zeroed hidden state distorts learning. R2D2, the influential agent here, fixes this by storing the recurrent states in the buffer and burning in a warm-up prefix of steps to re-establish the hidden state before computing losses. This made large-scale recurrent off-policy agents practical and set strong Atari records.