eligibility traces

When a reward finally arrives, which of the states you visited along the way deserve the credit? Eligibility traces are a short-term memory that answers this. Each state keeps a trace that jumps up when the state is visited and then fades over time. When a TD error appears, it is broadcast to every state in proportion to its current trace, so recently and frequently visited states receive the largest share.

This is the efficient, backward-looking way to implement TD(λ). Instead of waiting to assemble the lambda-return from the future, the agent carries the traces forward and updates many states at every step. The trace for each state decays by the product of the discount gamma and lambda between visits. The result is faster credit assignment — a single delayed reward can immediately adjust a whole trail of preceding states rather than only the last one.