discount factor (gamma)

The discount factor decides how much the agent cares about the future versus right now. With gamma near zero it is impatient, grabbing whatever reward is immediately on offer; with gamma near one it is far-sighted, happily forgoing a quick treat for a bigger payoff many steps later. It plays the role an interest rate plays for money: a reward later is worth a little less than the same reward today.

Concretely, a reward arriving k steps from now is multiplied by gamma raised to the power k before being summed into the return. A value below one keeps that infinite sum finite and well-defined, which is essential when interaction never ends. Setting gamma is a real modelling choice: too low and the agent is short-sighted, too high and credit gets spread so thin that learning becomes slow and unstable.