Rewards, Returns, and the Discount Factor

Reward versus return

The reward is a single number the environment hands over at each step — say, +1 for staying upright, −100 for falling. By itself it's short-sighted: it can't tell whether a step was a brilliant setup or a costly mistake. What the agent actually cares about is the return: the total reward collected from now until the end. One step is a frame; the return is the film.

This gap is the whole reason RL is hard: a great move now might only pay off twenty steps later. Optimising the return — not the next reward — is what separates RL from greedy one-step thinking. The formal target the agent maximises is the expected return, averaged over all the randomness in the environment.

The discount factor γ

If a task could run forever, the total reward might add up to infinity — and you can't compare two infinities. The fix is the discount factor γ, a number between 0 and 1. A reward arriving k steps from now is multiplied by γ to the power k, so distant rewards count for less. This makes the discounted return a finite, well-behaved number even over an endless horizon.

def discounted_return(rewards, gamma=0.99):
    G, scale = 0.0, 1.0
    for r in rewards:        # rewards in time order
        G += scale * r
        scale *= gamma       # each step counts a little less
    return G

# rewards = [1, 1, 1, 1]  with gamma=0.9
# -> 1 + 0.9 + 0.81 + 0.729 = 3.439

The discounted return: every step further into the future is multiplied by another factor of γ.

G_t = \sum_{k=0}^{\infty} \gamma^{k} R_{t+k+1}

The discounted return: every step further into the future is weighted by another factor of γ.

γ also encodes patience. Near 0 (say 0.1) the agent is impulsive, chasing immediate reward; near 1 (say 0.999) it's far-sighted, willing to sacrifice now for a bigger payoff later. The same discount idea shows up across machine learning whenever future quantities must be weighed against present ones.

Horizons: when does the story end?

How far into the future the return reaches is the horizon. The finite versus infinite horizon distinction is one of the first design choices you make:

Finite horizon — the task lasts a fixed number of steps, like a 200-move game. The agent should plan with the clock in mind.
Infinite horizon — the task could run indefinitely, like balancing a pole or managing a server. Here the discount factor is what keeps the return finite.

Many real tasks come in natural chunks called episodes. An episodic MDP restarts after each episode: a game ends, a robot reaches the goal or falls over, and the clock resets. Episodes give learning clean, comparable units — each one is a complete attempt from start to finish.

Absorbing states tie episodes off neatly

How does an episode actually end inside the math? Through an absorbing state — a terminal state the agent can enter but never leave, and which pays zero reward forever after. 'Game over' and 'goal reached' are absorbing states. Modelling them this way lets a single infinite-horizon formula describe episodic tasks too: the episode 'continues forever', but harmlessly, parked on the absorbing state collecting nothing.

P(s \mid s, a) = 1,\qquad R(s, a) = 0 \quad \forall a

An absorbing terminal state in symbols: every action keeps you in s and pays zero reward forever after.