From Rewards to Returns: What the Agent Actually Maximizes

A reward now, or more reward later?

Reinforcement learning starts from one signal: the reward, a number the environment hands the agent at each step. But a good agent does not chase the next reward — it chases the sum of rewards stretching into the future. This is the heart of sequential decision making: a move that looks great right now (grab the cookie) can be terrible later (skip dinner). To choose well, the agent needs one quantity that captures 'how good is this whole future, not just this step?'

The return: adding up the future

The return is simply the total reward collected from now until the end. If after this moment the agent receives rewards r1, r2, r3, …, the return Gt is their sum. In an episodic task (a game that ends, a maze that gets solved) this sum is finite and easy to picture. The agent's job, restated cleanly, is: act so that the return tends to be large.

This reframing is powerful. The reward hypothesis claims that any goal we care about — winning, surviving, being helpful — can be expressed as the maximization of expected cumulative reward. Returns are how that 'cumulative' becomes a concrete number.

Discounting: why far-off rewards count less

For tasks that never end (a continuing task like balancing a pole forever) an infinite sum of rewards can blow up. The fix is the discount factor γ (gamma), a number between 0 and 1. We weight a reward k steps in the future by γ to the power k, so the discounted return is r1 + γ·r2 + γ²·r3 + ….

The discounted return: every future reward summed, each weighted by γ raised to how far ahead it lands.

γ has a clean intuition. With γ near 0 the agent is myopic — it cares almost only about the immediate reward. With γ near 1 it is far-sighted, valuing distant rewards almost as much as present ones. Discounting also encodes uncertainty: a reward we might collect far away is worth less because the world may change before we get there.

The objective: expected return

There is one more subtlety. The world is usually random — transitions and even rewards can be stochastic — so the return itself is a random quantity. We cannot maximize a coin flip; we maximize its average. The expected return objective is therefore the true target of RL: choose behavior that makes the expected (average) discounted return as large as possible.

This single sentence — 'maximize expected discounted return' — is the formal goal behind every RL algorithm, from tabular methods to deep RL. Everything else is machinery for estimating and improving it.

Decomposing a return: reward-to-go

Returns have a beautiful recursive shape that we will lean on constantly. The return decomposition says: the return at time t equals the reward right now plus the discounted return from the next step onward — Gt = r(t+1) + γ·G(t+1). The future folds neatly into 'one step' plus 'the rest.'

The return's recursive shape: the return now equals the immediate reward plus the discounted return from the next step.

That 'the rest' has a name: the reward-to-go. When we ask 'how much credit does an action at time t deserve?', only the rewards that come after it matter — the past is sunk. Focusing on reward-to-go instead of the full episode return is a key idea that later sharpens credit assignment and cuts noise in policy-gradient methods.

1. Write the future reward stream after a decision: r(t+1), r(t+2), …
2. Discount each by how far ahead it is: γ⁰, γ¹, γ²…
3. Sum them to get the return Gt — the number the agent wants large.
4. Notice Gt = r(t+1) + γ·G(t+1): this recursion is the seed of every value function.

# Discounted return from a list of future rewards
def discounted_return(rewards, gamma=0.99):
    G = 0.0
    for r in reversed(rewards):    # walk backward
        G = r + gamma * G          # G_t = r + gamma * G_{t+1}
    return G

Recap