The Loop Up Close: State, Action, and Reward

Three signals flowing in a circle

Last time we sketched the interaction loop in broad strokes. Now let's name its three signals precisely, because the whole field is built on them. At each step the agent receives a state, sends back an action, and then receives a reward along with the next state. State in, action out, reward back — that triple repeats forever.

State vs observation: what the agent can actually see

It's tempting to think the agent sees "everything," but real agents almost never do. The state is the true, complete description of the situation; the observation is the possibly-partial slice of it that actually reaches the agent's senses. A chess engine sees the whole board — state and observation match. A robot with one camera sees only what's in front of it — it gets an observation, not the full state.

This gap matters enormously. When the observation hides part of the truth, the same observation can mean two different things, and an agent acting only on what it sees can be fooled. For this whole introductory track we'll mostly pretend observation equals state (the "fully observable" case), but keep the distinction in your back pocket — later tracks devote serious machinery to the partially observable world.

Actions and the action space

An action is the agent's lever on the world — the one thing it gets to choose each step. The full menu of choices available is the action space. That menu can be discrete (a small fixed set: up, down, left, right; or fold, call, raise) or continuous (any value in a range: steer the wheel to 12.7°, set the motor to 0.43 of full power).

The shape of the action space quietly decides which methods you can use, so it's worth identifying early. Counting discrete choices is easy; choosing well among infinitely many continuous ones needs different tools. For now, just get used to asking of any problem: *what exactly can the agent do, and how many options is that?*

Reward: the feedback that defines success

We met the reward signal last time; now place it precisely in the loop. After the agent acts and the world moves to its next state, the environment also emits a reward — a number scoring that transition. Crucially, the reward is part of the environment, not something the agent gets to set. The agent can only learn to earn it; it cannot simply declare itself successful.

Why it's a sequence, not a single shot

Each turn of the loop looks simple, but the deep difficulty is that the turns are linked. This is sequential decision-making: today's action changes the state you'll face tomorrow, which changes the actions available, which changes future rewards. A move that grabs a small reward now might steer you into a corner with no good options later. The agent isn't picking one answer — it's steering a path through a branching future.

state = env.reset()          # the starting situation
while not done:
    action = agent.choose(state)     # agent picks from the action space
    next_state, reward, done = env.step(action)  # world responds
    agent.learn(state, action, reward, next_state)
    state = next_state               # the loop carries forward

Read that loop a few times. Every algorithm in the rest of the domain — from simple tables to deep neural networks — is ultimately just a smarter way to fill in `agent.choose` and `agent.learn`.