JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Policies, Trajectories, and Rollouts

A policy is the agent's whole behaviour boiled down to a rule. Run it through an MDP and you get experience — the raw material every learning algorithm feeds on.

What a policy is

A policy is the agent's strategy: a rule that maps each state to an action (or to a distribution over actions). It's the answer to the MDP — once you have a good policy, the agent simply follows it. Everything the agent has learned is compressed into this one object. In broader ML the same word, policy, means exactly this decision rule.

A policy lives inside the agent: it reads the state and chooses the action that drives this loop.

Diagram of the reinforcement-learning loop: the agent takes an action, and the environment returns a reward and the next state.

Policies come in two flavours. A deterministic policy always picks the same action in a given state — clean and easy to read. A stochastic policy returns probabilities over the action space, so the same state can lead to different choices on different visits.

Why on earth would a policy be random?

It sounds odd to deliberately act randomly, but a stochastic policy earns its keep:

  1. Exploration. Trying varied actions is the only way to discover whether an untested choice is secretly better — the engine behind learning at all.
  2. Games of bluff. In rock-paper-scissors, any predictable rule loses. The optimal play is genuinely random.
  3. Hidden state. When two situations look identical but call for different actions, mixing them probabilistically is the best you can do.

Trajectories: the trail an agent leaves

Let a policy loose in an MDP and it produces a trajectory: the sequence of states, actions, and rewards it actually experiences — s₀, a₀, r₁, s₁, a₁, r₂, and so on. A trajectory is one story of what happened: the agent's lived experience for a single run.

Generating a trajectory by stepping the policy and environment forward is called a rollout. Rollouts are how learning algorithms get their data: you can't read the reward function directly, so you sample the world by acting in it and watching what comes back.

Run a policy through this gridworld to watch a rollout unfold — each path is one trajectory of states, actions and rewards.

Interactive Q-learning gridworld in which an agent moves cell to cell, tracing trajectories and collecting rewards.

def rollout(env, policy, max_steps=1000):
    s = env.reset()
    trajectory = []
    for _ in range(max_steps):
        a = policy.sample(s)          # stochastic: draw an action
        s_next, r, done = env.step(a) # the MDP's transition + reward
        trajectory.append((s, a, r))
        s = s_next
        if done:                      # hit an absorbing state
            break
    return trajectory
A rollout: alternate 'policy picks an action' and 'environment returns a reward and next state' until the episode ends.

From trajectories back to the objective

Each trajectory has a return — the discounted sum of its rewards. Because a stochastic policy and a random environment make every rollout a little different, the same policy yields a spread of returns. The agent's true goal is the expected return: the average return over all the trajectories the policy could generate.

J(\pi)=\mathbb{E}_{\tau\sim\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\,r_{t+1}\right],\qquad \pi^{\star}=\arg\max_{\pi}\,J(\pi)

The objective in one line: find the policy whose expected discounted return is largest.

That single line is the north star of the entire field: *find the policy whose expected return is largest.* Every algorithm you'll meet later — value-based, policy-gradient, model-based — is a different route to that one summit. They all begin by gathering trajectories.