JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Policies: How an Agent Decides What to Do

Deterministic or stochastic, greedy or exploring — the policy is the agent's behavior, and learning means improving it.

A policy is the agent's strategy

If the return is what the agent wants, the policy is how it acts to get it. A policy, written π (pi), is a rule that maps each state to an action — or to a probability over actions. Hand the agent a state, ask 'what now?', and the policy answers. Everything the agent does flows from π; learning, in the end, means improving the policy.

Deterministic policies: one state, one action

The simplest kind is a deterministic policy: each state maps to exactly one action, a = π(s). A thermostat ('if below 20°C, turn on the heat') is a deterministic policy. They are easy to reason about and are the natural output of value-based methods and of continuous-control actors like DDPG.

Their weakness: a purely deterministic agent that always picks the same action can never discover whether a different action was better. Without some variation it gets stuck — an instance of the exploration–exploitation tradeoff.

Stochastic policies: a distribution over actions

A stochastic policy outputs a probability for each action, π(a | s), and the agent samples from it. This is the same object as the stochastic policy in the MDP formalism. Randomness is not sloppiness — it is useful. It builds exploration directly into behavior, it is essential in games where being predictable is fatal (rock-paper-scissors), and it makes the objective smooth enough to optimize with gradients, which is why policy-gradient methods learn stochastic policies.

\pi(a\mid s)=\dfrac{\exp\!\big(Q(s,a)/\tau\big)}{\sum_{a'}\exp\!\big(Q(s,a')/\tau\big)}

A stochastic policy as a softmax over action values: every action gets a probability that sharpens toward the best one as the temperature τ falls.

Over the course of training, a stochastic policy typically starts broad (try many actions) and gradually sharpens toward the best ones as the agent gains confidence.

The greedy policy: act on what you believe

There is one especially important way to derive a policy from value estimates: the greedy policy picks, in every state, the action with the highest estimated value. If you already knew the value of each action, greedy behavior would be optimal. The catch is that early in learning your value estimates are wrong, so being purely greedy locks in mistakes — which is why we soften it with exploration (for example ε-greedy) until the estimates are trustworthy.

Interactive Q-learning on a gridworld: as the value estimates form, the greedy policy simply reads off the highest-value action in each state.

Interactive gridworld where Q-values are learned and arrows show the greedy action chosen in each cell.

From policy to behavior: rollouts

Run a policy in the environment and you generate a trajectory (a rollout): state, action, reward, next state, repeated until the episode ends. Trajectories are the raw experience every RL algorithm consumes — to estimate returns, to evaluate the policy, and to figure out which actions to make more or less likely.

One trip around the agent–environment loop — observe a state, take an action, receive a reward and the next state — is exactly what a rollout records.

Diagram of the reinforcement-learning loop: the agent sends an action to the environment, which returns a reward and the next state.

  1. Observe the current state s.
  2. Ask the policy for an action: deterministic a = π(s), or sample a ~ π(·|s).
  3. Send a to the environment; receive a reward and the next state.
  4. Repeat, recording (s, a, r) at each step — that record is the trajectory.
# A stochastic policy as a probability table, then one action
import random
def act(policy, state):
    actions, probs = zip(*policy[state].items())   # actions and probabilities
    return random.choices(actions, weights=probs)[0]   # sample one action
A stochastic policy maps a state to a distribution over actions; the agent samples to act.

Recap