JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The Markov Property and Designing States

'The future depends only on now' is a gift — but only if your state really captures 'now'. Here's how to make it true.

What the Markov property actually claims

The Markov property says one clean thing: the probability of the next state depends only on the current state and action — not on anything that happened before. Knowing the whole history gives you no advantage over knowing just the present. The present is a sufficient summary of the past.

P\!\left(S_{t+1}=s' \mid S_t=s,\, A_t=a\right) = P\!\left(S_{t+1}=s' \mid S_t=s,\, A_t=a,\, S_{t-1},\, A_{t-1},\, \ldots,\, S_0\right)

The Markov property in one line: the next-state probability conditions only on the current state and action, not on the full history.

Why care? Because it shrinks the bookkeeping from 'remember everything' to 'remember the current state'. Every value and every transition can be defined on states alone, and the whole MDP formalism rests on this single load-bearing assumption.

When it quietly breaks

The property is not a law of nature — it's a property of how you defined the state. It breaks whenever the state leaves out something that matters for the future. A few classic traps:

  1. Velocity is missing. A photo of a ball tells you where it is, not where it's going. The next frame depends on motion the single image hides.
  2. Hidden mode. A thermostat reading 21°C behaves differently if the heater is secretly on vs off. The mode matters but isn't in the reading.
  3. Partial view. A camera sees one corner of a room. What's behind the agent still affects outcomes — this is the realm of partial observability, covered in guide 5.
The agent only ever sees observations, not the hidden state — a Kalman-style tracker infers the missing piece (like velocity) the Markov property needs.

Interactive Kalman filter estimating a hidden moving state from noisy partial observations.

Designing a state that earns the property

The practical fix is to enrich the state until it captures enough of the past. The most common move is to fold recent history into the state itself.

  1. Stack frames: instead of one image, use the last four. Now position and velocity are visible.
  2. Add hidden variables you can infer: include 'heater on/off' in the state vector.
  3. Summarise history into features: a running average, a counter, a time-since-last-event.

Each addition grows the state space, which costs memory and data — so the craft is to add just enough to restore the Markov property without exploding the space. Designing the state is one of the highest-leverage choices in any RL project, and it sits entirely with you, not the algorithm.

Two relatives: random environments and reward processes

The Markov property does not require the world to be predictable. A deterministic environment always sends the same action to the same outcome; a stochastic one rolls dice. Both can be Markov — the property is about what the next-state distribution depends on (the current state), not about whether that distribution is a spike or a spread.

If you strip the actions out of an MDP — letting states evolve on their own while still collecting rewards — you get a Markov reward process. It's an MDP with no choices, useful as a stepping stone: you can study how rewards accumulate before adding the complication of deciding. Strip the rewards too, and you're left with a plain Markov chain (a close cousin of the hidden Markov model you may have met elsewhere).

A Markov reward process is just a Markov chain whose states evolve on their own while collecting rewards — no actions to choose.

Interactive Markov chain: nodes are states linked by arrows labelled with transition probabilities.