Markov reward process (MRP)

A Markov reward process is an MDP with the steering wheel removed. There are still states, the world still moves from one to the next by chance, and rewards still arrive along the way, but there are no actions and nothing to decide. It describes a system you can only watch unfold, like a board-game piece that drifts on its own and pays out as it goes.

Stripping away choice makes the MRP a clean place to study value before tackling control. Each state has a value equal to the expected discounted reward earned from it onward, and that value obeys a linear Bellman equation that, in the finite case, you can solve directly with linear algebra. An MDP is exactly an MRP with a layer of decisions added on top; fixing a policy in an MDP collapses it into an MRP, which is why policy evaluation is built on this idea.