stationary vs nonstationary MDP

An MDP is stationary when its rules hold still: the transition dynamics and reward function are the same on step one as on step one million. A nonstationary MDP changes underneath you, as prices drift, a game patches its mechanics, or machinery wears out, so a recipe that worked yesterday may quietly stop working tomorrow.

Most textbook theory assumes stationarity, because it lets a single fixed policy be optimal and lets the agent treat old and new experience as equally valid. Nonstationarity breaks both: the agent must keep adapting, weight recent data more heavily, or fold the changing quantity into the state so the larger process becomes stationary again. Note that a finite-horizon problem's optimal policy is itself nonstationary even when the dynamics are fixed, because there the changing thing is the time remaining.