deterministic vs stochastic environment

In a deterministic environment the same action in the same state always leads to the same outcome, like a puzzle where pressing a button always opens the same door. In a stochastic environment the same choice can land you in different places or pay different amounts, because dice, noise, or other unpredictable forces are at play. A robot on dry ground is roughly deterministic; the same robot on ice is decidedly stochastic.

Stochasticity is why RL leans so hard on expectations. The agent cannot judge an action by a single outcome, which might have been lucky or unlucky; it must reason about the average over all the outcomes that action could produce. This is also why a good policy is sometimes randomised, why we average over many rollouts to estimate value, and why exploration and patience matter so much more than in a clockwork world.