multi-armed bandit
/ MUL-tee-armd BAN-dit /
Imagine a row of slot machines, each with its own unknown payout rate. You have a fixed number of pulls and want to win as much as possible. Every pull is a dilemma: stick with the machine that's paid out best so far, or try a different one that might secretly be better? That, stripped to its bare essentials, is the multi-armed bandit — the simplest possible reinforcement-learning problem, with the explore-vs-exploit tension and nothing else. ('One-armed bandit' is old slang for a slot machine.)
What makes the bandit so clean is what it leaves out: there are no states. The world doesn't change based on what you do — pulling a lever never moves you to a new situation, it just pays out. This is reinforcement learning with the 'sequence of situations' stripped away, leaving only the pure question of how to allocate trials between known-good and possibly-better options. Because it's so simple, it can be analyzed rigorously, with strategies that come with mathematical guarantees.
Bandits are far from a toy. They are the everyday engine behind A/B testing, online ad selection, news-article recommendation, and clinical-trial design — anywhere you must choose repeatedly among options with uncertain payoffs and learn as you go. The crucial limitation to remember: a bandit assumes your choices don't change the world's future state. The moment an action shifts what situation you face next (most real games, robotics, dialogue), you've left bandit territory and need the full machinery of an MDP.
A website tests three headlines for the same article. Each visitor is shown one; clicks are the payoff. A bandit algorithm gradually sends more visitors to the headline that's clicking better — but keeps showing the others a little, in case an early lead was just luck. It learns and earns at the same time, unlike a fixed A/B test.
The purest explore-vs-exploit problem: choose repeatedly among uncertain payoffs, with no changing state.
The bandit is full reinforcement learning minus the notion of state: your actions don't change what you'll face next. That single simplification is why bandits have clean optimal strategies, and also why the lessons don't transfer wholesale — most real-world problems do have changing states, which is exactly the hard part bandits sidestep.