Gibbs sampling
/ GIBZ SAM-pling /
Gibbs sampling is a simple, popular flavor of Markov chain Monte Carlo for when you have many unknowns at once and updating them all together is hard, but updating one at a time — with the rest held fixed — is easy. The idea is like adjusting a room full of thermostats that affect each other: you can't set them all perfectly in one go, so you tweak one while pretending the others are settled, then move to the next, then the next, looping around. Each pass nudges the whole system toward balance.
Concretely, you cycle through your variables, and for each one you draw a fresh random value from its distribution conditioned on the current values of all the others. You never have to sample the whole tangled joint distribution directly — only these easier one-variable-at-a-time pieces. Round and round you go, and the sequence of full configurations you pass through becomes valid samples from the joint distribution you actually wanted, exactly as general MCMC promises.
Why it matters: Gibbs sampling is a default workhorse for fitting hierarchical Bayesian models, Markov random fields, and topic models, beloved because it needs no hand-tuned step size — every proposed move is automatically accepted. Its honest weakness shows when variables are strongly correlated: changing one at a time forces the sampler to shuffle in tiny diagonal-blocked steps, like trying to cross a room only by moving north-south or east-west. It then explores painfully slowly and can take an enormous number of iterations to cover the space, even though each individual step is cheap and easy.
Two unknowns that move together — say a person's true height and weight, which are correlated. Gibbs sampling fixes weight, draws a new height given it; fixes that height, draws a new weight given it; repeats. If height and weight are tightly linked, each move can only go straight up or straight sideways, so the sampler zigzags slowly along the diagonal cloud instead of striding across it.
Gibbs updates one variable at a time, so strongly correlated variables make it zigzag slowly instead of moving freely.
Gibbs sampling is appealing because it has no tuning knobs and accepts every move, but that convenience hides its Achilles' heel: when variables are highly correlated, one-at-a-time updates mix extremely slowly, and the chain may need vastly more iterations than it appears to.