the Gibbs sampler
/ GIBZ /
Suppose your unknown is not one number but several at once — say a mean and a variance, or thousands of parameters in a model. Sampling the whole joint distribution in one go is hard, but often you can sample EACH coordinate easily if you pretend the others are known. The Gibbs sampler exploits exactly this: it cycles through the variables one at a time, redrawing each from its full conditional distribution given the current values of all the rest.
Concretely, to sample a joint distribution of (X, Y, Z), you repeat a sweep: draw a new X from P(X given Y, Z) using the latest Y and Z; then draw a new Y from P(Y given X, Z) using the just-updated X; then draw a new Z from P(Z given X, Y). One full sweep updates every coordinate; many sweeps produce a sequence whose distribution converges to the joint target. It is a special case of Metropolis-Hastings in which the proposal is the full conditional itself — and that choice makes the acceptance probability exactly 1, so every proposed move is accepted. The reason it works is that the full conditionals are mutually consistent with the joint, so steering by them drives the chain to the right stationary distribution.
Gibbs is beloved because each step samples from a one-dimensional (or low-dimensional) conditional, which is frequently a standard distribution you can draw from directly — no tuning, no rejected moves. It pairs perfectly with conjugate priors, where the conditionals come out in closed form. The honest caveats: like all MCMC the draws are correlated and require burn-in and convergence checks, and Gibbs mixes badly when the variables are strongly correlated — it can only step along coordinate axes, so it crawls slowly up a narrow diagonal ridge. And it needs every full conditional to be available to sample from; if even one is intractable, Gibbs alone does not apply.
Fit a normal model with both mean mu and precision tau unknown. Given tau, the conditional for mu is a normal you can sample directly; given mu, the conditional for tau is a gamma you can sample directly. So you alternate: draw mu given the current tau, draw tau given the new mu, repeat. The chain of (mu, tau) pairs converges to their joint posterior with no rejections at all.
Cycle through variables, redrawing each from its full conditional; every move is accepted.
Gibbs only steps along coordinate axes, so it mixes painfully slowly when the variables are strongly correlated; reparameterizing to reduce correlation often helps more than running it longer.