best-of-n sampling

When something really matters, you write a few drafts and send the best one. Best-of-n sampling applies that habit to a language model: for a given prompt, generate n different answers, score each with a reward model, and return the single highest-scoring one. The model's weights are never changed; you simply spend extra computation at answer-time to filter for quality.

Concretely, with n set to, say, 16, the model samples 16 candidate replies, the reward model rates all 16, and the user sees only the winner; the other 15 are discarded. This is sometimes called rejection sampling. It is appealing because it is dead simple, needs no fragile RL training, and is easy to dial: bigger n usually means better answers but proportionally more compute. It also serves as a clean baseline for judging whether full RL fine-tuning was worth it, and the winning samples can be collected to fine-tune a model so it produces such answers on the first try (a form of distillation).

Best-of-n is a useful and controllable way to lift quality, but it does not escape the core RLHF pitfall. Because it selects whatever the reward model scores highest, a larger n gives the proxy more chances to surface its own blind spots, so pushing n very high drifts into reward over-optimization just as RL training does, picking answers that game the reward model rather than genuinely better ones. In analyses it is often measured against the same KL-divergence budget as RL fine-tuning, making the two directly comparable as ways to trade compute for alignment.