Greedy, Sampling, and Temperature

Two philosophies of choosing

Once you have a probability distribution, there are two broad ways to choose a token. The first is to exploit: always take the most likely one — that is greedy decoding. The second is to sample: treat the probabilities as a weighted lottery and draw randomly, so a token with 30% probability is chosen roughly 30% of the time. The tension between them is the heart of sampling versus greedy.

Why pure greedy gets dull

Greedy feels safe, but on open-ended writing it has a famous failure mode: it falls into loops and bland, repetitive phrasing — the model keeps choosing the same comfortable continuation and gets stuck saying “and then they went and then they went…”. This is degeneration, and it is a big reason creative tasks add randomness rather than chasing the single most-probable sentence.

Temperature: the diversity dial

The main knob on sampling is temperature (T). Before the softmax, every logit is divided by T. A low temperature (e.g. 0.2) sharpens the distribution — the top token towers over the rest, so behaviour edges back toward greedy. A high temperature (e.g. 1.3) flattens it — long-shot tokens get a real chance, giving surprising, sometimes incoherent text. At T → 0 sampling collapses into greedy; many APIs treat `temperature=0` as exactly that.

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Temperature T divides every logit before the softmax: low T sharpens the distribution, high T flattens it.

Beam search: hunting the likeliest sentence

Greedy picks the best token at each step, but the best token now can lead to a worse sentence later. Beam search tries to fix that by keeping the top k partial sequences (the beams) alive at once, expanding all of them, and finally returning the highest-probability whole sequence. It shines on tasks with a crisp target — machine translation, grammar correction, short factual answers.

But beam search is a poor fit for open-ended generation. The most probable long text tends to be the most generic one, so beams collapse into safe, repetitive prose — the same degeneration trap, now baked into the search. That is why chat and creative endpoints almost always use sampling instead.

Nudging individual tokens with logit bias

Sometimes you do not want to change the whole strategy, just shift a few specific tokens. Logit bias lets you add a fixed number to chosen tokens' logits before softmax. A large positive bias makes a token far more likely; a large negative bias (toward −100) effectively bans it — useful for forbidding a profanity, blocking a stray quotation mark, or forcing a yes/no token set.

p_i = \frac{\exp(z_i + b_i)}{\sum_j \exp(z_j + b_j)}

Logit bias adds a fixed number to chosen tokens' logits before the softmax, nudging their probability up or down.

# Pseudocode: bias applied to raw logits, before softmax
logits[token_id(" yes")] += 5.0     # strongly encourage
logits[token_id(" maybe")] -= 100   # effectively ban
probs = softmax(logits / temperature)
next_token = sample(probs)

Logit bias and temperature both reshape the logits before a token is drawn.

So far we have one global setting (greedy vs sampling), one dial (temperature), one search method (beam), and one surgical tool (logit bias). Next we add the safety net that makes sampling reliable: trimming the distribution before we draw.