When free text is a bug
Chatting with a person, you want prose. But when an LLM sits inside a program — feeding a database, calling an API, filling a form — free-form text is a liability. One stray sentence of preamble (“Sure! Here is the JSON you asked for:”) and your parser crashes. The cure is constrained decoding: instead of trusting the model to behave, you reshape the choose step so only output that fits your format can ever be produced.
Stop sequences, revisited
The simplest constraint is on where to stop. Stop sequences tell the decoder to halt the moment it produces a given string. In a few-shot setup where examples are separated by `"###"`, setting that as a stop sequence keeps the model from happily inventing a next fake example. Stop sequences trim the tail of an answer; they do not police what comes before — that is the next job.
Logit bias as a blunt constraint
For tiny, fixed output spaces, logit bias alone can do the job. If you only ever want one of five category labels back, ban every other token by pushing its logit far negative; the model is then physically unable to say anything else. It is crude and does not scale to rich structure, but for a one-token classifier it is bulletproof.
Logit bias adds a fixed term to each token's logit before the softmax; pushing it toward negative infinity bans the token outright.
Grammars: validity by construction
The real workhorse is grammar-based constrained decoding. You supply a formal grammar — a context-free grammar or a regular expression — describing every legal output. At each step a mask zeroes out every token that would break the grammar, so the model samples only among tokens that keep the output valid. Done this way, structured output is correct by construction: the result is guaranteed to parse, because no path through the grammar can produce anything that does not.
Grammar-based decoding masks every token the grammar forbids — driving its logit to negative infinity — so each step can only produce valid structure.
JSON mode and function calling
Most providers package this as JSON mode. JSON-mode decoding applies a built-in JSON grammar (often plus your schema) so the reply is always parseable JSON, no preamble. The same machinery underlies function calling: the model emits a structured call — a tool name and a JSON argument object that matches the tool's signature — which your code can run directly. Without constrained decoding, both features would be a flaky game of please format it right; with it, they become dependable.
# At each decode step, mask tokens that would break the JSON schema logits = model.next_logits(context) allowed = grammar.allowed_tokens(state) # valid next tokens only logits[~allowed] = -inf # mask everything else next_token = sample(softmax(logits / temperature)) state = grammar.advance(state, next_token)
Confidence and streaming under constraints
Even with valid structure you often want to know how sure the model was. The logprobs of the chosen tokens give a usable confidence signal — a low-probability field value is a flag to double-check or ask a human. And because streaming still works under constraints, you can render a structured object field-by-field as it forms, then validate the instant the closing brace arrives.