structured output decoding

If a downstream system expects strict JSON or a specific syntax, a model that is merely asked nicely to comply will occasionally produce a stray comma, a missing brace, or prose where data was wanted, and the parser breaks. Structured output decoding removes that risk by construction. Rather than hoping the model stays in format, it constrains generation so only tokens that keep the output valid are ever allowed — the very possibility of malformed output is engineered away.

The mechanism operates at decode time on the logits. At each step a validator derived from the target schema — a context-free grammar, a regular expression, or a JSON schema compiled into a finite-state machine — determines which next tokens could still lead to a well-formed string, and every other token's logit is masked to negative infinity before sampling. The model therefore samples freely among legal continuations only. Care is needed at token boundaries, where techniques like token healing reconcile the tokenizer's chunks with the grammar's character-level rules.

This guarantees syntactic validity and underpins reliable tool calling and function calling, where arguments must parse exactly. Two honest caveats: it constrains form, not truth — a schema-valid object can still be factually wrong — and forcing tokens the model finds unlikely can subtly distort its distribution, occasionally degrading content quality compared with a model that was simply trained to format itself well.