guided decoding with grammars

When you need a model's output to be valid JSON, to match a regular expression, or to conform to a formal grammar, hoping a prompt will keep it in line is unreliable — one stray token breaks a parser. Guided decoding makes invalid output impossible by construction. At each generation step it computes which tokens could legally extend the partial output under the target schema and masks every other token to probability zero before sampling, so the model can only ever choose a continuation the grammar permits.

Mechanically the schema — a regex, a context-free grammar, or a JSON schema — is compiled into a finite-state machine or pushdown automaton whose state advances as tokens are emitted. The hard part is that the legal symbols live in grammar space while the model samples in token space, and a single token may span several grammar symbols or split one, so the constraint must be projected onto the vocabulary. Modern engines precompute these token masks so the per-step overhead is near zero, making structured generation essentially free at inference time.

Constraining the surface form does not constrain meaning: the output is guaranteed parseable, but the values inside can still be wrong or hallucinated.