Grounding and Reliability: Retrieval-Augmented Fine-Tuning and Structured Output

Two reliability gaps

After SFT and alignment you have a model that is helpful and well-mannered — but still unreliable in two specific ways that block real products. First, it hallucinates: asked about facts outside its training, it answers confidently and wrongly. Second, it is unstructured: ask for JSON and it may wrap the JSON in prose, miss a field, or emit a trailing comma that crashes your parser. This guide closes both gaps — one with data, one with decoding.

Retrieval-augmented fine-tuning (RAFT)

Plain retrieval-augmented generation (RAG) bolts a retriever onto a frozen model at inference: fetch relevant passages, paste them into the prompt, and hope the model uses them. It often does not — a model never trained to read evidence may ignore the passages and fall back on its (possibly wrong) parametric memory, or get distracted by an irrelevant passage that the retriever returned by mistake.

Retrieval-augmented fine-tuning (RAFT) closes that gap by training the model to do RAG well. The key idea is to fine-tune on examples that contain both a golden document (which holds the answer) and deliberately-inserted distractor documents (which do not). The model learns to cite the relevant passage, quote it, and ignore the irrelevant ones — exactly the skill an inference-time retriever demands.

For each training question, build a context of one golden document plus several distractors, in random order.
Write target answers that cite and quote the golden passage with chain-of-thought, modeling the reasoning you want at inference.
Crucially, include some examples with no golden document — train the model to say 'the context does not contain the answer' instead of confabulating.
Pair RAFT with a good embedding retriever at inference; the model now reads what the retriever returns instead of fighting it.

Structured output: constrain the decoder

The second gap is format. You can ask nicely for JSON in the prompt and fine-tune for it, but prompting is a suggestion, not a guarantee — under distribution shift the model will eventually emit something unparseable. Structured output decoding makes invalid output impossible by changing how tokens are sampled rather than relying on the model to behave.

The mechanism rides on autoregressive decoding. At each step the model proposes a probability distribution over the whole vocabulary; constrained decoding intersects that with a grammar that encodes which tokens are legal right now given what has been emitted so far. Illegal tokens are masked to probability zero before sampling, so the output is guaranteed to conform — every brace closes, every required field appears.

p_{\text{mask}}(t \mid x) = \frac{p_\theta(t \mid x)\,\mathbb{1}\!\left[t \in \mathcal{A}(x)\right]}{\displaystyle\sum_{t' \in \mathcal{A}(x)} p_\theta(t' \mid x)}

Constrained decoding zeroes out every token the grammar forbids, then renormalizes over the allowed set A(x) — a guarantee, not a suggestion.

# Constrained decoding: mask illegal next-tokens to -inf
logits = model(prefix)                 # raw scores over vocab
allowed = grammar.allowed_tokens(prefix)   # set legal given JSON state
logits[~allowed] = -float('inf')       # forbid everything else
next_tok = sample(softmax(logits))     # guaranteed to keep JSON valid

A grammar (regex, JSON schema, or context-free grammar) decides the legal next tokens; the rest are masked out.

Modern engines compile a JSON schema or a grammar into a fast token mask, so the cost is negligible. The caveat worth knowing: an over-tight grammar can force the model down a path it dislikes, hurting answer quality — a model that wanted to say 'I don't know' but is grammar-forced into a JSON enum may pick a wrong-but-valid value. Constrain the structure, but leave the model freedom over the content wherever you can.

Putting it together: a tool-using assistant

These two techniques compound. A RAFT-trained model that reliably reads retrieved evidence, plus structured decoding that guarantees machine-readable output, is exactly the foundation for function-calling agents and agentic RAG: the model emits a valid tool call (structure guaranteed), the tool returns evidence, and the model reasons over that evidence without making things up (grounding learned). Reliability is not one trick — it is the disciplined stacking of data-side grounding and decode-side constraints.