Supervised Fine-Tuning: From Base Model to Assistant

Why a raw base model won't help you

A freshly pretrained model is a base model: it has only ever learned to continue text. Ask it "What is the capital of France?" and it might helpfully answer Paris — or it might continue with three more trivia questions, because that pattern was just as common in its training data. It predicts plausible continuations; it does not know it is supposed to assist.

Supervised fine-tuning (SFT) fixes this. We show the model thousands of examples of the format "here is a request, here is the ideal response," and train it to produce the response. It is supervised because every example carries a target answer we want it to imitate.

How SFT trains

Mechanically, SFT is the same next-token objective as pretraining — but on curated prompt-and-completion pairs. One crucial detail: the loss is usually computed only on the response tokens. We don't want the model wasting capacity learning to predict the user's words; we want it to get its own answer right, given the prompt.

\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t\,\in\,\text{completion}} \log p_\theta\!\left(y_t \mid y_{<t},\, x\right)

SFT keeps the next-token objective, but the loss is summed only over the completion tokens — the prompt is masked out.

{
  "messages": [
    {"role": "user", "content": "Summarize this email in one line: ..."},
    {"role": "assistant", "content": "Client approved the budget; kickoff is Monday."}
  ]
}
# Loss is masked to the assistant tokens only — the model
# learns to *produce* the answer, not to predict the question.

A single SFT example. The assistant text is the supervision signal.

Instruction tuning: teaching the meta-skill

If your SFT data covers one narrow task, you get a one-trick model. Instruction tuning is SFT done with a deliberately diverse mix — summarizing, translating, classifying, coding, answering, rewriting — each phrased as a natural-language instruction. The model doesn't just memorize those tasks; it learns the general habit of reading an instruction and doing what it says, which then generalizes to instructions it never saw in training.

Chat templates: the formatting contract

A chat model needs to know where the system instructions end and the user turn begins. That structure is encoded by a chat template — a fixed pattern of special tokens that wraps each role's text. You must fine-tune with the same template you'll use at inference time, or the model will see an unfamiliar format and behave erratically.

<|im_start|>system
You are a concise support assistant.<|im_end|>
<|im_start|>user
My order hasn't arrived.<|im_end|>
<|im_start|>assistant
I'm sorry to hear that. Could you share your order number?<|im_end|>

One common chat template. Special tokens mark roles and turn boundaries.

Curating the data — where quality is made

The dataset is the program. Curating an instruction dataset is mostly the unglamorous work of deduplicating, fixing inconsistent formatting, removing wrong or contradictory answers, and balancing task types. A few hundred clean, on-style examples routinely beat tens of thousands of noisy scraped ones.

When you can't collect enough by hand, synthetic instruction data — examples generated by a stronger model, then filtered — can fill the gaps cheaply. The catch is that synthetic data inherits the generator's blind spots and quirks, so always human-review a sample and keep a real held-out set for evaluation.

Decide the exact response style you want and write a short style guide — then make every example obey it.
Deduplicate and remove near-duplicates; repeated examples quietly overweight a behavior.
Balance task types so instruction tuning generalizes instead of overfitting one shape.
Hold out a real evaluation set before you generate or augment anything.