From Base Model to Assistant: Supervised Fine-Tuning

Why a base model is not an assistant

A base model is trained on a single objective: predict the next token of internet-scale text. It is a formidable continuation engine — give it half a recipe and it finishes the recipe; give it half a flame war and it finishes the flame war. What it does not know is that you want it to answer your question rather than imitate the kind of text where your question typically appears. Post-training is the craft of reshaping that raw language-modeling ability into something that follows instructions, stays helpful, and refuses what it should.

The first move of post-training is supervised fine-tuning (SFT), sometimes called instruction tuning. You show the model thousands of demonstrations — an instruction paired with an ideal response — and continue training so it learns to produce responses of that shape. SFT does most of the heavy lifting: a well-done SFT pass alone already gives you a usable assistant. The alignment methods later in this track are mostly about polishing what SFT establishes.

What SFT actually optimizes

Mechanically, SFT is the same cross-entropy next-token loss used in pretraining — the model is still just predicting tokens. The two differences are the data and the loss mask. The data is curated instruction→response pairs instead of scraped text. The mask tells the loss to ignore the prompt tokens and score the model only on the response tokens: you do not want to reward the model for predicting the user's question, only for producing the answer.

\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t=1}^{T} m_t\,\log P_\theta\!\left(x_t \mid x_{<t}\right),\qquad m_t = \begin{cases} 1 & x_t \in \text{completion} \\ 0 & x_t \in \text{prompt} \end{cases}

SFT reuses pretraining's cross-entropy next-token loss, but a mask m_t zeroes the gradient on prompt tokens so only the completion is learned.

# SFT loss = next-token CE, but only on response tokens
logits = model(input_ids)            # prompt + response concatenated
loss   = cross_entropy(logits, labels)
# labels = [-100, -100, ..., r1, r2, ...]  # -100 masks prompt tokens

The prompt tokens are masked with -100 so they contribute zero gradient; only response tokens are scored.

Data is the whole game

The single biggest lever in SFT is data quality, not data quantity. A famous result is that a few thousand carefully curated, diverse, high-quality demonstrations can outperform hundreds of thousands of mediocre ones. Each example teaches the model not just an answer but a style: how long to write, how to structure reasoning, when to ask a clarifying question, how to refuse. Garbage in that style propagates.

Choose a chat template and apply it consistently — special tokens marking system/user/assistant turns. The model learns these boundaries; mismatched templates at inference cause garbage.
Curate for diversity across tasks, lengths, and refusal cases — not just easy Q&A. Coverage of the behaviors you want matters more than raw count.
Decontaminate against your evaluation sets, and dedupe near-identical examples so the model does not over-memorize a template phrasing.
Mask the prompt, pack sequences to a fixed length for efficiency, and keep a held-out set to watch validation loss for overfitting.

The cost problem that shapes everything

Naive SFT updates every weight in the model — full fine-tuning. For a 70-billion-parameter model that means holding the weights, their gradients, and the optimizer's two moment buffers in memory at once: easily 16 bytes per parameter, well over a terabyte of accelerator memory. That is out of reach for most teams, and you would need to store a full copy of the model for every task you adapt to.

This single constraint is why the next guide exists. Parameter-efficient fine-tuning (PEFT) freezes the giant base model and trains a tiny set of extra weights instead — cutting memory by an order of magnitude and shrinking each task's artifact from a terabyte to a few megabytes. Almost all real-world LLM adaptation today is PEFT, not full fine-tuning.

The post-training pipeline, mapped

It helps to see where this whole track is going. A modern post-training pipeline is roughly: SFT establishes instruction-following behavior; preference alignment then nudges the model toward responses humans prefer; optional grounding teaches it to use retrieved knowledge; and serving engineering makes the result fast and cheap. The rest of this track walks that ladder.

You will meet reward modeling and rejection-sampling fine-tuning as ways to push quality past what SFT alone reaches, and lightweight alignment recipes that skip reinforcement learning entirely. But none of it works without a solid SFT base — so get the demonstrations right first.