JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Distillation, Merging & Choosing Your Recipe

Compress a big model into a small one, fold several fine-tunes into a single checkpoint, and pull the whole track together into a decision you can defend.

Distillation: teaching a small model from a big one

Distillation turns an expensive, capable "teacher" model into a cheap, fast "student" that mimics it. In the most common LLM version, you simply have the teacher generate high-quality responses to many prompts and fine-tune the student on those — the teacher becomes your data factory. Richer variants train the student to match the teacher's full probability distribution over next tokens, transferring more nuance than the single chosen word does.

\mathcal{L}_{\text{KD}} = (1-\alpha)\,\mathrm{CE}(y,\, p_s) \;+\; \alpha\,T^{2}\,\mathrm{KL}\!\left(p_t^{T}\,\big\|\,p_s^{T}\right),\qquad p^{T}=\mathrm{softmax}(z/T)

Distillation loss: the student matches the teacher's temperature-softened probabilities while still fitting the true labels.

# Distillation by imitation (the common LLM recipe)
for prompt in many_prompts:
    answer = teacher.generate(prompt)   # strong, expensive model
    dataset.add(prompt, answer)
student = sft(small_base_model, dataset)  # cheap to run afterward
The teacher writes the curriculum; the student learns to reproduce it cheaply.

Model merging: averaging weights, not retraining

Model merging is a surprising trick: take two or more models fine-tuned from the same base and combine their weights directly — often just a weighted average — to get a single model that inherits strengths from each, with no extra training at all. A model good at coding and one good at multilingual chat can sometimes merge into one that does both.

\theta_{\text{merge}} = \sum_{i} \lambda_i\,\theta_i,\qquad \sum_{i} \lambda_i = 1

Model merging in one line: a weighted average of the weights from fine-tunes that share a common base.

Choosing your recipe

You now have the whole toolkit. Picking among them is mostly about matching the method to the kind of gap you face — and stacking them in the right order when you need more than one.

  1. Format or style gap → supervised / instruction fine-tuning, almost always via PEFT.
  2. Missing durable domain knowledge → continued pretraining, then SFT to recover instruction-following.
  3. Missing fast-changing facts → don't fine-tune; use retrieval (RAG).
  4. Need it smaller / cheaper → distill a strong model into a small one.
  5. Have several good fine-tunes → try merging before you spend on retraining a combined one.

Evaluate, or you're guessing

Every method above can look like an improvement on a handful of hand-picked examples while quietly regressing somewhere else. Evaluating the fine-tune is non-negotiable: hold out a real test set the model never trained on, score the target task, and also score a general suite to catch forgetting. When responses are open-ended, an LLM-as-judge can grade them at scale — but spot-check its verdicts against human judgment.

Hold out untouched validation and test sets before fine-tuning — that split is what tells you the method actually worked.

Data divided into separate train, validation, and test partitions for honest held-out evaluation.

Hard-won advice

  1. Exhaust prompting and retrieval first. Fine-tuning is powerful but it's the most expensive lever to build and maintain.
  2. Data quality beats every clever method. A clean, consistent, well-curated dataset is the single biggest predictor of a good fine-tune.
  3. Default to PEFT and small ranks; scale up only when evaluation says you must.
  4. Always test against the original base model — sometimes the honest result is that you didn't need to fine-tune at all.