JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Domain Adaptation & Continued Pretraining

When the gap isn't behavior but a whole vocabulary — medicine, law, code, a new language — and how to teach it without erasing what the model knew.

The domain gap

Sometimes a model follows instructions perfectly but still flounders, because it has barely seen your domain. Clinical shorthand, statutory citations, a proprietary codebase, or a low-resource language can all be underrepresented in the pretraining mix. No amount of prompt polishing teaches vocabulary the model never absorbed. This is the realm of domain adaptation.

Before adapting weights, ask whether retrieval-augmented generation could just supply the missing domain facts at request time.

Pipeline: a query retrieves documents that are added to the prompt before the model generates an answer.

Continued pretraining

The tool for closing a knowledge gap is continued pretraining (also called domain-adaptive pretraining): you keep going with the same self-supervised next-token objective the base model was built with, but now on a large corpus of your domain's raw text. There are no labels here — just lots of in-domain documents the model learns to predict. This is transfer learning at work: the general language skill carries over, and the domain settles on top.

\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log P_\theta\!\left(x_t \mid x_{<t}\right)

Continued pretraining keeps optimizing the same self-supervised objective: predict each next token, minimizing the negative log-likelihood over your domain corpus.

A common high-end recipe is two stages: continued pretraining to install the domain, then supervised fine-tuning to restore instruction-following on top of it. Pretraining teaches what the world looks like in your domain; SFT teaches how to behave within it.

Catastrophic forgetting — the central danger

Push a model hard on narrow data and it can suffer catastrophic forgetting: as the weights move to fit your domain, they drift away from the general abilities that made the base model worth using. You end up with a model fluent in radiology reports that can no longer hold a normal conversation or do basic arithmetic. This is the failure mode that makes domain adaptation tricky.

  1. Mix in replay data — blend general-domain text back into the corpus so old skills keep getting reinforced. Tuning this mixture is a data-mixture-weighting decision.
  2. Use a low learning rate and few epochs — gentle nudges preserve more of the original model than aggressive runs.
  3. Prefer PEFT when you can — adapters confine most changes to a small add-on, leaving the frozen base intact and far less prone to forgetting.

Adaptation vs the knowledge cutoff

Domain adaptation is also one way to move a model past its knowledge cutoff for a specific area — continued pretraining on recent in-domain text bakes newer facts into the weights. But baked-in facts go stale the moment the world changes again, and retraining is slow. For information that updates frequently, retrieval is usually the better answer; reserve adaptation for durable domain knowledge and idiom.

A domain-adaptation recipe

  1. Confirm the gap is really domain knowledge, not prompt or format — and that retrieval can't cover it more cheaply.
  2. Gather a large, clean corpus of in-domain text and a slice of general text for replay.
  3. Continued-pretrain at a low learning rate for a few epochs, watching general benchmarks for signs of forgetting.
  4. Follow with supervised fine-tuning to restore instruction-following and your target response style.
  5. Evaluate on both a domain test set and a general one, so an in-domain win can't hide a general loss.