continued pretraining

Continued pretraining takes a model that has already been pretrained on a broad corpus and keeps training it with the same self-supervised objective on a new distribution — a specialized domain like code, biomedicine, or law; a new language; or much longer sequences than it originally saw. It sits between pretraining and fine-tuning: it still uses next-token prediction over large unlabeled text, but starts from learned weights rather than random ones, so it adapts the model's knowledge rather than installing a narrow skill.

The central difficulty is the stability–plasticity trade-off. Train too hard on the new data and the model suffers catastrophic forgetting, losing general competence it cost a fortune to acquire; train too gently and it never genuinely absorbs the new domain. The practical levers are a replay mixture (blending a fraction of the original distribution back in), a re-warmed but modest learning rate, and careful data curation. For context extension specifically, continued pretraining on long documents is paired with positional-encoding adjustments like RoPE base-frequency scaling so attention generalizes to lengths beyond the original window.

Continued pretraining is the economical path to a specialized or longer-context model — far cheaper than pretraining from scratch and far more capable than fine-tuning alone for deep domain shifts. Its honest limitation is that it cannot add a fundamentally new capability the base model lacks the scale to support; it redistributes and extends existing competence rather than creating it.