continued pretraining (domain)

Before you teach a model to follow instructions in a specialised field, it helps to let it simply read a lot of that field first. Continued pretraining does exactly that: you take a finished base model and keep training it with the same self-supervised next-token objective, but on a large pile of domain text — legal opinions, medical literature, a codebase, or a new language.

Unlike instruction fine-tuning, there are no curated question-and-answer pairs here, just raw domain documents and the original predict-the-next-token loss. This soaks up vocabulary, conventions, and facts that the original pretraining under-covered. It usually comes before instruction tuning in a domain pipeline: first broaden the model's knowledge of the field, then shape how it responds. Watch the learning rate and mix in some general data, or you invite catastrophic forgetting.