pretraining

Pretraining is the long, expensive first stage where a model reads an enormous amount of raw data and learns the general shape of it — before anyone tells it what job it will eventually do. For a language model, this usually means a deceptively simple game played trillions of times: cover the next word in a sentence and make the model guess it. "The cat sat on the ___." Guess wrong, nudge the model; guess right, leave it be. Repeat across a huge slice of the internet.

What makes this powerful is that the data labels itself — every sentence already "knows" what its next word is, so no humans need to hand-mark examples. This is called self-supervised learning. By grinding through that prediction game on a vast scale, the model is forced to absorb grammar, facts, styles, and threads of reasoning, simply because all of those help it predict the next word a little better. The result is a foundation model brimming with general competence but not yet pointed at any task.

It is worth being honest about what pretraining is and isn't. It teaches the model what is statistically likely to come next, not what is true or kind or helpful. A freshly pretrained model will happily continue a toxic rant or invent a fake citation, because both are patterns it has seen. The polish that makes a chatbot useful and safe comes afterward, from fine-tuning and human feedback. Pretraining builds raw capability; it does not build judgment.