self-supervised pretraining (LLM)

Self-supervised pretraining means the data labels itself. Ordinary supervised learning needs people to tag each example — this is a cat, this review is positive — which is slow and expensive. Here the trick is simpler: take raw text and hide part of it, then ask the model to recover the hidden part. The answer is already sitting in the text, so no human annotation is required at all.

For an LLM the hidden part is always the next token, and the visible part is everything before it. Each position in a document quietly becomes a training example: context as the input, the following word as the correct answer. A single web page yields thousands of such pairs for free. That is what makes it possible to train on trillions of tokens — there simply aren't enough human labels in the world to do otherwise.

Because the model is only ever predicting text, it learns whatever regularities help: spelling, syntax, world facts, even rough reasoning. But it also faithfully soaks up the biases and errors of its data, since nothing tells it which patterns are good.