pretraining objective (LLM)

An LLM learns by playing one enormous game of guess-the-next-word. Show it a sentence with the continuation hidden, let it guess, and nudge it whenever the guess is wrong. Repeat this across trillions of words and the model slowly absorbs grammar, facts, and patterns of reasoning — never handed a single rule, only pushed again and again toward predictions that match real text.

Concretely, the objective is to maximise the probability the model assigns to each true next token given everything before it — equivalently, to minimise the average cross-entropy (negative log-likelihood) of those tokens. Every weight is adjusted by gradient descent so the correct continuation becomes a little more likely. This one simple loss, applied at web scale, is what turns random weights into a capable language model.

The beauty is that no human labels are needed: the text supervises itself. The catch is that the objective only rewards reproducing text as written — not being truthful, safe, or helpful. Those qualities are added afterwards by fine-tuning and alignment.