JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Pretraining: Teaching a Model to Predict the Next Token

Before a model can chat, it spends months doing one thing: guessing the next word. Meet the self-supervised objective that quietly builds everything else.

One task to rule them all

When you talk to a large language model, it feels like it understands questions, writes code, and explains physics. But none of those skills were taught directly. They are side effects of a single, almost embarrassingly simple game played trillions of times: given some text, predict what comes next. This game is the pretraining objective — see the pretraining objective — and almost everything an LLM can do is downstream of getting very, very good at it.

Concretely, the model reads a chunk of text one token at a time and, at every position, outputs a probability for every possible next token. If the next real token was "ocean" but the model gave it only a 2% chance, it was surprised — and surprise is exactly what we train it to reduce. This is just language modeling taken to enormous scale.

The model first splits text into tokens, maps them to ids, then to embedding vectors — the raw input it predicts the next token from.

Diagram: text split into tokens, converted to token ids, then to embedding vectors.

Why it needs no labels: self-supervision

Most machine learning needs humans to label data: this photo is a cat, that review is positive. Pretraining sidesteps that entirely. The text is its own answer key. To make a training example, you take a sentence, hide the next token, and ask the model to predict it — the hidden token is the label, supplied for free by the text itself. This trick is called self-supervised pretraining, and it is the reason LLMs can learn from the entire internet without an army of annotators.

Because every position in every sentence becomes a free training example, a single document of 1,000 tokens yields roughly 1,000 prediction problems. Multiply that across a corpus of trillions of tokens and you have an astronomical supply of supervision — all without anyone hand-labeling a thing.

Causal language modeling: only look left

There is one rule that makes the game honest: when predicting the token at position i, the model may only look at tokens before it, never after. Otherwise it would be peeking at the answer. This left-to-right setup is causal language modeling (also called autoregressive modeling), and it is enforced inside the network by causal masking — a mask that blocks attention from reaching into the future.

This is exactly the same machinery as next-token prediction at inference time. The beautiful consequence: the model trains and generates with identical logic. During pretraining it predicts the next token to learn; later, when you chat with it, it predicts the next token to write. Nothing changes but the source of the prefix.

Autoregressive generation: each predicted token is fed back in to predict the next — the same left-to-right logic used during pretraining.

Diagram of an autoregressive next-token generation loop feeding each output back as input.

context = ["The", "capital", "of", "France", "is"]
# model only sees the words to the left, then scores every token
P(next = "Paris")  -> 0.71
P(next = "Lyon")   -> 0.05
P(next = "banana") -> 0.0001
# the real next token was "Paris"; loss rewards the 0.71
At each position the model scores the whole vocabulary; the true next token is the label.

Measuring surprise: cross-entropy loss

How do we score a prediction? With cross-entropy loss. For each position, the loss is the negative logarithm of the probability the model assigned to the correct next token. Give the right token a high probability and the loss is small; give it a tiny probability and the loss is large. Averaged over a batch, this single number is the entire signal that drives learning.

\mathcal{L} = -\log p_\theta\!\left(x_t \mid x_{<t}\right)

Cross-entropy loss at each position: the negative log-probability the model assigned to the true next token.

Cross-entropy has a friendly cousin you will see everywhere: perplexity, which is just the exponential of the loss. A perplexity of 10 means the model is, on average, as unsure as if it were choosing uniformly among 10 options. Watching loss (and perplexity) fall steadily is how a team knows a multi-million-dollar run is healthy.

Why such a simple game works

Here is the deep idea. To predict the next token well, across cooking blogs, legal contracts, and Python tracebacks, a model is quietly forced to learn grammar, facts, arithmetic, code structure, and even a rough model of how people reason. You cannot reliably finish "The boiling point of water at sea level is" without storing a fact. Prediction is a Trojan horse: aim at it hard enough, at enough scale, and understanding sneaks in.

This is also why people argue about LLMs. A skeptic calls a next-token predictor a stochastic parrot that merely remixes its data; an optimist sees genuine, if alien, competence emerging from scale. Both are looking at the same loss curve. The rest of this track follows that curve in detail: the data the game is played on, the budget of tokens and compute, the optimizer that nudges the weights, and the distributed systems that make a months-long run possible.