JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The One Big Idea: Guess the Next Word

Everything an LLM does grows from a single trick — predicting what text comes next. Here's why that trick is so much more powerful than it sounds.

Language modeling in plain words

Strip away the marketing and an LLM is doing one mathematical task, called language modeling: given a stretch of text, estimate how likely each possible next word is. "The sky is ___" makes "blue" very likely, "falling" somewhat likely, and "refrigerator" almost impossible. A model that can score those possibilities well, for any text, is a language model.

This idea is old. Early language models were simple n-gram counters that just tallied which words tend to follow which. They worked for short phrases but fell apart on anything long or subtle. What changed with LLMs is the quality of the prediction: with enough scale, predicting the next word well requires the model to track meaning, grammar, facts, tone, and context — not just surface word pairs.

P(w_1,\ldots,w_n)=\prod_{t=1}^{n}P\!\left(w_t \mid w_1,\ldots,w_{t-1}\right)

Language modeling factorizes a whole text into a chain of next-word probabilities.

Next-token prediction, step by step

When an LLM writes, it doesn't compose a whole answer at once. It performs next-token prediction over and over: it looks at all the text so far, predicts the single most fitting next chunk, adds it on, and then looks again — now including the chunk it just wrote. (A "token" is roughly a word or word-piece; we'll meet tokens properly later. For now, read it as "the next bit of text.")

  1. Read the prompt and everything generated so far.
  2. Score every candidate next token by how well it fits.
  3. Pick one (usually a high-scoring one) and append it to the text.
  4. Repeat from step 1 until the answer is complete.
Autoregressive generation: each new token is predicted, appended, and fed back in — the loop you watch when text streams out word by word.

Diagram of an autoregressive loop: the context goes into the model, the next token is sampled and appended to the context, then the loop repeats.

It's a text-completion engine

Because of that loop, the cleanest way to describe the underlying machine is a text-completion engine: hand it any text, and it extends it. This reframes every task as completion. "Translate to French: Hello" is completed with "Bonjour." "Summarize this article: …" is completed with a summary. A question is just text whose most natural completion happens to be the answer. That single trick — completion — is doing all of the work.

Before it completes anything, the engine splits text into tokens, maps them to ids, and turns them into embedding vectors it can extend.

Diagram: a sentence is split into tokens, each token mapped to a numeric id, and each id to an embedding vector.

Seen this way, the kinship with phone keyboards in the autocomplete-to-LLM story is exact, not just a metaphor. The keyboard completes your sentence; the LLM completes your document, your question, or your conversation. Same shape, wildly different scale.

Why "guess the word" gives you so much

Here's the part that surprises people. To predict the next word in all kinds of human text, the model has to implicitly learn a staggering amount. To finish a math derivation, it must track the numbers. To complete a sentence in a legal document, it must respect that register. To answer a trivia question correctly, it must have stored the fact. Prediction, pushed to a high enough standard across enough text, quietly forces the model to acquire skills nobody programmed in by hand.

That is the quiet miracle behind LLMs and behind every generative text model: a deceptively dull objective — predict the next token — turns out to be a doorway to writing, reasoning, and conversation, once you make the predictor big enough and feed it enough of the world's text. Keep this in your back pocket; almost everything else is an elaboration of it.

Just from predicting nearby words, the model learns vectors where meaning becomes geometry — famously, king − man + woman ≈ queen.

Word vectors arranged so analogies become vector arithmetic, showing king minus man plus woman landing near queen.