JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

From prompt to answer: how an LLM actually runs

Every reply is built one token at a time in two distinct phases. Understanding prefill and decode is the key that unlocks everything else about serving.

Inference is just running the model forward

Training is the expensive one-time act of building a model from data. Inference is what happens every time you actually use it: you hand the model some text, and it hands back more text. LLM inference is the everyday workhorse — the same trained weights get run forward billions of times, once for every chat message, autocomplete, and API call in the world.

At heart the model does one tiny thing: predict the next token. Given everything so far, it outputs a probability for every possible next token, picks one, appends it, and repeats. A 500-word answer is roughly 700 of these steps stacked end to end.

p(x_t \mid x_{<t}) = \mathrm{softmax}(W\,h_t)

Next-token prediction in one line: the model turns its hidden state into a probability over every possible next token, then samples one.

Phase one: prefill reads your whole prompt at once

When your prompt arrives, the model processes all of it in a single parallel pass. This is prefill. Because every token of the prompt is available up front, the GPU can crunch them together as one big matrix multiply — extremely efficient, and what GPUs are built for. Prefill versus decode is the single most important split to internalize.

Before prefill can crunch them in one parallel pass, your whole prompt is split into tokens and turned into embedding vectors.

Diagram: text is tokenized into token ids, then mapped to embedding vectors.

Prefill is compute-bound: with 4,000 prompt tokens the GPU is busy doing real arithmetic and its cores run hot. The time prefill takes is what the user experiences as time to first token (TTFT) — the pause between hitting Enter and seeing the first character appear.

Phase two: decode writes one token at a time

Once the prompt is digested, generation begins. Decode produces the answer one token per forward pass: predict a token, append it, feed the slightly-longer sequence back in, predict the next. You cannot skip ahead — token 50 depends on token 49 — so decode is inherently sequential.

Decode is exactly this autoregressive loop: predict a token, append it, feed the slightly-longer sequence back in.

Diagram: an autoregressive next-token generation loop that appends one token each forward pass.

tokens = prefill(prompt)         # one big parallel pass over the prompt
while not finished:
    logits = model.step(tokens)  # one forward pass -> scores for next token
    nxt    = sample(logits)      # pick a token (greedy or sampling)
    tokens.append(nxt)           # grow the sequence by one, then repeat
Prefill happens once; the decode loop runs once per generated token.

Decode is memory-bound, not compute-bound: each step touches only one new token, so the GPU's math units sit mostly idle while it waits to stream the model's weights in from memory. That asymmetry — fat parallel prefill, skinny sequential decode — explains almost every serving optimization you will ever meet.

The two numbers users actually feel

Two metrics describe the experience. TTFT is the wait before anything appears (dominated by prefill). Tokens per second (TPS) is how fast text streams out afterward (dominated by decode). A snappy assistant has low TTFT and high TPS; these are two different problems with two different fixes.

Why this matters for everything ahead

Hold on to one sentence: *prefill is compute-bound and parallel; decode is memory-bound and sequential.* The KV cache exists to make decode bearable, batching exists to fill the idle math units during decode, and throughput-versus-latency trade-offs all live in the gap between these two phases. Next we open up the memory wall that decode runs into.