JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Two phases, one wall: the anatomy of LLM inference

Before you can speed inference up, you need to know what it is actually doing. Meet prefill, decode, the memory-bandwidth wall, and the latency metrics that everything else optimizes.

Why serving is its own discipline

You already understand training versus inference at a conceptual level: training adjusts weights, inference just runs the forward pass. It is tempting to think inference is therefore the easy part. For large language models the opposite is true. A single training run is a one-time cost amortized over months; inference is a cost you pay on every token of every request, forever. Squeezing that cost is where a huge fraction of applied-AI engineering now lives, and it has its own vocabulary, its own bottlenecks, and its own surprising failure modes.

The reason LLM inference is hard is not raw compute — modern GPUs have enormous arithmetic throughput. The reason is that autoregressive generation is sequential and memory-bound. To produce the 200th token you must have produced the 199th, and each step reads a large amount of state for a tiny amount of math. This guide builds the mental model that the rest of the track depends on.

Prefill and decode: two very different jobs

Generating from a transformer splits into two phases. Prefill processes the entire prompt at once: every prompt token attends to every earlier one, so the whole prompt flows through the network in a single, highly parallel matrix-heavy pass. Decode then emits output tokens one at a time via autoregressive decoding: each new token requires a forward pass over a single position that attends back to all previous tokens.

Decode runs this autoregressive loop one token at a time — the sequential bottleneck that makes inference hard.

Diagram of an autoregressive generation loop feeding each output token back in as the next input.

The KV cache and why decode reads so much

Naively, generating token t would re-attend over all *t-1* previous tokens by recomputing their keys and values. That would make generation quadratic. Instead we store every layer's keys and values in the KV cache and reuse them: at step t we compute only the new token's query, key, and value, append the latter two to the cache, and attend. This turns the per-step compute from quadratic to linear — but it shifts the problem to memory. The cache grows with sequence length and must be read in full at every single decode step.

The KV cache stores exactly these per-layer keys and values so decode never recomputes them.

Diagram of attention as a soft query–key–value lookup with softmax weights.

# Decode step, conceptually
q, k, v = project(x_t)          # one token's worth
kv_cache.k.append(k)            # grow the cache
kv_cache.v.append(v)
attn = softmax(q @ kv_cache.k.T) @ kv_cache.v   # read the WHOLE cache
x_next = mlp(attn)
Each decode step does tiny math (one query) but reads the entire cache and all weights — the essence of being memory-bound.

The memory-bandwidth wall

A useful number is arithmetic intensity: floating-point operations performed per byte read from memory. Decode has terribly low arithmetic intensity — for batch size one, you read billions of weight bytes to do a few billion FLOPs, so you finish the math long before the bytes have arrived. The kernel is memory-bandwidth-bound, and the GPU's tensor cores sit mostly idle. See arithmetic intensity for the formal picture.

\text{Arithmetic intensity} = \frac{\text{FLOPs performed}}{\text{bytes moved from memory}}

Arithmetic intensity — FLOPs per byte moved — is the single number that explains the decode bottleneck.

This single fact explains the whole field. Batching raises arithmetic intensity by amortizing each weight read over many requests. KV-cache compression and paging shrink the bytes you must move. Speculative decoding does more useful math per memory pass. Quantization makes each byte carry more. Keep the wall in mind and every later trick has an obvious motive.

The metrics everything optimizes

Serving has two latency metrics, captured by TTFT and TPOT. Time to first token (TTFT) is dominated by prefill and measures how long the user waits before anything appears. Time per output token (TPOT), sometimes called inter-token latency, measures the steady-state decode speed once streaming begins. These are user-facing service-level objectives (SLOs), and they trade off against throughput.

T_{\text{total}} = \text{TTFT} + (N - 1)\cdot \text{TPOT}

End-to-end latency decomposes into prefill-dominated TTFT plus one TPOT per remaining generated token.

The eternal tension is latency versus throughput. Bigger batches use the GPU more efficiently and raise tokens-per-second across all users (throughput, which drives inference cost) but can hurt any single user's TPOT. A serving system is fundamentally a scheduler trying to hit latency SLOs while keeping the expensive hardware busy.