JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Serving a crowd: batching, throughput, and latency

One GPU should answer hundreds of users at once. The trick is batching — and continuous batching is the idea that made modern LLM serving economical.

Idle silicon is the enemy

Recall that decode is memory-bound: serving one user, the GPU's math units are starved, waiting on memory. The classic fix is batching — run many users' decode steps together in one forward pass. The weights get read from memory once and reused across the whole batch, so adding users is nearly free until the math units finally fill up.

This is the whole economic argument for GPU serving: a card answering one person and a card answering sixty cost almost the same to run, so packing the batch is how you drive down the price of every answer.

The trade you can never escape

Bigger batches mean more answers per second (throughput) but each individual answer can wait longer behind the others (latency). Throughput versus latency is the master dial of serving: a chat product guards latency so replies feel instant, while an overnight document-processing job maximizes throughput and ignores the clock.

L_{\text{user}} = \text{TTFT} + \dfrac{N_{\text{out}}}{\text{TPS}_{\text{user}}}

A user's total wait is the time to first token plus the rest streamed at their per-user token rate.

Concretely, this is the split between batch and online inference. Online (interactive) serving optimizes TTFT and per-user TPS; batch (offline) serving optimizes total tokens-per-dollar across an entire job. Same model, opposite tuning.

Static batching wastes the GPU

The obvious way to batch is static: gather, say, 16 requests, run them as a group until all 16 finish, then start the next group. The flaw is that answers have wildly different lengths. A 20-token reply and a 2,000-token reply in the same batch means the short one finished long ago, but its slot sits locked and idle until the long one is done.

Continuous batching: a revolving door

Continuous batching (also called in-flight batching) fixes this by working at the granularity of a single token step rather than a whole request. After every decode step the scheduler checks: did anyone finish? Free their slot and admit a waiting request right now, mid-flight, without draining the batch.

The batch becomes a revolving door: finished sequences leave, new ones enter, the GPU stays packed every step. This one change typically multiplies throughput several-fold over static batching on real workloads, and it is the reason a single card can keep hundreds of conversations alive at once. It leans entirely on paging from the last guide — flexible GPU memory management is what lets requests come and go cleanly.

What to measure when you serve a crowd

  1. TTFT (per request): how long before the first token streams. Mostly prefill + queue wait. Watch the p95, not just the average — tail latency is what users complain about.
  2. Per-user TPS: how fast each person's text streams. This must comfortably beat human reading speed (~5–10 tokens/sec) or the reply feels sluggish.
  3. Aggregate throughput: total tokens/sec across all concurrent users on the card — the number that decides cost. Continuous batching pushes this up without wrecking the two above.
  4. Concurrency at the memory limit: how many requests fit before the KV cache fills the GPU. This caps the batch, so it caps everything else.
\text{Throughput}_{\text{agg}} = N_{\text{concurrent}} \times \text{TPS}_{\text{user}}

Aggregate throughput is just how many users you serve at once times each one's token rate — which is why packing the batch pays off.