Into production: serving frameworks and the cost per token

You don't build this from scratch

Everything in this track — prefill/decode scheduling, the KV cache, paging, continuous batching, quantization, speculative decoding — is hard to implement well. So in practice you reach for a serving framework that bundles all of it: you point it at a set of weights and it exposes a fast, batched, streaming API. Popular open-source engines (such as vLLM, TGI, SGLang, and TensorRT-LLM) differ in tuning but share this core feature set.

# bring a model up with an OpenAI-compatible server (vLLM example)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \          # weight-only quantization
  --max-model-len 8192 \        # context length cap (bounds KV cache)
  --gpu-memory-utilization 0.9  # how much VRAM to use for weights + cache

A single command turns weights into a production endpoint with paging and continuous batching already on.

Pick your serving mode first

Before tuning anything, decide where you sit on batch versus online. Online: real users waiting, so guard TTFT and per-user speed, cap batch size, accept a higher cost per token. Batch: a million documents to summarize overnight, so max out batch size, saturate the GPU, and let individual latency balloon. The same framework serves both; you are choosing a point on the throughput–latency curve.

Cost per token: the bottom line

Strip away the engineering and one number decides whether a deployment is viable: the cost per token. It is almost pure arithmetic — what you pay per hour for the GPU, divided by how many tokens that GPU produces per hour. Everything in this track is, in the end, a tactic to push that fraction down.

\text{cost/token}=\frac{\text{GPU price per hour}}{\text{tokens per hour}}=\frac{p_{\$/\text{hr}}}{3600\cdot\text{throughput}_{\text{tok/s}}}

Cost per token is almost pure arithmetic: the GPU's hourly price divided by how many tokens it produces in that hour.

gpu_cost_per_hour      = 2.00        # USD for the GPU instance
aggregate_tokens_per_s = 3000        # all users combined, after batching
tokens_per_hour        = 3000 * 3600 # = 10,800,000

cost_per_million_tokens = gpu_cost_per_hour / (tokens_per_hour / 1_000_000)
#                       = 2.00 / 10.8  ->  about $0.19 per million output tokens

Throughput is the denominator — double it (paging, batching, quantization) and the price per token halves.

A capacity-planning walkthrough

Fix the model and context length. These set the weight footprint and the per-request KV cache size — together the GPU memory you need just to admit one user.
Compute the concurrency ceiling: (VRAM − weights) ÷ per-request cache = how many users fit. Quantizing the weights frees room and raises this ceiling.
Load-test to the real throughput and p95 TTFT at that concurrency. Benchmarks lie; measure with traffic shaped like your own.
Divide GPU price by measured throughput to get cost per token. If it is too high, your levers are: quantize harder, raise batch size (if latency allows), add speculative decoding for interactive traffic, or shard the model across cards.
Size the fleet: requests-per-second ÷ per-GPU capacity = number of GPUs, then autoscale on queue depth so you pay for idle silicon as little as possible.

N_{\text{concurrent}} \le \dfrac{\text{VRAM} - \text{weights}}{\text{KV cache per request}}

The concurrency ceiling from the walkthrough: whatever VRAM is left after the weights, divided by one request's KV cache, caps how many users fit at once.

Where this track leaves you

You now hold the full mental model of serving an LLM: an answer is built in a parallel prefill and a sequential decode; decode is memory-bound, so the KV cache and GPU memory are the real budget; paging and continuous batching pack that budget tight; quantization and speculative decoding stretch it further; and a serving framework delivers all of it behind one API, judged by a single honest number — the cost per token.