Serving is an SLO problem
A production model serving system is judged not on average speed but on tail behavior against a contract. Typical SLOs read like: "p99 TTFT under 500 ms and p95 TPOT under 50 ms at a given queries-per-second." Every knob from the earlier guides — batch size, chunk size, how aggressively to admit prefill — is set to satisfy these percentiles, not the mean. Optimizing the average while blowing the tail is the classic rookie mistake.
Because TTFT and TPOT pull in opposite directions, real schedulers prioritize by SLO class: latency-critical interactive chat gets small batches and prompt admission, while throughput-oriented batch jobs are packed aggressively. The same hardware serves both by routing them into different scheduling regimes.
Guided decoding: making structure reliable
Applications need JSON that parses, function calls with valid arguments, SQL that runs. Prompting for format works most of the time, which in production means it fails enough to page someone. Guided decoding makes malformed output structurally impossible: at each step it computes which tokens a formal grammar or regular expression permits and masks the rest to probability zero before sampling. The model can only walk paths the grammar allows.
Grammar-constrained decoding multiplies the logits by a 0/1 mask, so only tokens the grammar permits keep nonzero probability.
Cascades: spend big-model money only when needed
Most real traffic is easy; a small model answers it perfectly well, and only a minority of queries truly need the flagship. Cascade inference exploits this by routing each query to the cheapest model that can handle it: a fast small model answers first, and only escalates to a larger model when a confidence signal — its own uncertainty, a verifier, or a lightweight router — says the answer is unreliable.
Because cost is dominated by the expensive tier, sending even half the traffic to a small model can roughly halve spend at near-equal quality. The engineering art is the routing signal: too timid and you escalate everything and save nothing; too bold and quality slips on the hard tail. Cascades pair naturally with streaming — the small model's answer can start streaming immediately and be superseded only if escalation triggers.
A cascade's expected cost is the small-model cost plus the escalation probability times the large-model cost — which is why moving the routing threshold moves spend so directly.
Observability and autoscaling
You cannot defend an SLO you do not measure. LLMOps observability for serving tracks the latency percentiles, the acceptance rate of your speculation, cache hit rates, queue depth, GPU utilization, and per-request token counts — and connects them to cost and quality. When load rises, inference autoscaling adds replicas, but LLM serving makes this hard: replicas have slow cold starts (gigabytes of weights to load) and warm KV-cache state that does not migrate, so naive request-rate autoscaling overshoots and thrashes. Effective policies scale on queue depth and SLO headroom, and pre-warm capacity ahead of predictable demand.
Little's Law ties queue depth to arrival rate and latency — the basis for sizing replicas when autoscaling a serving fleet.
The frontier
Serving is moving fast. Active threads include: smarter KV-cache eviction and offloading so practically unbounded contexts spill gracefully to CPU memory or storage; tighter co-design of model architecture and engine, where attention variants are chosen for what they cost at serving time; speculation that adapts its draft length to the live acceptance rate; and disaggregation pushed further into per-layer and per-expert placement for sparse mixture-of-experts models. The unifying theme has not changed since guide one: inference is memory-bound and sequential, and every advance is a new way to do more useful work per byte moved and per step taken.