Scaling, Governance, and LLMOps

Autoscaling inference under SLOs

Production traffic is not flat — it has daily peaks, weekend lulls, and the occasional viral spike. Inference autoscaling adjusts the number of serving replicas to demand so you meet a latency SLO (e.g. p99 under 200 ms) without paying for idle accelerators at 3 a.m. The core tension is economic: too few replicas and requests queue past the SLO; too many and you burn money on idle GPUs that are far more expensive than ordinary web servers.

ML autoscaling is harder than web autoscaling for two reasons. First, scaling up is slow: loading a multi-gigabyte model onto a fresh GPU is a cold start of seconds-to-minutes, so you must scale on a leading signal (queue depth, request rate) and keep warm headroom, not react after latency already breached. Second, the right scaling unit is the batch: GPUs are most efficient when many requests are processed together, so autoscaling interacts tightly with batching policy.

The autoscaler's core objective made precise: choose the fewest replicas N that keep p99 latency within the SLO at minimum serving cost.

Model governance and audit

As ML moves into hiring, credit, healthcare, and regulated domains, 'it works' is no longer enough — you must prove how it works, who approved it, and that it stays within bounds. Model governance and audit is the layer of approval workflows, documentation, and immutable audit trails that makes a model accountable. It rests directly on the lineage backbone from guide 1: governance is impossible without reproducible versions, because you cannot audit a model you cannot reconstruct.

1. Documentation: a model card per version — intended use, training data, evaluated fairness slices, known limitations.
2. Approval workflow: a named human sign-off gating the staging → production promotion, recorded immutably.
3. Audit trail: who deployed what, when, trained on which data, and why — answerable months later.
4. Controls: access restrictions, change logs, and the documented data lineage a regulator can inspect.

This is no longer optional in many jurisdictions: regimes like the EU AI Act impose documentation, risk-management, and human-oversight duties on high-risk systems. The teams that survive an audit are the ones for whom governance was a byproduct of good versioning and metadata — not a panicked scramble to reconstruct history that was never recorded.

LLMOps: when the artifact is a prompt

Operating a large language model application breaks several assumptions baked into classical MLOps. You often did not train the model — you call it, possibly a third party's. The behavior-defining artifact is frequently not weights but the prompt, the tool definitions, and the retrieval context. Outputs are open-ended text, so 'accuracy' is no longer a single number, and a single user request may fan out into a chain of model calls, tool invocations, and retrieval steps.

LLMOps observability meets this by tracing: capturing the full tree of a request — every prompt sent, every token returned, every tool call and its result, latency and cost at each node. A trace is the LLM-era analogue of the joined observability record from guide 4, and it is indispensable: when an agent gives a wrong answer, the bug is almost always somewhere in the chain — a bad retrieval, a malformed tool result, a prompt that drifted — and only the trace lets you see which step failed.

Evaluation also transforms: with no single gold label, teams increasingly score open-ended outputs with LLM-as-a-judge — using a strong model to grade responses against a rubric — run continuously in production over sampled traces. It is powerful but fallible (judges have biases of their own), so treat its scores as a monitored signal to be calibrated against human review, never as unquestioned ground truth.

Prompt versioning and the new release loop

If the prompt is a behavior-defining artifact, it deserves everything we gave models in guide 1. Prompt versioning treats prompts as first-class deployable artifacts: each is versioned, stored, linked to the eval results it produced, and released through the same safe-release ladder — you canary a new prompt to 5% of traffic, compare on judged quality and cost, and roll back instantly if it regresses. A prompt edit is a deployment, with all the discipline that word now carries.

The research frontier

Production ML is a young, fast-moving field, and its open problems are where research meets the operational reality you have just toured. A few live frontiers: efficient serving — how do techniques like continuous batching and disaggregated prefill/decode squeeze more throughput per accelerator as model sizes outrun hardware? Drift for generative systems — what does 'drift' even mean when outputs are free text with no label, and how do you detect quality regressions in an agent without ground truth?

And agent operations — when one user request expands into dozens of interdependent model and tool calls, the questions of cost attribution, reliability, evaluation, and rollback from this whole track must be re-answered for a system of models rather than a single one. The throughline of this track holds even at the frontier: a model is only trustworthy in production if you can reproduce it, release it carefully, watch it honestly, scale it economically, and account for it fully. Everything else is detail — important, evolving, but detail. You now have the map.