inference autoscaling

Traffic to a deployed model is rarely flat: it surges at peak hours and ebbs overnight, and a fleet sized for the peak wastes money while one sized for the average misses its latency target when demand spikes. Inference autoscaling adjusts the number of serving replicas (or accelerators) up and down in response to load, chasing a target where the latency service-level objective is met at the lowest sustainable cost. It is the control loop that keeps a model both fast enough and affordable.

An autoscaler observes a signal, requests per second, queue depth, GPU utilization, or measured tail latency, and compares it to a target, adding replicas when the signal runs hot and removing them when it cools. ML serving complicates the textbook web-autoscaling story: model containers can take minutes to load large weights onto a GPU, so cold-start latency is severe; accelerators are coarse-grained and costly, so you cannot scale by single small units; and throughput depends on batching, where larger batches raise utilization but also per-request latency. Effective autoscaling therefore co-designs batch size, replica count, and scale-out timing.

The defining trade-off is the triangle of latency, cost, and headroom. Scale aggressively and you burn money holding idle accelerators against a surge that may not come; scale lazily and a real surge blows the latency SLO before new replicas warm up. Practical systems keep a warm buffer sized to expected burstiness, scale on predictive as well as reactive signals, and may degrade gracefully (smaller model, longer queue) rather than violate the SLO outright.