LLM serving frameworks

You could write a simple loop that calls a model token by token, but doing it efficiently for thousands of simultaneous users is a serious systems problem. Serving frameworks are the production engines — vLLM, TensorRT-LLM, TGI, SGLang and others — that package the hard parts so you do not have to reinvent them.

These systems bundle a paged KV cache, continuous batching, optimized attention and matrix-multiply kernels, quantization, speculative decoding, tensor parallelism, and an OpenAI-style HTTP API into one server. They turn raw model weights into a service that maximizes GPU utilization and throughput while still meeting latency targets. Choosing one trades off peak performance, hardware support, ease of use, and feature coverage; in practice they are the default way LLMs are deployed at scale, and much serving research now ships first as a feature inside them.