GPU memory for LLMs

A GPU has a fixed pool of fast memory, its VRAM, and an LLM must fit two big things into it: the model's weights and the KV cache for every active request. If they do not fit, the model simply will not run on that device — it has to be split across several GPUs instead.

The weights dominate the static footprint — roughly the parameter count times the bytes per parameter, so a seventy-billion-parameter model needs about 140 GB in fp16 and about 35 GB in int4 — plus some activation scratch space. On top of that sits the KV cache, which grows with context length and batch size and eats whatever memory is left at serving time. When a model exceeds one GPU's VRAM it is sharded across devices with tensor or pipeline parallelism. Memory, more often than raw compute, is the binding constraint that decides how big a model and how long a context you can actually serve.