CUDA graphs

Launching a GPU kernel from the CPU costs a few microseconds of overhead — negligible for one big kernel, but crippling when you fire thousands of tiny ones, as small-batch LLM inference does. CUDA graphs let you record a whole sequence of kernel launches once, as a graph of operations, and then replay that entire sequence with a single launch, paying the per-kernel CPU cost only at capture time.

You capture a stream of CUDA operations — kernels, memory copies, and their dependencies — into a graph object, instantiate it once, and thereafter launch the instantiated graph repeatedly. Replay submits the whole directed acyclic graph to the GPU with one CPU call, eliminating per-kernel launch latency and letting the driver schedule the fixed dependency structure efficiently. The constraint is that the graph is static: the same kernels, the same shapes, and typically the same memory addresses each replay, so inputs must be written into pre-allocated buffers and control flow cannot change between iterations.

For decode-heavy LLM serving and other CPU-launch-bound workloads, graph replay removes the launch bubbles that otherwise leave the GPU idle between tiny kernels — often a large end-to-end win. The price is rigidity, which is why graphs pair naturally with static-shape, fixed-batch inference rather than dynamic training loops.