tensor and pipeline parallelism (LLM)

When a model is too big to fit on one device even as a single copy, you must split the model itself — not just the data. Tensor and pipeline parallelism are the two ways to cut it. Tensor parallelism slices across a layer: a single huge matrix multiply is divided among several devices, each computing part of the output, with results stitched back together. Pipeline parallelism cuts the other way, placing different layers on different devices.

Each has a characteristic cost. Tensor parallelism demands very frequent, high-bandwidth communication mid-layer, so it is kept within a tightly connected group of GPUs sharing a fast interconnect. Pipeline parallelism passes activations from one layer-group to the next like an assembly line, which risks idle bubbles while early or late stages wait for work; clever micro-batch scheduling keeps the pipeline full and the bubbles small.

Neither is used alone. Real systems layer tensor parallelism inside a node, pipeline parallelism across nodes, and data parallelism on top — the three-dimensional parallelism that lets a model with hundreds of billions of parameters train on a cluster where no single device could hold even a fraction of it.