pipeline parallelism

Pipeline parallelism cuts the model the other way from tensor parallelism: rather than splitting each layer across devices, it assigns consecutive groups of whole layers to different devices, called stages. A batch flows through stage 0, whose output becomes stage 1's input, and so on — like an assembly line. The appeal is that the only communication is a point-to-point hand-off of activations at each stage boundary, far cheaper than the all-reduces tensor parallelism needs, so pipeline stages can span across nodes.

The catch is the pipeline bubble: while stage 0 processes the first input, every later stage sits idle, and symmetrically at drain time. To shrink the bubble, the batch is split into many micro-batches that are pumped through in a schedule. GPipe fills then flushes; the 1F1B (one-forward-one-backward) schedule interleaves forward and backward micro-batches to bound activation memory; interleaved and zero-bubble schedules push idle time lower still by assigning each device several non-contiguous stage chunks. Bubble fraction shrinks roughly as the number of stages over the number of micro-batches.

Pipeline parallelism is the cross-node scaling axis of choice because its communication is light, but it is sensitive to load imbalance — an uneven layer-to-stage split leaves some stages waiting — and its activation memory grows with the number of in-flight micro-batches, which is why it is paired with activation recomputation.