tensor-parallel inference

A model too large to fit on one GPU, or one whose single-GPU latency is too high, can be split across several GPUs that cooperate on each forward pass. Tensor parallelism does this within a layer: it shards the big weight matrices across devices so each GPU holds a slice and computes a partial result. For a feed-forward block one matrix is split column-wise and the next row-wise, so each device runs its share of the matrix multiply in parallel; for multi-head attention, different heads are placed on different devices. The aggregate compute and memory of the layer are divided across the group.

The price is communication. Because every device computes only a partial output, the shards must be combined with a collective operation — an all-reduce or all-gather — typically twice per transformer layer, and that synchronization happens on the critical path of every single token. Tensor parallelism therefore demands very high-bandwidth, low-latency links such as NVLink and is normally confined to GPUs inside one server; stretched across slower network it stalls. It is the standard way to drive down the per-token latency of a large model, and is composed with pipeline and data parallelism for the largest deployments.

Choosing the tensor-parallel degree is a latency-versus-efficiency trade: more GPUs cut latency but add communication overhead and can leave each device underutilized.