sequence parallelism

Tensor parallelism shards the parts of a transformer block that contain large matrix multiplies, but it leaves the dropout, layer-norm, and residual regions replicated — and for long sequences those replicated activations dominate memory. Sequence parallelism plugs that hole: it shards those replicated regions along the sequence (token) dimension, so each device holds only a slice of the tokens' activations there. Combined with tensor parallelism, the replicated activation footprint falls by the parallel degree with no extra weight memory.

Mechanically it changes which collectives appear: where pure tensor parallelism does an all-reduce at the block boundary, the tensor-plus-sequence arrangement replaces it with an all-gather entering the attention/MLP region (to reassemble the full sequence the matmuls need) and a reduce-scatter leaving it (to re-shard along the sequence). The total bytes moved are the same as the original all-reduce, so the memory saving is essentially free. A related variant, context parallelism, shards the attention computation itself across the sequence using ring-style passing of keys and values so that even the attention activations and the KV cache scale to very long contexts.

Sequence parallelism is what makes million-token context training tractable: attention's activation and memory cost grow with sequence length, and sharding the token axis is the only way to keep per-device memory bounded as context windows stretch into the hundreds of thousands.