Megatron-LM

Megatron-LM is the NVIDIA research codebase that turned large-transformer parallelism from a research curiosity into a reproducible recipe. Its lasting contribution is a clean, communication-minimal scheme for tensor parallelism: split the MLP's first matrix column-wise and its second row-wise, shard attention by heads, and arrange the cuts so each transformer block needs only two all-reduces forward and two backward. This made it practical to spread a single layer across the GPUs of one node without rewriting the model.

Over successive papers Megatron added the other axes and the tricks that make them efficient: sequence parallelism to shard the leftover replicated activations, interleaved pipeline schedules to shrink the bubble, and the empirical guidance for combining tensor, pipeline, and data parallelism that became the canonical 3D-parallelism playbook. Its kernels and scheduling were refactored into Megatron-Core, a library many downstream training stacks build on, and its tensor-parallel layout is mirrored by most modern frameworks even when they are not Megatron-based.

Megatron-LM matters less as a specific program than as the source of a shared vocabulary — column/row parallel, 1F1B, sequence parallelism — that the whole field now uses. When people say a model was trained with Megatron-style parallelism they mean this particular factorization of the transformer, not necessarily that exact code.