data, model, and pipeline parallelism
/ DAY-tuh, MOD-ul, and PIPE-line PAR-uh-lel-iz-um /
These are the three main ways to split a training job across many chips, each answering a different question. Picture a giant restaurant order. Data parallelism is the simplest: give every cook the same full recipe but a different batch of ingredients, and at the end they compare notes. Each chip holds a complete copy of the model and processes a different chunk of the training examples, then they average what they learned.
Model parallelism (often called tensor parallelism) is for when the recipe itself is too big to fit in one cook's head. You split a single dish across several cooks — one chops, one sears, one plates — so the model's own layers or even individual layers are sliced across chips, with each chip holding only a piece of the network. Pipeline parallelism is the assembly line: cook A always does the first course, hands it to cook B for the second, and so on; the model is cut into sequential stages, each living on a different chip, with data flowing through like cars down a production line.
Why it matters: today's biggest models are too large for any one chip, so all three are usually combined at once — this is what 'training a model on a thousand GPUs' actually means under the hood. The honest tradeoff is that each strategy adds communication and complexity. Pipeline parallelism in particular leaves chips idle at the start and end of each batch (the 'bubble') while the line fills and drains. Choosing the right mix is a delicate balancing act that depends on the model's shape, the hardware, and the network connecting it all.
A model too big for one GPU is sliced 4 ways (model parallel) and laid out as 4 stages (pipeline). That whole 4-chip setup is then copied 64 times, each copy fed different examples (data parallel) — 256 chips, all three strategies at once.
Real frontier training blends all three at once — this layout is the norm, not the exception.
Data parallelism is easy but useless once the model itself no longer fits in one chip's memory — copying it everywhere is then impossible. That memory wall is exactly why model and pipeline parallelism exist, despite being fiddlier to set up.