DeepSpeed (ZeRO)

DeepSpeed is Microsoft's training library, best known for introducing and popularizing ZeRO, the sharding of optimizer states, gradients, and parameters across data-parallel ranks. Wrapped around an existing model with minimal code change, it lets a team push parameter counts far past what fits replicated, while keeping the easy data-parallel programming model. It packages the three ZeRO stages behind a JSON config rather than requiring model surgery.

Beyond core sharding, DeepSpeed's distinctive moves are about spilling state down the memory hierarchy when the GPUs alone cannot hold it. ZeRO-Offload moves optimizer states and the optimizer step to CPU memory and the CPU's cores; ZeRO-Infinity extends this to NVMe, treating disk as another tier so that models far larger than aggregate GPU memory can be trained, at the cost of bandwidth-bound steps. The library also bundles fused kernels, a curriculum of mixed-precision and quantization tools, and an inference engine, but its scaling identity remains ZeRO plus offload.

DeepSpeed and PyTorch FSDP now offer overlapping capability — both implement the ZeRO-3 algorithm — and the choice between them is largely ecosystem and tooling. DeepSpeed's offload tiers remain its differentiator for training oversized models on memory-constrained clusters, with the understanding that NVMe-bound steps trade throughput for feasibility.