N:M sparsity (e.g. 2:4)

N:M sparsity is a middle ground between the irregular zeros of unstructured pruning and the coarse blocks of structured pruning. It demands that within every contiguous group of M weights, at most N are nonzero — the popular 2:4 pattern keeps two out of every four. This fine but regular structure is exactly what some hardware can accelerate.

NVIDIA Ampere and later GPUs include sparse tensor cores that, given a 2:4 weight matrix stored as the kept values plus 2-bit indices, run the matrix multiply at up to twice the dense throughput. Producing such weights applies a magnitude or importance criterion within each group of M, usually followed by fine-tuning; a prune-train-reprune schedule helps recover accuracy. It also composes cleanly with quantization for stacked compression.

The catch is that the 2× figure is a hardware ceiling rarely realized end-to-end once other layers and overheads are counted, and forcing the N:M mask can cost accuracy on sensitive layers, so it is applied selectively rather than everywhere.