SwiGLU activation

SwiGLU is a gated feed-forward unit that replaces the usual linear-then-activation-then-linear block of a transformer. Instead of one path through an activation, it splits into two parallel projections: one is passed through a smooth SiLU/Swish nonlinearity and used as a gate that multiplies the other element by element. The gate lets the network modulate how much of each feature to let through, which empirically gives a quality boost at equal parameter count.

Concretely the block computes (SiLU(xW) elementwise-times xV) times W_out, where SiLU(z) = z times sigmoid(z) is the Swish activation and W, V, W_out are learned. Because gating adds a third weight matrix, implementations shrink the hidden dimension, often to two-thirds of the equivalent dense feed-forward, to keep the parameter and floating-point budget matched for a fair comparison. The variant comes from Shazeer's family of gated linear units, GEGLU, ReGLU, and SwiGLU.

SwiGLU is now standard in LLaMA, PaLM, and most modern LLMs, having displaced plain ReLU and GELU feed-forwards. The gain is consistent though modest, and as Shazeer wryly noted, the success is partly empirical, these architectures simply work better in practice without a fully first-principles explanation.