SwiGLU feed-forward

After attention mixes information across tokens, each token is processed on its own by a small two-layer network, the feed-forward block. SwiGLU upgrades this block with a gate. Instead of one linear layer followed by an activation, it computes two projections of the input: one becomes a smooth on/off gate via the SiLU function, the other is the raw signal, and the model multiplies them element by element. The gate lets each hidden unit decide how much of the signal to pass through, giving the layer a sharper, learnable filter.

The name packs in two ideas: SiLU (also called Swish) as the smooth nonlinearity, and GLU, the gated linear unit pattern of multiplying a signal by a gate. Because gating uses an extra projection, implementations shrink the hidden width to about two-thirds so the parameter count matches a plain feed-forward. Empirically this gated MLP trains to lower loss than a ReLU or GELU block of the same size, which is why it is the default in most recent open LLMs.