vector-quantized VAE (VQ-VAE)

VQ-VAE learns to compress an image or audio clip into a grid of discrete symbols drawn from a learned vocabulary, much like turning a photo into a small page of text. This discreteness is the point: once data is a sequence of tokens, you can train a powerful autoregressive or masked transformer over those tokens to generate new data, which is exactly how image and audio token models are built.

An encoder produces a continuous feature map; each spatial vector is then snapped to its nearest entry in a learned codebook (the quantization step), and the decoder reconstructs the input from these quantized vectors. The nearest-neighbour lookup has no gradient, so training uses the straight-through estimator: the forward pass quantizes but the backward pass copies the decoder's gradient straight past the quantizer to the encoder. The loss combines reconstruction, a codebook term that moves chosen codes toward the encoder outputs, and a commitment term that keeps encoder outputs from wandering away from their codes. VQGAN later added an adversarial and perceptual loss for sharper reconstructions.

The classic failure mode is codebook collapse, where only a handful of code vectors ever get used and the rest go dead, shrinking the effective vocabulary. Tricks like EMA codebook updates, code resets, and lower-dimensional codes mitigate it. Discrete latents trade a little reconstruction fidelity for a representation that downstream sequence models find far easier to learn over.