latent diffusion
/ LAY-tunt dih-FYOO-zhun /
Latent diffusion is the clever shortcut that made image generators fast and cheap enough to run on ordinary hardware. A plain diffusion model does all its noisy work directly on the pixels — and a high-resolution image has millions of them, which is enormously expensive to denoise step after step. Latent diffusion sidesteps that by doing the whole denoising process in a compressed, shrunken representation of the image instead of on the raw pixels.
It works in three moves. First, a separate network (an autoencoder) learns to squeeze an image down into a small "latent" code — a compact numerical summary that keeps the meaningful content while throwing away pixel-level bulk — and to expand such a code back into a full image. Diffusion then runs entirely inside that small latent space, which might be dozens of times smaller than the image, so each denoising step is far quicker. Finally the cleaned-up latent code is decoded back into a full-resolution picture. The popular Stable Diffusion family works exactly this way.
The honest trade-off: working in a compressed space saves enormous compute but throws some information away, so very fine detail — small faces in a crowd, the exact letters on a sign, intricate textures — can come out smudged or wrong, because the latent code never carried that detail precisely. It is a brilliant efficiency win that democratized image generation; it is not a free lunch, and the compression leaves visible fingerprints.
A 512×512 image has about 260,000 pixels. The autoencoder compresses it to a 64×64 latent grid — roughly 48 times smaller. Diffusion runs all its steps on that tiny grid, then the decoder blows the result back up to full size. The same picture, generated at a fraction of the cost.
Denoise in a tiny compressed space, then decode back to full size.
Latent diffusion is why image generation got cheap enough for consumer GPUs (Stable Diffusion, 2022). The catch is the compression: fine details the latent code can't hold — readable small text, faces in a crowd — often come out garbled, no matter how good the prompt.