latent diffusion model

A latent diffusion model (LDM) makes diffusion affordable by refusing to do the hard work on raw pixels. Running a full denoising chain directly on a 512x512x3 image means the network must process roughly 800,000 numbers at every one of dozens of steps — enormously expensive. The LDM insight is that most of those pixels are perceptually redundant: a separate autoencoder can compress the image into a small latent grid that keeps the meaningful structure and discards imperceptible detail, and you run the entire diffusion process in that compact space instead.

Concretely an LDM has two stages. First, a variational autoencoder (VAE) is trained to encode an image into a latent tensor — for example compressing 512x512x3 down to 64x64x4, an ~48x reduction in spatial elements — and to decode that latent back into a faithful image. This VAE is trained once and then frozen. Second, a normal diffusion model (forward noising, learned reverse denoising, the same DDPM/DDIM machinery) is trained entirely on those latents. To generate, you sample a latent from noise via the diffusion reverse chain, then run it through the VAE decoder to get pixels. The split is deliberate: the VAE handles high-frequency perceptual detail (sharp edges, texture) where diffusion is wasteful, and the diffusion model handles the semantic, compositional content where it excels.

This factorization is what made high-resolution, text-conditioned generation practical on consumer hardware and is the architecture underneath Stable Diffusion. It also makes conditioning cheap: because the denoiser is smaller, you can afford cross-attention layers that inject text embeddings at every block. The main caveat is that the VAE imposes a quality ceiling and characteristic failure modes — fine text, tiny faces, and intricate textures can be smeared by the encode/decode round-trip regardless of how good the diffusion model is, because that information was thrown away before diffusion ever ran.