stable diffusion

Stable Diffusion is the open-weights latent text-to-image system, first released in 2022 by Stability AI with CompVis and Runway, that put high-quality image generation on ordinary GPUs and into the hands of the public. Its significance is as much sociological as technical: because the weights were openly released, an entire ecosystem of fine-tunes, plugins (LoRA adapters, ControlNet), and tools grew around it, in contrast to closed systems like DALL-E or Midjourney accessible only through an API.

Architecturally Stable Diffusion is a latent diffusion model with three cooperating components. A VAE encodes images to and decodes them from a compact latent space. A denoising U-Net runs the diffusion reverse process in that latent space, predicting noise at each step. And a frozen text encoder turns the prompt into embeddings that condition the U-Net via cross-attention. At generation time the text encoder reads the prompt, the U-Net iteratively denoises a random latent under classifier-free guidance toward that prompt, and the VAE decoder converts the final latent into a picture. SD 1.x used OpenAI's CLIP text encoder; SD 2 switched to OpenCLIP; SDXL (2023) scaled up the U-Net, used two text encoders concatenated for richer prompt understanding, and added a refinement stage.

Understanding which component does what is the key to using and debugging the system. Prompt not being followed? That is the text encoder and cross-attention. Output blurry or low-detail at small sizes? Often the VAE or generating below the resolution the model was trained for. Want pose or edge control? Attach a ControlNet to the U-Net. Want a new style cheaply? Fine-tune with a LoRA on the U-Net attention layers. The clean separation of VAE, U-Net, and text encoder is precisely what makes Stable Diffusion so modular and hackable.