Painting from Words: Latent Diffusion and Text-to-Image

Working in latent space, not pixels

In Guides 2 and 3 we learned to add noise to an image and then train a U-Net to undo that noise one small step at a time. It works beautifully — but it is slow. To make one picture, the sampler runs the U-Net hundreds or even a thousand times, and each run has to process every single pixel. A modest 512×512 colour image is a grid of 512×512×3 = 786,432 numbers. Pushing nearly a million numbers through a deep network a thousand times, just to produce one image, is enormously expensive — far too heavy for an ordinary graphics card.

The trick that made high-resolution generation affordable is almost embarrassingly simple: don't run diffusion on the pixels at all. Back in Guide 1 we hinted that diffusion would one day borrow a tool from the autoencoder family — here it is. We take a pretrained VAE encoder and use it to squeeze the big image into a small, dense summary called a latent. Instead of a 512×512×3 pixel grid, we now work with something like a 64×64×4 latent. We run the entire forward-and-reverse diffusion process inside this little latent space, and only at the very end do we hand the finished latent to the VAE decoder, which expands it back into full-resolution pixels.

Think of it like editing a compact blueprint instead of repainting an entire mural pixel by pixel. The latent keeps the meaningful structure of the image — shapes, layout, broad textures — while throwing away the redundant fine detail the decoder can faithfully reconstruct anyway. A model built this way, doing all of its diffusion work in a compressed latent rather than in raw pixels, is called a latent diffusion model. The shrink is about 8× per side (512→64), which cuts the number of values the U-Net touches by roughly 48× (786,432 down to 16,384) — and that single change is what let high-resolution image generation run on consumer GPUs. It is the backbone of Stable Diffusion.

Turning a prompt into numbers

So far our denoiser only knows how to clean up noise — it has no idea what we want the picture to be of. To paint from words, we first have to turn those words into something a neural network can consume: numbers. A prompt like 'a corgi wearing sunglasses' starts life as plain text. We first tokenize it — chop it into small pieces called tokens (roughly words or word-fragments) — and then feed those tokens through a pretrained text encoder, such as the text transformer from CLIP.

The text encoder outputs a sequence of embedding vectors: one vector per token. Each vector is a point in a high-dimensional space whose directions have learned to encode meaning — there are directions that capture 'dog-ness', 'corgi-ness', 'wearing', and 'sunglasses'. So our sentence becomes an ordered list of meaning-vectors. Using this text to steer the image generator is called prompt conditioning, and the whole task — describe a scene in words and get back a matching picture — is text-to-image generation.

# Turn a text prompt into a sequence of embedding vectors
prompt = "a corgi wearing sunglasses"

tokens = tokenizer(prompt)            # -> token ids, e.g. [a][corgi][wearing][sun][glasses]
text_embeddings = text_encoder(tokens)
# text_embeddings shape: (num_tokens, embed_dim)
# one meaning-vector per token; the U-Net consumes THIS, never the raw string

Here is the subtle but crucial point: the diffusion U-Net never reads your sentence. It cannot see the letters c-o-r-g-i. It only ever sees these embedding vectors — the numerical fingerprint of your prompt. That raises the obvious question: how does a list of word-vectors actually reach in and change which pixels get painted where? The answer is a mechanism called cross-attention, and it is the heart of the next section.

Cross-attention: where words meet pixels

Attention is a way for one set of things to look up information in another set, using three roles. A query is a question — 'what am I looking for?'. A key is a label that advertises 'here is what I contain'. A value is the actual payload that gets handed over. Each query is compared against every key to see how well they match; the better the match, the more of that key's value the query pulls in. Picture asking a roomful of people a question and listening hardest to whoever answers most relevantly.

Ordinary self-attention has all three roles come from the same source. Cross-attention is the twist that lets words talk to pixels: the queries Q come from the image features flowing through the U-Net, while the keys K and values V come from the text embeddings. Concretely, every spatial location in the half-formed image asks 'which words matter for me?', compares itself against each token, and pulls in the content of the tokens that match best. The patch of latent destined to become sunglasses ends up attending strongly to the 'sunglasses' token and absorbing its meaning. This mechanism is called cross-attention conditioning.

Scaled dot-product attention. In cross-attention, Q is a projection of the U-Net image features; K and V are projections of the text embeddings.

Read it from the inside out. Q is a matrix of queries — one row per image location, each a linear projection of the U-Net's image features asking 'which words matter here?'. K stacks the keys, one row per text token, where each token advertises itself; V stacks the matching values, the text content to be passed along. The product QKᵀ takes the dot product of every image-query with every text-key, giving a grid of raw similarity scores — a big score means 'this location and this word strongly match'. We divide by √dₖ, where dₖ is the dimension of each key vector: without it, large dₖ makes the dot products huge and shoves softmax into a region where its gradients vanish, so √dₖ keeps the numbers in a stable range. The softmax then turns each row of scores into positive weights that sum to 1 — a soft selection of which tokens this location cares about. Finally we multiply by V, blending the chosen text contents in proportion to those weights, and inject the result back into the image features. For example, with dₖ = 64, if a sunglasses-patch's query scores 9.0 against the 'sunglasses' key but only 1.0 against 'corgi', dividing by √64 = 8 gives 1.125 vs 0.125; softmax turns those into about 73% vs 27%, so that patch's update is dominated by the sunglasses value. Crucially, these cross-attention layers are inserted at multiple resolutions inside the U-Net, so the prompt steers both the coarse layout and the fine detail — this is the engine of prompt conditioning.

Classifier-free guidance: making prompts stick

We can now feed a prompt into the denoiser through cross-attention — but in practice that alone often isn't enough. A model that is merely conditioned on the prompt tends to treat it as a loose suggestion: ask for 'a corgi wearing sunglasses' and you might get a generic dog with the sunglasses forgotten. We need a way to dial up how strongly the prompt is obeyed. That technique is classifier-free guidance.

The setup is clever. During training, we randomly drop the prompt some fraction of the time (say 10–20%), replacing it with an empty/null prompt. As a result the very same network learns two jobs at once: a conditional noise predictor εθ(x, c) that uses the prompt c, and an unconditional one εθ(x, ∅) that predicts noise with no prompt at all. No separate classifier is needed — hence 'classifier-free'. At sampling time we run both predictions and then extrapolate away from the unconditional one toward the conditional one. This is classifier-free guidance.

Classifier-free guidance combines the conditional and unconditional noise predictions, pushing w times further along the prompt's effect.

Unpack the formula. εθ(x_t, c) is the noise the network predicts when it can see the prompt c at noisy step x_t; εθ(x_t, ∅) is what it predicts with the prompt removed. Their difference, εθ(x_t, c) − εθ(x_t, ∅), is exactly the part of the prediction the prompt is responsible for — the direction in which mentioning the prompt nudged the model. The guidance weight w controls how far we travel along that direction: we start from the unconditional prediction and add w times the prompt's effect. The intuition is simply 'amplify whatever the prompt changed'. Note w = 1 gives back ordinary conditioning — the formula collapses to εθ(x_t, c) — while w > 1 exaggerates the prompt. The cost is that every denoising step now needs two network evaluations instead of one, conditional and unconditional. This is the heart of strong prompt conditioning.

The guidance scale knob

If you have ever used an image generator and seen a slider labelled 'guidance scale', 'CFG scale', or just 'prompt strength', you have already met w. It is the same w from the previous formula, and it is the single most-tweaked hyperparameter in text-to-image. Turning it changes the balance between two things you want but cannot fully have at once: faithfulness to the prompt, and natural-looking variety.

Picture sweeping w while keeping everything else fixed. At low w (about 1–3) the unconditional and conditional predictions are barely separated, so images are diverse and natural but follow the prompt only loosely — details drift or go missing. At medium w (about 7–8) you hit the sweet spot most tools default to: the prompt is clearly obeyed while the picture still looks believable. At very high w (above ~15) the model overshoots the prompt direction: colours over-saturate, hard edges and artifacts creep in, and diversity collapses as every sample is shoved toward the same exaggerated interpretation. This dial is the guidance scale.

# Change ONLY the guidance scale and compare results
prompt = "a corgi wearing sunglasses"
for w in [1.5, 3, 7.5, 15, 30]:
    image = generate(prompt, guidance_scale=w, seed=42)  # fix the seed to isolate w
    save(image, f"corgi_w{w}.png")
# expect: low w -> loose/varied, ~7.5 -> sweet spot, high w -> over-saturated artifacts

Because it is the same w that multiplies the prompt direction in the guidance formula, the cleanest way to build intuition is to change only w and watch the fidelity-versus-diversity trade move. Two caveats: w interacts with the rest of the recipe. A very strong or wordy prompt may need a lower w to avoid over-cooking, and the number of sampling steps matters too — high guidance with very few steps tends to amplify artifacts. Tune w first, then revisit it after you change the prompt or step count. This whole knob only exists because of classifier-free guidance.

Stable Diffusion: the whole pipeline assembled

We now have every piece. Let's assemble them into the full machine behind Stable Diffusion and trace a single image from prompt to pixels. You built each part yourself across Guides 1–4, so every step should now read like a familiar face.

1. Encode the prompt: tokenize the text and run it through the pretrained text encoder to get a sequence of embedding vectors — one meaning-vector per token (Section 2).
2. Start from noise: sample a random latent — pure Gaussian noise, but in the small latent space (e.g. 64×64×4), not in pixels (Section 1).
3. Denoise with guidance: run the conditioned U-Net for N steps with a DDIM-style sampler. At every step, inject the prompt via cross-attention and combine the conditional and unconditional predictions with classifier-free guidance at scale w (Sections 3–4).
4. Decode to pixels: hand the final clean latent to the VAE decoder, which expands it into the full-resolution image (Section 1).

def stable_diffusion(prompt, w=7.5, steps=50):
    # 1) Encode the prompt (and an empty prompt for classifier-free guidance)
    c     = text_encoder(tokenizer(prompt))
    empty = text_encoder(tokenizer(""))

    # 2) Start from pure noise IN LATENT SPACE (e.g. 64x64x4), not pixels
    x = sample_gaussian_noise(shape=(64, 64, 4))

    # 3) Iteratively denoise with cross-attention + classifier-free guidance
    for t in ddim_schedule(steps):
        eps_cond   = unet(x, t, context=c)       # prompt injected via cross-attention
        eps_uncond = unet(x, t, context=empty)   # second pass: no prompt
        eps = eps_uncond + w * (eps_cond - eps_uncond)
        x = ddim_step(x, eps, t)                  # one reverse step (Guide 3's sampler)

    # 4) Decode the finished latent back to full-resolution pixels
    image = vae_decoder(x)
    return image

Two names are worth keeping straight. A latent diffusion model is the architecture — the general recipe of running diffusion inside a VAE latent with text injected by cross-attention. Stable Diffusion is the famous open-weights model built with that recipe. Together they are the workhorse of modern text-to-image generation.