prompt conditioning

Prompt conditioning is the whole pipeline by which your typed words become a control signal the diffusion model can use. It is broader than cross-attention (the injection mechanism) — it starts the moment you write the prompt and includes how the text is tokenized, encoded into embeddings, and fed in, plus the practical craft of writing prompts that get what you want. The conditioning is what turns an unconditional 'generate any plausible image' model into a controllable 'generate this specific thing' model.

Mechanically, the prompt is split into tokens (sub-word pieces) and passed through a frozen text encoder — CLIP's text transformer in Stable Diffusion 1.x, OpenCLIP in SD2, often two encoders in SDXL, and large language models like T5 in systems such as Imagen. The encoder outputs a sequence of embedding vectors (for example 77 vectors for CLIP's fixed context length) that summarize the prompt's meaning. These embeddings condition the denoiser, normally through cross-attention. Because the model was trained to associate captions with images, embeddings of unseen prompts compose meaningfully — which is why novel combinations like 'a teapot made of glass jellyfish' can work.

A crucial, widely used extension is the negative prompt: a second prompt describing what you do not want, plugged in as the unconditional branch of classifier-free guidance instead of the empty embedding. Guidance then pushes away from the negative prompt while pushing toward the positive one — say 'blurry, extra fingers, watermark' as a negative to suppress common artifacts. Prompt design as a craft (ordering, emphasis weighting, style tokens, negatives) exists because the conditioning is powerful but blunt: the model attends to what is salient in the embeddings, so what you emphasize and how you phrase it materially changes the result.