text-to-image generation

Text-to-image generation is the task of synthesizing a brand-new image that matches a natural-language description — type 'a watercolor fox reading a book under a streetlamp' and get a picture of exactly that, one that has never existed before. It is not retrieval (finding an existing matching photo) and not editing (modifying a given image); it is generation from scratch, conditioned on language. The hard part is aligning two very different spaces: the discrete, compositional structure of words and the continuous, high-dimensional space of plausible images.

Modern systems solve this with two pieces working together. A text encoder (such as CLIP's text tower, or a large language model like T5 in some systems) maps the prompt into a sequence of embedding vectors capturing its meaning. A conditional generative model — today almost always a diffusion model — then produces an image steered by those embeddings, typically via cross-attention and amplified by classifier-free guidance. The text encoder's quality largely determines compositional understanding (does the model grasp 'a red cube on top of a blue sphere'?), while the generator's quality determines visual fidelity. The whole system is trained on hundreds of millions of image-caption pairs scraped from the web.

Capabilities are now striking: photorealistic scenes, coherent art styles, novel combinations of concepts the model never saw paired in training. But the failure modes are equally characteristic and worth knowing. Legible text within images is often garbled. Counting and exact spatial relations are unreliable ('three cats' may give two or four). Compositional binding fails ('a red ball and a blue box' may swap colors). Hands, fine anatomy, and reflections are classic weak spots. Outputs inherit the biases and content of web training data, and there are real questions about copyright, consent, and provenance. Knowing both the power and these reliable cracks is essential to using these systems well.