text-to-video generation

Text-to-video generation extends text-to-image from a single still to a moving clip: from a prompt the model must synthesise a sequence of frames that are not only individually plausible but coherent over time, so the same dog runs across the same yard with consistent lighting and a believable trajectory rather than flickering into a different dog each frame. Temporal consistency, not per-frame quality, is the hard part.

Most modern systems are video diffusion models operating in a learned latent space. A spatiotemporal autoencoder compresses the clip into latent tokens, and a transformer or U-Net denoiser, equipped with temporal attention or 3D convolutions in addition to the usual spatial layers, learns to denoise the whole sequence conditioned on the text. Treating the video as a sequence of spacetime latent patches and denoising them jointly, as in transformer-based diffusion, is the approach behind the strongest current models. Frame rate, length, and resolution are usually extended by cascaded or interpolation stages.

The frontier challenges are length and physics: maintaining identity and motion over many seconds strains memory and consistency, and the models learn appearance far better than dynamics, so objects can morph, limbs can multiply, and intuitive physics is routinely violated. Evaluation is also unsettled, since no single metric captures temporal coherence, prompt fidelity, and motion realism together.