interleaved image-text modelling

Interleaved image-text modelling treats a document as it actually appears in the wild, with images and text woven together in arbitrary order, a paragraph, then a figure, then a caption, then more prose, all as one sequence. Rather than the clean single-image-plus-caption format of early datasets, this matches webpages, manuals, and articles, and learning from it lets a model handle multiple images in context and reason about how text and pictures refer back and forth to one another.

Technically the model ingests a single token stream in which visual tokens, produced by an encoder and resampler or projector, are spliced in at their original positions among the text tokens, and it is trained autoregressively to predict the next text token given everything, images included, that came before. Careful attention masking ties each text span to the images that precede it. Training on large interleaved corpora is what unlocked few-shot multimodal in-context learning in Flamingo and its successors, since the format naturally provides image-text-image-text demonstrations the model can imitate at inference.

Interleaving is the substrate for genuinely multi-image reasoning, comparing two charts, following an illustrated procedure, narrating a sequence, which single-image VLMs cannot do. Its costs are data and length: high-quality interleaved corpora are harder to curate than caption pairs, and packing many images into one sequence strains the context window.