Perceiver architecture

The Perceiver is a transformer designed to swallow inputs of any size and any modality, images, audio, point clouds, video, without the quadratic blow-up that ordinary self-attention suffers when the input is huge. Its idea is to keep a small, fixed array of learned latent vectors as the working memory, and let that small array repeatedly cross-attend into the enormous input, distilling the high-dimensional data into a compact representation it can then reason over with cheap self-attention.

Architecturally each block is a cross-attention from the latents (as queries) into the byte array of inputs (as keys and values), followed by a stack of latent self-attention layers, with these blocks repeated and often weight-shared. Because the expensive cross-attention is linear in the number of inputs rather than quadratic, the cost is decoupled from input length: a thousand or a million pixels are handled by the same fixed-size latent bottleneck. Perceiver IO extends this with a learned output-query array so the model can emit structured outputs of arbitrary shape.

This latent-bottleneck pattern is the conceptual ancestor of the resampler in Flamingo and the Q-Former in BLIP-2, which use a handful of learned queries to compress a variable number of image patches into a fixed token budget before handing them to a language model. The trade-off is that the bottleneck can discard fine detail if the latent array is too small for the task.