Q-Former (querying transformer)

The Q-Former is the lightweight bridge that BLIP-2 places between a frozen image encoder and a frozen language model so the two never-jointly-trained giants can cooperate. It carries a small set of learned query vectors, on the order of thirty-two, which cross-attend into the frozen visual features and come out the other side as a compact bundle of tokens that summarise the image in a form the language model can read. Only this little module is trained, which makes building a capable VLM remarkably cheap.

It is a small transformer whose queries do double duty across two stages. In the first, representation-learning stage it is trained on the frozen image features with three objectives at once, image-text contrastive matching, image-grounded text generation, and image-text matching, sharing the query tokens but masking attention differently per objective so the queries learn to extract language-relevant visual information. In the second stage its output queries are linearly projected and prepended to the language model's input as soft visual prompts, and only the Q-Former and that projection are tuned.

The querying design is a direct descendant of the Perceiver latent bottleneck and the Flamingo resampler: a fixed, small number of queries decouples the language model's cost from image resolution. Its limitation is exactly that compression, since thirty-two tokens cannot preserve dense text or fine spatial detail, which is why later VLMs often abandon it for higher-resolution patch projections.