multimodal tokenization

Tokenization is how any input gets chopped into the discrete units a transformer reads. For text it splits a sentence into subword pieces. Multimodal tokenization extends the same idea to everything else: how do you cut an image, a sound, or a video into a clean sequence of tokens the model can consume? Getting this right is the hinge on which every multimodal model turns.

There are two broad styles. Continuous tokenization keeps each image patch or audio frame as a real-valued feature vector from an encoder, projected next to word embeddings — flexible, and standard for understanding tasks. Discrete tokenization instead quantises the signal into entries from a fixed codebook, using a learned tokenizer like a VQ autoencoder, so a picture literally becomes a string of integer tokens just like text — which is what lets one model both read and generate across modalities with shared machinery.

Every scheme trades detail for token count. Too few tokens and fine information is lost; too many and a single image swamps the context window and inflates cost — so resolution, patch size, and codebook design are constant, consequential knobs.