byte-level tokenization

What happens when a model meets a character it has never seen — a rare emoji, an unusual script, a corrupted byte? Byte-level tokenization makes sure this never breaks by working from the raw bytes underneath the text. Since every character on a computer is ultimately stored as a sequence of bytes, and there are only 256 possible byte values, every conceivable input can be represented with no exceptions and no special unknown token.

Tokenizers like GPT-2's byte-level byte-pair encoding first turn text into its UTF-8 bytes, then learn merges over those bytes. The base vocabulary of 256 bytes guarantees coverage: anything, from an ancient script to raw binary, can be spelled out, in the worst case one byte at a time. The cost is that unusual text fragments into many tokens, which is exactly why some non-English languages and long emoji sequences are surprisingly expensive to process.