JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Splitting Text into Subwords

Whole words make the vocabulary explode and still miss new ones. Single characters make sequences endlessly long. Subword tokenization is the elegant compromise — and BPE, WordPiece, and Unigram are how it is learned.

Two bad extremes

Imagine you must pick the vocabulary yourself. Option one: one token per word. Clean, but English alone has millions of words, names, and typos — your list would be gigantic, and the first time a user writes unfriending or *Llama-3* you have no token for it. Option two: one token per character. Now the list is tiny (just letters and symbols), but *"internationalization"* becomes 20 tokens, sequences get very long, and long sequences are slow and expensive for the model to process.

Subword tokenization threads the needle: keep frequent whole words as single tokens (the, cat), but break rare or novel words into reusable pieces (token + ization, Llama + *-* + *3*). Common things are short; nothing is ever truly unknown, because in the worst case you fall back to characters or bytes. This is the scheme every major LLM uses.

Subword tokenization turns text into token ids, which the model then looks up as embedding vectors.

A pipeline: a sentence is split into subword tokens, each mapped to an integer id, then each id mapped to an embedding vector.

BPE: learn the vocabulary by merging

The most common recipe is Byte-Pair Encoding (BPE). You do not write the vocabulary by hand — you learn it from a big pile of text. Start with every word split into single characters, then repeatedly find the most frequent adjacent pair and glue it together into a new token. Each glue-together is recorded as a merge rule, and the ordered list of these is the merges table.

  1. Start: the word "lower" is the characters l · o · w · e · r. In the corpus, the pair (e, r) is very common.
  2. Merge the most frequent pair (e, r) → "er". Record the rule. Now "lower" = l · o · w · er.
  3. Next most frequent pair might be (l, o) → "lo". Record it. "lower" = lo · w · er.
  4. Keep going for thousands of merges until you reach your target vocabulary size. The recorded merges ARE the tokenizer.
(a^\star,\,b^\star)=\operatorname*{arg\,max}_{(a,b)}\;\operatorname{count}(a,b)

Each BPE step merges the single most frequent adjacent pair in the corpus.

At inference time you do the same merges in the same order on new text. Because er and lo are now single tokens, a word the tokenizer never saw — say lowering — still splits cleanly into known pieces. That is the whole magic: a finite list with infinite reach.

WordPiece and Unigram: two other dialects

BPE is not the only scheme. WordPiece (used by BERT) is very BPE-like, but instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training text — a slightly more statistical criterion. It famously marks word-continuation pieces with `##`, so playing becomes `play` + `##ing`.

\mathrm{score}(a,b)=\frac{\operatorname{count}(a,b)}{\operatorname{count}(a)\,\operatorname{count}(b)}

WordPiece instead merges the pair with the highest likelihood ratio, not just the most frequent one.

Unigram works the opposite way round. It starts with a huge candidate vocabulary and trims it down, repeatedly throwing away the tokens that hurt the model's likelihood the least, until the target size is reached. At inference it can consider several ways to split a word and pick the most probable one. BPE builds up; Unigram cuts down — different paths to the same goal of useful subword pieces.

Bytes and SentencePiece: never get stuck

What if the input contains an emoji, a rare Chinese character, or a symbol the tokenizer never learned? Byte-level tokenization solves this once and for all: before learning merges, treat the text as its raw UTF-8 bytes (256 possible values). Since every possible character is some sequence of bytes, there is genuinely no input the tokenizer cannot represent — there are no truly unknown tokens. Modern GPT-style tokenizers are byte-level BPE for exactly this robustness.

SentencePiece is the popular toolkit (used by Llama, T5, and many multilingual models) that packages BPE or Unigram in a language-agnostic way. Its key idea: treat the input as a raw stream of characters including spaces — it encodes the space itself as a visible symbol `▁` — so it works identically for English, Japanese, or Thai, none of which agree on what a "word" even is. No language-specific pre-splitting required.

Picking a vocabulary size

The number of merges you run sets the final vocabulary size, and it is a genuine trade-off. A larger vocabulary means each token covers more text, so sequences are shorter and the model processes a prompt in fewer steps — but the embedding table (next guide) grows, eating memory. A smaller vocabulary is lean but chops text into more, finer pieces, making sequences longer. Production tokenizers usually land somewhere between 32k and 256k as the sweet spot for their target languages.