tokenization pitfalls

Tokenization stays invisible until it suddenly bites. Because the model sees tokens rather than letters, it struggles with tasks that need character-level awareness: counting the r's in "strawberry", reversing a string, or spelling a word that happens to be a single opaque token. Numbers are a notorious case — "1234" might split as "12" plus "34", so arithmetic gets tangled not by bad reasoning but by where the digits were chopped.

Other quirks share the same root cause. A trailing space, a capital letter, or an unusual Unicode form can change which tokens appear and quietly shift the answer. Rare strings shatter into many tokens, inflating cost and occasionally triggering strange behaviour from so-called glitch tokens that were almost never seen in training. None of this means the model is stupid; it literally never sees the underlying characters. Knowing how the tokenizer works explains a surprising number of baffling failures.