tokenization
/ TOH-ken-eye-ZAY-shun /
Before a computer can do anything with a sentence, it has to chop the stream of text into bite-sized pieces. Tokenization is that chopping step: turning "I don't like cats." into a list of units the machine can count and look up — say ["I", "do", "n't", "like", "cats", "."]. Each piece is called a token. It sounds trivial, but it is the very first decision in almost every language program, and getting it wrong ripples through everything after.
The hard part is that "a word" is fuzzier than it looks. Do you split "don't" into one token or two? Is "New York" one thing or two? In Chinese and Japanese there are no spaces at all, so the machine must guess where one word ends and the next begins. There is no single correct answer — only choices that suit a particular task. Simple schemes split on spaces and punctuation; smarter ones (see byte-pair encoding) break rare words into reusable fragments so the system is never stumped by a word it has never seen.
Tokenization matters because everything downstream — counting words, building a vocabulary, feeding a neural network — operates on tokens, not raw letters. A model's "vocabulary" is just its fixed list of known tokens. If your tokenizer mangles "COVID-19" into nonsense, no amount of clever modeling afterward can fully recover. It is the quiet plumbing of NLP: invisible when it works, the source of baffling bugs when it doesn't.
Sentence: "Unbelievable!" A whitespace tokenizer gives one token: ["Unbelievable!"]. A subword tokenizer might give ["Un", "believ", "able", "!"], reusing fragments it already knows so the rare word still has meaning.
Same text, two tokenizations — the choice shapes everything downstream.
There is no universally "correct" tokenization — it is a design choice tied to the task and language. Modern systems mostly use subword tokenizers so they never hit a word that is completely unknown.