TF-IDF
/ TEE-eff-EYE-dee-eff /
TF-IDF is a simple, beautifully effective recipe for scoring how important a word is to a particular document within a larger collection. The intuition: a word matters to a document if it appears often there but rarely elsewhere. "The" appears constantly in every document, so it's worthless for telling them apart. "Mitochondria" appears a lot in one biology article and almost nowhere else — so it's a strong fingerprint of what that document is about.
The name spells out the two halves you multiply together. TF — term frequency — counts how often a word appears in this document: more mentions, more weight. IDF — inverse document frequency — looks across the whole collection and downweights words that show up in many documents: the rarer a word is across the collection, the more distinctive it is. Multiply them and a word scores high only when it is both frequent here and rare overall. This automatically squashes common filler (handling stop words gracefully without a hand-built list) and lifts the words that actually characterize each document.
For decades TF-IDF was the backbone of search engines and document classifiers, and it's still a strong, fast, transparent baseline today — easy to compute, easy to explain. Its honest limits are the same as bag-of-words, on which it builds: it treats words as separate symbols, so it can't tell that "car" and "automobile" mean the same thing, and it ignores word order entirely. Embeddings capture meaning that TF-IDF cannot — but TF-IDF remains a remarkably sturdy first thing to try.
In a collection of news articles, "said" appears in nearly all of them, so its IDF is tiny and its TF-IDF score stays low everywhere. "earthquake" appears heavily in one article and rarely elsewhere — high TF there, high IDF overall — so it scores high and flags that article's topic.
High TF-IDF = frequent here, rare overall = a word that characterizes this document.
TF-IDF down-weights ubiquitous words automatically, so it handles stop words without a manual list — but it still treats words as separate symbols, blind to synonyms and word order.