naive Bayes
/ nah-EEV BAYZ /
Naive Bayes is a fast, surprisingly effective classifier built on a 250-year-old rule for updating beliefs with evidence. The classic job is spam filtering: each word in an email is a clue. The word "viagra" nudges your belief toward spam; the words "meeting" and "invoice" nudge it back toward legitimate. Naive Bayes tallies up all these little nudges and asks: given this exact bag of words, which category — spam or not — was more likely to have produced it?
It rests on Bayes' theorem, the formula for flipping a probability around: it knows from training data how often "viagra" appears in spam, and uses that to reason backward about how likely an email is spam given that it contains "viagra." The "naive" part is a deliberately crude shortcut: it pretends every clue is independent of the others — that seeing "free" tells you nothing about whether "money" also appears. This is plainly false in real language, yet the model works astonishingly well anyway, because for picking the single most likely category, the rough probabilities are usually good enough.
Its virtues are speed, simplicity, and a hunger for very little training data — it was the backbone of early spam filters and remains a strong baseline for text classification. Where it struggles is wherever its independence pretense bites hard, or when you need the output probability itself to be accurate rather than just the winning category: because it double-counts correlated clues, its confidence numbers are often wildly overstated, even when the final label is right.
Training data shows "prize" appears in 30% of spam but only 1% of real mail. A new email contains "prize," "winner," and "claim" — each a word far more common in spam. Naive Bayes multiplies these clues (pretending they're independent) and concludes: spam, with overwhelming odds.
Each word casts a vote; the "naive" leap is treating the words as unrelated.
The independence assumption is almost always false — words travel in correlated company — yet the classifier still works because the right category often wins even on rough math. Trust its label more than its probability: those confidence percentages are typically overconfident.