Bayes' theorem

Bayes' theorem is the rule for flipping a conditional probability around. You often know P(evidence given hypothesis) — for instance, how often a test reads positive when someone truly has a disease — but what you actually want is P(hypothesis given evidence): now that the test is positive, how likely is the disease? These two are different, and Bayes tells you exactly how to convert one into the other.

The formula is P(A given B) = P(B given A) P(A) / P(B). It follows in one line from the definition of conditional probability written two ways: P(A and B) equals both P(B given A) P(A) and P(A given B) P(B), so set them equal and divide. The denominator P(B) is usually expanded with the law of total probability over the hypotheses: P(B) = P(B given A) P(A) + P(B given not A) P(not A). Read the parts as belief-updating: P(A) is the prior (your belief before the evidence), P(B given A) is the likelihood (how well the evidence fits the hypothesis), and P(A given B) is the posterior (your updated belief). Bayes says posterior is proportional to prior times likelihood.

This is arguably the single most useful idea in probability: it is how a spam filter weighs words, how a doctor reads a test, how a search algorithm locates a lost submarine, and the foundation of an entire school of statistics. The most important honesty point is that the answer depends heavily on the prior P(A). A very accurate test for a very rare disease can still leave the posterior surprisingly low, because most positives come from the huge healthy majority — ignoring this is the base-rate fallacy.