Bayesian thinking

Two philosophies of probability

Imagine I flip a fair coin, slap it onto the back of my hand, and cover it before either of us can look. What is the probability it shows heads? Most people happily answer “50%.” But pause for a second: the coin has already landed. It is definitely heads or definitely tails — there is nothing random left about it. So in what sense is the answer 50%? That little flicker of discomfort is the doorway to the two great philosophies of probability.

The first view, called frequentist, says a probability is a long-run frequency: the fraction of times an event happens if you could repeat the situation forever. “The coin is 50% heads” means that over millions of flips, about half come up heads. This is the view quietly assumed by everything in this track so far — p-values, confidence intervals, and the whole machinery of hypothesis testing. It is powerful, but it has an awkward edge: it struggles to talk about a one-off event that is already decided, like the coin under my hand, or “the probability this particular drug works.”

The second view, called Bayesian (after the 18th-century minister Thomas Bayes), says a probability is a degree of belief — a number between 0 and 1 measuring how confident you are, given everything you currently know. Under this view, “50% heads” is a perfectly sensible statement about the covered coin: you have no information favoring either side, so your belief splits evenly. Crucially, belief is something you update as evidence arrives. That is exactly how humans reason in everyday life — you start with a hunch and revise it the moment you see new facts.

This guide builds the Bayesian view from one concrete medical example, names its three moving parts — the prior, the likelihood, and the posterior — shows how beliefs update as data streams in, and finally delivers the payoff: an interval you are actually allowed to read the natural way. Along the way we will be honest about the costs, because the Bayesian approach asks something of you that the frequentist approach does not.

Bayes' theorem with a real medical-test example

Here is a puzzle that has fooled doctors in published studies, so do not feel bad if it catches you too. A certain disease affects 1 person in 1,000. There is a test for it that is very good: if you have the disease, it comes back positive 99% of the time (this is called the sensitivity). If you do not have the disease, it correctly comes back negative 95% of the time — meaning it gives a false positive just 5% of the time. You take the test. It is positive. What is the probability you actually have the disease?

Most people's gut says something like “99%” or “95%” — the test is accurate, after all. The real answer is about 2%. To see why, forget formulas and just count people. This trick — turning probabilities into imagined head-counts — is called using natural frequencies, and it makes Bayesian reasoning almost obvious.

1. Start with a crowd: imagine 100,000 people who all take the test.
2. Apply the base rate: 1 in 1,000 have the disease, so 100 people are sick and 99,900 are healthy.
3. Test the sick: 99% of the 100 sick people test positive → 99 true positives.
4. Test the healthy: 5% of the 99,900 healthy people test positive → 4,995 false positives.
5. Count all positives: 99 + 4,995 = 5,094 people see a positive result.
6. Read off the answer: of those 5,094, only 99 are truly sick, so the chance you are sick given a positive test is 99 ÷ 5,094 ≈ 0.019, about 1.9%.

The surprise comes from the base rate — the background rate of the disease before any test, here 1 in 1,000. Because the disease is so rare, the few real cases are swamped by the vastly larger pool of healthy people, and even a small 5% false-positive rate applied to that huge healthy group produces far more positives than the disease itself does. The test did move your belief — from 0.1% up to 1.9%, a nearly twentyfold jump — but it started from such a low base that the result is still reassuringly small. (In practice this is exactly why one positive screening test is followed by a second, more specific confirmatory test.)

What we just did by counting is precisely Bayes' theorem. It is the rule for flipping a conditional probability around: we knew the probability of a positive test given the disease, but we wanted the probability of the disease given a positive test. In symbols, writing D for “has the disease,” ¬D for “does not,” and + for a positive result:

Bayes' theorem. The numerator is the path to a positive test through the disease; the denominator is every path to a positive test (through the disease plus through false positives).

The same calculation with the numbers plugged in — it returns the 1.9% we found by counting people. The formula and the head-count are two faces of one idea.

Prior, likelihood, posterior

The medical example already contains the three ingredients of every Bayesian calculation; we just need to name them. Each is a probability, and Bayes' theorem is simply the recipe that combines them.

The prior is what you believe before seeing the new data — here, the base rate of 0.1%. It encodes background knowledge: how common the disease is, what past experiments found, what a domain expert expects. The prior is where the Bayesian approach is both most powerful and most controversial, because two reasonable people can bring two different priors to the same problem.

The likelihood is how probable the observed data is under each hypothesis you are considering. Here it is the test's behavior: a positive result is 99% probable if you are sick and 5% probable if you are healthy. Note the subtlety — the likelihood is read as a function of the hypothesis, not of the data. It asks, “If this hypothesis were true, how expected would what I actually saw be?”

The posterior is your updated belief after combining prior and data — the 1.9% answer. It is the whole point: a single probability about the very thing you care about, ready to feed into a decision. And, as the next section shows, today's posterior becomes tomorrow's prior.

Posterior is proportional to likelihood times prior. The symbol ∝ means “proportional to” — we drop the bookkeeping constant (the total probability of the data) that just rescales everything to add up to one.

Read that proportionality as a plain sentence: your belief in a hypothesis after seeing the data equals your belief before, multiplied by how well the hypothesis predicted the data. A hypothesis is rewarded when it expected what actually happened and penalized when it did not. That single line is the engine of all Bayesian inference.

Updating beliefs as data arrives

Bayes' theorem is not a one-shot trick; it is an engine you can run again and again. The rule is beautifully simple: today's posterior becomes tomorrow's prior. Each new batch of data updates your belief, and that updated belief is the starting point for the next batch. This is how a Bayesian learns continuously, rather than waiting for one big analysis at the end.

Suppose you run a website and want to estimate the click-through rate of a new button — the fraction of visitors who click it. Your old button converted at about 5%, so it is reasonable to start from a prior belief centered near 5% rather than pretending you know nothing. Then visitors arrive. Out of the first 200, 14 click — an observed rate of 7%. What should you now believe the true rate is?

A pure frequentist would report the raw 7% (the point estimate) with a confidence interval around it. The Bayesian blends the 7% from the data with the 5% from the prior. The result lands in between — pulled toward 7% by the evidence, but anchored near 5% by prior experience — and the more data you collect, the more the data wins. With a standard prior worth “about 100 visits of experience,” the posterior mean works out to roughly 6.3%.

Blending prior and data for a rate. The prior acts like α prior “successes” and β prior “failures” (here 5 and 95, a 5% rate from 100 imagined visits); the data adds k = 14 clicks out of n = 200. The posterior, 6.3%, sits between the prior 5% and the observed 7%.

from scipy.stats import beta
# Prior Beta(5, 95): like having already seen 5 clicks in 100 visits (about 5%)
a, b = 5, 95
clicks, visits = 14, 200          # the new data
a_post = a + clicks
b_post = b + (visits - clicks)
print(beta.mean(a_post, b_post))               # 0.0633...  posterior click-rate
print(beta.ppf([0.025, 0.975], a_post, b_post)) # 95% credible interval

Two honest lessons fall out of this. First, with lots of data the prior washes out: after a million visitors your starting hunch barely matters, and the Bayesian and frequentist answers nearly coincide. Second, with little data the prior matters a great deal — which is a feature when your prior is good (it stops you over-reacting to a tiny, noisy sample) and a bug when your prior is bad (it can drag a correct signal toward a wrong preconception). We return to this honesty shortly.

Credible intervals vs confidence intervals

Now the payoff that makes all this worth it. Recall the most common misunderstanding in statistics: people see a 95% confidence interval like “the rate is between 4.1% and 6.3%” and conclude “there is a 95% probability the true rate is in that range.” For a confidence interval, that reading is simply wrong. The frequentist 95% describes the procedure: if you repeated the whole study many times, about 95% of the intervals it generates would contain the true value. For your one particular interval, the true value is either in it or not — there is no probability left to assign.

The Bayesian interval — called a credible interval — gives you exactly the reading everyone wanted all along. A 95% credible interval is a range holding 95% of the posterior probability, so you are genuinely entitled to say, “Given my prior and the data, there is a 95% probability the true rate lies between these two numbers.” It is a direct probability statement about the unknown quantity itself, not about a repeated procedure.

A 95% credible interval, stated honestly: the posterior probability that the unknown parameter θ lies in [a, b] is 0.95. This is the sentence people wrongly attach to a confidence interval — and rightly attach to a credible one.

Two honest caveats keep this from being a free lunch. First, that clean interpretation is bought with a prior — change the prior and you change the interval, especially when data is scarce. The frequentist interval needs no prior, which is precisely why some people prefer it. Second, with plenty of data and a mild prior, the 95% credible interval and the 95% confidence interval are often numerically almost identical; they just carry different warranties on the label. The two terms to keep straight are the credible interval (Bayesian — a probability about the parameter) and the confidence interval (frequentist — a property of the procedure).

Strengths, costs, and when to go Bayesian

So when should you reach for the Bayesian tool? Its strengths are real. It hands you a probability statement you can act on directly. It lets you fold in genuine prior knowledge instead of throwing it away. It is graceful with small samples, shrinking wild estimates toward something sensible. It updates naturally as data streams in. And it gives you a full posterior distribution, which is far richer for decision-making than a single estimate plus a yes/no test — you can read off the probability that the effect exceeds any threshold you care about, such as “probability the new button beats the old by at least 1 point.”

The costs are equally real, and pretending otherwise is how Bayesian analysis earns a bad name. You must choose a prior, and that choice is a judgment others can challenge — in an adversarial or regulated setting (a drug trial, a court case) “I assumed the effect was probably small” may be unacceptable. The computation can be hard: realistic models need numerical methods such as MCMC (Markov chain Monte Carlo — a way of drawing samples from a posterior too complicated to write down neatly), which take care to run and to check. And the results take more words to communicate honestly than a single p-value does.

1. Reach for Bayesian when you have real, defensible prior information you would be foolish to ignore.
2. Reach for Bayesian when data is scarce or arrives sequentially and you want to update as you go.
3. Reach for Bayesian when the decision needs a direct probability — “how likely is it that B beats A by at least 2%?”
4. Lean frequentist when you must avoid any argument about priors — regulated trials, audits, adversarial reporting.
5. Lean frequentist when a simple, well-understood test already answers the question and a prior would add only controversy.
6. Either way, state your assumptions out loud — the prior for Bayesians, the model and stopping rule for everyone.

It is worth knowing that the two worlds touch. The frequentist method of maximum likelihood — picking the hypothesis under which the data was most probable — is the same as finding the peak of a Bayesian posterior when the prior is flat. In that sense maximum likelihood is Bayes with the prior switched off. Many staple methods, including the bootstrap and regularized regression, have natural readings in both languages.

Wrapping up the inference toolkit

You have now seen the full arc of statistical inference, so let us connect the dots. It began with a problem: we can never measure a whole population, only a sample. The central limit theorem told us that averages of samples behave predictably, which let us attach a standard error to an estimate. From there we built confidence intervals to express our uncertainty, and hypothesis tests with their p-values to decide whether an effect is more than noise. This guide added the Bayesian alternative: start from a prior, update with the likelihood, and read off a posterior — and the honest, intuitive credible interval that comes with it.

Underneath every one of these tools is a single idea: reasoning carefully about uncertainty when all you have is a sample of a larger, noisier world. The frequentist and Bayesian schools are two dialects of the same conversation. Fluent data scientists switch between them without fuss, choosing whichever makes the next decision clearest and most defensible.

1. Want to express how uncertain an estimate is? Report an interval, not just a point — confidence or credible.
2. Want to know if an effect is real? A hypothesis test or an interval that excludes “no effect” both work; prefer the interval, because it also shows the size.
3. Have strong prior knowledge or streaming data? Go Bayesian and update as you go.
4. Need a result no one can accuse of a biased prior? Stay frequentist.
5. Always ask the last question: is the effect big enough to matter, not merely big enough to detect?

That is the core toolkit. The next tracks put it to work: designing trustworthy experiments and A/B tests, fitting and validating regression models, and — hardest of all — making honest causal claims about what actually causes what. You now hold the language of uncertainty that every one of them speaks. Onward.