The Bayesian View & Empirical Bayes

Credibility was Bayes' theorem in disguise

By now the credibility formula feels familiar: blend the policyholder's own experience with the portfolio average, weighting the personal data by Z and the average by (1 minus Z). The earlier guides motivated Z two ways — first by demanding the data be stable enough (limited fluctuation), then by minimising squared error (Bühlmann credibility). Both arrived at a weighted average almost by engineering instinct. This guide reveals the deeper truth: that blend is not an approximation or a clever heuristic. For a wide and important family of models, it is exactly what the laws of probability demand. Credibility is Bayesian updating wearing a working actuary's coat.

Recall Bayes' theorem from the probability rung. It is a rule for changing your mind in the face of evidence. You begin with a prior — your belief about an unknown quantity before seeing this policyholder's data. Here the unknown is the policyholder's own true risk level — call it θ — which is hidden from you because risk is invisible. The prior is what the whole portfolio teaches you about how θ varies from one customer to the next. Then the customer generates some claims; that is the evidence. Bayes' theorem mechanically combines prior and evidence into a posterior — your updated belief about this person's θ now that you have watched them for a while.

Look at what the posterior is. The prior is centred on the portfolio average — that is the manual rate every newcomer starts at. The evidence pulls toward the customer's own observed experience. The posterior settles somewhere between the two, leaning toward the data when you have a lot of it and toward the prior when you have little. That is precisely the credibility blend. So Bayesian credibility is not a rival to what you already learned — it is its foundation. The Z you computed was always the posterior quietly speaking.

The clean conjugate story: Poisson meets gamma

Bayes' theorem is honest but, in general, an arithmetic nightmare — the posterior is an integral that rarely has a tidy form. Yet there is one magical pairing where it stays simple, and it is the textbook example precisely because it is the one you can do by hand. Model each policyholder's annual claim count as Poisson with their own personal rate θ. Across the portfolio, those θ values themselves vary, and we describe that spread with a gamma distribution as the prior. The gamma is the prior shaped to match a Poisson likelihood, and the two were made for each other.

Here is the small miracle. When the prior is gamma and the data are Poisson, the posterior is again a gamma — same family, just with updated parameters. A prior whose form survives the update like this is called a conjugate prior, and the conjugate prior for Poisson data is exactly the gamma. The update is almost absurdly simple: the gamma's two parameters are often nicknamed a 'shape' and a 'rate.' To get the posterior, you add the total number of claims you observed to the shape, and add the number of years you watched to the rate. That is the entire calculation — addition.

Prior belief about a driver's true rate theta:  Gamma(shape=3, rate=2)
   prior mean = shape / rate = 3 / 2 = 1.5 claims per year  (the manual rate)

Observe this driver for 4 years: 2 + 1 + 0 + 1 = 4 claims total

Posterior  = Gamma(shape + claims, rate + years)
           = Gamma(3 + 4, 2 + 4) = Gamma(7, 6)
   posterior mean = 7 / 6 = 1.167 claims per year   <- the new premium rate

It is a credibility blend:  1.167 = Z * (own rate) + (1-Z) * 1.5
   own observed rate = 4 claims / 4 years = 1.0
   1.167 = Z*1.0 + (1-Z)*1.5  ->  Z = 4 / (4 + 2) = 0.667
   Z = years / (years + rate-parameter)  <- exactly the Buhlmann form, k = 2

Stare at the worked numbers and the punchline lands. The posterior mean of 1.167 is literally a weighted average of the driver's own rate (1.0) and the manual rate (1.5), with weight Z = 0.667 — and that Z equals years divided by (years plus a constant). That constant is none other than Bühlmann's credibility constant k. So the exact Bayesian premium and the approximate Bühlmann credibility premium are not merely close; for the Poisson-gamma model they are the identical number. Bühlmann's linear shortcut, it turns out, was never a shortcut at all in this case — it was the truth.

Why this matters — and where it stops being exact

The agreement is reassuring, but be honest about why Bühlmann is still worth knowing. The exact Bayesian posterior is only this clean when the prior and likelihood happen to be conjugate. Swap in a different claim distribution or a less obliging prior and the posterior turns into an integral with no closed form, demanding either heavy numerical work or simulation. Bühlmann's genius was to ask for less: instead of the full posterior, find the best straight-line approximation to it. That linear estimate is computable for any model from just a few moments, and in the conjugate cases it costs you nothing because it happens to be exact.

There is also a subtler honesty point. The whole Bayesian story rests on the prior being right — on the gamma genuinely describing how risk varies across your customers. If the prior is badly chosen, the posterior inherits that flaw no matter how much data arrives (though, mercifully, lots of data eventually swamps a wrong prior). And where, in practice, does a freshly hired actuary obtain a trustworthy prior? Nobody walks in carrying one. That gap — a beautiful theory that needs an input you do not have — is exactly what the final idea of this guide fills.

Empirical Bayes: let the data hand you its own prior

Here is the practical bind. The Bayesian recipe needs a prior — for Bühlmann, that means two structural numbers: the overall portfolio mean, and the constant k, which itself is the ratio of two quantities you met last guide, the expected process variance and the variance of the hypothetical means. A purist would supply these from outside belief. But where would such belief come from? The honest answer for an actuary staring at a fat database is: estimate them from the data itself. That move — using the portfolio to estimate the very prior the portfolio is then judged against — is called empirical Bayes.

The idea is wonderfully down-to-earth, and it is where credibility plugs straight back into the statistics rung. You have hundreds of policyholders, each with a few years of data. Two kinds of variation jostle in that database. Within a single policyholder, claims bounce around year to year purely by chance — that scatter, averaged over everyone, estimates the expected process variance. Between policyholders, the long-run averages genuinely differ because people carry different risk — that spread of the per-person averages estimates the variance of the hypothetical means. The first is noise; the second is signal. Empirical Bayes credibility extracts both straight from the data, no prior belief required.

1. Estimate the portfolio mean — the grand average claim across every policyholder and every year. This is the manual rate that newcomers, and thin data, will be pulled toward.
2. Estimate the expected process variance — average the year-to-year scatter within each policyholder. This is the pure noise that makes any one person's record an unreliable witness.
3. Estimate the variance of the hypothetical means — how much the per-person long-run averages truly differ from one another, after subtracting out the noise above. This is the real risk-to-risk signal.
4. Form k as the ratio (process variance over variance of means), then Z = n / (n + k) for each policyholder, and finally blend their own average with the portfolio mean. The whole credibility machine now runs on nothing but the data you already had.

Honest cautions, and the thread back to statistics

Notice how neatly this closes the loop. The structural quantities are estimated by something very close to the method of moments from the statistics rung — match observed scatter to theoretical variances and solve. The Bayesian premium is a posterior mean, a creature of Bayes' theorem from probability. Credibility is not a strange island of its own; it is probability and statistics applied to the one question that obsesses insurers — how much to trust thin data — and dressed in the working vocabulary of premiums and portfolios. The mathematics you climbed earlier was the ladder up to this room.

So this rung ends where it secretly began. The opening guides framed the question — thin data versus a broad average — and offered a weighted answer that felt almost too convenient. The Bayesian view explained why the answer is right; the conjugate Poisson-gamma case showed it being exactly right; empirical Bayes showed how to run the whole apparatus when, as always, nobody hands you a prior. What you carry forward is a habit of mind that outlasts the formulas: never fully trust a small sample, never fully ignore it, and let the breadth of the portfolio set the exchange rate between the two.