Bayesian neural network
/ BAY-zee-un NOOR-ul NET-wurk /
A Bayesian neural network is a neural network that, instead of committing to one fixed value for each of its internal weights, keeps a probability distribution over them — a sense of 'this weight is probably around here, but I'm not certain.' An ordinary network is like one expert giving one firm answer; a Bayesian neural network is like a whole committee of plausible networks voting, where their agreement or disagreement becomes a built-in measure of confidence.
The payoff is principled uncertainty. When all the committee's networks agree on an input, the prediction comes with high confidence; when they scatter — especially on inputs unlike anything seen in training — the spread of their votes signals genuine doubt. This is meant to fix the dangerous overconfidence of standard deep networks, which will happily slap a 99% label on something utterly foreign to them. A Bayesian neural network aims to say 'I don't know' when it truly doesn't.
Why it matters and where reality bites: in principle this is the gold-standard way to get trustworthy uncertainty from deep learning, valuable wherever a confident wrong answer is costly. In practice, doing it exactly is computationally impossible for real networks, so every method is an approximation — and a coarse one. The popular shortcuts (like keeping 'dropout' on at prediction time, or training an ensemble of ordinary networks) give a useful but imperfect estimate, and their uncertainty can still be miscalibrated. A Bayesian neural network is a serious, honest attempt at knowing what it doesn't know — not a solved guarantee of it.
A digit recognizer trained only on clean handwritten numbers is shown a scribble that's not a digit at all. An ordinary network might confidently shout '8, 97% sure'. A Bayesian neural network's committee of weight-settings disagree wildly on this unfamiliar input — some say 8, others 3, others 5 — and that scatter produces a low, honest confidence, flagging the input as one it shouldn't trust itself on.
On unfamiliar input, the 'committee' of weight settings disagrees — and that disagreement is the honest uncertainty signal.
Exact Bayesian inference over a real network's millions of weights is computationally out of reach, so every Bayesian neural network in practice is an approximation. The uncertainty it reports is therefore an estimate that can itself be miscalibrated — better than a plain network's false confidence, but not a guarantee.