neural-linear models

Full Bayesian inference over a deep net is hard; over a linear model it is closed-form. Neural-linear models get the best of both by drawing the line in between: treat the deep network as a fixed feature extractor that maps inputs to a learned representation, then do exact Bayesian linear regression or classification on just that last layer. The deep part supplies expressive features; the Bayesian last layer supplies cheap, principled uncertainty.

Train the network normally, freeze all but the final layer, and treat the penultimate-layer activations as basis functions. With a Gaussian prior on the last-layer weights you get a conjugate Bayesian linear model whose posterior and predictive variance are available in closed form. This is equivalent to a Gaussian process with a kernel learned by the network, and it is the same object that the linearized Laplace and deep-kernel methods build toward.

It is fast, exact where it counts, and a strong baseline for Bayesian optimization and contextual bandits. The catch is that the features are point-estimated rather than Bayesian, so the uncertainty ignores ambiguity in the representation itself; if the features collapse or are poorly suited to a new region, the last-layer uncertainty can be confidently wrong.