deep kernel learning (DKL)

Gaussian processes give beautiful, calibrated uncertainty but rely on a kernel, a similarity measure, that you usually have to hand-design, and standard kernels work poorly on raw images or text. Deep kernel learning lets a neural network learn that similarity measure: pass inputs through a deep net to get a representation, then apply a standard GP kernel in that learned space. The net supplies flexible features; the GP supplies principled uncertainty on top.

Formally the kernel becomes a base kernel evaluated on deep-network embeddings of the two inputs, and all of it (network weights, base-kernel hyperparameters, observation noise) is trained jointly by maximizing the GP marginal likelihood, with scalable GP machinery such as inducing points and KISS-GP to handle large data. Wilson and colleagues introduced it as a way to get the non-parametric uncertainty of a GP with the representational power of a deep net.

It marries calibrated GP uncertainty with learned features and is closely related to neural-linear models, which are the last-layer-only special case. The danger is overfitting the kernel: because the marginal likelihood is optimized over a very flexible feature map, DKL can collapse distances and produce overconfident, miscalibrated uncertainty unless regularized, which later work has documented and tried to fix.