neural tangent kernel (NTK)

Take a very wide network and train it with gradient descent. The neural tangent kernel says that, in the infinite-width limit, the function the network computes barely strays from its first-order Taylor expansion around the initialization. Training then behaves like kernel regression with one fixed kernel — the geometry is frozen the moment you start, and learning is just solving a linear problem in a high-dimensional feature space you never actually move.

Concretely, define the kernel as the inner product of parameter-gradients of the output at two inputs. Under the NTK parameterization and as width goes to infinity, this kernel is deterministic at initialization and stays constant throughout training. Gradient descent on the squared loss becomes a linear ordinary differential equation — kernel gradient flow — and the trained predictor converges to the kernel (ridgeless) regression solution. This gives clean global-convergence guarantees for grossly overparameterized networks.

The catch is that the constant-kernel regime is exactly the regime with no feature learning: the network's internal representations do not adapt. Much of deep learning's empirical power lives outside this lazy regime, so the NTK is best read as a tractable boundary case that explains trainability while quietly omitting what makes finite, feature-learning networks special.