influence functions for attribution

Influence functions answer a counterfactual without retraining: if I had upweighted (or removed) this one training point, how would a specific prediction have changed? Rather than the brutally expensive leave-one-out retrain, classical statistics gives a first-order approximation by perturbing the training point's weight infinitesimally and tracing the effect through the model's optimality conditions. It turns 'which training data is responsible for this output' into a calculus question.

The estimate combines two gradients through the inverse Hessian of the training loss: the gradient of the test loss and the gradient of the training point's loss. The Hessian captures how the loss landscape curves at the optimum, so the inverse-Hessian-vector product converts a local nudge into a global change in parameters and hence in the test prediction. Because forming and inverting the Hessian is infeasible for large models, practitioners use stochastic estimators (LiSSA), Kronecker-factored (EK-FAC) approximations, or Arnoldi iterations; recent work scales these to large language models.

The caveats are sharp: the first-order theory assumes a strictly convex optimum, so for deep non-convex networks the estimates are approximate and sometimes fragile, and the Hessian approximation dominates both cost and error. Still, influence is a leading tool for finding mislabeled data, explaining predictions, and auditing memorization.