structural risk minimization (SRM)

Pure empirical risk minimization, left unchecked, picks the most flexible model that fits the training data best — and overfits. Structural risk minimization adds a conscience. Arrange your hypotheses into a nested ladder of classes of increasing complexity, then choose not the model with the lowest training error, but the one that best balances training error against the complexity of the rung it came from. It operationalizes Occam's razor as an explicit optimization.

Given a sequence of nested classes of increasing capacity, measured by VC dimension or another complexity measure, SRM minimizes, over all rungs and all hypotheses within them, the sum of the empirical risk and a complexity penalty derived from a generalization bound for that rung. Because the bound holds uniformly across the whole ladder — via a union bound that allocates confidence to each rung — the chosen hypothesis carries a valid guarantee on true risk. The penalty grows with the rung index, so SRM automatically trades fit against capacity and selects the right model size from data.

SRM is the principled ancestor of penalized model selection, regularization paths, and complexity-based criteria; the support vector machine's margin maximization is a continuous SRM over norm-bounded classes. Its strength is a non-asymptotic guarantee with automatic capacity selection. Its weakness is practical: the bound-derived penalties are often too conservative, so in practice cross-validation usually replaces the theoretical penalty for tuning.