Support-Vector Networks
Separate two classes by the widest margin — and only the borderline points decide where the line falls.
Of all the lines that could split two groups of dots, a support-vector machine picks exactly one — the one with the most elbow room — and that single choice is why it generalizes so well.
The idea, unpacked
Suppose you have two kinds of things — spam and not-spam, say — each described by some numbers, and you want a rule that tells them apart. Geometrically, you are hunting for a line (in higher dimensions, a flat sheet) that puts one kind on each side. Usually many lines would work. Which is best?
The support-vector machine answers: the line that leaves the widest empty gap on both sides. It pushes the boundary as far as it can from the nearest examples of each class, so there is the most possible room before a new, slightly different example would be misjudged. Remarkably, only the closest few examples — the support vectors — matter; the rest could vanish and the line would not move.
Where it came from
The method grew at AT&T Bell Labs, the same storied lab behind the transistor and information theory. Vladimir Vapnik had spent decades in the Soviet Union building a mathematical theory of learning; after emigrating, he and his colleagues turned it into an algorithm. A 1992 paper handled the clean case where the two classes can be separated perfectly. Then in 1995 Corinna Cortes and Vapnik published "Support-Vector Networks," which added the crucial ability to cope with messy, overlapping real data — and the method took off.
Why it mattered
Before deep learning's resurgence, the support-vector machine was the workhorse of practical machine learning. It was accurate, it came with genuine mathematical guarantees about when it would generalize, and — through a clever shortcut called the kernel trick — it could draw curved, complicated boundaries while still solving a simple, well-behaved problem with a single best answer. For fifteen years it was the method to beat.
A precise analogy
Think of building the widest possible road between two neighbourhoods of houses. You don't care about the houses deep inside each neighbourhood; only the ones nearest the other side constrain how wide the road can be and where it must run. Those few border houses are the support vectors. Move a house in the interior and the road is unchanged; nudge a border house and the whole road shifts. Try it on the dots below.
Where it sits
This is one of the great branches of machine learning: statistical learning, where the goal is provable generalization rather than brain-inspired wiring. It shares the Library's AI shelf with the neural-network lineage — Rosenblatt's perceptron, Rumelhart's backpropagation, and the deep networks of AlexNet and the Transformer that later overtook it. For much of the 1990s and 2000s, though, the support-vector machine quietly won.
The support-vector network is a new learning machine for two-group classification problems.
The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine.