Support vector machine
Support vector machines (SVMs) are a set of related supervised learning methods, applicable to both classification and regression.
The basic classification SVM creates a maximum-margin hyperplane that lies in a transformed input space.
Given training examples labeled either "yes" or "no", a maximum-margin hyperplane splits the "yes" and "no" training examples, such that the distance from the closest examples (the margin) to the hyperplane is maximized.
The use of the maximum-margin hyperplane is motivated by statistical learning theory, which provides a probabilistic test error bound which is minimized when the margin is maximized.
The parameters of the maximum-margin hyperplane are derived by solving a quadratic programming (QP) optimization problem. There exist several specialized algorithms for quickly solving the QP problem that arises from SVMs.
The original optimal hyperplane algorithm proposed
by Vladimir Vapnik in 1963 was a linear classifier.
However, in 1992, Bernhard Boser, Isabelle Guyon and Vapnik suggested applying the kernel trick (originally proposed by Aizerman) to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a non-linear kernel function. This causes the linear algorithm to operate in a different space. Using the kernel trick makes the maximum margin hyperplane be fit in a feature space. The feature space is a non-linear map from the original input space, usually of much higher dimensionality than the original input space. In this way, non-linear classifiers can be created. If the kernel used is a radial basis function, the corresponding feature space is a Hilbert space of infinite dimension. Maximum margin classifiers are well regularized, so the infinite dimension does not spoil the results.
In 1995, Corinna Cortes and Vapnik suggested a modified
maximum margin idea that allows for mislabeled examples.
If there exists no hyperplane that can split the "yes" and "no" examples,
the Soft Margin method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. This work popularized the expression
Support Vector Machine or SVM.
A version of a SVM for regression was proposed in 1997 by Vapnik, Steven Golowich, and Alex Smola. This method is called CiteSeer: )
- Bernhard Schölkopf and A. J. Smola: Learning with Kernels. MIT Press, Cambridge, MA, 2002. (Partly available on line: .) ISBN 0-262-19475-9
