# KNOWLEDGE DISCOVERY WITH SUPPORT VECTOR MACHINES PDF

- Contents:

Knowledge Discovery with Support Vector Machines. Author(s). Lutz Hamel. First published July Print ISBN |Online. Maximum margin classifiers. Support vector machines. Elements of statistical learning theory. Multi-class classification. Regression with support vector machines. 7 SUPPORT VECTOR MACHINES. The Lagrangian Dual. Dual Maximum -Margin Optimization. The Dual Decision Function. Linear Support.

Author: | CLARETTA CHAPKO |

Language: | English, Indonesian, French |

Country: | Haiti |

Genre: | Personal Growth |

Pages: | 291 |

Published (Last): | 13.09.2016 |

ISBN: | 579-3-15950-896-7 |

ePub File Size: | 29.49 MB |

PDF File Size: | 11.28 MB |

Distribution: | Free* [*Registration Required] |

Downloads: | 45539 |

Uploaded by: | ANNETTA |

Views 2MB Size Report. DOWNLOAD PDF Knowledge discovery with support vector machines / Lutz Hamel. p. cm. — (Wiley series on methods and. Knowledge Discovery with Support Vector Machines: Computer Science Books @ longmogedwapor.cf C. J. C. Burges A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and. Knowledge Discovery 2, , pages

C, and Smola A. Google Scholar Joachims T. Kluwer Academic Publishers, Google Scholar Law M. Bayesian support vector regression. Google Scholar Lin C. Formulations of support vector machines: a note from an optimization point of view. Neural Computation ; 13 2 — On the convergence of the decomposition method for support vector machines. C, and Malik J. Learning to detect natural image boundaries using brightness and texture.

Google Scholar Mukherjee S. Nonlinear prediction of chaotic time series using a support vector machine. In Principe J. Google Scholar Muller K. An improved training algorithm for support vector machines. Google Scholar Platt J. Fast training of support vector machines using sequential minimal optimization. Google Scholar Ratsch G. Soft margins for AdaBoost. Machine Learning ; 42 3 — Google Scholar Scholkopf B.

Oldenbourg Verlag, Munich, Extracting support data for a given task. In Fayyad U. Prior knowledge in support vector kernels. In Jordan M. Learning with Kernels. C, and Bartlett P. New support vector algorithms. Neural Computation ; — Kernel Methods for Pattern Analysis. We are now ready to construct our Lagrangian dual. Substituting 7. Proposition 7. That means 7. This point represents a constraint on the margin in that the supporting hyperplanes cannot be moved beyond it.

We call points with nonzero Lagrangian multipliers support vectors, and a close inspection of equations 7. We can relate this to our primal maximum-margin algorithm. Also recall that the points in the training set that limit the size of the margin were called support vectors. We can now make the following statement: The primal maximum-margin optimization computes the supporting hyperplanes whose margin is limited by support vectors.

The dual maximum-margin optimization computes the support vectors that limit the size of the margin of the supporting hyperplanes. This is illustrated in Figure??. Here the primal optimization computes the two supporting hyperplanes that are limited by the support vectors, and the dual optimization computes the support vectors that limit the margin of the supporting hyperplanes. Rather than searching for the points from each class that lie closest to the decision surface as we did above, we already know which training set points constitute the constraints on the supporting hyperplanes: Furthermore, it is considered a linear support vector machine since it is based on a linear decision surface.

Here we take as our model the linear support vector machine from equation 7. What makes support vector machines so remarkable is that the basic linear framework is easily extended to the case where the data set is not linearly separable. The fundamental idea behind this extension is to transform the input space where the data set is not linearly separable into a higher-dimensional space called a feature space, where the data are linearly separable. Remarkably, if we choose these transformations carefully, all the computations associated with the feature space can be performed in the input space.

That is, even though we are transforming our input space so that the data become linearly separable, we do not have to pay the computational cost for these transformations. The functions associated with these transformations are called kernel functions, and the process of using these functions to move from a linear to a nonlinear support vector machine is called the kernel trick. Consider the following example.

Here our data set in Figure 7.

## Support Vector Machines

With this mapping any point on the nonlinear decision 7. The point clearly lies on the nonlinear decision surface 7. We now show that this point also lies on the plane in feature space: The fact that the plane 7. This is illustrated in Figure 7. It is revealing to study the structure of this decision function in more detail.

We do this by expanding the function using the identities in equation 7. However, it is necessary from a more technical point of view because to construct kernels you must have the third dimension. As we consider more complex nonlinear decision surfaces in the input space, we would expect that we need higher-and higherdimensional feature spaces to be able to continue to construct linear decision surfaces. Plugging this dual representation into our decision function gives us!

If we simplify equation 7. Furthermore, we have an expression that computes 7. We can now rewrite our decision function given in equation 7.

By selecting the kernel function judiciously, we can control the complexity of this model. We have already encountered two kernels. This is called the linear kernel, and here the feature space is simply the same as the input space.

Table 7. These constraints simply have to hold in whatever feature space we are working. For more complex decision surfaces in the input space, we might try polynomial kernels of higher degrees or even more complex kernels, such as the Gaussian kernel, to induce linear decision surfaces in some appropriate feature space. The process of selecting a kernel and the associated values of its free parameters, such as the degree d for the polynomial kernel, is called a feature search.

A feature search is, in general, not trivial and requires some trade-offs in model complexity and model accuracy.

Many packages provide tools that automate some aspects of the feature search, and this search is often referred to as a grid search. Most support vector machine packages include a set of fairly standard kernels from which to choose. But what about the more complex kernels such as the Gaussian kernel function?

What are the mappings and the feature spaces associated with these types of kernels? Here we show that every kernel has an associated canonical or standard mapping and feature space. The existence of these canonical structures is guaranteed by a set of assumptions on the kernel.

One interesting corollary of this is that the mappings and feature spaces associated with kernels are not unique. But this is of no consequence to us, since due to the kernels, we never need explicitly to evaluate the mappings. This also characterizes the class of kernels at the core of support vector machines. We need another property of kernels in order to construct our canonical feature spaces. Let k: Theorem 7.

## Customers who viewed this item also viewed

In other words, a kernel can be evaluated at points x and y by taking the dot product of the two partially evaluated kernels at these points. Finally, we need one more identity which will help us in the investigation of the structure of our canonical feature spaces. At a very high level we construct our canonical feature space as follows: Turn our feature space into a vector space.

Now we need to turn our feature space into a vector space. We do this by allowing arbitrary functions to be represented by linear combinations of our partially evaluated kernels over the given set of points. That is, some function h: Finally, we need to show that our construction preserves the kernel condition 7. A direct consequence of our construction is that feature spaces for kernels are not unique. This shows that feature spaces are not unique, but the dot product values they compute are unique in the sense that given a pair of input space elements, the dot products in the various spaces will evaluate to the same value for this pair.

By essentially allowing the training algorithm to ignore certain training points which are thought to be due to noise gives rise to much simpler decision surfaces in noisy data than would otherwise be possible. This is desirable because simpler decision surfaces tend to generalize better. This construction is only partially successful in the case of noisy training data where the size of the margin is limited by a few noisy training points.

That is, a slack variable measures how much of an error is committed by allowing the supporting hyperplane to be unconstrained by that point. Figure 7. If, on the other hand, the point x i lies on the wrong side of its respective supporting hyperplane, equation 7. Putting all this together, we can rewrite our maximum-margin objective function that takes slack variables into account as in the following proposition.

The larger we make the margin, the more training points will be on the wrong side of their respective supporting hyperplanes, and therefore the larger the error, and vice versa. More precisely, if we make the margin large, this will probably introduce a large number of nonzero slack variables.

If we make the margin small, we can reduce the number of nonzero slack variables, but we are also back to where we started in the sense that noisy points will dictate the position of the decision surface.

The constant C, called the cost, allows us to control the trade-off between margin size and error. More precisely, a large value for C forces the optimization to consider solutions with small margins.

This gives us the following relation between the cost and margin size: To quantify these possible 7. Otherwise, the point x j lies below the decision surface.

However, since we assume that the points that violate the constraints are due to noise and therefore unreliable, misclassifying such a point does not do as much damage to our model as perhaps assumed. We start with the primal objective function 7. In particular, equation 7. The last four conditions are the constraints of the respective primal and Lagrangian optimization problems.

## Knowledge Discovery with Support Vector Machines

As in the hard-margin case, we can solve this optimization problem much more readily by computing the Lagrangian dual. To accomplish this we apply the KKT conditions and differentiate the Lagrangian with respect to the primal variables and then evaluate the derivatives at the saddle point.

Therefore, the basic nature of the optimization problem has not changed—what has changed are the constraints. We have picked up the additional constraints 7.

This implies that we can rewrite the constraints 7. Here C is the cost constant. It is remarkable that the only difference between the maximum-margin optimization problem given in Proposition 7. To interpret this result we can go back to the complementarity condition 7. From the constraint 7.

That is, for any point that lies on the wrong side of its respective supporting hyperplane, the corresponding Lagrangian multiplier is bound to the value C. Here we illustrate these implementations through examples using the linearly separable biomedical data set given in Exercise?? Also, before proceeding, it might be a good idea to visualize these data sets in order to get a feel for the structure of the data. Load this data set into WEKA using the explorer interface. You should see a window as in Figure 7.

Now we want to build support vector machines. To do this you will need to open the Classify tab. Clicking on the resulting SMO box brings up a parameter dialog box. There are many parameters which allow the user to tune the training algorithm.

At this point we should be able to recognize a fair number of parameters from our support vector machine development above. There is also the exponent parameter, which represents the degree d for a homogeneous polynomial kernel. Finally, the useRBF parameter together with the gamma parameter enables Gaussian kernels Gaussian kernels are sometimes also called radial basis functions—therefore the parameter name.

This means that WEKA supports three kernels, the homogeneous polynomial kernel together with the linear kernel, by setting the polynomial exponent to 1, and the Gaussian kernel.

We leave the cost constant at 1. We want to evaluate our support vector machine on the training data; therefore, we will need to set the Test options to the value Use training set.

Once we have set the parameters and test options we are ready to build a model by pressing the Start button. When the training has completed you should see a window as shown in Figure 7. Let us test this theory by making our cost constant in WEKA smaller. For our next experiment we leave all the parameters as they were from the previous experiment with the exception of C, which we set to the value 1. Now we build a new model by pressing the Start button.

The margin has grown to such an extent that one of the training instances is now located on the wrong 7. If we make the cost even smaller, say 1. On the other hand, increasing the value of C beyond the default of 1. This is easily explained by again referring to equation 7. A large value for C results in a small margin.

However, increasing the cost C to a value beyond the default 1. Nonlinear Support Vector Machines We now turn our attention to data sets that are not linearly separable. For the next set of experiments we use the data set given in Table 7. Now return to the Classify tab. Again we use the training set in the test options. This implies that the linear kernel is not the appropriate kernel to construct a model for this data set. Let us try a homogeneous polynomial kernel of degree 2.

To select this kernel we set the exponent parameter to the value 2. When we build the model now, the model summary shows that a support vector machine with a homogeneous polynomial kernel of degree 2 and the default margin size separates the data set perfectly.

We have just performed a simple feature search to determine a kernel that will allow the support vector machine to classify all the instances correctly. C-classification linear 1 Number of Support Vectors: If we experiment with the value of the cost constant, we can observe that the cost C has the same effect on the model as it did in WEKA: R allows us to visualize the decision surface.

Notice that the decision surface is equidistant from the two classes. Also notice that we have two support vectors, one for each class data points represented as crosses. That is, the model shown is a model with a large margin. Here all data points are support vectors; that is, all data points are either on or in the margin.

Nonlinear SupportVector Machines Let us turn to data sets that are not linearly separable. Here we assume that the data set from Table 7. The key insight here is that in the Lagrangian dual all data points in the input space appear in the context of dot products.

By taking advantage of the kernel trick we are able to replace these dot products with appropriate 7. Consider, for example, the dual perceptron learning algorithm depicted in Algorithm??. Applying our kernel trick would make this dot product appear as k x j , x i , where k is some appropriate kernel see Table 7.

In Algorithm?? The resulting kernel-perceptron algorithm is shown in Algorithm 7. A variant of the kernel-perceptron algorithm is shown in Algorithm 7. Furthermore, just because the decision surface goes through the origin of the feature space does not necessarily imply that the decision surface goes through the origin of the input space.

Which kernel did you use to achieve reasonable accuracy? What value of the cost constant C did you use? What were the effects of changing the value of C? Does the kernel-perceptron construct a decision surface that separates the two classes?

Karush in the s [41] and later, Kuhn and Tucker [49] extended this theory to inequality constraints. Boyd and Vandenberghe [14] discuss the Lagrangian dual in some detail. However, these methods were not truly appreciated until the seminal paper published by Boser et al. Support vector machines themselves in the formulation given here were introduced by Cortes and Vapnik in [22]. Our construction of canonical feature spaces follows closely a proof given in [65].

A nice overview of support vector machines with a more geometric slant is [6]. This geometric interpretation is discussed even further in [5]. Kernel-perceptrons are discussed by Freund and Schapire [32] and Herbrich [38].

General kernel methods in the area of pattern recognition are discussed in [67]. When we refer to the implementation of support vector machines, we usually mean the implementation of the training algorithm that produces the necessary values for the Lagrangian multipliers and the offset term for a support vector machine model.

We start our implementation discussion by taking a look at a simple gradient ascent optimization algorithm, also known as the kernel-adatron algorithm. This straightforward optimization technique solves the Lagrangian dual optimization problem for support vector machines with some simplifying assumptions.

We then take a look at the use of quadratic programming solvers. The fact that data matrices associated with a quadratic program for support vector machines grow quadratically with the size of the training data limits the straightforward use of quadratic programming solvers in many knowledge discovery projects.

However, a technique called chunking which takes advantage of the sparseness of the optimization problem associated with support vector machines only a few training instances are actual support vectors remedies this situation and allows us to apply quadratic programming solvers to fairly large knowledge discovery projects. We conclude the chapter with a discussion of sequential minimal optimization which has become the defacto standard implementation technique of the Lagrangian dual optimization associated with support vector machines.

The values C and k are free parameters and represent the cost and the kernel function, respectively. Perhaps the most straightforward implementation of the Lagrangian dual optimization problem 8.

The gradient of a differentiable function is a vector composed of the partial derivatives of that function with respect to the dimensions of the underlying vector space. The length of the vector represents the increase of the function at that point.

In gradient ascent we take advantage of the gradient and use it to point the way toward the maximum value of the function. Should the gradient become zero at a particular point, we know that we are at a maximum because there is no further increase in the value of the function at that point.

To use gradient ascent as an optimization procedure for some objective function, we pick a random starting point on the surface of the objective function and then iteratively move along the surface of the function in the direction of the gradient until the gradient evaluated at some point becomes zero. Following is a sketch of the gradient ascent algorithm for our Lagrangian optimization problem: That is, we initially assume that none of the training points are support vectors. The algorithm continues to iterate until we have converged on the maximum value.

The version of gradient ascent described here is called stochastic gradient ascent, since we use the components of the gradient as soon as they become available during the iteration over the training points.

Algorithm 8. One is that we have treated our optimization problem as an unconstrained optimization. That is, we ignored the constraints given in 8. By addressing these two problems in the simple gradient ascent algorithm above, we obtain the kernel-adatron algorithm.

Recall that the constraint 8. This constraint is impossible to satisfy at each optimization step in an algorithm that updates only one Lagrangian multiplier at a time. In fact, to maintain constraint satisfaction at each step of the algorithm, we would need to modify at least two Lagrangian multipliers simultaneously at each update.

This insight is the cornerstone of the sequential minimal optimization algorithm discussed below. Here we take a different approach; rather than trying to optimize the offset term, we set it to zero i. This takes care of the constraint 8. These constraints are easily implemented, however. We accomplish this by rewriting our update rule for the gradient ascent 8.

It is interesting to note the similarities between the kernel-perceptron algorithm given in Algorithm?? The big difference is that the kernel-perceptron algorithm stops iterating as soon as it has found some decision surface, whereas the kernel-adatron algorithm terminates when it has found the optimal decision surface.

However, for the implementation of support vector machines it is important that the optimization package supports both the equality constraints 8.

The shape of the generalized objective function optimization 8. The biggest difference is that the generalized form is expressed as a minimization, whereas the optimization of the Lagrangian dual is a maximization. However, applying identity?? This shows that the generalized optimization problem can be instantiated as our Lagrangian dual optimization problem.

We summarize this construction in Algorithm 8. Here the function solve is assumed to be a quadratic programming solver that operates on the generalized representation of an optimization problem according to 8. This matrix grows quadratically with the size of the training set. This poses problems even for moderately sized knowledge discovery projects. Consider a training set with 50, instances.

To implement a training algorithm for a support vector model using a quadratic programming solver we would have to construct a kernel matrix with 2.

Not only will the memory requirements for a matrix of this size exceed the capabilities of many machines, but the size of this kernel matrix also implies that solution of the optimization problem will be very slow. We can take advantage of the sparseness of the support vector model to reduce the memory requirements of the kernel matrix.

That is, we can take advantage of the fact that typically very few instances in a training set are support vectors that constrain the position of the decision surface. Delete observations from W that are not support vectors. We will see that the chunking algorithm makes repeated use of this property. The training algorithm based on chunking is shown in Algorithm 8.

Here, the constant k is the chunk size and the set W is often referred to as the working set. The algorithm iterates over a succession of smaller optimization problems characterized by the training observations in W. At each iteration the solution to the Lagrangian dual optimization of W is used to estimate how far the overall optimization has progressed. To accomplish this we discard points that are not support vectors in W , and the remaining support vectors are used to construct a model.

This model is then applied to all observations in the training set. That is, we have converged on a global solution if all observations in the training set satisfy the KKT conditions. As can be seen in the algorithm, in the last step of the loop we identify the k worst offenders of the KKT conditions. This is done by measuring how far away each training observation is from satisfying the KKT conditions.

**You might also like:**

*BRAIN TEASERS WITH ANSWERS PDF*

This allows us to sort the offenders and extract the k top offenders from this list. These top offenders are then added to the working set W , which now constitutes a new optimization problem for the next iteration. The algorithm monitors the two complementarity conditions?? From condition 8. Let 0 8. Constraint 8. Using equation 8. Putting this all together gives us the following conditions that we need to monitor in Algorithm 8.

Compute b. A typical value for k is , which implies that each optimization subproblem uses a kernel matrix on the order of , elements. It is interesting to observe that if our sparseness assumption does not hold, that is, almost all training points are considered to be support vectors, the algorithm degenerates into the original overall optimization problem.

However, most real-world data sets give rise to sparse solutions, and therefore the latter is not of great concern.

Furthermore, the chunking algorithm has been shown to converge in a robust fashion in real-world situations. This algorithm works similar to the chunking algorithm of Section 8. As mentioned before, updating a single Lagrangian multiplier does not work since it is not guaranteed that the constraint 8. If so, the algorithm terminates.

Otherwise, it will continue to iterate. That is, a viable Lagrangian multiplier is always optimized against a nonviable Lagrangian multiplier. With this it can be shown that the algorithm is guaranteed to converge. What is remarkable and sets this algorithm apart from previous implementations is the fact that the optimization subproblem over the two training instances can be solved analytically, and therefore a call to a computationally expensive optimization library is not necessary.

We can simplify this optimization problem even further by rewriting constraint 8. Some care needs to be taken that this optimization respects the inequality constraints 8. Refer to the bibliographic notes for pointers to the relevant literature. All three strategies share the characteristic that they represent incremental improvements of the global optimization problem until the global maximum is reached. Of the three strategies discussed, the quadratic programming solution lends itself to a straightforward implementation of support vector machines, and SMO is the most popular.

SMO is the foundation of many support vector machine packages available on the Web.

It is also the foundation of the support vector machine implementation in WEKAand in the R package that we use. Today, the implementation of support vector machines remains an active area of research. With respect to SMO, much of the research concentrates on the selection of appropriate Lagrangian multipliers to optimize.

Implement it in R and compare the results to the simple gradient ascent in part c. Test the implementation on your favorite data set using an appropriate kernel from Table??. Test the implementation on a data set that has at least instances, using an appropriate kernel from Table??.

Then implement the algorithm in R. We based our discussion here on a book by Kreyszig [48]. Gradient ascent optimization methods are discussed in [14] and [69]. Again, any introductory book on mathematical optimization will have a discussion on gradient ascent methods. The kernel-adatron was introduced by Friess et al. Here we present a simpler version of this algorithm as given in [67]. Another, more sophisticated chunking algorithm is given by Osuna et al. There are many industrial-strength optimization packages available that can be used to solve the convex optimization problem due to support vector machines.

Two packages that have been used routinely in this context are LOQO http: Platt proposed his sequential minimal optimization algorithm in [58]. There is a lot of information with respect to SMO on his Website http: More recently, advanced selection heuristics for the Lagrangian multipliers have been proposed for SMO e.

A nice summary of different implementations of support vector machines in R is a paper by Karatzoglou et al. A recent collection of papers dealing with support vector machine implementations for very large knowledge discovery projects is [13]. It is therefore necessary to use subsets of the data universes as training sets.

As soon as we use only a subset of a data universe as a training set, we are faced with the question: How does our model perform on instances of the data universe that are not part of the training set? In this chapter we introduce techniques that will allow us to quantify the performance of our models in the context of this uncertainty. We then introduce the confusion matrix, which gives us a more detailed insight into model performance. In particular, the confusion matrix characterizes the types of errors that models make.

With these metrics and tools in hand, we discuss formal model evaluation. In particular, we discuss the difference between training and test error. We show that we can use test error as a way to estimate model parameters which promise the best model performance on instances of the data universe that are not part of the training set.

The testing techniques we discuss include the hold-out method and N -fold crossvalidation. Model evaluation is not complete without a discussion of the uncertainty of the estimated model performance. This function compares the output of a model for a particular observation with the label of this observation.

If the model commits a prediction error on this observation, the loss function returns a 1; otherwise, it returns a 0. With this we can rewrite the expression of the model error in a formal fashion. The model error is computed by summing the number of errors committed by the model on the data set D and dividing by the size of the data set D. In other words, the model error is the average loss over the data set D. The subscript D indicates that we use the data set D to compute the error.

We have also parameterized the error with respect to the model. We can compute the error using 9. Then we have the following four possibilities when the model is applied to the observation: If the model output does not match the label observed, we have either a false positive or a false negative outcome, both of which are error outcomes. In many situations it is important to distinguish these two error outcomes when evaluating model performance.

Consider the following clinical example. Suppose you are developing a model that, given the parameters of a tissue biopsy, will predict whether or not this tissue is cancerous. Now, from the discussion above, your model can commit two types of errors. It can commit a false positive error; that is, it predicts that the tissue sample is cancerous when it is not.

Here the model predicts that the tissue sample is not cancerous when in reality it is. In a clinical setting the latter is a much more serious error than the former since a false negative implies that the patient will remain untreated, whereas a false positive usually results in more tests until the false positive error is detected and the patient is discharged appropriately. When analyzing model performance in these types of situations we would like to understand the different types of errors that our model commits.

Unfortunately, the simple performance metrics that we just discussed, based on the 0—1 loss function, do not allow us to distinguish these errors. However, a representation of model performance called the confusion matrix does distinguish between the two types of errors and is therefore the tool of choice when analyzing model performance where one or the other type of error can have serious implications.

That is, for each observation we obtain the pair of labels y, y.

True positive predictions are mapped into the top left corner of the confusion matrix, and true negative predictions are mapped into the bottom right corner of the matrix. False positives and false negatives are mapped into the bottom left and top right corners of the matrix, respectively. For models that do commit errors, we see that the errors will be mapped into the confusion matrix according to the type of error the model commits. Table 9. On this set of observations the model commits 7 false negative errors and 4 false positive errors in addition to the 95 true positive and 94 true negative predictions.

Here it would be advisable to build a new model with more balanced errors. Only the confusion matrix is able to provide this type of insight. However, in the context where model errors can have serious consequences, we will have to look more closely at the types of errors that a model commits.

Given a confusion matrix of a model as in Table 9. In addition to these metrics we have two other metrics that are commonly used to characterize model performance: Going back to the confusion matrix of our model given in Table 9.

This iterative model evaluation process stops when we obtain a model with a satisfactory performance. We need to introduce some additional notation to formalize our model evaluation process. Recall that soft-margin support vector machine models have several free parameters that need to be set by the user.

These free parameters include the cost constant and the kernel function, with its corresponding parameters for details on various kernels, see Table?? In addition, the subscript D indicates that the model was trained on set D. On the right side of the identity 9. Therefore, the values of the Lagrangian multipliers depend on the cost constant. For convenience we make this dependence explicit with our notation.

Figure 9. These three model parameters control the complexity of the models. For example, we consider a model with a linear kernel a low complexity model because of its limited ability to model complex class boundaries. On the other hand, we consider a model with a high-degree polynomial kernel or a model based on a Gaussian kernel to be a complex model because of its ability to model complex class boundaries.

Low-complexity models appear on the left side of the horizontal axis; high-complexity models appear on the right. The training error is mapped onto the vertical axis. Notice that the training error decreases with the growing complexity of the models. That is, the more complex a model is, the better it can model individual observations in the training data, and the fewer mistakes it makes. Here the error curve is depicted in an idealized fashion. In real knowledge discovery projects the curve would exhibit many local maxima and minima.

But unfortunately, training data sets are never perfect representations of the corresponding data universes since they typically consist of only a small fraction of a data universe. Therefore, the fact that we can reduce the training error to zero is meaningless since it does not allow us to draw any conclusions with respect to the performance of the model over the remaining data universe.

Besides the fact that training sets represent only a small fraction of the overall underlying data universes, there are a number of other errors that can pollute the construction of training data. A second source of errors is due to noise when observing the labels for the training set.

Here the target function produces an erroneous label for an instance while constructing our training set. A third source of errors are accidental misrepresentations of the sample points in the training set. Please check your email for instructions on resetting your password. If you do not receive an email within 10 minutes, your email address may not be registered, and you may need to create a new Wiley Online Library account.

If the address matches an existing account you will receive an email with instructions to retrieve your username. Skip to Main Content. Lutz Hamel. First published: Print ISBN: About this book An easy-to-follow introduction to support vector machines This book provides an in-depth, easy-to-follow introduction to support vector machines drawing only from minimal, carefully motivated technical and mathematical background material.

It begins with a cohesive discussion of machine learning and goes on to cover:The dot product of the two unit vectors is equal to the cosine of the angle between them.

As in the hard-margin case, we can solve this optimization problem much more readily by computing the Lagrangian dual. The attribute is immediately removed from the data table.

We can observe that the bars for two and three legs only contain observations that do not belong in the class of mammals light gray. Least Squares Support Vector Machines.