Part IIPart II

Support Vector Machine AlgorithmsSupport Vector Machine Algorithms

OutlineOutline

Some variants of SVMSome variants of SVMRelevant algorithmsRelevant algorithmsUsage of the algorithmsUsage of the algorithms

SVM AlgorithmsSVM Algorithms

Famous SVM Implementations Famous SVM Implementations LibSVMLibSVM

• http://http://www.csie.ntu.edu.tw/~cjlin/libsvmwww.csie.ntu.edu.tw/~cjlin/libsvm// SVM-LightSVM-Light

• http://svmlight.joachims.org/http://svmlight.joachims.org/

General Data Mining tools that contain SVMGeneral Data Mining tools that contain SVM Spider toolbox (matlab)Spider toolbox (matlab)

• http://http://www.kyb.tuebingen.mpg.de/bs/people/spiderwww.kyb.tuebingen.mpg.de/bs/people/spider WEKA toolbox (java)WEKA toolbox (java)

• http://http://www.cs.waikato.ac.nz/~ml/weka/index.htmlwww.cs.waikato.ac.nz/~ml/weka/index.html

Variants in LibSVMVariants in LibSVM

We use libSVM as an exampleWe use libSVM as an example C-SVCC-SVC Nu-SVCNu-SVC One class SVMOne class SVM Epsilon-SVREpsilon-SVR Nu-SVR Nu-SVR

C-SVC C-SVC

The dual is

nu-SVC nu-SVC

The dual is

One class-SVM One class-SVM

The dual is

Epsilon-SVR Epsilon-SVR

The dual is

nu-SVRnu-SVR

The dual is

Commands of libSVMCommands of libSVM

SVMtoy : an Illustration toolSVMtoy : an Illustration tool SVMtrain: the training algorithmSVMtrain: the training algorithm SVMpredict: the test algorithmSVMpredict: the test algorithm SVMscale: an algorithm for scallingSVMscale: an algorithm for scalling

Usage of svm-trainUsage of svm-train Usage: svmtrain [options] training_set_file [model_file]Usage: svmtrain [options] training_set_file [model_file]options:options: -s svm_type : set type of SVM (default 0)-s svm_type : set type of SVM (default 0) 0 -- C-SVC0 -- C-SVC 1 -- nu-SVC1 -- nu-SVC 2 -- one-class SVM2 -- one-class SVM 3 -- epsilon-SVR3 -- epsilon-SVR 4 -- nu-SVR4 -- nu-SVR -t kernel_type : set type of kernel function (default 2)-t kernel_type : set type of kernel function (default 2) 0 -- linear: u'*v0 -- linear: u'*v 1 -- polynomial: (gamma*u'*v + coef0)^degree1 -- polynomial: (gamma*u'*v + coef0)^degree 2 -- radial basis function: exp(-gamma*|u-v|^2)2 -- radial basis function: exp(-gamma*|u-v|^2) 3 -- sigmoid: tanh(gamma*u'*v + coef0)3 -- sigmoid: tanh(gamma*u'*v + coef0)

-d degree : set degree in kernel function (default -d degree : set degree in kernel function (default 3)3)

-g gamma : set gamma in kernel function -g gamma : set gamma in kernel function (default 1/k)(default 1/k)

-r coef0 : set coef0 in kernel function (default 0)-r coef0 : set coef0 in kernel function (default 0) -c cost : set the parameter C of C-SVC, epsilon--c cost : set the parameter C of C-SVC, epsilon-

SVR, and nu-SVR (default 1)SVR, and nu-SVR (default 1) -n nu : set the parameter nu of nu-SVC, one--n nu : set the parameter nu of nu-SVC, one-

class SVM, and nu-SVR (default 0.5)class SVM, and nu-SVR (default 0.5) -p epsilon : set the epsilon in loss function of -p epsilon : set the epsilon in loss function of

epsilon-SVR (default 0.1)epsilon-SVR (default 0.1)

-m cachesize : set cache memory size in MB (default -m cachesize : set cache memory size in MB (default 100)100)

-e epsilon : set tolerance of termination criterion (default -e epsilon : set tolerance of termination criterion (default 0.001)0.001)

-h shrinking: whether to use the shrinking heuristics, 0 or -h shrinking: whether to use the shrinking heuristics, 0 or 1 (default 1)1 (default 1)

-b probability_estimates: whether to train a SVC or SVR -b probability_estimates: whether to train a SVC or SVR model for probability estimates, 0 or 1 (default 0)model for probability estimates, 0 or 1 (default 0)

-wi weight: set the parameter C of class i to weight*C, for -wi weight: set the parameter C of class i to weight*C, for C-SVC (default 1)C-SVC (default 1)

-v n: n-fold cross validation mode-v n: n-fold cross validation mode

Example of svm-train(cont.)Example of svm-train(cont.) svmtrain -s 0 -c 1000 -t 2 -g 0.5 -e 0.00001 data_filesvmtrain -s 0 -c 1000 -t 2 -g 0.5 -e 0.00001 data_file

Train a classifier with RBF kernel exp(-0.5|u-v|^2) and Train a classifier with RBF kernel exp(-0.5|u-v|^2) and stopping tolerance 0.00001stopping tolerance 0.00001

svmtrain -s 3 -p 0.1 -t 0 -c 10 data_filesvmtrain -s 3 -p 0.1 -t 0 -c 10 data_file Solve SVM regression with linear kernel u'v and C=10, Solve SVM regression with linear kernel u'v and C=10,

and epsilon = 0.1 in the loss function.and epsilon = 0.1 in the loss function. svmtrain -s 0 -c 10 -w1 1 -w-1 5 data_filesvmtrain -s 0 -c 10 -w1 1 -w-1 5 data_file

Train a classifier with penalty 10 for class 1 and penalty Train a classifier with penalty 10 for class 1 and penalty 50 for class -1.50 for class -1.

svmtrain -s 0 -c 500 -g 0.1 -v 5 data_filesvmtrain -s 0 -c 500 -g 0.1 -v 5 data_file Do five-fold cross validation for the classifier usingthe Do five-fold cross validation for the classifier usingthe

parameters C = 500 and gamma = 0.1> svm-train -s 0 -parameters C = 500 and gamma = 0.1> svm-train -s 0 -b 1 data_fileb 1 data_file

Usage of SVM-predictUsage of SVM-predict

Usage: svmpredict [options] test_file Usage: svmpredict [options] test_file model_file output_filemodel_file output_file options: -b probability_estimates: whether to options: -b probability_estimates: whether to

predict probability estimates, 0 or 1 (default predict probability estimates, 0 or 1 (default 0); one-class SVM not supported yet0); one-class SVM not supported yet

ExamplesExamples svmtrain -c 100 -t 0 .\data\breast_cancer.trainsvmtrain -c 100 -t 0 .\data\breast_cancer.train svmpredict .\data\breast_cancer.test .\data\svmpredict .\data\breast_cancer.test .\data\

breast_cancer.train.model breast_cancer.resbreast_cancer.train.model breast_cancer.res

Usage of SVM-scaleUsage of SVM-scale

Usage: svmscale [-l lower] [-u upper] [-y Usage: svmscale [-l lower] [-u upper] [-y y_lower y_upper] [-s save_filename] [-r y_lower y_upper] [-s save_filename] [-r restore_filename] filenamerestore_filename] filename

(default: lower = -1, upper = 1, no y scaling)(default: lower = -1, upper = 1, no y scaling) svmscale -l 0 -u 1 -s range datasetx\svmscale -l 0 -u 1 -s range datasetx\

svmguide3> svmguide3.scalesvmguide3> svmguide3.scale svmscale -r range datasetx\svmguide3.t> svmscale -r range datasetx\svmguide3.t>

svmguide3t.scalesvmguide3t.scale

Usage of SVM-lightUsage of SVM-light

Commands:Commands: svm_learnsvm_learn svm_classifysvm_classify

Usage of SVM-lightUsage of SVM-light svm_learn [options] example_file model_filesvm_learn [options] example_file model_file Available options are: Available options are: Learning options: Learning options:

-z {c,r,p} - select between classification (c), regression (r), and -z {c,r,p} - select between classification (c), regression (r), and preference ranking (p) (default classification) preference ranking (p) (default classification)

-c float - C: trade-off between training error and margin (default -c float - C: trade-off between training error and margin (default [avg. x*x]^-1)[avg. x*x]^-1)

-w [0..] - epsilon width of tube for regression (default 0.1) -w [0..] - epsilon width of tube for regression (default 0.1) -j float - Cost: cost-factor, by which training errors on positive -j float - Cost: cost-factor, by which training errors on positive

examples outweight errors on negative examples (default 1) examples outweight errors on negative examples (default 1) -b [0,1] - use biased hyperplane (i.e. x*w+b0) instead of -b [0,1] - use biased hyperplane (i.e. x*w+b0) instead of

unbiased hyperplane (i.e. x*w0) (default 1) unbiased hyperplane (i.e. x*w0) (default 1) -i [0,1] - remove inconsistent training examples and retrain -i [0,1] - remove inconsistent training examples and retrain

(default 0)(default 0)

Usage of SVM-lightUsage of SVM-light

Transduction options Transduction options -p [0..1] - fraction of unlabeled examples to be classified into the -p [0..1] - fraction of unlabeled examples to be classified into the

positive class (default is the ratio of positive and negative positive class (default is the ratio of positive and negative examples in the training data)examples in the training data)

Kernel options: Kernel options: -t int - type of kernel function: 0: linear (default) 1: polynomial (s -t int - type of kernel function: 0: linear (default) 1: polynomial (s

a*b+c)^d 2: radial basis function exp(-gamma ||a-b||^2) 3: a*b+c)^d 2: radial basis function exp(-gamma ||a-b||^2) 3: sigmoid tanh(s a*b + c) 4: user defined kernel from kernel.h sigmoid tanh(s a*b + c) 4: user defined kernel from kernel.h

-d int - parameter d in polynomial kernel -d int - parameter d in polynomial kernel -g float - parameter gamma in rbf kernel -g float - parameter gamma in rbf kernel -s float - parameter s in sigmoid/poly kernel-s float - parameter s in sigmoid/poly kernel -r float - parameter c in sigmoid/poly kernel -r float - parameter c in sigmoid/poly kernel -u string - parameter of user defined kernel -u string - parameter of user defined kernel

Example of SVM-LightExample of SVM-Light

SVM-Light considers the sparsity of dataSVM-Light considers the sparsity of data Especially useful for Text ClassificationEspecially useful for Text Classification

svm_learn –c 100 .\data\train.dat .\data\svm_learn –c 100 .\data\train.dat .\data\model.datmodel.dat

svm_classify .\data\test.dat .\data\model.dat .\svm_classify .\data\test.dat .\data\model.dat .\data\res.datdata\res.dat

Illustration of SVM in SpiderIllustration of SVM in Spider% Clear all variables/definitions % Clear all variables/definitions 

>> clear classes  >> clear classes  %% Generate 100 points from a toy data object and lets have a look to it.  Generate 100 points from a toy data object and lets have a look to it. 

>> d=gen(toy2d('uneven_gauss','l=200'));>> d=gen(toy2d('uneven_gauss','l=200'));>> plot(d); >> plot(d);

Illustration of SVM in SpiderIllustration of SVM in Spider % Create a C-SVM object with RBF kernel (sigma = 1) and C=Inf % Create a C-SVM object with RBF kernel (sigma = 1) and C=Inf

>> s=svm({kernel('rbf',1)});>> s=svm({kernel('rbf',1)});% We now train the SVM and obtain a trained object and the training result % We now train the SVM and obtain a trained object and the training result

>> [r,a]=train(s,d);>> [r,a]=train(s,d);% We evaluate the training error and visualize the trained object% We evaluate the training error and visualize the trained object

>> loss(r)>> loss(r)>> plot(a) >> plot(a)