Direct Convex Relaxations of Sparse SVM

Antoni B. Chan, Nuno Vasconcelos, and Gert R. G. LanckrietThe 24th International Conference on Machine Learning (ICML 2007)

Presented by Shuiwang Ji

Outline

• Introduction;

• Quadratically Constrained Quadratic Programming (QCQP) formulation;

• Semidefinite Programming (SDP) formulation;

• Experiments;

Sparsity of SVM

x1, …, xd

SVM is sparse w.r.t. data points,

but not sparse w.r.t. features.

R

kkkk yα

12

1xw

( ) sgn( )Tf x w x b

Motivations & Related Work

• Features may be noisy, redundant;

• Sparsity enhance interpretability;

• Sparse PCA (Zou et al. & d'Aspremont et al.);

• Sparse Eigen Methods by D.C. Programming (ICML07);

An Example

Vector Norm

1 1

2 22 1

0

|| || | | ... | |

|| || ...

|| || ( )

n

Tn

x x x

x x x x x

x Card x

Number of nonzero entries in x

C-SVM Primal and Dual

2-norm

R

kkkk yα

12

1xw

( ) sgn( )Tf x w x b

LP-SVM Primal and Dual

1-norm

R

kkkk yα

12

1xw

( ) sgn( )Tf x w x b

Convex QCQP Relaxation

2|| ||w t

Interpretations of QCQP-SSVM

• Problem 6 and 7 are equivalent;

• QCQP-SSVM is a combination of C-SVM and LP-SVM, 1-norm encourages sparsity and 2-norm encourages large margin;

QCQP-SSVM Dual

QCQP-SSVM

• QCQP-SSVM automatically learns an adaptive soft-threshold on the original SVM hyperplane.

SDP Relaxation

SDP-SSVM Dual

• The optimal weighting matrix increases the influence of the relevant features while demoting the less relevant features;

• SDP-SSVM learns a weighting on the inner product such that the hyperplane in the feature space is sparse.

Results on Synthetic Data

Results on 15 UCI data sets

Results on 15 UCI data sets