MURI Meeting July 2002 Gert Lanckriet ( gert@eecs.berkeley.edu ) gert@eecs.berkeley.edu L. El...

MURI MeetingJuly 2002

Gert Lanckriet (gert@eecs.berkeley.edu)L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P.

BartlettU.C. Berkeley

Convex Optimization in Machine Learning

Convex Optimization in Machine Learning

QPLP

QCQP

SDPSOCP

Advanced Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning

Linear Programming (LP)

Second Order Cone Programming (SOCP)

Semi-Definite Programming

Advanced Convex Optimization in Machine Learning

MPM: Problem Sketch (1)

aT z = b : decision hyperplane

MPM: Problem Sketch (2)

MPM: Problem Sketch (3)

Probability of misclassification…

… for worst-case class-conditional density…

… should be minimized !

MPM: Main Result (5)

MPM: GeometricInterpretation

Robustness to Estimation Errors: Robust MPM (R-MPM)

Robust MPM (R-MPM)

Robust MPM (R-MPM)

MPM: Convex Optimization to solve the problem

Linear Classifier

Nonlinear Classifier Kernelizing

Convex Optimization:Second OrderCone Program (SOCP)

) competitive with Quadratic Program

(QP) SVMs

LemmLemmaa

MPM: Empirical results=1– and TSA (test-set accuracy) of the MPM, compared to BPB (best performance in Breiman's report (Arcing classifiers, 1996)) and SVMs. (averages for 50 random partitions into 90% training and 10% test sets)

• Comparable with existing literature, SVMs• =1- is indeed smaller than the test-set accuracy in all cases (consistent with as worst-case bound on probability of misclassification)• Kernelizing leads to more powerfull decision boundaries (linear

decision boundary < nonlinear decision boundary (Gaussian kernel))

Advanced Convex Optimization in Machine Learning

The idea (1)

Machine learning

Kernel-based machine learning

The idea (2)

training set (labelled)

test set (unlabelled)

The idea (4)

Hard margin SVM classifiers (3)

Hard margin SVM classifiers (4)

SDP !

Hard margin SVM classifiers (5)

training set (labelled)

test set (unlabelled)

Learning the kernel matrix !

Hard margin SVM classifiers (7)

?

Hard margin SVM classifiers (8)

Hard margin SVM classifiers (11)

Learning Kernel Matrix with SDP !

Empirical results hard margin SVMs

See also