Embed Size (px)
Citation preview
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1
Support Vector Machines
Find a linear hyperplane (decision boundary) that will separate the data
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2
Support Vector Machines
One Possible Solution
B1
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 3
Support Vector Machines
Another possible solution
B2
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 4
Support Vector Machines
Other possible solutions
B2
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 5
Support Vector Machines
Which one is better? B1 or B2? How do you define better?
B1
B2
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 6
Support Vector Machines
Find hyperplane maximizes the margin => B1 is better than B2
B1
B2
b11
b12
b21
b22
margin
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 7
| |
b
w
X1
X2
x
w
0w x b
| || | cos 0
| | cos| |
x w b
bx
w
| |
b
w
此條線為由原點沿 w
方向走 與 垂直的線w
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 8
Support Vector Machines
B1
b11
b12
0 bxw
1 bxw 1 bxw
1bxw if1
1bxw if1)(
xf
2Margin
|| ||w
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 9
w
dx1
x2
x1-x2
w•x+b = -1
w•x+b=1
W•X +b =0
w•x+b=0
0
此線在沿法向量走 -b/|w| 的距離，與法向量垂直
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 10
開始推導 w•x+b=0 這是一條怎樣的線呢？ xa , xb 為線上兩點 w•xa+b=0,
w•xb+b=0
兩式相減 w •(xa-xb)=0 線上兩點所形成的向量與 w 垂直 所以 w 是法向量
w •xs+b=k ， xs 在上面的線 k > 0
w •xc+b=k’ ， xc 在下面的線 k’ < 0
方塊分類為 y=1 ， 圓圈分類為 y=-1 分類公式 z 是 y=1 類或 y=-1 類依據：
1,~ ~ 0
1,~ ~ 0
if w z by
if w z b
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 11
調整 w 與 b 得到
w•(x1-x2)=2 |w|*d=2 ∴
1
2
: 1,
: 1i
i
b w x b
b w x b
2
| |d w
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 12
Support Vector Machines
We want to maximize:
– Which is equivalent to minimizing:
– But subjected to the following constraints:
This is a constrained optimization problem– Numerical approaches to solve it (e.g., quadratic programming)
2
2Margin
| |w
1bxw if1
1bxw if1)(
i
i
ixf
2| |( )
2
wL w
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 13
Min f(w)=
Subject to
Change to Min f(w)=
Subject to
2| |
2
w
( ) 1, 1,2,..., .i iy w x b i N
( ( ) 1) 0, 1,2,...,i iy w x b i N
2| |
2
w
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 14
2
1
1| | ( 1)
2
N
P i i ii
L w y w x b
1
0N
Pi i i
i
Lw y x
w
1
0 0N
Pi i
i
Ly
b
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 15
Nonlinear Support Vector Machines
What if decision boundary is not linear?
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 16
Nonlinear Support Vector Machines
Transform data into higher dimensional space
