Introduction to Kernel Methods - GitHub Pages · to Kernel Methods F. Gonz alez Introduction The...

Introductionto KernelMethods

F. Gonzalez

Introduction

The KernelTrick

The KernelApproach toMachineLearning

A KernelPatternAnalysisAlgorithm

KernelFunctions

KernelAlgorithms

Kernels inComplexStructuredData

Introduction to Kernel Methods

Fabio A. Gonzalez Ph.D.

Depto. de Ing. de Sistemas e IndustrialUniversidad Nacional de Colombia, Bogota

September 11, 2015

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

1 IntroductionMotivation

2 The Kernel TrickMapping the input space to the feature spaceCalculating the dot product in the feature space

3 The Kernel Approach to Machine Learning

4 A Kernel Pattern Analysis AlgorithmPrimal linear regressionDual linear regression

5 Kernel FunctionsMathematical characterisationVisualizing kernels in input space

6 Kernel Algorithms

7 Kernels in Complex Structured Data

F. Gonzalez

Introduction

Motivation

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

Motivation

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

Motivation

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem 1

How to separate these two classes using a linear function?

F. Gonzalez

Introduction

Motivation

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem 2

How to do symbolic regression?

Σ = {A,C ,G ,T}

f : Σd → RACGTA 7→ 10.0GTCCA 7→ 11.3GGTAC 7→ 1.0CCTGA 7→ 4.5

......

F. Gonzalez

Introduction

The KernelTrick

Mapping the inputspace to the featurespace

Calculating the dotproduct in thefeature space

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem 1

• How to separate these two classes using a linear function?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• Map to R3:

φ : R2 → R3

(x , y) 7→ (x 2, y2, xy)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• Map to R3:

φ : R2 → R3

(x , y) 7→ (x 2, y2, xy)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Input space vs. feature space

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Dot product in the feature space

φ : R2 → R3

(x1, x2) 7→ (x 21 , x

22 ,√

2x1x2)

〈φ(x ), φ(z )〉 =⟨

(x 21 , x

22 ,√

2x1x2), (z21 , z

22 ,√

2z1z2)⟩

= x 21 z

21 + x 2

2 z22 + 2x1x2z1z2

= (x1z1 + x2z2)2

= 〈x , z 〉2

• A function k : X ×X → R such thatk(x , z ) = 〈φ(x ), φ(z )〉 is called a kernel

• Morale: you don’t need to apply φ explicitly tocalculate the dot product in the feature space!

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

φ : R2 → R3

(x1, x2) 7→ (x 21 , x

22 ,√

2x1x2)

〈φ(x ), φ(z )〉 =⟨

(x 21 , x

22 ,√

2x1x2), (z21 , z

22 ,√

2z1z2)⟩

= x 21 z

21 + x 2

2 z22 + 2x1x2z1z2

= (x1z1 + x2z2)2

= 〈x , z 〉2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

φ : R2 → R3

(x1, x2) 7→ (x 21 , x

22 ,√

2x1x2)

〈φ(x ), φ(z )〉 =⟨

(x 21 , x

22 ,√

2x1x2), (z21 , z

22 ,√

2z1z2)⟩

= x 21 z

21 + x 2

2 z22 + 2x1x2z1z2

= (x1z1 + x2z2)2

= 〈x , z 〉2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

φ : R2 → R3

(x1, x2) 7→ (x 21 , x

22 ,√

2x1x2)

〈φ(x ), φ(z )〉 =⟨

(x 21 , x

22 ,√

2x1x2), (z21 , z

22 ,√

2z1z2)⟩

= x 21 z

21 + x 2

2 z22 + 2x1x2z1z2

= (x1z1 + x2z2)2

= 〈x , z 〉2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Kernel induced feature space

• The feature space induced by the kernel is not unique:The kernel

k(x , z ) = 〈x , z 〉2

also calculates the dot product in the four dimensionalfeature space:

φ : R2 → R4

(x1, x2) 7→ (x 21 , x

22 , x1x2, x2x1)

• The example can be generalised to Rn

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Kernel induced feature space

• The feature space induced by the kernel is not unique:The kernel

k(x , z ) = 〈x , z 〉2

also calculates the dot product in the four dimensionalfeature space:

φ : R2 → R4

(x1, x2) 7→ (x 21 , x

22 , x1x2, x2x1)

• The example can be generalised to Rn

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

The Process

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

The Approach

• Data items are embedded into a vector space called thefeature space

• Linear relations are sought among the images of the dataitems in the feature space

• The pattern analysis algorithm are based only on thepairwise dot products, they do not need the actualcoordinates of the embedded points

• The pairwise dot products in the feature space could beefficiently calculated using a kernel function

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

The Approach

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

The Approach

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

The Approach

F. Gonzalez

Introduction

The KernelTrick

Primal linearregression

Dual linear regression

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem definition

• Given a training set S = {(x1, y1), . . . , (xl , yl )} of pointsxi ∈ Rn with corresponding labels yi ∈ R the problem isto find a real-valued linear function that best interpolatesthe training set:

g(x) = 〈w, x〉 = w′x =

n∑i=1

• If the data points were generated by a function like g(x),it is possible to find the parameters w by solving

Xw = y

x′1...

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem definition

• Given a training set S = {(x1, y1), . . . , (xl , yl )} of pointsxi ∈ Rn with corresponding labels yi ∈ R the problem isto find a real-valued linear function that best interpolatesthe training set:

g(x) = 〈w, x〉 = w′x =

n∑i=1

• If the data points were generated by a function like g(x),it is possible to find the parameters w by solving

Xw = y

x′1...

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Graphical representation

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Loss function

• Minimize

L(g ,S ) = L(w,S ) =

l∑i=1

(yi − g(xi))2 =

l∑i=1

L(g , (xi , yi))

• This could be written as

L(w,S ) = ‖ξ‖2 = (y −Xw)′(y −Xw)

• Optimization problem:

minwL(w,S ) = min

w(y −Xw)′(y −Xw)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Loss function

• Minimize

L(g ,S ) = L(w,S ) =

l∑i=1

(yi − g(xi))2 =

l∑i=1

L(g , (xi , yi))

L(w,S ) = ‖ξ‖2 = (y −Xw)′(y −Xw)

minwL(w,S ) = min

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Loss function

• Minimize

L(g ,S ) = L(w,S ) =

l∑i=1

(yi − g(xi))2 =

l∑i=1

L(g , (xi , yi))

L(w,S ) = ‖ξ‖2 = (y −Xw)′(y −Xw)

minwL(w,S ) = min

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

∂L(w,S )

∂w= −2X′y + 2X′Xw = 0,

thereforeX′Xw = X′y,

andw = (X′X)−1X′y

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Dual representation of the problem

• w = (X′X)−1X′y = X′X(X′X)−2X′y = X′α

• So, w is a linear combination of the training samples,w =

∑li=1 αixi .

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Dual representation of the problem

• w = (X′X)−1X′y = X′X(X′X)−2X′y = X′α

• So, w is a linear combination of the training samples,w =

∑li=1 αixi .

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• From the solution of the primal problem:

X′Xw = X′y,

• thenXX′Xw = XX′y,

• using the dual representation

XX′XX′α = XX′y,

• thenα = (XX′)−1y,

• andg(x) = w′x = α′Xx.

• Note: XX′ may be close to singular, or singular accordingto machine precision.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

X′Xw = X′y,

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

X′Xw = X′y,

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

X′Xw = X′y,

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

X′Xw = X′y,

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

X′Xw = X′y,

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Ridge regression

• If XX′ is singular, the pseudo-inverse could be used: tofind the w that satisfies X′Xw = X′y with minimal norm.

• Optimisation problem:

minwLλ(w,S ) = min

wλ ‖w‖2 +

l∑i=1

(yi − g(xi))2,

where λ defines the trade-off between norm and loss. Thiscontrols the complexity of the model (the process is calledregularization).

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Ridge regression

• If XX′ is singular, the pseudo-inverse could be used: tofind the w that satisfies X′Xw = X′y with minimal norm.

• Optimisation problem:

minwLλ(w,S ) = min

wλ ‖w‖2 +

l∑i=1

(yi − g(xi))2,

where λ defines the trade-off between norm and loss. Thiscontrols the complexity of the model (the process is calledregularization).

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• Taking the derivative and making it equal to zero:

X′Xw + λw = (X′X + λIn)w = X′y,

where In is an identity matrix of n × n dimension,

• then,w = (X′X + λIn)−1X′y.

• In terms of α:

w = λ−1X′(y −Xw) = X′α,

• thenα = λ−1(y −Xw) = (XX′ + λIl )

−1y.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• In terms of α:

w = λ−1X′(y −Xw) = X′α,

−1y.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• In terms of α:

w = λ−1X′(y −Xw) = X′α,

−1y.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• In terms of α:

w = λ−1X′(y −Xw) = X′α,

−1y.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Prediction function

g(x) = 〈w, x〉 =

⟨l∑

αixi, x

l∑i=1

αi 〈xi, x〉

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Ridge regression as a kernelmethod

• The Gram matrix G = XX′ is the matrix of dot products

G = XX′ =

x′1...

[x1 · · · xl ] =

〈x1, x1〉〈x1, xl 〉

〈xl , x1〉〈xl , xl 〉

• G may be replaced by a general kernel matrix, K, with

kij = k(xi, xj) = < φ(xi), φ(xj) >

• The α’s are calculated as:

α = (K + λIl )−1y

• The predicted function is approximated as:

g(x) =

l∑i=1

αik(x, xi) = y ′(K + λIl )−1

k(x, x1)...

k(x, xl)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

G = XX′ =

x′1...

[x1 · · · xl ] =

〈x1, x1〉〈x1, xl 〉

〈xl , x1〉〈xl , xl 〉

α = (K + λIl )−1y

g(x) =

l∑i=1

αik(x, xi) = y ′(K + λIl )−1

k(x, x1)...

k(x, xl)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

G = XX′ =

x′1...

[x1 · · · xl ] =

〈x1, x1〉〈x1, xl 〉

〈xl , x1〉〈xl , xl 〉

α = (K + λIl )−1y

g(x) =

l∑i=1

αik(x, xi) = y ′(K + λIl )−1

k(x, x1)...

k(x, xl)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

G = XX′ =

x′1...

[x1 · · · xl ] =

〈x1, x1〉〈x1, xl 〉

〈xl , x1〉〈xl , xl 〉

α = (K + λIl )−1y

g(x) =

l∑i=1

αik(x, xi) = y ′(K + λIl )−1

k(x, x1)...

k(x, xl)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

Mathematicalcharacterisation

Visualizing kernels ininput space

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Characterisation

Theorem(Mercer’s Theorem)A function

k : X ×X → R,

which is either continuous or has a countable domain, can bedecomposed

k(x, z) = 〈φ(x), φ(z)〉

into a feature map φ into a Hilbert space F applied to both itsarguments followed by the evaluation of the inner product in Fif and only if it satisfies the finitely positive semi-definiteproperty.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Some kernel functions

Assume k1 and k2 kernels:

• k(x, z) = p(k1(x, z)). p a polynomial with positivecoefficients.

• k(x, z) = exp(k1(x, z)).

• k(x, z) = exp(−‖x− z‖2 /(2σ2)). Gaussian kernel.

• k(x, z) = k1(x, z)k2(x, z)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• k(x, z) = exp(k1(x, z)).

• k(x, z) = k1(x, z)k2(x, z)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• k(x, z) = exp(k1(x, z)).

• k(x, z) = k1(x, z)k2(x, z)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• k(x, z) = exp(k1(x, z)).

• k(x, z) = k1(x, z)k2(x, z)

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Embeddings corresponding tokernels

• It is possible to calculate the feature space induced by akernel (Mercer’s Theorem)

• This can be done in a constructive way

• The feature space can even be of infinite dimension.

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

How to visualize?

• Choose a point in input space p0

• Calculate the distance from another point x to p0 in thefeature space:

‖φ(p0)− φ(x )‖2F = 〈φ(p0)− φ(x ), φ(p0)− φ(x )〉F= 〈φ(p0), φ(p0)〉F + 〈φ(x ), φ(x )〉F−2 〈φ(p0), φ(x )〉F

= k(p0, p0) + k(x , x )− 2k(p0, x )

• Plot f (x ) = ‖φ(p0)− φ(x )‖2F

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

How to visualize?

= k(p0, p0) + k(x , x )− 2k(p0, x )

• Plot f (x ) = ‖φ(p0)− φ(x )‖2F

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

How to visualize?

= k(p0, p0) + k(x , x )− 2k(p0, x )

• Plot f (x ) = ‖φ(p0)− φ(x )‖2F

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Identity kernel

k(x , z ) = 〈x , z 〉

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Quadratic kernel (1)

k(x , z ) = 〈x , z 〉2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Identity kernel (2)

k(x , z ) = 〈x , z 〉2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Gaussian kernel

k(x, z) = e−‖x−z‖22σ2

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Basic computations in featurespace

• Means

• Distances

• Projections

• Covariance

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Means

• Distances

• Projections

• Covariance

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Means

• Distances

• Projections

• Covariance

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Means

• Distances

• Projections

• Covariance

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Classification and regression

• Support Vector Machines

• Support Vector Regression

• Kernel Fisher Discriminant

• Kernel Perceptron

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Dimensionality reduction andclustering

• Kernel PCA

• Kernel CCA

• Kernel k -means

• Kernel SOM

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Kernel PCA

• Kernel CCA

• Kernel k -means

• Kernel SOM

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Kernel PCA

• Kernel CCA

• Kernel k -means

• Kernel SOM

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Kernel PCA

• Kernel CCA

• Kernel k -means

• Kernel SOM

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Outline

6 Kernel Algorithms

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Kernels in complex structured data

• Since kernel methods do not require an attribute-basedrepresentation of objects, it is possible to perform learningover complex structured data (or unstructured data)

• We only need to define a dot product operation (similarity,dissimilarity measure)

• Examples:

• Strings• Texts• Trees• Graphs

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Examples:

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

• Examples:

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Problem 2

How to do symbolic regression?

Σ = {A,C ,G ,T}

f : Σd → RACGTA 7→ 10.0GTCCA 7→ 11.3GGTAC 7→ 1.0CCTGA 7→ 4.5

......

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

• Define a kernel on strings

k : Σd × Σd → R

• Use the kernel along with a kernel learning regressionalgorithm to find the regression function

• What is a good candidate for k?

• a function that measures string similarity• higher value for similar strings, smaller value for different

strings• k(s1 . . . sd , t1 . . . td) =

∑ni=1 equal(si , ti)

equal(si , ti) =

{1 if si = ti

0 otherwise

• k(ACTAG ,CCTCG) =?

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

strings• k(s1 . . . sd , t1 . . . td) =

equal(si , ti) =

{1 if si = ti

0 otherwise

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

strings• k(s1 . . . sd , t1 . . . td) =

equal(si , ti) =

{1 if si = ti

0 otherwise

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

strings• k(s1 . . . sd , t1 . . . td) =

equal(si , ti) =

{1 if si = ti

0 otherwise

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

strings• k(s1 . . . sd , t1 . . . td) =

equal(si , ti) =

{1 if si = ti

0 otherwise

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Solution

strings• k(s1 . . . sd , t1 . . . td) =

equal(si , ti) =

{1 if si = ti

0 otherwise

• Is it a kernel?

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Induced Feature Space

• What is the feature space induced by k?

φ : Σd → R4d

s1 . . . sd 7→ (x 11 , . . . , x

14 , x

21 , . . . , x

24 , . . . , x

d1 , . . . , x

(x j1 , . . . , x

j4 ) =

(1, 0, 0, 0) if sj = ′A′

(0, 1, 0, 0) if sj = ′C′

(0, 0, 1, 0) if sj = ′G′

(0, 0, 0, 1) if sj = ′T′

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

Induced Feature Space

• What is the feature space induced by k?

φ : Σd → R4d

s1 . . . sd 7→ (x 11 , . . . , x

14 , x

21 , . . . , x

24 , . . . , x

d1 , . . . , x

(x j1 , . . . , x

j4 ) =

(1, 0, 0, 0) if sj = ′A′

(0, 1, 0, 0) if sj = ′C′

(0, 0, 1, 0) if sj = ′G′

(0, 0, 0, 1) if sj = ′T′

F. Gonzalez

Introduction

The KernelTrick

KernelFunctions

KernelAlgorithms

References

Shawe-Taylor, J. and Cristianini, N. 2004 Kernel Methodsfor Pattern Analysis. Cambridge University Press.

Introduction to Kernel Methods - GitHub Pages · to Kernel Methods F. Gonz alez Introduction The...

Documents

Kernel Methods - Alexander J. Smola

Basic Kernel Methods

Intro ML: Tutorial on Kernel Methods

Francesco Camastra- Kernel Methods for Clustering

Kernel Methods for Pattern Recognition 2018

Sparse Kernel Methods 1 Sparse Kernel Methods for Classification and Regression October 17, 2007 Kyungchul Park SKKU

Kernel methods & sparse methods for computer visionfbach/INRIA_summer_school_2011...Kernel methods & sparse methods for computer vision Francis Bach Sierra project, INRIA - Ecole Normale

Bias / Variance Analysis, Kernel Methods

Kernel methods in machine learning

Module 2: Splines and Kernel Methods

Introduction: Overview of Kernel Methodsfukumizu/Kyushu2008/Kernel_intro_1.pdf · Basic idea of kernel methods Some examples of kernel methods Outline Basic idea of kernel methods

Image Categorization Using Kernel Methods

Kernel Approximation Methods for Speech Recognition

Kernel-based machine learning methods

Kernel Methods

Kernel Smoothing Methods - GitHub Pages

Heat kernel methods for Lifshitz theories

Overview of Kernel Methods (Part 2)

› ~fbach › INRIA_summer_school_2011... · Kernel methods & sparse methods for computer vision2011-07-25 · Kernel methods & sparse methods for computer vision Francis Bach Sierra

Kernel methods in machine learning - Kernel Machines