Correntropy as a similarity measure

Weifeng Liu, P. P. Pokharel, Jose Principe

Computational NeuroEngineering LaboratoryUniversity of Florida

http://www.cnel.ufl.eduweifeng@cnel.ufl.edu

Acknowledgment: This work was partially supported by NSF grant ECS-0300340 and ECS-0601271.

Outline

What is correntropy Interpretation as a similarity measure Correntropy Induced Metric robustness Applications

Correntropy: General Definition

For random variables X, Y correntropy is

where K is Gaussian kernel

Sample estimator

( , ) [ ( )]V X Y E k X Y

1

1ˆ( , ) ( , )N

i ii

V X Y k x yN

2

2

( )1( ) exp( ).

22i

i

x xk x x

Correntropy = ‘Correlation’ + ‘Entropy’

Correlation with high order moments– Taylor expansion of Gaussian kernel– Kernel size large, second order moment

dominates

Average over dimensions is the argument of Renyi’s quadratic entropy

Reproducing Kernel Hilbert Space induced by Correntropy- (VRKHS)

V(t,s) is symmetric and positive-definite Defines a unique Reproducing Kernel Hilbert

Space---VRKHS– Wiener filter is an optimal projection in RKHS

defined by autocorrelation– Analytical nonlinear Wiener filter framed as an

optimal projection in VRKHS

Probabilistic Interpretation

Integration of joint PDF along x=y line Probability of Probability density of X=Y

, ,

,

( , ) ( ) ( , ) ( , ) |

(| | / 2 )( )

2

X Y X Y x y u

x y

V X Y k x y f x y dxdy f x y du

P Y Xp X Y

| |Y X

Probabilistic Interpretation

Geometric meaning

Two vectors

Define a function CIM

1 2( , ,..., )TNX x x x 1 2( , ,..., )TNY y y y

1/ 2ˆ( , ) ( (0) ( , ))CIM X Y k V X Y

Correntropy Induced Metric

CIM is Non-negative CIM is Symmetric CIM obeys the triangle inequality

Therefore it is a metric that is induced in the input space when one operates with correntropy

Metric contours

Contours of CIM(X,0) in 2D sample space

close, like L2 norm Intermediate, like L1

norm far apart, saturates with

large-value elements (direction sensitive)

0.05

0.1

0.1

0.15

0.15

0.2

0.2

0.2

0.25

0.25

0.250.25

0.3

0.3

0.3

0.3

0.30.35

0.35

0.35

0.35

0.35

0.35

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.4

0.45

0.45

0.45

0.45

0.45

0.45

0.45

0.45

0.5

0.5

0.5

0.5

0.55

0.55

0.55

0.55

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

CIM versus MSE as a cost function

Localized similarity measure

2

2

,

2

( , ) [( ) ]

( ) ( , )

( )

XY

x y

E

e

MSE X Y E X Y

x y f x y dxdy

e f e de

,

( , ) [ ( )]

( ) ( , )

( ) ( ) .

XY

x y

E

e

V X Y E k X Y

k x y f x y dxdy

k e f e de

CIM is robust to outliers

measure similarity in a small interval; Do not care how different outside the interval

Resistant to outliers (in the sense of Huber’s M-estimation)

(| | / 2 )P Y X

Application 1: Matched filter

S transmitted binary signal

N channel noise Y received signal

Y S N

Application 1: Matched filter

Sampled (1,-1) received signal Linear matched filter

Correntropy matched filter

1

1ˆ ( ).

M

ii

s sign yM

1 1

1 1ˆ ( ( ,1) ( , 1)).

M M

i ii i

s sign k y k yM M

{ , 1,2,..., }iy i M

Application 1: Matched filter

-80 -70 -60 -50 -40 -30 -20 -10 0 1010

-6

10-5

10-4

10-3

10-2

10-1

100

SNR(dB)

Err

or R

ate

MSE Error Rate M=10

Correntropy Error Rate(<1e-6) M=10, 20

MSE Error Rate M=20

SNR (dB)

BER

Application 2: Robust Regression

X input variable f unknown function N noise Y observation

( )Y f X N

Application 2: Robust Regression

Maximum Correntropy Criterion (MCC)1

1max ( ) ( ( ) )

M

i ii

J w k g x yM

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-2

0

2

4

6

8

10

x

y,f,

g

Regression Results

f(x)

Yg(x) MSE

g(x) Correntropy

X

y=g(x)

MCC is M- Estimation

2 2( ) (1 exp( / 2 )) / 2e e

1

min ( )N

ii

e

2

1

min ( )N

i ii

w e e

2 2 3( ) exp( / 2 ) / 2w e e

MCC

Significance

Correntropy is a building block of– correntropy nonlinear Wiener filter– correntropy matched filter– correntropy nonlinear MACE filter– correntropy Principal Component Analysis– Renyi’s quadratic entropy

This understanding is crucial to explain the behavior of nonlinear algorithms and high-order statistics!

References

[1] I. Santamaria, P. P. Pokharel, J. C. Principe, “Generalized correlation function: definition, properties and application to blind equalization,” IEEE Trans. Signal Processing, vol 54, no 6, pp 2187- 2186

[2] P. P. Pokharel, J. Xu, D. Erdogmus, J. C. Principe, “A closed form solution for a nonlinear Wiener filter”, ICASSP2006

[3] Weifeng Liu, P. P. Pokharel, J. C. Principe, “Correntropy: Properties and Applications in Non-Gaussian Signal Processing”, submitted to IEEE Trans. Signal Proc.