Dorin Comaniciu, Visvanathan Ramesh and Peter Meer “Kernel-Based Object Tracking”, IEEE...

Dorin Comaniciu, Visvanathan Ramesh and Peter Meer “Kernel-Based Object Tracking”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.25, No 5, May 2003

Dorin Comaniciu and Peter Meer, “Mean shift: a robust approach toward feature space analysis”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.25, no 5, May 2002

Dorin Comaniciu and Peter Meer, “Mean shift analysis and applications", The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol.2, pp.1197-1203, September 1999

Mean-shift and its application for object tracking

Andy {andrey.korea@gmail.com}

2Intelligent Systems

Lab.

What is mean-shift?

Non-parametricDensity Estimation

Non-parametricDensity GRADIENT Estimation

(Mean Shift)

Data

Discrete PDF Representation

PDF Analysis

A tool for: finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in RN

3Intelligent Systems

Lab.

Intuitive description

Region ofinterest

Center ofmass

Mean Shiftvector

Objective : Find the densest region

4Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

Mean Shiftvector

5Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

Mean Shiftvector

6Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

Mean Shiftvector

7Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

Mean Shiftvector

8Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

Mean Shiftvector

9Intelligent Systems

Lab.

Intuitive descriptionObjective : Find the densest region

Region ofinterest

Center ofmass

10Intelligent Systems

Lab.

Modality analysisTessellate the space with windows

Run the procedure in parallel

11Intelligent Systems

Lab.

Modality analysis

The blue data points were traversed by the windows towards the mode

12Intelligent Systems

Lab.

Modality analysis

Window tracks signify the steepest ascent directions

13Intelligent Systems

Lab.

Mean shift for data clustering

14Intelligent Systems

Lab.

Data clustering example

Simple Modal Structures

Complex Modal Structures

15Intelligent Systems

Lab.

Data clustering example

Initial windowcenters

Modes found Modes afterpruning

Final clusters

Feature space:L*u*v representation

16Intelligent Systems

Lab.

Image segmentation examples

17Intelligent Systems

Lab.

General framework: target representation

Current frame

… …

Choose a feature space

Represent the model in the

chosen feature space

Choose a reference

model in the current frame

18Intelligent Systems

Lab.

General framework: target localization

Search in the model’s

neighborhood in next frame

Start from the position of the model in the current frame

Find best candidate by maximizing a similarity func.

Repeat the same process in the next pair

of frames

Current frame

… …Model Candidate

19Intelligent Systems

Lab.

Target representation

Choose a reference

target model

Quantized Color Space

Choose a feature space

Represent the model by its PDF in the

feature space

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 . . . m

color

Pro

bab

ility

20Intelligent Systems

Lab.

PDF representation

,f y f q p y Similarity

Function:

Target Model(centered at 0)

Target Candidate(centered at y)

1..1

1m

u uu mu

q q q

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 . . . m

color

Pro

bab

ility

1..

1

1m

u uu mu

p y p y p

0

0.05

0.1

0.15

0.2

0.25

0.3

1 2 3 . . . m

color

Pro

bab

ility

21Intelligent Systems

Lab.

Smoothness of the similarity function

f ySimilarity Function: ,f y f p y q

Problem:Target is

represented by color info only

Spatial info is lost

Solution:Mask the target with an isotropic kernel in the spatial domain

f(y) becomes smooth in y

f is not smooth

Gradient-based

optimizations are not robust

Large similarity variations for

adjacent locations

22Intelligent Systems

Lab.

PDF with spatial weighting

1..i i nx

Target pixel locations

( )k x A differentiable, isotropic, convex, monotonically decreasing kernel• Peripheral pixels are affected by occlusion and background interference

( )b x The color bin index (1..m) of pixel x

2

( )i

u ib x u

q C k x

Normalization factor

Pixel weight

Probability of feature u in model

0

0.05

0.1

0.15

0.2

0.25

0.3

1 2 3 . . . m

color

Pro

bab

ility

Probability of feature u in candidate

0

0.05

0.1

0.15

0.2

0.25

0.3

1 2 3 . . . m

colorP

rob

abili

ty

2

( )i

iu h

b x u

y xp y C k

h

Normalization factor Pixel weight

0

model

y

candidate

23Intelligent Systems

Lab.

Similarity function

1 , , mp y p y p y

1, , mq q q

Target model:

Target candidate:

Similarity function: , ?f y f p y q

1 , , mp y p y p y

1 , , mq q q

q

p yy1

1

1

cosT m

y u uu

p y qf y p y q

p y q

The Bhattacharyya Coefficient

m

uuu qpqypff

1

)(]),([)( yy

24Intelligent Systems

Lab.

Target localization algorithm

,f p y q

Start from the position of the model in the current frame

q

Search in the model’s

neighborhood in next frame

p y

Find best candidate by maximizing a similarity func.

25Intelligent Systems

Lab.

Approximating similarity function

01 1 0

1 1

2 2

m mu

u u uu u u

qf y p y q p y

p y

Linear

approx.(around y0)

0yModel location:yCandidate location:

2

12

nh i

ii

C y xw k

h

2

( )i

iu h

b x u

y xp y C k

h

Independent of y

Density estimation

(as a function of y)

1

m

u uu

f y p y q

26Intelligent Systems

Lab.

Computing gradient of similarity function

i

n

i

ii

hn

i

ii

hm

uuu w

h

xykxy

h

C

h

xykw

Cqypyf

hh

12

110 '

22ˆ)(ˆ

2

1

xkxg '

y

hxy

gw

hxy

gwx

h

xygw

h

Cw

h

xygyx

h

Ch

h

hh

n

i

ii

n

i

iiin

i

ii

hi

n

i

ii

h

1

1

12

12 22

2

1

2

1

( )

ni

ii

ni

i

gh

gh

x - xx

m x xx - x

Simple Mean Shift procedure:• Compute mean shift vector

•Translate the Kernel window by m(x)

27Intelligent Systems

Lab.

Kernels and kernel profiles

y

h

xygw

h

xygwx

h

xygw

h

Cw

h

xykxy

h

Cyf

h

h

hh

n

i

ii

n

i

iiin

i

ii

hi

n

i

ii

h

1

1

12

12 2

'2

xkxg ' k – kernel profile

2xckxK K– kernel

• Epanechnikov Kernel

• Uniform Kernel

• Normal Kernel

21 1

( ) 0 otherwise

E

cK

x xx

1( )

0 otherwiseU

cK

xx

21( ) exp

2NK c

x x

28Intelligent Systems

Lab.

Epanechnikov kernel

Epanechnikov kernel:

1 if x 1

0 otherwise

xk x

A special class of radially symmetric kernels: 2

K x ck x

11

1

n

i iin

ii

x wy

w

2

0

1

1 2

0

1

ni

i ii

ni

ii

y xx w g

hy

y xw g

h

1 if x 1

0 otherwiseg x k x

Uniform kernel:

29Intelligent Systems

Lab.

Tracking with mean-shift1. Initialize the location of the target in the current frame (y0) with location of model on previous frame, compute candidate target model p(y0) and evaluate �����������������

2. Compute weights3. Find the next location of the target candidate according to

4. Compute candidate model at new position p(y1) �����������������5. If ||y0-y1||<ε Stop. Else – set y0 = y1 and go to step 2.

m

uuu qypqypf

100 )(]),([

2

0

1

1 2

0

1

ni

i ii

ni

ii

y xx w g

hy

y xw g

h

30Intelligent Systems

Lab.

Conclusions

Advantages :

• Application independent tool

• Suitable for real data analysis

• Does not assume any prior shape (e.g. elliptical) on data clusters

• Can handle arbitrary feature spaces

• Only ONE parameter to choose

• h (window size) has a physical meaning, unlike K-Means

Disadvantages:

• The window size (bandwidth selection) is not trivial

• Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes Use adaptive window size