Junseok Kwon* and Kyoung Mu lee

Computer Vision Lab.Dept. of EECS Seoul National University, Korea

Homepage: http://cv.snu.ac.kr

Goal of Visual Tracking

Robustly tracks the target in real-world scenarios

Frame #1 Frame #43

Bayesian Tracking Approach Maximum a Posteriori (MAP) estimate

)Y|X(pmaxarg t:1tXt

Intensity

edge

}X,X,X{X st

yt

xtt

State Sampling

MAP estimate by Monte Carlo sampling

),Y|X(pmaxarg t:1)l(

tX )l(

t

N,,1l

X position

Y po

sitio

n

Sca

le

State space

Visual tracker

Guided by

Problem of previous works Conventional trackers have difficulty in

obtaining good samples.

Visual tracker

Tra

ckin

g e

nvir

onm

ent

changes

Fixed

can not reflect the changing

tracking environment

well.

Tracker space

Tracker sampling

Our approach : Tracker Sampling Sampling tracker itself as well as state

X positionY

posit

ion

Sca

le

X positionY

posit

ion

Sca

le

Tracker

#2

Tracker

#M

X positionY

posit

ion

Sca

le Tracker

#1

State sampling

Two challenges

How the tracker space is defined?

When and which tracker should be sampled?

Tracker space

Tracker

#1

Tracker

#2

Tracker

#M

Tracker space

Challenge 1 : Tracker Space Tracker space

Nobody tries to define tracker space. Very difficult to design the space because

the visual tracker is hard to be described.

Tracker space

Bayesian Tracking Approach

)Y|X(pmaxarg t:1tXt

1t1t:11t1tttt dX)Y|X(p)X|X(p)X|Y(p

Go back to the Bayesian tracking formulation

Updating rule

Bayesian Tracking Approach

1t1t:11t1tttt dX)Y|X(p)X|X(p)X|Y(p

What is important ingredients of visual tracker?

1. Appearance model

2. Motion model

3. State representation type

4. Observation type

Appearance

model

Mot

ion

mod

el

Stat

e

repr

esen

tatio

n

Observation

Tracker Space

Motion model

)M( t

Observation type )O( t

State representation type )S( t

Appearance model )A( t

Challenge 2 : Tracker Sampling Tracker sampling

When and which tracker should be sampled ? To reflect the current tracking environment.

Tracker space

Tracker

#m

Reversible Jump-MCMC

We use the RJ-MCMC method for tracker sampling.

1tA

2tA

|A|t

tA

Add Delete

Set of sampled appearance

models

1tM

2tM

|M|t

tM

Add Delete

Set of sampled motion models

1tS

2tS |S|

ttS

Add Delete

Set of sampled state

representation types

1tO

2tO

|O|t

tO

Add Delete

Set of sampled observation types

Sampled basic trackers

1tT 2

tT |O|S||M||A|t

ttttT

Sampling of Appearance Model Make candidates using SPCA*

The candidates are PCs of the target appearance.

Appearance models

* A. d’Aspremont et. al. A direct formulation for sparse PCA using semidefinite programming. Data Min. SIAM Review, 2007.

Sparse Principle Component Analysis*

Accept an appearance model With acceptance ratio

*tA

1i

*t

1t

5tj

itjjt:1t

*t

t*tt:1tt

*ttt:1t

*t

Alog)),X(Y(DD)Y,X|A(plogwhere

)A;A(Q)Y,X|A(p

)A;A(Q)Y,X|A(p,1min

Sampling of Appearance Model

Our method has the limited number of

models

The accepted model increase the total likelihood scores for recent frames

When it is adopted as the target reference

*tA

1i

1t

5tj

itjj )),X(Y(DD

Sampling of Motion Model

Make candidates using KHM* The candidates are mean vectors of the

clusters for motion vectors.

Motion models

K-Harmonic Means Clustering (KHM)*

* B. Zhang, M. Hsu, and U. Dayal. K-harmonic means - a data clustering algorithm. HP Technical Report, 1999

Sampling of Motion Model

Accept a motion model With acceptance ratio

*tM

1i

*titt:1t

*t

t*tt:1tt

*ttt:1t

*t

Mlog),D(VAR)Y,X|M(plogwhere

)M;M(Q)Y,X|M(p

)M;M(Q)Y,X|M(p,1min

Our method has the limited number of

models

The accepted model decreases the total clustering error of motion vectors for recent frames

When it is set to the mean vector of the cluster

*tM

1iit ),D(VAR

Sampling of State Representation

Make candidates using VPE* The candidates describe the target as the

different combinations of multiple fragments.

Vertical Projection of Edge (VPE)*

EdgePosi

tion

Intensity

* F.Wang, S. Yua, and J. Yanga. Robust and efficient fragments-based tracking using mean shift. Int. J. Electron. Commun., 64(7):614–623, 2010.

State representatio

n

Fragment 1

Fragment 2

Accept a state representation type With acceptance ratio

*t

itS

1i

F

1j

*tjt:1t

*t

t*tt:1tt

*ttt:1t

*t

Slog)f(VAR)Y,X|S(plogwhere

)S;S(Q)Y,X|S(p

)S;S(Q)Y,X|S(p,1min

Sampling of State Representation

Our method has the limited number of

types

The accepted type reduce the total variance of target appearance in each fragment for recent frames

*t

itS

1i

F

1jj )f(VAR

Sampling of Observation

Make candidates using GFB* The candidates are the response of multiple

Gaussian filters of which variances are different.

Gaussian Filter Bank (GFB)*

* J. Sullivan, A. Blake, M. Isard, and J. MacCormick. Bayesian object localisation in images. IJCV, 44(2):111–135, 2001.

Sampling of Observation Accept an observation type

With acceptance ratio

*tO

1i

1t

5tk,j

ik

ij

O

1i

1t

5tk,j

ik

ij

t:1t*t

t*tt:1tt

*ttt:1t

*t

Olog

),(DD

),(DD

)Y,X|O(plogwhere

)O;O(Q)Y,X|O(p

)O;O(Q)Y,X|O(p,1min

*t

*t

Our method has the limited number of

types

The accepted type makes more similar between foregrounds, but more different with foregrounds and backgrounds for recent frames

*t

*t

O

1i

1t

5tk,j

ik

ij

O

1i

1t

5tk,j

ik

ij

),(DD

),(DD

Tracker space

Overall Procedure

X positionY

posit

ion

Sca

le

X positionY

posit

ion

Sca

le

X positionY

posit

ion

Sca

le

Tracker #1

Tracker #2

Tracker #M

Tracker sampling

State sampling

Inte

racti

on

Qualitative Results

Qualitative Results

Iron-man dataset

Qualitative Results

Matrix dataset

Qualitative Results

Skating1 dataset

Qualitative Results

Soccer dataset

Quantitative Results

MC IVT MIL VTD Ourssoccer 53 116 41 23 17

skating1 172 213 85 8 8

animal 26 21 30 22 10

shaking 98 150 38 20 5

Soccer* 72 225 147 34 24

Skating1*

126 291 87 16 8

Iron-man 78 104 122 30 15

Matrix 123 50 57 80 12Average center location errors in pixels

IVT : Ross et. al. Incremental learning for robust visual tracking. IJCV 2007.MIL : Babenko et. al. Visual tracking with online multiple instance learning. CVPR 2009.

MC : Khan et. al. MCMC-based particle filtering for tracking a variable number of interacting targets. PAMI 2005.

VTD: Kwon et. al. Visual tracking decomposition. CVPR 2010.

Summary

Visual tracker sampler New framework, which samples visual

tracker itself as well as state. Efficient sampling strategy to sample the

visual tracker.

http://cv.snu.ac.kr/paradiso