Junseok Kwon* and Kyoung Mu lee

Computer Vision Lab.Dept. of EECS Seoul National University, Korea

Homepage: http://cv.snu.ac.kr

Goal of Visual Tracking

Robustly tracks target in real-world scenarios

Frame #1 Frame #60

Real-World Scenarios

Occlusions

Illumination changes

Abrupt motions

Pose variationsMixed

Previous Works

In the real-world scenarios, conventional tracking methods frequently fail.

OAL Tracker [2]MIL Tracker [1]Our method

[2] Ross et. al. Incremental learning for robust visual tracking. IJCV 2007.

[1] Babenko et. al. Visual tracking with online multiple instance learning. CVPR 2009.

Bayesian Tracking Approach Maximum a Posteriori (MAP) estimate

)Y|X(pmaxarg t:1tXt

Position, scalecolor

edge

Bayesian Tracking Approach

)Y|X(pmaxarg t:1tXt

)X|Y(p tt )X|X(p 1tt 1t1t:11t dX)Y|X(p

Observation model Motion model

Update rule

Compound Model

Compound Observation Model

Pose variation

Occlusion

Illumination change

Clutters

Smooth

Abrupt

Compound Motion Model

Need for real-world scenarios But difficult to design

Our Approach Observation Model

Decomposition

+ + +Basic

Observation Model

1

Basic Observation Model r

Basic Observation Model 2

Compound Observation Model

1w,)X|Y(pw)X|Y(pr

1ii

r

1ittiitt

Our Approach

+ + +Basic

Motion Model 2

Basic Motion Model s

Basic Motion Model

1

Motion Model Decomposition

1w,)X|X(pw)X|X(ps

1jj

s

1j1ttjj1tt

Compound Motion Model

Our Approach

Basic Tracker 1

Basic Tracker 2

Basic Tracker rs

Tracker DecompositionBasic

Observation Model 1

Basic Motion Model 1

Basic Motion Model 2

Basic Motion Model s

Basic Observation Model r

Basic Observation Model 2

Basic Observation Model 1

Basic Motion Model 1

Basic Observation Model 1

Basic Motion Model 2

Basic Observation Model r

Basic Motion Model s

Our Approach

Basic Tracker 1

Basic Tracker 2

Basic Tracker rs

Tracker Decomposition Each tracker takes charge of a certain

change in the object.

Our Approach

Sampling based Tracker Markov Chain Monte Carlo (MCMC)

Basic Tracker

Sampling…

Basic Observation Model i

Basic Motion Model j

Basic Observation Model i

Basic Motion Model j

)Y|X(p t:1tji

Remaining Tasks

How to determine the basic models ?

How to estimate weights of the models ?

Basic Observation Model 1

Basic Observation Model r

Basic Motion Model 1

Basic Motion Model s

Basic Observation Model 1

Basic Observation

Model r

Basic Motion Model 1

Basic Motion Model s

Remaining Tasks

How to determine the basic models ?

Sparse PCA [1]

How to estimate weights of the models ? Interactive MCMC [2]

[2] J. Corander et. al. Parallell interacting MCMC for learning of topologies of graphical models. Data Min. Knowl. Discov., 2005.

[1] A. d’Aspremont et. al., A direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49(3), 2007.

Design of Basic Observation Models

Template set

tS

Hue

Saturation

Value

Edge

4 recent frames

1 initial frame

1tM 2

tM rtM

Object models

A subset of the template set

)X|Y(p tt1 )X|Y(p tt2 )X|Y(p ttr

Basic observation models

Diffusion distance

Object Model

Three conditions Representativeness

The model has to cover most appearance changes in an object over time.

Compactness

The formation of it should be as compact as possible.

Complementary relation The relations between models should be

complementary.

1tM 2

tM rtM

Object Model

Sparse Principal Component Analysis (SPCA)

1c to

2

2

subject

ccAcMaximize tT PCA Sparsenes

s

tAc

: Gram matrix of the template set

: Principal component

1tM 2

tM rtM

Object Model

1subject

ccAcMaximize

2

2

tT

c to

tA

Tem

pla

te

set

Template set

Object Model

Sparse

PC 10 0 0 0 0 0 0 0

Object model 1

Representativeness

Sparse

PC 2 Object model 20 0 0 0 0 0 0 00 0

Compactness

Sparse

PC r Object model r0 0 0 0 0 00 0 0

Complementary relation

1subject

ccAcMaximize

2

2

tT

c to

Basic Observation Model

Diffusion distance [3]

[3] H. Ling and K. Okada. Diffusion distance for histogram comparison. CVPR, 2006.

)M,λDD(Y ittexp

Edge

Object model

Diffusion distance [3]

Hue Saturation

Value

Edge

)X|(Yp tti

Two conditions Exploitation ( for smooth motions )

Further simulating the seemingly good moves near the local minima

Exploration ( for abrupt motions ) Further simulating moves that have not

been explored much

ExplorationExploitation

Design of Basic Motion Models

),(| 211 jtttj XG)X(Xp

Weights of Basic Models

Parallel Mode Interaction Mode

Basic Tracker 1

Basic Tracker 2

Basic Tracker rs

Basic Observation Model 1

Basic Motion Model 2

Basic Observation Model 1

Basic Motion Model 1

Basic Observation Model r

Basic Motion Model s

State

State

Experimental Results

The number of models Basic observation models : #4 Basic motion models : #2 Basic tracker models: #8(=4X2)

Settings for comparison Standard MCMC (MC) : 800 samples Mean Shift (MS) On-line Appearance Learning (OAL) : 800

samples Multiple Instance Learning (MIL)

OAL : Ross et. al. Incremental learning for robust visual tracking. IJCV 2007.MIL : Babenko et. al. Visual tracking with online multiple instance learning. CVPR 2009.

Experimental Results

Abrupt Motions and Illumination Changes

Illumination Changes and Pose Variations

Occlusions and Pose Variations

Background Clutters

Quantitative ResultsMC MS OAL MIL VTD

tiger1 27 93 65 15 13

david 49 88 4 23 7

face 19 45 19 27 7

shaking

97 241 95 38 5

soccer 47 97 151 41 21

animal 32 207 23 30 11

skating1

111 141 174 85 7

- Average center location errors in pixels

OAL : Ross et. al. Incremental learning for robust visual tracking. IJCV 2007.MIL : Babenko et. al. Visual tracking with online multiple instance learning. CVPR 2009.

MS : Comaniciu et. al. Real-time tracking of nonrigid objects using mean shift. CVPR 2000.

Summary

Visual tracking decomposition (VTD) Our method successfully tracks an object

whose motion and appearance change at the same time

Since VTD is easy to extend by adding new features or trackers, our method can be more improved.

http://cv.snu.ac.kr/paradiso