Upload
michael-palmer
View
215
Download
0
Embed Size (px)
Citation preview
Prakash ChockalingamPrakash Chockalingam
Clemson UniversityClemson University
Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models
Committee MembersCommittee Members
Dr Stan Birchfield (chair)Dr Stan Birchfield (chair)Dr Robert SchalkoffDr Robert Schalkoff
Dr Brian DeanDr Brian Dean
Tracking OverviewTracking Overview
Tracker Tasks
Feature Descriptors
Object Model
Update / LearningMechanism
TrackingFramework
Color
Gradients
Texture
Shape
Motion
Template
Contour
Active Appearance
ProbabilityDensities
Mean Shift
Pixel-wiseClassification
Optical Flow
Filtering techniques
No Update
Adaboost
Expectation Maximization
Re-weightingStrategy
Object Detection
Manual
Segmentation
Feature Points
ApproachApproach
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
Tracking FrameworkTracking Framework
Bayesian Formulation:Bayesian Formulation:
0: 0: 1 0: 1
arg
( | , ) ( | ) ( | ) ( | )t t t t t t t t t
t et background shape
p I p I p I p
Image data of all frames
Contour at time t Previously seen contours
*
**( | ) ( | )t t t
y R
p I p y
Assuming conditional independence among pixels,
Feature vector
* { , }
Object ModelingObject Modeling
f1
f2
*( | )tp y ?
Gaussian Mixture Model (GMM):
*
*1
( | ) ( | , )k
t tj
prior likelihood
p j p y j
( | )tp j * * 1 **
1( | , ) exp{ ( ) ( ) ( )}
2T
t j j jp y j y y
Strength Image:
( )( ) log
( )
p xx
p x
>0 for Foreground<0 for Background
y
Strength ImageStrength Image
GMM Linear Classifier Single Gaussian
Strength Image (contd…)Strength Image (contd…)
…
Linear ClassifierSingle Gaussian
Individual Fragments
Final Strength Strength Without Spatial Information
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
Contour ExtractionContour Extraction
Implicit representation of growing region
Likelihood term(Strength image) Regularization term
Energy Functional:
(strength image) (frontier)
( ) ( ) * ( )x x
E x x G x
> 0 Inside< 0 Outside
Contour Extraction (contd…)Contour Extraction (contd…)
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
(Region to be shrunk)
(Region already grown)
(Region to be grown)
(Region that need not be considered)
Contour Extraction (contd…)Contour Extraction (contd…)
4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x
4{ : ' ( )x x N x ( ) 0, ( ') 0}gx x such that
x x’
xx’
4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x
4{ : ' ( )x x N x ( ) 0, ( ') 0}g gx x such that
Dilation
Contraction
Contour Extraction (contd…)Contour Extraction (contd…)
Expand
Remove interior points
Contract
Remove exterior points
Contour Extraction (contd…)Contour Extraction (contd…)
Likelihood
Final Region
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
Region SegmentationRegion Segmentation
Mode-seeking region growing algorithm:
do {
• Pick a seed point that is not associated to any fragment
• Grow the fragment from the seed point based on the similarity of the pixel and its neighbor’s appearance
• Stop growing the fragment if no more similar pixels are present in the neighborhood of the fragment
} until all pixels are assigned
Seed point:
3
1
( ) ( )ii
x x
Eigen values of 3x3 RGB covariance matrix
1,..., nS where ( , , )i i i ix y
Region Segmentation (contd…)Region Segmentation (contd…)
• Pick the minimum element in S. Create a region to hold the pixel and add the neighbors in a fixed window.
• Compute Mean μj and Covariance Σj of the region.
• Likelihood:
• Grow the region as before with two additional steps: Update μj, and Σj, as a new pixel is added Remove the corresponding element in S if a pixel is added
• Continue above steps if S is not empty.
( ) ( ( ), ( , ))j j jx MD f x
Initial region
Mahalanobis distance
Configurable parameter
Region Segmentation (contd…)Region Segmentation (contd…)
Region Growing Graph-Based Mean-Shift
Region Segmentation (contd…)Region Segmentation (contd…)
Region Growing Graph-Based Mean-Shift
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
Update MechanismUpdate Mechanism
f1
f2
* *, ,,j t j t
• Update parameters of existing fragments
• Detect fragment occlusion
• Find new fragments
Initial Frame Initial Model Fragment Association
Update Mechanism (contd…)Update Mechanism (contd…)
* * * * *, ,0: ,0(1 )j t j j t j j
( ) *,
* 0,0:
( )
0
tt
j
j t tt
e
e
Initial Model
(function of past and current values)
Weight computed by comparing Mahalanobis distance
Updating parameters of existing fragments:
Update Mechanism (contd…)Update Mechanism (contd…)
Occluded fragments:
If a fragment is associated with less than 0.2% of the image pixels, then the fragment is declared as occluded.
Finding new fragments:
( ){ : log 0}
( )
p xT x
p x
Helps in handling self-occlusion
Spatial AlignmentSpatial Alignment
The spatial parameters are updated using the motion vectors from Joint Lucas-Kanade approach
Lucas-Kanade Joint Lucas-Kanade
Algorithm summaryAlgorithm summaryInitial frame:
• The user marks the object to be tracked.
• The target object and background scene are segmented based on their appearance similarity.
• The target object and background scene are modeled using a mixture of Gaussians where each Gaussian correspond to a fragment in the joint feature-spatial space
Subsequent frames:
• Update the spatial parameters of GMM using the motion vectors of Joint Lucas-Kanade
• Each pixel is classified into either foreground or background by generating a strength map using the Gaussian mixture model (GMM) of the object and background.
• The strength map is integrated into a discrete level set formulation to obtain accurate contour of the object.
• Using the tracked data, the appearance parameters of the GMM are updated.
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Extract contour using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
Experimental ResultsExperimental Results
Elmo Sequence Monkey Sequence
Experimental Results (Contd…)Experimental Results (Contd…)
Person Sequence Fish Sequence
Experimental Results: Self-Occlusion Experimental Results: Self-Occlusion
Without Self-Occlusion Module With Self-Occlusion Module
ConclusionConclusion
• A tracking framework based on modeling the object as mixture of Gaussians is proposed
• An efficient discrete implementation of level sets is employed to extract contour.
• A mode-seeking region growing algorithm is used to segment the image.
• A simple re-weighting strategy is proposed to update the parameters of Gaussians.
Future Directions:
• Incorporate shape priors.
• Utilize the extracted shapes to learn more robust priors.
• An offline or online evaluation mechanism during the initialization phase.
• Adding global information into the region segmentation process.
• Automating the object detection and initialization.
Questions ?Questions ?
Thank you !Thank you !