Upload
vuthuy
View
218
Download
0
Embed Size (px)
Citation preview
NEAR-DUPLICATE KEYFRAME RETRIEVAL BY NONRIGID IMAGE MATCHING
Jianke Zhu, CUHKSteven C.H. Hoi, NTUMichael R. Lyu, CUHKShuicheng Yan, NUS
The Chinese University of Hong Kong
Near-Duplicate Keyframes
ProblemDuplicate video searchCopyright detection.
Challenges Different capture devices under a variety of illumination conditions.Video editing: photometric and geometric transformations, and occludes original video by adding captions.
Outline
Related work
Nonrigid Image MatchingFormulationAlgorithmCase Studies
Multi-level Near-Duplicate Keyframe (NDK) RetrievalFrameworkFormulation as a Machine Learning TaskSemi-supervised Support Vector Machine
Experiements
Conclusion
Related work I: Appearance-based Methods
Extend content-based image retrieval techniques for NDK detection and retrieval
Measure the similarity between two keyframes based on the extracted global features:
Color histogram and color moment (Zhang and Chang ACM MM’04, A. Qamra et al. TPAMI’05).Grid color moment (Zhao et al. CIVR’06)
Very efficient for finding identical copies
Not very robust to the variations of illumination changes, geometric transformations and occlusions.
Related work II: Feature-based Methods
Detect local keypoints in two keyframes
Measure the similarity by counting the number of correct correspondences between two keypoint sets
Overcome the limitations of global appearance based methods
Incur a heavy computational cost for point matching
Related work II: Feature-based Methods
PCA-SIFT with LSH indexing (Ke et al. ACM MM’04)Efficient feature matching with a rigid projective geometry assumption
Attributed Relational Graph (ARG) (Zhang and Chang, ACM MM’04)Involves the process of stochastic belief propagation
One-to-One Symmetric matching (OOS) (Zhao et al. TMM’07)Bipartite graph matchingA local smoothing constraint to remove the outlier matches
OOS with Pattern Entropy (PE) (C-W. Ngo et al. ACM MM’06)
Visual Keywords (VK) (Wu et al. CIVR’07)
Nonrigid Image Matching (NIM)
Nonrigid transformation between the two NDKsProgressive Finite Newton approach (CVPR’07, TPAMI’08)
A closed-form solution for nonrigid image matching
PCA-SIFT NIMOOS OOS-PE
Nonrigid Image Matching
Estimate the explicit mesh model with a few deformation parameters
Recover the local deformations from salient feature correspondences between two images and reject the outlier matches simultaneously
Choose the total number of inlier matches as a confidence measure to judge wheather the two keyframes are near-duplicate or not.
2D Nonrigid Shape Model
where are the barycentriccoordinates for the point m.
m
(xi,yi)
(xj,yj) (xk,yk)
Nonrigid Image Matching
Energy function:
Ec(s) is the sum of the weighted square residual errors for the matched points.
Er(s) represents the surface deformation energy, is composed of the sum of the squared second-order derivatives of the mesh vertex coordinates
where K is a sparse and banded matrix which is determined by the structure of the mesh model
is a regularization coefficient.
Error term Regularization term
Robust Estimator
Finite Newton Formulation
Modified Finite Newton optimization schemeUpdate equation:Set step size r = 1, and obtain the linear equation:
Multi-level NDK Retrieval
Local feature matching in NIM is computational intensiveTo improve the efficiency and scalability:
Formulation as a Machine Learning Task
Query set of label image examples ( )
Gallery set of unlabeled image examples
DifficultiesNo negative samples availableSmall sample learning: very few labeled examples
SolutionPseudo-negative example (Rong et al. ACM MM’03).Semi-Supervised Learning
Semi-Supervised SVM
SVMLearn an optimal hyperplane with maximal marginSVM in a regularization framework
Semi-Supervised SVMUtilize the unlabeled dataUnified kernel learning: data-dependent Kernel
Learning a Data-dependent Kernel
Kernel deformation (Vikas et al. ICML’05)First estimate the geometry of the underlying marginal distribution Derive a data dependent kernel by incorporating the estimated geometry
AssumeNew kernel functionwhere
New kernel function
Overview of Experiments
Experimental testbeds
Feature extraction
Experiment I: ranking on global feature
Experiment II: re-ranking with NIM on local features
Computational cost
Experimental testbeds
Columbia’s TRECVID2003 dataset600 keyframes size of 352x264150 near-duplicate image pairs 300 non-duplicate images
CityU’s TRECVID 2004 dataset7006 keyframes3388 near-duplicate image pairs involve a total of 1953 keyframesOne keyframe may be associated with several near-duplicate pairs
Evaluation protocolsEach query contain single query image, remaining keyframes are regarded as the gallery setAverage cumulative accuracy metric
Feature Extraction
Global Feature: 297-dimensional vectorGrid Color moment:
3x3 grid, color mean, variance, skewness for R, G, B
Local Binary Pattern (LBP)59-dimensional LBP histogram
Gabor wavelets texture5 scales and 8 orientations, 3 moments for each subimage
EdgeEge orientation histogram, Canny edge detector
Local Feature ExtractionSIFT, PCA-SIFT v.s. SURF (H. Bay et al. ECCV’06)SURF is used for NIM matching:
takes advantage of fast feature extraction using integral imagesmore compact than SIFT
Ranking on Global Feature: Similarity and Features
L1>Cos>L2
Color Moments>Gabor, LBP>Edge
5 10 15 20 25 30
40
50
60
70
80
90
Rank
Cum
ulat
ive
Acc
urac
y (%
)
L1
L2
CosColorGaborLBPEdge
5 10 15 20 25 3030
35
40
45
50
55
60
65
70
75
Rank
Cum
ulat
ive
Acc
urac
y (%
)
L1
L2
CosColorGaborLBPEdge
TRECVID2003 dataset TRECVID2004 dataset
Ranking on Global Feature
5 10 15 20 25 3050
55
60
65
70
75
80
85
90
95
100
Rank
Cum
ulat
ive
Acc
urac
y (%
)
S3VMSVMNNColor Moment−Zhao CIVR06Color Histogram−Zhang MM04
TRECVID2003 dataset
S3VM>NN, SVM
Re-ranking with NIM on Local Features
5 10 15 20 25 3076
78
80
82
84
86
88
90
92
94
96
98
Rank
Cum
ulat
ive
Acc
urac
y (%
)
NIMS3VMOOS−SIFT −Wu CIVR07OOS−PCA−SIFT −Zhao TMM07VK −Wu CIVR07
TRECVID2003 dataset
NIM>OOS>S3VM
Re-ranking with NIM on Local Features
5 10 15 20 25 3052
56
60
64
68
72
76
80
82
Rank
Cum
ulat
ive
Acc
urac
y (%
)
NIM
S3VMSVMNNColor Moment−Zhao CIVR06Color Histogram−Zhang MM04
TRECVID2004 dataset
NIM>S3VM>NN>SVM
Efficiency and performance on TRECVID 2003
5 10 15 20 25 30 35 40 45 5082
84
86
88
90
92
94
96
98C
umul
ativ
e A
ccur
acy
(%)
5 10 15 20 25 30 35 40 45 500
200
400
600
800
1000
1200
1400
1600
Number of top ranked examples τk
Tim
e
Time
Accuracy
Conclusion
A novel nonrigid image matching method for NDK retrieval.In contrast to the traditional approaches,
Recover the explicit nonrigid mapping between two NDK.A robust coarse-to-fine optimization schemeNot only detect the NDK pairs, but also recover the local deformations between them simultaneouslyAn effective multi-level ranking scheme A semi-supervised ranking technique to improve the ranking performance with the unlabeled data
Extensive experimental evaluations on two testbeds extracted from TRECVID corpora.More effective than conventional approaches