05 - Recognizing a Large Number of Object Classes

8/8/2019 05 - Recognizing a Large Number of Object Classes

1/27

LargeScale

Recognition and

Retrieval


2/27

What does the world look like?

High level image statistics -Object Recognition for large scale searchFocus on scaling rather than

understanding image

Scaling to billions of images


3/27

Content-Based ImageRetrieval

Variety of simple/hand-designed cues:

Color and/or Texture histograms, Shape,PCA, etc.

Various distance metrics

Earth Movers Distance (Rubner et al.98)

QBIC from IBM (1999)


4/27

Some vision techniques forlarge scale recognition

Efficient matching methods

Pyramid Match Kernel

Learning to compare images

Metrics for retrieval

Learning compact descriptors


5/27








6/27

Matching features in-category level recognition


7/27

Comparing sets of localfeatures

Previous strategies: Match features

individually, vote onsmall sets to verify

[Schmid, Lowe, Tuytelaars et

al.]

Explicit search for one-to-one correspondences

[Rubner et al., Belongie etal., Gold & Rangarajan,

Wallraven & Caputo, Berg etal., Zhang et al.,]

Bag-of-words: Comparefrequencies ofprototype features

[Csurka et al., Sivic &Zisserman, Lazebnik & Ponce]:S lid e cre d it K riste n G ra u m a n


8/27

Pyramid match kernel

o p tim a lp a rtia lm a tch in g

: (ptimal match O m3): ( )yramid match O mL= #m features= #L levels in pyramid

[ & ,Grauman Darrell ICCV ]2005

:Slide credit Kristen Grauman


9/27

Pyramid match: main idea

descriptr space

Feature space partitions serve to match the localdescriptors within

successively wider.regions



10/27

Pyramid match: main idea

H isto g ra m in te rse ctio nco u n ts n u m b e r o fp o ssib le m a tch e s a t a

.g ive n p a rtitio n in g:S lid e cre d it K riste n G ra u m a n


11/27

Computing the partial matching

Earth Movers Distance [Rubner, Tomasi, Guibas 1998]

Hungarian method [Kuhn, 1955]

Greedy matching

Pyramid match

for sets with features of dimension

[Grauman and Darrell, ICCV 2005]


12/27

Recognition on the ETH-80

Recogn

itiona

ccuracy

(%)

Tes

tingtime( s

)

Mean number of features per set (m) Mean number of features per set (m)

ComplexityKernel

Pyramid match

Match [Wallraven et al.]

Slide credit: Kristen Grauman


13/27

Spatial Pyramid Match Kernel, , , .Lazebnik Schmid Ponce 2006

-Dual spacePyramid Matching., .Hu et al 2007

Representing Shape with aPyramid Kernel

& , .Bosch Zisserman 2007

=L 0 =L 1 =L 2

Pyramid match kernel: examples ofextensions and applications by other

groups

cenerecognition haperepresentation edical imageclassification



14/27

wave it downSingle View Human Action

Recognition using Key Pose, & , .Matching Lv Nevatia 2007

-Spatio temporalPyramid Matching for Sports, ., .Videos Choi et al 2008

From OmnidirectionalImages to Hierarchica

,Localization Murillo. .et al 2007

:Pyramid match kernel examples of extensions and applications by other

groups

ction recognition ideo indexing obotlocalization:Slide Kristen Grauma


15/27








16/27

Learning how to compare images

dissimilar

similar

Exploit( )dis similarity

constraints toconstruct more

useful distancefunction

Number of existingtechniques for

metric learning[Weinberger et al. 2004,Hertz et al. 2004, Frome etal. 2007, Varma & Ray2007, Kumar et al. 2007]


17/27

Example sources of similarity constraints

Partially labeled imagedatabases

Fully labeled imagedatabases

Problem-specificknowledge


18/27

Locality Sensitive Hashing(LSH)

Gionis, A. & Indyk, P. & Motwani, R. (199Take randomprojections of dataQuantize each projection with few bits

0

1

0

1 0

1

101

Descriptor in

high D space


19/27

Fast Image Search for Learned Metrics

Jain, Kulis, & Grauman, CVPR 2008

Less likely to split pairs like those

with similarity constraint

More likely to split pairs like those

with dissimilarity constraint

h( ) = h( ) h( ) h( )

Slide : Kristen Grauman

Learn a Malhanobis metric for LSH


20/27

Results: Flickr dataset

slower search faster search

30% of data 2% of data

Error

rate

18 classes, 5400 imagesCategorize scene based on

nearest exemplars

Base metric: Ling &Soattos Proximity

Distribution Kernel (PDK)

Query time:



21/27

Results: Flickr dataset

slower search faster search

30% of data 2% of data

Error

rate

18 classes, 5400 imagesCategorize scene based on

nearest exemplars

Base metric: Ling &Soattos Proximity

Distribution Kernel (PDK)

Query time:



22/27


Efficient matching methods Pyramid Match Kernel

Learning to compare images Metrics for retrieval



23/27

Semantic Hashing

Address Space

Semanticallysimilar

images

Query address

QueryImage

Binarycode

Images in database

[ & , ]Salakhutdinov Hinton 2007 for text documents

Quite different( )to a conventional

randomizing hash


24/27

of

semantic hash function

QueryImage

.3 RBM

ComputeGist

Binary code

Gist descriptor

Image 1

Semantic Hash

Retrieved images


25/27

Learn mapping

Neighborhood Components Analysis [Goldberger etal., 2004]

Adjust model parameters to move:

Points ofSAME class closer Po in ts o f D IFF E R E N T cla ss

a w a yPo in ts in co d e sp a ce

LabelMe retrieval


26/27

LabelMe retrievalcomparison

Size of retrieval

set

%

of

50

true

ne

ig

hb

ors

in

re

tr

ie

va

l

se

t

, ,0 2 000 10 000,20 0000

32-bit learnedcodes do aswell as 512-dim real-valuedinputdescriptor

Learningmethodsoutperform

LSH


27/27

Review: constructinga good metric from data

Learn the metric from training data

:Two approaches that do this

, , & , :Jain Kulis Grauman CVPR 2008 Learn Malhanobis.distance for LSH

, , , :Torralba Fergus Weiss CVPR 2008 Directly learn mapping.from image to binary code

( )Use Hamming distance binary codes for speedLearning metric really helps over plain LSH

Learning only applied to metric not

Documents

05 - Recognizing a Large Number of Object Classes