Shape Matching

and Object Recognition

using Low Distortion Correspondence

Alexander C. Berg, Tamara L. Berg, Jitendra Malik

U.C. Berkeley

Object Category Recognition

Deformable Template Matching with Exemplars for Recognition

● Use exemplars as deformable templates

● Find a correspondence between the query image and each template

Query

Image

Database of

Templates

Deformable Template Matching with Exemplars for Recognition

● Use exemplars as deformable templates

● Find a correspondence between the query image and each template

Query

Image

Database of

Templates

Best matching template is a helicopter

Correspondence for Deformable Template Matching

● Evaluate correspondence based on:

– Similarity of appearance near feature points

– Similarity in configuration of the feature points

QueryTemplate

Correspondence for Deformable Template Matching

● Evaluate correspondence based on:

– Similarity of appearance near feature points

– Similarity in configuration of the feature points

QueryTemplate

Correspondence for Deformable Template Matching

● Evaluate correspondence based on:

– Similarity of appearance near feature points

– Similarity in configuration of the feature points

QueryTemplate

Correspondence for Deformable Template Matching

● Evaluate correspondence based on:

– Similarity of appearance near feature points

– Similarity in configuration of the feature points

QueryTemplate

Correspondence Result

QueryTemplate

Interpolated CorrespondenceUsing Thin Plate Splines

QueryTemplate

Correspondence for Deformable Template Matching

If i i' and j j' then rijri'j'

If i i' then pipi'

i i'

j j'

i i'

j j'QueryTemplate r

ijr

i'j'

Geometric Blur(Local Appearance Descriptor)

Geometric Blur Descriptor

~

Compute sparse

channels from image

Extract a patch

in each channel

Apply spatially varying

blur and sub-sample

(Idealized signal)

Descriptor is robust to small affine distortions

Berg & Malik '01

Geometric Blur(Local Appearance Similarity)

Geometric Blur Descriptor

Geometric Blur Descriptor~

Are Features Enough?

Not Quite...

Color indicates

similarity using

Geometric Blur

Descriptor

Measuring Distortion(Similarity in Configuration)

i i'

j j'

i i'

j j'QueryTemplate R

ij Si'j'

Measure distortion in vectors between pairs of feature points

- R and S same length for rotations

- R and S same direction for scalings

Cost Function as IQP

Appearance costif i -> j

Distortion cost ifi -> j and k -> l

Integer Quadratic Programming Problem...

iff template point i maps to query point j, If binary vector x represents a correspondence

cf. Maciel & Costeira '03

Optimization

● Integer Quadratic Programming is NP hard

● The instances we generate seem easy

● Using a linear bound to initialize gradient descent provides good results

– (In fact better than the guarantee of Goemans &

Williamson's randomized algorithm)

● Varying the linear constraints on x allows

– one-one, one-many, or fixed number of outliers, etc.

Correspondence Result

Interpolated CorrespondenceUsing Thin Plate Splines

Quadratic Assignment(Using IQP)

Linear Assignment(e.g. Hungarian)

Correspondence Examples (Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Correspondence Examples(Shape Matching)

Application to RecognitionCaltech 101

– 101 classes of objects + “background”

● Large Scale

● Roughly aligned

● Large intra-class variation

● Fei-Fei, Fergus, Perona '04

Caltech 101 Recognition Results

Chance ~1%

N.N. whole image 16%

Discriminative version ofConstellation Model 27%

N.N. Geometric BlurDescriptors 38%

Low Distortion Correspondence (GB+IQP) 45%

102 way Alternative Forced Choice test

(15 training examples per class)

102 way confusion matrix

100%

0%

Model Building for Segmentation

Rough correspondenceto each example image

Average quality of alignment

Threshhold

Automatic vs Hand Segmentation

Application to RecognitionFaces

● Face dataset from Berg et al '03

– Medium to large scale faces

– AP News photos

● 20 face exemplars

● Same methodology as Caltech 101, but multiple objects / image

– After one face is identified its features are removed and the

search continues

● Compared to a detector from Mikolajczyk based on

Schneiderman & Kanade, that is quite successful on this dataset

Application to RecognitionFaces

Conclusion

● Use rich descriptors that are insensitive to typical

transformations

– Geometric Blur

● Enforce relationship constraints among corresponding

features

– Integer Quadratic Programming

● Estimate smooth transform

– Thin Plate Splines

Thank You

Acknowledgements

Charless FowlkesXiaofeng RenDavid Forsyth