Multiscale Symmetric Part Detection and GroupingAlex Levinshtein, Sven Dickinson, University of Toronto and Cristian Sminchisescu, University of Bonn

IntroductionMotivationMotivation DemonstrationDemonstration

Input image

1a. Multiscale superpixel segmentation

1b. Superpixel affinity (for each scale)

1c. Group superpixels into symmetric parts using the affinities

2a. Part affinity 2b. Group parts

1. Part detection

2. Part grouping

Key IdeaKey Idea

• Symmetric shape decompositions offer an effective representation for shape indexing and matching.

• Skeleton-based approaches assume correct figure-ground segmentation, precluding their application to cluttered scenes.

• Filter-based approaches are imprecise and offer poor precision-recall in both part detection and part grouping.

• Contour-based methods can suffer from high computational complexity and typically stop short of part grouping.

• How can we extract and group the symmetric parts of an object from a cluttered scene in a computationally efficient manner?

• Traditional skeletonization algorithms decompose a closed contour into symmetric parts and attachment relations – a highly top-down process.

• In contrast, we will detect symmetric parts locally and then assemble them into an object – a highly bottom-up process.

• We begin by segmenting the image into compact regions at multiple scales, representing a large set of (deformable) maximal disc hypotheses from which skeletal parts can be detected.

• Adjacent maximal disc hypotheses that form a symmetric part with strong image evidence are grouped.

• Finally, detected parts whose attachments are deemed nonaccidental are assembled to form objects.

Features to compute part attachment affinity:

• Boundaryevidence

• Appearance similarity

• Attachment category (based on main part axes)

• Cluster parts using affinities.• Use standard graph clustering algorithm.

(We use Felzenszwalb’s and Huttenlocher’s greedy clustering algorithm (2004) )

• Select non-redundant parts from each cluster.Formulate as a quadratic program that minimizes overlap, # of selected parts, while maximizing the covered area.

Probability part not redundant

• Define an adjacency graph for each superpixel scale.

• Each edge is assigned an affinity representing the likelihood that the two superpixels represent maximal discs of the same symmetric part.

• Affinity has shape and appearance components:– Shape: check the presence of symmetric edges– Appearance: homogeneity

• Shape features:

• Learn the affinity function from training data.

• Trained part detection and grouping on images from Weizmann Horse dataset (Borenstein and Ullman 2002).

• Quantitative evaluation of part detection on images from the Weizmann Horse dataset (compare to Lindeberg and Bretzner, 2003).

• Qualitative evaluation on images from other domains.

1. Part DetectionObject part Maximal

discsSuperpixel

approximation

SVM Logistic

Shape features

Superpixel affinity

Logistic

),( jiAS

shapeA appearanceA

Appearance features

Redundant parts affinity

Probability that part is

redundant

Attached parts affinity

Part affinity

∙ ∙+

Results

image main part clusters

Too coarse Too fineGood scale• Parts appear at different scales in the image

• Need at least one superpixel scale that captures each part well

• Solution: Use multiscale superpixel segmentation

1a. Multiscale superpixel 1a. Multiscale superpixel segmentationsegmentation

1b. Superpixel affinity1b. Superpixel affinity

1c. Grouping superpixels into parts1c. Grouping superpixels into parts

edge histogramedges histogram bins

• Cluster superpixels using affinities.• Use standard graph clustering algorithm.

(We use Felzenszwalb’s and Huttenlocher’s greedy clustering algorithm (2004) )

2a. Part affinity2a. Part affinity

2b. Grouping parts into objects2b. Grouping parts into objects

Features to determine part redundancy:

• Area overlap

• Boundary overlap

• Appearance similarity

image main part clusters

ApproachApproach• A symmetric part is the locus of maximal

disc centers (Blum 1967).

• Use superpixels at multiple scales as data-driven hypotheses of maximal discs.

• Identify groups of superpixels that havestrong symmetric edge support.

Input image Region skeleton

Blobs/Ridges(Lindeberg and Bretzner)

Contour Groups(Stahl and Wang)

ApproachApproach

• Part affinity defined differently for redundant and attached parts.

• Redundant parts are assigned high affinity, while non-redundant parts are assigned an affinity based on evidence of attachment.

• Learn redundancy and attachment affinity from labeled training data.

• Goal: Group together parts likely to belong to the same object, based on detecting part attachments.

• Issue: Since the same part can be detected at different scales, resulting groups may contain redundant parts.

• Solution: – Greedily cluster parts based on computed part attachment affinities.– Decide which redundant parts to remove in the context of each cluster.

2. Part Grouping