Category Independent Region Proposals
Ian Endres and Derek HoiemUniversity of Illinois at Urbana-Champaign
Finding Objects
Scanning Window
HorseDogCatCarTrain… 10,000+ windows
Category Independent Search
~100 regions
Finding Unfamiliar Objects
Finding Objects
Objectives:1. Minimize number of proposed regions2. Maintain high recall of all objects3. Provide detailed spatial support (i.e. segmentation)
Challenges
• Objects extremely diverse– Variety of shapes, sizes– Many different appearances
• Within object variation– Multiple materials and textures– Strong interior boundaries
• Many objects in an image
Overview
1 2 3 4 ...
Generate Proposals:Maximize recall
Rank Proposals:Small diverse set of object regions
Generating Proposals1. Select Seed 2. Compute affinities for seed
3. Construct binary CRF
+
Unary term:Affinities
Pairwise term:Occlusion Boundaries
4. Compute proposal
5. Change parametersRepeat
Generating Seeds
• Compute occlusion boundaries (Hoiem et al. ICCV ‘07)
• Generate hierarchal segmentation– Incrementally merge regions of oversegmentation
• Use regions with sufficient size and boundary strength– Avoids redundant or uninformative seeds
Region Affinity
• Learned from pairs of regions belonging to an object– Computed between the seed and each region of
the hierarchy
– Features: color and texture similarity, boundary crossings, layout agreement
Color/Texture Similarity•Color, texture histograms for each region•Compute histogram intersection distance between two regions
Boundary Crossing•Draw line between region centers of mass
•Compute strength of occlusion boundaries crossed
Layout Agreement•Predict object extent from each region
•Compute strength of agreement between two regions
CRF Segmentation
• Binary segmentation• Graph composition:– Nodes: Superpixels– Edges: Adjacent superpixels
+
CRF Segmentation
• Graph Potentials– Unary Potential: affinity values for each superpixel– Edge Potential: occlusion boundary strength
• Parameters (25 combinations)– Node/Edge weight tradeoff– Node bias
+
Unary potential:Affinities
Edge potential:Occlusion Boundaries
Ranking Proposals
wT X1
wT X3
Appearance scores
wT X4
1.
2.
3.
4.
wT X2Sort
scores
GeneratedRanking
Lacks Diversity
• But in an image with many objects, one object may dominate 1
2
3
4
…
20
150
100
…
50…
…
Encouraging Diversity
• Suppress regions with high overlap with previous proposals
…
1
2
3
10
4
…
20
50
100
…
…
Ranking as Structured Prediction
• Find the max scoring ordering of proposals
• Greedily add proposals with best overall score
Appearance score
Overlap penalty
Gives higher weight to higher ranked proposals
Overall score
Learning to Rank(Max-margin Structured Learning)
• Score of ground truth ordering (R(n)) should be greater than all other orderings (R):
• Loss ( ) encourages good orderings:– Higher quality proposals should have higher rank– Each object should have a highly ranked proposal
Experimental Setup• Train on 200 BSDS images
• Test 1: 100 BSDS images
• Test 2: 512 Images from Pascal 2008 Seg. Val.
Evaluation
• Region overlap
• Recall at 50% region overlap– Typically more strict that 50% bounding box overlap– Measures detection quality and segment quality
Ai Aj
Qualitative Results
Pascal
BSDS(Rank, % overlap)
Vs. Standard Segmentation
Standard: 53%3000 proposals
Ours: 53%18 proposals
Standard: 80%70,000 proposals
(merge 2 adjacent regions)
Ours: 80%180 proposals
Recalling Pascal Categories
Future work
• Object Discovery• Incorporate into detection systems– Label regions directly– Voting from proposed regions
• Refine proposals with domain knowledge– i.e. wheel or head models