Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Object Proposal Estimation in Depth Images using Compact 3D Shape Manifolds
18/10/2015
Shuai Zheng, Victor Prisacariu, Melinos Averkiou, Ming-Ming Cheng, Niloy Mitra, Jamie Shotton, Philip Torr, Carsten Rother
Goal
18/10/2015 Object Proposals using Compact 3D Shape Manifolds 2
Depth image 100 proposal windows for chairs (few shown)
Four Classes: Chair, Sofa, Toilet, Monitor
Motivation
18/10/2015 3
Dr. Hicks, Oxford University(Aid for partially sighted)
100 Chair Proposals(few shown)
Heat map
Chair detection
Object Proposals using Compact 3D Shape Manifolds
Challenges
18/10/2015 4
• Chairs (Sofas, etc.) have very different shape and appearance
• Collecting and annotating enough real data is expensive
Object Proposals using Compact 3D Shape Manifolds
Related Work
• Proposal Generation
• Not class-specifice.g. Objectness [Alexe, Deselaers, Ferrari PAMI 12],
Selective Search [Uijlings, van de Sande, Gevers, Smeulders IJCV 13],
• Class specifice.g. Cascade SVM [Zhang, Warell, Torr CVPR 11], [Cheng, Zhang, Lin, Torr, CVPR 14], Note that both can also be applied in generic objectness proposal.
• Utilizing 3D Database for Training
• Seeing 3D chairs [Aubry, Maturana, Efros, Russel, Sivic, CVPR 14]
• Sliding Shapes [Song, Xiao, ECCV 2014]
18/10/2015 5Object Proposals using Compact 3D Shape Manifolds
Google 3D Warehouse
Our approach
18/10/2015 6
Contribution 1: This two-stage pipeline
Contribution 2: Training Data generation
BING (0,0009sec)
CNN(0.88sec)1000 proposals
(few shown)
Positive examples(> 50% IoU)
Negative examples(< 50% IoU)
100 proposals(few shown)
Object Proposals using Compact 3D Shape Manifolds
Our approach
18/10/2015 7
Contribution 1: This two-stage pipeline
Contribution 2: Training Data generation
BING (0,0009sec)
CNN(0.88sec)1000 proposals
(few shown)
Positive examples(> 50% IoU)
Negative examples(< 50% IoU)
100 proposals(few shown)
Object Proposals using Compact 3D Shape Manifolds
First Stage: BING [Cheng, Zhang, Lin, Torr, CVPR 14]
18/10/2015 8
Binarized Normalized Gradients for Objectness Estimation at 300fps:
1. Extract ~7𝑀 rectangles from 0.3𝑀 Pixel Image (quantized position×scale×aspect ratio)
2. Resize to 8 × 8 pixel and compute normalized gradient
3. Classify with 2-Stage Linear SVM
4. Test-time: 1000 proposals using SVM score
Chair SVM weights
Normalized Gradient (8 × 8 window)
Object Proposals using Compact 3D Shape Manifolds
Second Stage: CNN
18/10/2015 9
CNN (Network in Network) [Lin, Chen, Yan ICLR 2013]
Re-train for Chair Proposals
Train new Linear SVM
{chair, no chair}
{dog, frog,ship,bird,truck,…}
{chair, no chair}
Lin
ear
SV
M
Test Time: Use SVM score to take best 100 out of 1000 proposals
Object Proposals using Compact 3D Shape Manifolds
Our approach
18/10/2015 10
BING (0,0009sec)
CNN(0.88sec)1000 proposals
(few shown)100 proposals
(few shown)
Positive examples(> 50% IoU)
Negative examples(< 50% IoU)
Object Proposals using Compact 3D Shape Manifolds
Contribution 1: This two-stage pipeline
Contribution 2: Training Data generation
Real Training Data
18/10/2015 11
495 NYU V2 Training Images (404 NYU V2 Test images)
Dataset Detection Rate(1000 proposals)
NYU 85.6 %
Detection Rate (Recall): How many true rectangles are covered by a proposal?(Inlier: IoU > 50%)
Bathroom
Classroom
Dining room Livingroom Office
Object Proposals using Compact 3D Shape Manifolds
Google 3D Warehouse
18/10/2015 12
37,400 Synthetic Images (374 Warehouse Chairs)
Dataset Detection Rate(1000 proposals)
NYU 85.6 %
Warehouse Low number
NYU+Warehouse 87.8 %
Detection Rate (Recall): How many true rectangles are covered by a proposal?(Inlier: IoU > 50%)
Conclusions:1) Data distribution of NYU and Warehouse
do not coincide2) Can we massage the Warehouse data to
make it work better?
Object Proposals using Compact 3D Shape Manifolds
Idea 1: Shapesynth [Averkiou, Kim, Zheng, Mitra, Computer Graphics Forum 2014]
18/10/2015 13
374 Warehouse Chairs 3D Bbox Templates
Filter
2D Embedding (MDS)
manual
~150 synthetic chairs(37,400 images)
Dataset Detection Rate(1000 proposals)
NYU 85.6 %
Warehouse Low number
NYU+Warehouse 87.8 %
NYU+ShapeSynth 88.5 %
Detection Rate (Recall): How many true rectangles are covered by a proposal?(Inlier: IoU > 50%)
Object Proposals using Compact 3D Shape Manifolds
Idea 2: Gaussian Process Latent Variable Model [Lawrence JLMR 2005]
18/10/2015 14
Low Dimensional Latent Space~150 Synthetic chairs
(37,400 images)374 Warehouse Chairs
Dataset Detection Rate(1000 proposals)
NYU 85.6 %
Warehouse Low number
NYU+Warehouse 87.8 %
NYU+ShapeSynth 88.5 %
NYU+GPLVM 89.7 %
Detection Rate (Recall): How many true rectangles are covered by a proposal?(Inlier: IoU > 50%)
Object Proposals using Compact 3D Shape Manifolds
How does it work?
18/10/2015 15
𝑁 = 374Warehouse Chairs
ICP
Aligned Chairs 2563 Volume (Signed Distance Function for each chair)
DCT
64,000𝐷vector
IDCT
𝒀
GPLVMEmbedding
1 2 𝑁
𝑿Latent Space • 2D Projection of 5D 𝑿
• For 𝒙𝑖∗ derive 𝒚𝑖
∗, 𝜎𝑖2
• Sample low variance shapes
𝒙𝑖
𝑀=64,000𝐷
1 2 𝑁
𝐿=5𝐷
Object Proposals using Compact 3D Shape Manifolds
𝑃 𝒀 𝑿 =
𝑚=1
(𝒚(𝑚)𝑇 |𝟎,𝑲 𝑿 )
𝑀
𝒚𝑛 = 𝑾𝒙𝑛 + 𝝁𝑛
GPLVM – Fine Tuning
18/10/2015 16
GPLVM Dimension 𝐿 = 5
Number sampled 3D Models
Detection Rate(1000 proposals)
100 87.8 %
150 89.7 %
200 89.2 %
374 88.4 %
GPLVM Dimensions 𝐿 Detection Rate(1000 proposals)
3 88.0 %
4 89.6 %
5 89.7 %
6 89.3 %
7 89.2 %
150 3D Models
Object Proposals using Compact 3D Shape Manifolds
GPLVM Manifolds: Chairs
18/10/2015 17
New Chairs (low variance)
New Chairs (high variance)
Reconstruct Training Shapes
Warehouse GPLVM
Object Proposals using Compact 3D Shape Manifolds
GPLVM Manifolds: Toilets
18/10/2015 18
New Toilets (low variance)
New Toilets (high variance)
Reconstruct Training Shapes
Warehouse GPLVM
Object Proposals using Compact 3D Shape Manifolds
Comparison
18/10/2015 19
Method Detection Rate(1000 proposals)
seconds
Ours 89.7 % 0.0009
Selective Search (no training)[Uijlings, van de Sande, Gevers, Smeulders IJCV 2013]
85.9 % 2.6
Cascade SVM (trained on NYU)[Zhang, Warell, Torr IJCV 2013]
84.5 % 1.2
Objectness (trained on NYU RGB)[Alexe, Deselaers, Ferrari PAMI 2012]
83.0 % 2.1
Random 42.0 % N/A
Object Proposals using Compact 3D Shape Manifolds
Our approach
18/10/2015 20
BING (0,0009sec)
CNN(0.88sec)1000 proposals
(few shown)100 proposals
(few shown)
Object Proposals using Compact 3D Shape Manifolds
Comparison – Full Pipeline
18/10/2015 21
82.9%
56.3 %59.2 %
26.6 %Improvement
Number of Proposals
Det
ecti
on
Rat
e
Selective Search[Uijlings et al. IJCV 2013]
Ours
Ours w/o CNN
Object Proposals using Compact 3D Shape Manifolds
Results: Chair
18/10/2015 22Object Proposals using Compact 3D Shape Manifolds
Results: Toilet
18/10/2015 23Object Proposals using Compact 3D Shape Manifolds
Result: Sofa
18/10/2015 24Object Proposals using Compact 3D Shape Manifolds
Result Monitor
18/10/2015 25Object Proposals using Compact 3D Shape Manifolds
Take-Home Messages
• Our 2-Stage pipeline (BING, CNN) gives state-of-the-artresults for object proposal generation
• GPLVM is useful for training with 3D shape collections
18/10/2015 26Object Proposals using Compact 3D Shape Manifolds
Advertisement
• Learning Analysis-by-Synthesis (Nectar Track Thursday 15-17)
• Join our 6D Object Pose Estimation Challenge at ICCV 2015
18/10/2015 27
[Krull et al. ICCV 2015]
CNN Energy
Object Proposals using Compact 3D Shape Manifolds