Upload
milkers
View
164
Download
5
Embed Size (px)
Citation preview
Outline
• Why priors?
• Prior enforcement through MRF
• Prior enforcement through primitives
• Prior enforcement through shortest-path
Brief Introduction of Multi-View Stereo (MVS)
Camera pose 3d reconstruction
Input: Calibrated photographs
Output: 3D geometry
1995 1999 1998 1998
2004 2005 2005 2007Computer Vision: Algorithms and Applications [Richard Szeliski]
A decade ago... A Theory of Space by Space Carving
[ Kutulakos and Seitz, 1999 ]
Reconstructions w/ and w/o color
Volumetric graph cutsby Vogiatzis, Torr, and Cipolla
CVPR 2005
1. This is called Haniwa. 2. This is Roberto Cipolla’s Haniwa.
This will be a final quiz!
Computer Vision in Industry
• Amazon
• Apple
• Microsoft
• Nokia
Mountain View Seattle Zurich NYC LA …
Computer Vision in Industry
• Amazon
• Apple
• Microsoft
• Nokia
Mountain View Seattle Zurich NYC LA …
Google Seattle - 3D Vision Team
• Bundling streetview data (by Sameer Agarwal)
• Photo Tours (by the team)
• Picasa face movie (by Ira Kemelmacher-Shlizerman and Rahul Garg)
• Lens Blur (by Carlos Hernandez)
Steve Seitz, Sameer Agarwal, Carlos Hernandez, David Gallup, Changchang Wu, Li Zhang
Why priors?
• Fails in some important cases
• Noise suppression
• Compression
• Uncanny valley for 3D reconstruction
Uncanny Valley for 3D Reconstruction
Ten Things You Should Know AboutLarge Scale 3D Reconstruction
(3DV 2013)
Martin Byröd Apple Maps
Why priors?
• Fails in some important cases
• Noise suppression
• Compression
• Uncanny valley for 3D reconstruction
Outline• Why priors?
• Prior enforcement through MRF
• Prior enforcement through primitives
• Prior enforcement through shortest-path
8.0
8.2
8.5
8.9
9.3
camera center image
screen
Depthmap Problem
Discretized depth values{ 0.1, 0.2, …, 8.0, 8.1, … 9.8, 9.9}
Standard DepthmapMarkov Random Field (MRF)
A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]
A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors [Szeliski et al., PAMI 2008]
Markov Random Field (MRF)
Piecewise planarity from MRF1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
Advanced MRF for Depthmap Estimation
Global Stereo Reconstruction under Second Order Smoothness Priors[Woodford et al., CVPR 2008] Best Paper Award
Reference image
Ground truth
Optimization becomes a challenge
Standard MRF (submodular)
MRF with a triple clique (non-submodular)
• Graph-cuts(alpha-expansion)
• Belief propagation(TRW)
• QPBO • Belief propagation
(TRW?)
Optimization becomes a challenge
Standard MRF (submodular)
MRF with a triple clique (non-submodular)
• Graph-cuts(alpha-expansion)
• Belief propagation(TRW)
• QPBO • Belief propagation
(TRW?)
Fusion moves…
Comparative experimentReference image Ground truth
Standard MRF MRF with a triple clique
[Woodford et al.]
How to enforce piecewise planar1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
PlanemapPossible plane ids a
b
c
d
e f g h
e
e
e
e
e
1. Extract Manhattandirections
2. Extract planes
How to enforce piecewise planar1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
How to enforce piecewise planar1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
Relaxing Mahnattan
Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Use sparse lines + sparse points to detect planes
• MRF + Graph-cuts
Enable Curved Surfaces• Building reconstruction from a top down view
Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
How to enforce piecewise planar1. Advanced MRF and optimization
• Global Stereo Reconstruction under Second Order Smoothness Priors [Woodford et al., CVPR 2008] Best Paper Award
2. Integrate with top-down (primitive) approach
• Manhattan World Stereo [Furukawa et al., CVPR 2009]
• Piecewise Planar Stereo for Image-based rendering [Sinha et al., ICCV 2009]
• Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery [Zebedin et al., ECCV 2008]
• Piecewise Planar and Non-Planar Stereo for urban Scene Reconstruction [Gallup et al., CVPR 2010]
Piecewise Planar and Non-‐Planar Stereo for Urban Scene Reconstruction
David Gallup Jan-‐Michael Frahm Marc Pollefeys
University of North Carolina ETH Zurich
First step: Planar and Non-‐Planar Stereo
108
Video
Real-‐Time Stereo Plane Detection Planemap (w/ non-‐planar))
Labels =
planes non-‐plane discard
Classification• divide image into regular grid • RGB, HSV, hue histogram, edge orientation histogram • 5000 human-‐labeled examples (5 images) • k-‐nearest-‐neighbor classifier
non-‐planar planar110
Hoiem et al. 2005, Xiao et al. 2009
AlgorithmVideo
Real-‐Time Stereo
Planar/Non-‐Planar Classification
Plane Detection
Planar and Non-‐Planar MRF
111
non-‐planar planar
Summary: Shape priorsin Depthmap MVS
MRF with a triple term
Manhattan planemap Planemap
Planemap w/ curved surfaces Mix of planemap and depthmap
Outline
• Why priors?
• Prior enforcement through MRF
• Prior enforcement through primitives
• Prior enforcement through shortest-path
Priors through primitives for large-scale indoor modeling
“Reconstructing the World’s Museums” [Xiao and Furukawa, 2012]
(Best Student Paper Award)
Technical Contributions
• Architectural shape priors through primitives
• Single consistent 3D model
Technical Contributions• Architectural shape priors through primitives
• Single consistent 3D model
Inverse Constructive Solid Geometry (CSG)
Technical Contributions
Inverse Constructive Solid Geometry (CSG)
• Architectural shape priors through primitives!
• Single consistent 3D model (real merging)
+
Technical Contributions
Inverse Constructive Solid Geometry (CSG)
• Architectural shape priors through primitives
• Single consistent 3D model (real merging)
+
+
-
+
+
-
Ground Ground+Aerial
Outdoor
Indoor
Ground vs. Aerial à Ground+Aerial
Google Streetview Google MapsGL
Furukawa et al.
Aerial
Google/Bing/NASA …
This paper This paper
Millions of small/medium-scale businesses
• Image-based for scalable deployment(especially for emerging markets)
• Compact model(for browser on low-end PCs)
Millions of small/medium-scale businesses
• Image-based for scalable deployment(especially for emerging markets)
• Compact mesh model(for browser on low-end PCs)
Outline
• Why priors?
• Prior enforcement through MRF
• Prior enforcement through primitives
• Prior enforcement through shortest-path
Why priors?• Only reconstruct planar outline
• Texture mapping looks better
• Challenge is how to IGNORE clutter
1. Panorama images 2. MVS points
3. Point evidence 4. Free-space evidence
5. 2D room outline 6. 3D model
2D room outline reconstruction3. Point evidence 4. Free-space evidence
Outline should pass through this point
Threshold
2D room outline reconstruction5. Shortest path formulationModel complexity penalty
added to each edge
Features
• Regularize on {# of line segments}
• Piecewise planarity enforcement
• With Dynamic Programming,exactly control the number of vertices
Summary• Why priors?
• Prior enforcement through MRF
• Prior enforcement through primitives
• Prior enforcement through shortest-path
A. Who owns the Haniwa in the volumetric graph-cuts paper?
1. Yasutaka Furukawa.
2. Roberto Cipolla.
3. British Museum.
B. What is Uncanny valley for 3D reconstruction?
1. As the number of authors in a paper goes up, the model quality goes down.
2. As the face realism goes up, the model looks creepier.
3. As the model accuracy goes up, the rendering quality goes down.
C. How can one enforce piecewise planarity with a standard MRF formulation easily?
1. Use super-pixel segmentation.
2. Use primitive detection.
3. Use object recognition.