Upload
chinghang-chen
View
115
Download
1
Embed Size (px)
Citation preview
Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth conversion by
Poisson Equation Ching-Hang Chen
Cheng-An Hou
Introduction• 3D properties by 2D RGB images
• 3D scene understanding• Object geometry• Object location (near/far) • Segmentation
• Leverage Deep Learning to solve 3D from monocular images• Surface normal• Depth
Related Work• Traditional approaches• Machine learning, deep learning
Traditional Approach• Structure from motion(SfM)
Imaging Feature Point Matching
Solve Fundamental Matrix
SceneShape
by Epi-polar geometry
Traditional Approach• Photometric Stereo
Imaging Surface Normal Estimation
Surface Normal Integration
Scene Shape
…
𝑵=𝑰 𝑳−1
Normal field
Limitation of Traditional Methods• Need Calibration: camera or light source• In general, assume Lambertian surface• The imaging step is usually impractical
Depth from 2D Images: Challenges • Ambiguity from 2D to 3D, and size• Complex light properties:
• Reflection• Refraction• Inter-Reflection• Sub-surface Scattering
Reflection
Refraction
Inter-reflections
Subsurface Scattering
• How can machine learning help?• How human infer the 3D properties from 2D images?
Machine Learning Approach
Fouhey, David F., Abhinav Gupta, and Martial Hebert. "Data-driven 3D primitives for single image understanding." Proceedings of the IEEE International Conference on Computer Vision. 2013.
Deep Learning Approach
Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.
Bansal, Aayush and Russell, Bryan and Gupta, Abhinav. "Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. " CVPR 2016.
Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015
Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.
Deep Learning Approach
Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015
Deep Learning Approach
Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.
Problem Formulation• Pixel-wise predict surface normal for monocular images by CNN
features• Explore the relation between surface normal and depth, conversion
between the two
Dataset: NYU v2• Indoor Scenes collected by Microsoft Kinect, 407,024 frames• RGB, Depth, Surface Normal maps
• Why predict Surface Normal instead of Depth?
Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.
Proposed Network
… … … …
NN Upsampling
…
Pretrained VGG
Convolutional Layers
128 x 128
64 x 64
Hypercolumn
128 256 512 512
1408
64
4096 40
Training• Classification instead of Regression• 40 classes of surface normal• Loss: negative log likelihood• Optimization: SGD
Conv1 layer Visualization (AlexNet)
Experiment Result: Surface Normal• Quantitative results:
Mean Median 11.25 22.50 30.00
AlexNet (Pretrain) 29.2 24.6 22.5 46.0 58.7
AlexNet (Scratch) 29.6 24.2 23.6 46.9 58.8
Ladicky et al. [3] 35.5 25.5 24.0 45.6 55.9
Wang, et al. [2] 28.8 17.9 35.2 57.1 65.5
Eigen, et al. [1] 25.9 18.2 33.2 57.5 67.7
Skip-Net (ours) 24.7 17.7 35.6 58.6 68.0
[1] Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV 2015.
[2] Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.
[3] Zeisl, Bernhard, and Marc Pollefeys. "Discriminatively trained dense surface normal estimation." Computer Vision–ECCV 2014.
Experiment Result: Surface Normal
RGB image
Surface Normal
Depth Map & Surface Normal
• Theoretically, Depth Map can be differentiated to derive Surface Normal, and Surface Normal can be integrated to obtain Depth Map.
• Depth Map has ambiguity of scale, and Surface Normal does not.
Depth Map Surface Normal
• NYU v2 Surface Normal Acquisition
Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.
Depth from Surface Normal by Orthogonality
Depth from Surface Normal by Poisson Equation
• Minimize the objective J(v), where v is the depth function v(x,y)
M. Breuß, Y. Qu´eau, M. B¨ahr, and J.-D. Durou. Highly efficient surface normal integration. In Proceedings of the Conference Algoritmy, pages 204–213, 2016
Experiment Result: ComparisonOrthogonality Poisson
Experiment Result: Depth Map from Surface Normal
Experiment Result: Depth Map from Surface Normal
More Experiment Examples
Discussion • To solve task such as surface normal prediction from monocular
images, information insufficiency could be resolved by learning from dataset
• Limitation of depth conversion from surface normal: discontinuous boundaries, and wrong surface normal prediction
• The objective function for predicting depth is to minimize the overall depth prediction in the scene, object’s local structure might not be discovered
•Thank you for your attention!