Segmentation Based Multi- V iew Stereo

Center for Machine PerceptionDepartment of Cybernetics

Faculty of Electrical EngineeringCzech Technical University in Prague

Segmentation Based Multi-View Stereo

Michal Jančošek, Tomáš Pajdlajancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz

Problem formulation : input34

jancom1@cmp.felk.cvut.cz, pajdla@cmp.felk.cvut.cz

OutputOutput

Problem formulation : output34

InputInput

Output

Previous work

[Furukawa07] Y. Furukawa and J. Ponce. Accurate, Dense, and Robust Multi-View Stereopsis, CVPR 2007 prematching, growing, filtering

[Tao00] H. Tao and H.S. Sawhney. Global Matching Criterion and Color Segmentation Based Stereo, ACV 2000 using segmentation for hypothesizing the continuous parts of a

[Felzenszwalb04] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation, IJCV 2004. color segmantation

[Jancosek08] Jancosek M. and Pajdla T. Effective seed generation for 3D reconstruction, CVWW 2008 optimal 3D segment orientation

Pipeline overview : Prematching34

InputInput Prematching(3D seeds)

Hypothesizing(3D segments)Hypothesizing(3D segments)

Final mesh constructionFinal mesh

constructionFiltering

(3D patches)Filtering

(3D patches)OutputOutput =

Pipeline overview : 3D segments34

Prematching(3D seeds)

Hypothesizing(3D segments)

Pipeline overview : Filtering34

Pipeline overview : Final mesh construction 34

Final mesh construction

Filtering(3D patches)

Filtering(3D patches)Output =

Poisson Surface Reconstruction

Pipeline overview : Prematching34

Prematching

Cameras are known (or computed [Martinec et al.]) Feature points detection (harris on a grid [Furukawa07]) Feature points are matched in each image pair (guided matching

by epigeoms). Matching is based on the NCC score of two 5x5 windows Mutually best matches are selected Tracks are constructed by grouping mutually best matches

r = ²i( )

= center of gravity of the points in S

Pipeline overview : 3D segments34

Hypothesizing(3D segments)

3D segments34

For each segment with some seed projections, we create an optimal 3D segment and set it as explored

Next we do a greedy searching of unexplored segments to explore more

Optimal 3D segment creation34

TP FPTP

TP – true positiveFP – false positive - explored segment - unexplored segment

Optimal 3D segment creation34

MNCC( )=

Ф = 45

Ѳ = 90

Ф = 75

Ѳ = 120

The goal is to find the global maximum of the criteria function

)),,(),,,((,,

,180,0180,0:

kvpTkvpTMNCCkf

Optimal 3D segment creation

First, we estimate 3D segment orientation 3D segment orientation is estimated by gradient descent optimization

from the best sample out of 10x10 regular orientation samples [Jancosek08]

Next, we find the optimum position of 3D segment on the ray from reference camera center for the estimated orientation

Greedy searching of unexplored segments34

TP FPTPTPTP

Greedy searching of unexplored segments34

TP FPTPTP TP TP

Pipeline overview : Filtering34

Filtering34

TP FPTP

Pipeline overview : Final mesh construction 34

Final mesh construction

Filtering(3D patches)

Filtering(3D patches)Output =

Poisson Surface Reconstruction

Results : Strecha’s evaluation34

Results : Homogenous regions34

Conclusions

Advantages Complete models Lack of texture is explained by planes Speed Possible to implement on GPU

Disadvantages Low accuracy

Future work MRF on volume around 3D segments

THANK YOU FOR YOUR ATTENTION

Segmentation Based Multi- V iew Stereo

Documents

Structure Style - IEW

V iew Maintenance

Mori 2004 Iew

Sample ideasfromtheirperspecve. - IEW

Scene Segmentation Assisted by Stereo Vision

IEW 2012 Magalog

Valley V iew School

IEW 2021 -brochure

Joint Optimization for Object Class Segmentation and Dense ... · problems of object class segmentation and dense stereo reconstruction. Section 3 describes the formulation al-lowing

2014 USC IEW Report

Object Stereo- Joint Stereo Matching and Object Segmentation

Fast Multi-frame Stereo Scene Flow with Motion Segmentation · Fast Multi-frame Stereo Scene Flow with Motion Segmentation ... We also report a thorough evaluation on challenging

IEW ostatic kshop - ESD A · kshop A IEW eiall oooe b Register Early and Save! IEW International ESD Workshop May 18-21, 2009 Stanford Sierra Camp Lake Tahoe, CA 2009 IEW Highlights

segmentation€¦ · A segmentation could be used for object recognition, occlusion bound-ary estimation within motion or stereo systems, image compression, image editing, or image

Petter Strandmark Fredrik Kahl Lund University. Image denoising Stereo estimation Segmentation Shape fitting from point clouds

Stereoscopic Segmentation - Web Hosting Default Page · STEREOSCOPIC SEGMENTATION STEREO LOCAL CORRESPONDENCE TEXTURE DENSITY NOISE INTENSITY Figure 1. Domain of operation of shape

IEW 2012 at ACB

iew ?, iew

Online 3D Reconstruction and Ground Segmentation using ... · Online 3D Reconstruction and Ground Segmentation using Drone based Long Baseline Stereo Vision System Prashant Kumar

Preparing for IEW 2013