視覚情報処理論 (Visual Information Processing )G. Turk, M. Levoy, Proc. SIGGRAPH ‘94, In...

視覚情報処理論(Visual Information Processing )

開講所属: 学際情報学府

水(Wed)5 [16:50-18:35]

3D data processing 2

Modeling procedures

1. Data acquisition3. Merging

2. Alignment

3. Merging

Merging methods�Point-based

• Directly generate mesh from point cloud�Mesh-based

• Connect the boundaries of range data�Volume-based

• Convert range data to volume data (implicit function) and re-convert to mesh data

Point-based method[Chang et al. ‘07]

Delaunay Triangulation

• every triangle has an empty circumscribing circle

• find and remove all triangles with non-empty circumcircles

• triangulate by connecting new point

Streaming Computation of 2D Delaunay Triangulations�streaming, out-of-core, I/O and memory efficient Delaunay

triangulation in 2D and 3D

[Isenburg et al. TOG 2006]

Streaming Delaunay in 3D

Ball-Pivoting Algorithm [Bernardini et al. TVCG 99]

Surface Reconstruction from Point Clouds by Transforming the Medial Scaffold

• Relate the sampled shape with the underlying (unknown) surface by a sequence of shape deformations (growing from samples).

• Represent (2D) shapes by their medial “shock graphs”. [Kimia et al.]

• Handle shock transitions across different shock topologies to recover gaps.

[CVIU 2009]

Costa minimum surface (courtesy of H. Hoppe)Mobius strip

Result: Videos on Meshing Algebraic Surfaces

Result:

The rocker arm data courtesy of Cyberware.Flat smooth regions are meshed prior to the ridges/corners.

Dealing with sampling qualityInput of non-uniform and low-density sampling:

Response to additive noise:

50% 100% 150%

Mesh-based method

Range data 1Range data 2

Zippered Polygon Meshes from Range ImagesG. Turk, M. Levoy, Proc. SIGGRAPH ‘94, In Computer Graphics Proc., Annual Conf. Series, pp.311-318, 1994.

Zippering polygon meshes

Range data 1Range data 2

Removal ofOverlapping area

Zippering

Procedures of Zipper method

Process at boundary

Refinement of data

�Estimate surface normal vectors�Compute corresponding points along the

surface normal�Move points to weighted average positions

Merging result 1

Merging result 2

Volumetric method

Signed Distance

FieldAlignment result

Connect highestprobability points

Merging result

Compute probabilityof each voxel

A Volumetric method for building complex modelsfrom range imagesB. Curless, M. Levoy, Proc. SIGGRAPH'96. ACM, pp.303-312, 1996.

Volumetric merging�Represent a space by voxels�Computer distance from voxel and represent as implicit

function (Signed Distance Field)

Example of SDF

Computation of SDF

[Curless et al. ‘96]

Space Curving

29

Inside

Outside

Mesh generation• Marching Cubes algorithm [Lorensen ’87]

32

Experimental result (Hole filling)

Zipper vs. Volume

Zipper Volume

Consensus surfaces for modeling 3d objects from multiple range imagesM. D. Wheeler, Y. Sato and K. Ikeuchi, International Conference on Computer Vision (ICCV), pp.917-924,1998.

Signed Distance Field (SDF)

35

Signed distance Field

Marching Cubes [Lorensen et al., 1987]

Computation of SDF• Naïve algorithm

• Value: distance to nearest point on the surface• Sign: inside or outside of the surface

f(x’)

f(x)

Issues of conventional method�Bumpy surface is generated�Instability of signs: in-out-in transit

Consensus surface�Grouping by surface normal and distance between

corresponding points (consensus group)

f(x)f(x’)

( )( )( )

≥⋅∧≤−

=

otherwiseFalse

cosTrue

,,,eSameSurfac

1010

1100

　　 nd nnpp θδ ))

))

npnp

Experimental result

39

Closest Signed DistanceConsensus Surfaces

Volume-based Merging�Merging methods

• Curless et al., 1996• Wheeler et al., 1998• Masuda et al., 2003• Turk et al., 1999• Rocchini et al., 2001�Practical ideas & methods

• Sagawa et al., 2002• Nakao et al., 2008

41

Registration and Integration of Multiple Range Images byMatching Signed Distance Fieldsfor Object Shape ModelingT. Masuda, Computer Vision and Image Understanding, 2002

43

Matching Signed Distance [Masuda et al., 2003]

[ ] [ ]( )( ) [ ] [ ]( )

( )∫∫ +−+

=pp

pppp

xx

xxpxpx

pp

dw

dBAw

BAd2

2

,SD,SD

,SDF,,SDF

[ ][ ] ( )[ ]( )∑ ∑

∑ ∑≤≤

≤≤=

p

pp

p

Tpp

T

N

N

SIdw

wIS

α

α

ααα

ααα

1

1

2

1

,SDF,,SDF

,,;E

Error function

T: affine matrixw: weightI: merged model

Alignment result

Merging result

The Marching Intersections algorithm for merging range imagesClaudioRocchini, PaoloCignoni, FabioGanovelli, ClaudioMontani, Paolo Pingi, Roberto Scopigno, Visual Computer 2004.

Marching Intersections [Rocchini et al. 2001]

47

Surface Reconstruction

48

Experimental result

Robust and Adaptive Integration of Multiple Range Images with Photometric AttributesR. Sagawa K. Nishino K. Ikeuchi, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2001

Adaptive Integration [Sagawa et al., 2002]�Splitting voxel by Octree

51

Geometric featureOptical feature

More adaptive

Adaptive merging�Geometric feature

�Photometric feature

52

( )( )nn

nii

TN >

<⋅ δnnarccosmax

( )( )cc

cjiji

TN

cc

>

< δ,Distancemax,

Experimental results

53

Only geometric feature With optical feature

Result of adaptive merging

Variational implicit surfacesG. Turk. J. F. O'Brien, 1999

Variational implicit functions

four zero-valued and one positive constraint

(left) + three new zero-valuedconstraints

Variational Implicit Surfaces [Turk et al., 1999]

57

0)()()(1

=−+= ∑=

N

iiivf xxxx φλ

zcycxccv 3210)( +++=x

( ) ( ) ( )( )∑

∑=

xw

xdxwxD

i

ii fi(x): VIF of the ith imageVariational Implicit Function

Low-degree polynomial Radial basis

Variational Implicit Functions

58

kihf ii ,,2,1 ,)( K==p

=

0

0

0

0

0000

0000

0000

0000111

1

1

1

2

1

3

2

1

0

2

1

21

21

21

21

22222221

11111211

kk

k

k

k

kkkkkkk

k

k

h

h

h

c

c

c

c

zzz

yyy

xxx

zyx

zyx

zyx

MM

L

L

L

L

L

MMMMMMM

L

L

λ

λλ

φφφ

φφφφφφ

zcycxccfN

iii 3210

1

)()( ++++−= ∑=

xxx φλConstraint pointsRange image

Surfaces by VIF

Surface Reconstruction from Point Clouds by Transforming the Medial ScaffoldM. –C. Chang*, F. F. Leymarie † and B. B. Kimia, Computer Vision and Image Understanding (CVIU), 2009

4. Online and Dynamic Modeling

Surround Structured Lighting for Full Object ScanningD. Lanman, D. Crispell, and G. Taubin, 3D Digital Imaging and Modeling (3DIM), 2007

Data acquisition

Multiple scans and alignment operations are required

System architecture�Multi-view: digital camera + planar mirrors�Orthographic: DLP projector + Fresnel lens

Mechanical Alignment ProcedureManual Mirror Alignment�Parallel to the orthographic illumination rays�Gray code patterns assist adjusting

Step 1: Alignment using a Flat Surface�Cover each mirror with a blank surface�Adjust the uncovered mirror so that the reflected and projected stripes coincide

Step 2: Alignment using a Cylinder�Place a blank cylindrical object in the center of the scanning volume�Adjust both mirrors until the reflected stripes coincide on the cylinder surface

Gray Code Sequence

Reconstruction Algorithm

Step 1: Recover Projector Rows� Project Gray code image sequence�Recover projector scanline illuminating each pixel� Post-process using image morphology

Step 2: Recover 3D point cloud�Reconstruct using ray-plane intersection�Consider each real/virtual camera separately�Assign per-point color using ambient image

Recovered Projector Rows

Real and Virtual Cameras

Camera Centers Optical Rays

Experimental reconstruction results

Ambient Illumination Gray Code Sequence Recovered Projector Rows

In-Hand Scanning with Online Loop ClosureThibaut Weise (EZH), Thomas Wismer (EZH), Bastian Leibe (RWTH Aachen), Luc Van Gool (EZH), ICCV Workshop on 3D Digital Imaging and Modeling (3DIM), 2009

Real-Time 3D model acquisition

[Rusinkiewicz et al. ’02]

Time-Coded Light Patterns�Assign each stripe a unique illumination code over time

[Posdamer 82]

Space

Time

Illumination history = (WB),(BW),(WB)

Code

Codes for Moving Scenes�Assign time codes to stripe boundaries�Perform frame-to-frame tracking of corresponding boundaries�Propagate illumination history

[Hall-Holt & Rusinkiewicz, ICCV 2001]

ICP in the Real-Time Pipeline�Potential problem with ICP: local minima

• In this pipeline, scans close together• Very likely to converge to correct (global) minimum�Basic ICP algorithm too slow (~ seconds)

• Point-to-plane minimization• Projection-based matching• With these tweaks, running time ~ milliseconds

[Rusinkiewicz & Levoy, 3DIM 2001]

Issues on Online Modeling�Loop closure problem

• Accumulation of alignment errors�Sensor noise

Approach�Fast ICP�Surfel representation with topology graph�As-rigid-as-possible deformation

Scanning system�Structured Light(Phase shifting)

Registration�Fast ICP

• Correspondence search by projection• Point-surface distance• Rigid transformation

Fast ICP

�Point-to-Plane error metric

( )∑≠

+−+⋅=kji

iikijijkjikitR

txRtyRnR,

2

,)}(){(minˆ

rr

rrr

ε

)( yxn −⋅y y

nyx

Surfel representation

Surfel

Hexagonal surfel forefficient rendering

Surfel visibilityconfidence

Integration�Surfel Update

�Surfel Addition�Surfel Removal�Surface Growing

• All surfels having low confidences deleted• Lost parts are rescanned

Topology graph�A surfel has at least one node attached in range: attached

nodes are closer than a threshold

Topology graph update�Surfel Addition�Node Addtion�Node Deletion�Surfel / Node Update

Online loop closure

Only registering and integrating new input data with one of the ’meeting’ bordersThe model is closed by an as-rigid-as-possible deformation

Loop closure procedure1. Determine the currently visible nodes2. Intersect the visible nodes with the topology graph and

decompose into connected components3. Register each component individually to the input scan4. Deform the model surface

Loop closure procedure

As-rigid-as-possible Deformation�Deformation is represented by a collection

of rigid transformations�Each graph node gj is assigned a

transformation

�Deformed position pi of surfel i is computed as weighted sum (surfel belongs to multiple nodes)

Estimation of rigid transformations

Position term (distance between corresponding points)

Regularization term (smoothness)

Energy function

Result

Other results

Elastic Convolved ICP for the Registration of Deformable ObjectsRyusuke Sagawa, Kyotaka Akasaka, Yasushi Yagi, Henning Hamer, Luc Van Gool

Registration of deformable object

Rigid object

Deformable object

Approach�Iterative manner (ICP-based)�Point-to-Plane error metric�Search along Line-of-sight�Convolution of the differences

Basic error metric�Point-to-Plane

)( yxn −⋅yy

x yn

T: Rigid transformation

Correspondence search

Similar to window matching

Convolved error

: Neighboring vertices of i

Data term Smoothness term

Weighting SchemesSmoothness (distance between neighboring points)

Error (distance between corresponding points)

Gaussian

Tukey

Linealization

is small

Solving by Least Square method

Smoothness weight

To avoid the shrinkage of the surface

KinectFusion

https://msdn.microsoft.com/ja-jp/library/dn188670.aspx

[Newcombe et al. ISMAR 2011]

Frame to model tracking

DynamicFusion

DynamicFusion [Newcombe et al. CVPR 2015]

Summary (3D Data Processing)�3D geometric modeling pipeline

• Data acquisition• Alignment• Merging

• Mesh-based• Volume-based• Point-based

�Online and Dynamic Modeling