Temporally Coherent Completion of Dynamic Shapes

AUTHORS:HAO LI,LINJIE LUO,DANIEL VLASICPIETER PEERS,JOVAN POPOVIC,MARK PAULY,SZYMON RUSINKIEWICZ

Presenter:Zoomin(Zhuming) Hao

Previous WorkPrevious Work

1.Based on Template How to obtain the template?

①a separate rigid reconstruction step

(e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008])

Previous WorkPrevious Work

Robust Single-View Geometry and Motion Reconstruction[Li et al. 2009]

Previous WorkPrevious Work

1.Based on Template How to obtain the template?

①a separate rigid reconstruction step

(e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008])

②globally aggregating all surface samples through time

(e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008])

Previous WorkPrevious Work

Efficient Reconstruction of Nonrigid Shape and Motion from Real-Time 3D Scanner Data

[Wand et al. 2009]

input ==> a sequence of point clouds sampled at different time instances

automatically assembles them into a common shape that best fits allof the input data

a deformation field is computed that approximates the motion ofthis shape to match all the data frames

limitations： occurs if objects disappear inan acquisition hole and come out in avery different pose

Previous WorkPrevious Work

1.Based on Template How to obtain the template?

①a separate rigid reconstruction step

(e.g., [Li et al. 2008; de Aguiar et al. 2008; Vlasic et al. 2008])

②globally aggregating all surface samples through time

(e.g., [Wand et al. 2009; Mitra et al. 2007; S¨ußmuth et al. 2008])

Disadvantage？ fix the topology

geometric details are limited to those in the template

Previous WorkPrevious Work

2.Based on the assumption:

Dynamic performance consists of rigid parts [Pakelny and Gotsman2008] manual segmentation,an optimal ri

gid motion is computed for each part

[Chang and Zwicker 2009] limits to subjects that exhibit articulated motion

[Zheng et al.2010] automatically extract a consensus skeleton to derive a consistent temporal topology

Previous WorkPrevious Work

Consensus skeleton for nonrigid space-time registration [Zheng et al.2010]

input==>a sequence of point clouds acquired over time

extract per-frame skeletons

consolidate them into a skeleton structure (consistent across time and accounts for all the frames)

Limitations： It assumes that the underlyingshape is clearly articulated which is not always the case forsubjects wearing loose clothing

Articulated Mesh Animation from Multi-view Silhouettes [Vlasic et al. 2008]

System OverviewSystem Overview

Framework -- 1Pairwise CorrespondencesFramework -- 1Pairwise Correspondences

Coarse-scale Correspondences：non-rigid ICP algorithm[Li et al.2009]

Framework -- 1Pairwise CorrespondencesFramework -- 1Pairwise Correspondences

Fine-scale Correspondences：Improvement based on two observations: 1.far-away points can bias the local alignment(local-support)

2.stability of ICP matching algorithm depends on the local geometry

Three-step Algorithm provided by this paper:

1.Sampling

2.Matching:

non-rigid locally weighted ICP algorithm[Brown and Rusinkiewicz 2007]

employ a CSRBF for point selection near feature point

3.Warping

Framework -- 1Shape AccumulationFramework -- 1Shape Accumulationfi

'(merged) and fi+1(original)==>Corrsepondences

fi+1‘

merge

warp

from first frame to the last framefrom last frame to the first frame

interleaved registration/merging scheme in a forward&backward fashion

System OverviewSystem Overview

Framework -- 2Hole FillingFramework -- 2Hole Filling

visual Hull prior [Vlasic et al. 2009] + weighted Poisson surface reconstruction [Kazhdan et al. 2006]

Surface Fairing: Minimizing bending energy of the patch’s vertices using bi-Laplacian[Botsch and Sorkine 2008]

System OverviewSystem Overview

Framework -- 3Temporal FilteringFramework -- 3Temporal Filtering

1. Warp two neighboring frames to current frame based on the pairwise correspondences

2. Combine them using Poisson reconstruction with different weight for different region

Poisson reconstruction

warp to warp to

System OverviewSystem Overview

Framework -- 4Detail ResynthesisFramework -- 4Detail Resynthesis

1. Resynthesize high frequency detail

[Nehab et al.2005]

2. Acquire normal maps [Vlasic et al.2009]

Conclusions Contribution:

• A framework to automatically fill holes with temporal coherent patches without relying on a geometrical template.

• some little improvements on previous algorithms

Limitation:

1. Topology of our meshes will always match the(changing and sometimes incorrect)topology of the visual hull.(Ideally, we need to extract a single consistent topology)

2. Temporal correspondences are valid between nearby frames only

3. each frame should cover most part of the object surface ( limited to multi-view scans),the unobserved regions have no geometric details in them.

Future work:

• To take physical properties into account