Embed Size (px)
Citation preview
Dense Variational Reconstruction of Non-RigidSurfaces from Monocular Video
Ravi Garg Anastasios Roussos∗ Lourdes Agapito∗
Queen Mary, University of London∗Now at UCL
Before This Paper
1 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
NRSfM: ill posed problem
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17
Traditional Sparse Non Rigid Structure from Motion
...
3 / 17
Traditional Sparse Non Rigid Structure from Motion
...Feature Tracking
...
3 / 17
Traditional Sparse Non Rigid Structure from Motion
...Feature Tracking
...
...3D ShapeInference
3 / 17
Traditional Sparse Non Rigid Structure from Motion
Priors
...Feature Tracking
... +
...3D ShapeInference
3 / 17
Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
Bregler et al. CVPR’00, Brand CVPR’01, Xiao et al. IJCV’06, Torresani et al. PAMI’08, Akhter et al.CVPR’09, Bartoli et al. CVPR2008, Paladini et al. IJCV’12,Dai et al. CVPR’12...
4 / 17
Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
(Park et al. ECCV’10)
4 / 17
Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
5 / 17
Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
Key features
Variational approach.
Use of smoothness priors.
Per pixel reconstruction.
Scalable and GPU friendly algorithm.
5 / 17
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
6 / 17
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.Variational framework from dense rigid SfM.
6 / 17
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.Variational framework from dense rigid SfM.
Our contribution
First variational formulation to dense NRSfM.
Scalable algorithm which can be ported on GPU.
6 / 17
Our Approach in a Nutshell
...
7 / 17
Our Approach in a Nutshell
...
...
Step 1: DenseVideo Registration ...
7 / 17
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
7 / 17
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
Priors +
7 / 17
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
Low rank.Spatial smoothness.+
7 / 17
Our Approach in a Nutshell
...
... Low rank.Spatial smoothness.
Step 1: DenseVideo Registration ...
+
Garg, Roussos, Agapito, A variational approach to video registration with subspace constraints, IJCV’13.7 / 17
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
... Low rank.Spatial smoothness.+
7 / 17
Orthographic Projection Model
8 / 17
Orthographic Projection Model
8 / 17
Orthographic Projection Model
8 / 17
Orthographic Projection Model
8 / 17
Orthographic Projection Model
W = RS
8 / 17
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
Reprojection ErrorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´Edata (R,S) = ‖W − RS‖2F
10 / 17
Spatial Smoothness PriorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´Ereg
(S)
=∑
i
TV (Si)
−−−−−−→
Without regularisation With regularisation11 / 17
Low Rank PriorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´
Etrace
(S)
= ‖S‖∗ =∑
i
σi(S)
lies in−−−−→ span
K � F
Angst et al. ECCV’12, Dai et al. CVPR’12, Angst et al. ICCV’11, Dai et al. ECCV’10
12 / 17
Minimisation of E(R,S
)
minR,Sλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
13 / 17
Minimisation of E(R,S
)
minR,Sλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Our Algorithm
Initialize R and S using rigid factorisation.
Minimize energy via alternation:
Step 1: Rotation estimation.Step 2: Shape estimation.
Efficient and highly parallelizable algorithm → GPU-friendly
13 / 17
Minimisation of E(R,S
)
minRλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Step 1: Rotation estimation
Robust estimation by using dense data.
Solved via Levenberg-Marquardt algorithm.
Rotations are parametrised as quaternions.
13 / 17
Minimisation of E(R,S
)
minSλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Step 2: Shape estimation
Convex sub-problem.
Optimisation via alternation between:
Per frame shape refinement: using primal dual algorithmEnforcing low rank: using soft impute algorithm.
13 / 17
Results on real sequences
14 / 17
Quantitative Evaluation
Average RMS 3D reconstruction errors.
Sequence TB MP Ours Ours(τ = 0)
Non-smooth rotations 4.50% 5.13% 2.60% 3.32%Smooth rotations 6.61% 5.81% 2.81% 3.89%
- TB: Akhter et al, Trajectory space: A dual representation for non-rigid structure from motion, PAMI’11.
- MP: Paladini et al, Optimal metric projections for deformable and articulated structure-from-motion,IJCV’12.
15 / 17
Conclusions and future work
Conclusions:
First dense, template-free approach to Non-rigid Structurefrom Motion.Unified energy minimization for both rotation and shapeestimation.Combination of low-rank and spatial regularization prior.Using variational methods, we can do much more withmonocular sequences that one could expect
Future work:
Direct estimation from pixel intensitiesTowards real-time: online formulationOcclusion modelling
16 / 17
Thank You for Your Attention!
For more details and data
Visit: www.eecs.qmul.ac.uk/~rgargOR Come to our poster!
Questions?
Authors thank
Chris Russell and Sara Vicente for valuable discussions.17 / 17
![Air Cooler Industry AC-LN / ACA-LN / ACAF-LN 8-141].pdf · Noise levels See technical data AC-LN and ACA-LN /ACAF-LN The noise levels are only reference values as the acoustic properties](https://img.dokumen.tips/doc/110x75/5b443be77f8b9a64608bbcbc/air-cooler-industry-ac-ln-aca-ln-acaf-ln-8-14-1pdf-noise-levels-see-technical.jpg)
![Samsung Ln-s3292d Ln-s4092d Ln-s4692d Bn94-01037a Schematic Diagram [Sch]](https://img.dokumen.tips/doc/110x75/563db88d550346aa9a94b946/samsung-ln-s3292d-ln-s4092d-ln-s4692d-bn94-01037a-schematic-diagram-sch.jpg)
