Multiple View Reconstruction Class 24 Multiple View Geometry Comp 290-089 Marc Pollefeys

Embed Size (px)

Text of Multiple View Reconstruction Class 24 Multiple View Geometry Comp 290-089 Marc Pollefeys

  • Slide 1
  • Multiple View Reconstruction Class 24 Multiple View Geometry Comp 290-089 Marc Pollefeys
  • Slide 2
  • Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Self-Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality
  • Slide 3
  • Multi-view computation
  • Slide 4
  • practical structure and motion recovery from images Obtain reliable matches using matching or tracking and 2/3-view relations Compute initial structure and motion sequential structure and motion recovery hierarchical structure and motion recovery Refine structure and motion bundle adjustment Auto-calibrate Refine metric structure and motion
  • Slide 5
  • Sequential structure and motion recovery Initialize structure and motion from 2 views For each additional view Determine pose Refine and extend structure
  • Slide 6
  • Initial structure and motion Epipolar geometry Projective calibration compatible with F Yields correct projective camera setup (Faugeras92,Hartley92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space
  • Slide 7
  • Compute P i+1 using robust approach (6-point RANSAC) Extend and refine reconstruction 2D-2D 2D-3D mimi m i+1 M new view Determine pose towards existing structure
  • Slide 8
  • Non-sequential image collections 4.8im/pt 64 images 3792 points Problem: Features are lost and reinitialized as new features Solution: Match with other close views
  • Slide 9
  • For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm Refine existing structure Initialize new structure Relating to more views Problem: find close views in projective frame For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm For all close views k Compute two view geometry k/i and matches Infer new 2D-3D matches and add to list Refine pose using all 2D-3D matches Refine existing structure Initialize new structure
  • Slide 10
  • Refining and extending structure Refining structure Extending structure Triangulation (Iterative linear) (Hartley&Sturm,CVIU97) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion
  • Slide 11
  • Structure and motion: example 190 images 7000points Input sequence Viewpoint surface mesh calibration demo
  • Slide 12
  • ULM demo
  • Slide 13
  • Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM
  • Slide 14
  • Stitching 3-view reconstructions Different possibilities 1. Align (P 2,P 3 ) with (P 1,P 2 ) 2. Align X,X (and CC) 3. Minimize reproj. error 4. MLE (merge)
  • Slide 15
  • Refining structure and motion Minimize reprojection error Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) Huge problem but can be solved efficiently (Bundle adjustment)
  • Slide 16
  • Non-linear least-squares Newton iteration Levenberg-Marquardt Sparse Levenberg-Marquardt
  • Slide 17
  • Newton iteration Taylor approximation Jacobian normal eq.
  • Slide 18
  • Levenberg-Marquardt Augmented normal equations Normal equations solve again accept small ~ Newton (quadratic convergence) large ~ descent (guaranteed decrease)
  • Slide 19
  • Levenberg-Marquardt Requirements for minimization Function to compute f Start value P 0 Optionally, function to compute J (but numerical derivation ok too)
  • Slide 20
  • Sparse Levenberg-Marquardt complexity for solving prohibitive for large problems (100 views 10,000 points ~30,000 unknowns) Partition parameters partition A partition B (only dependent on A and itself) typically A contains camera parameters, and B contains 3D points
  • Slide 21
  • Sparse bundle adjustment residuals: normal equations: with
  • Slide 22
  • Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:
  • Slide 23
  • Sparse bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization
  • Slide 24
  • Sparse bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms
  • Slide 25
  • Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:
  • Slide 26
  • Sparse bundle adjustment Covariance estimation
  • Slide 27
  • Degenerate configurations Camera resectioning Two views More views (H&Z Ch.21)
  • Slide 28
  • Camera resectioning Cameras as points 2D case Chasles theorem
  • Slide 29
  • Ambiguity for 3D cameras Twisted cubic (or less) meeting lin. subspace(s) (degree+dimension