Computer Vision Multiple View Geometry Marc Pollefeys COMP 256

Embed Size (px)

Text of Computer Vision Multiple View Geometry Marc Pollefeys COMP 256

  • Slide 1

Computer Vision Multiple View Geometry Marc Pollefeys COMP 256 Slide 2 Computer Vision 2 Last class Gaussian pyramid Laplacian pyramid Gabor filters Fourier transform Slide 3 Computer Vision 3 Histograms : co-occurrence matrix Not last class Slide 4 Computer Vision 4 Texture synthesis [Zalesny & Van Gool 2000] 2 analysis iterations 6 analysis iterations 9 analysis iterations Slide 5 Computer Vision 5 View-dependent texture synthesis [Zalesny & Van Gool 2000] Slide 6 Computer Vision 6 Efros & Leung 99 Assuming Markov property, compute P(p|N(p)) Building explicit probability tables infeasible Instead, lets search the input image for all similar neighborhoods thats our histogram for p To synthesize p, just pick one match at random p non-parametric sampling Input image Synthesizing a pixel Slide 7 Computer Vision 7 p Efros & Leung 99 extended Observation: neighbor pixels are highly correlated Input image non-parametric sampling B Idea: unit of synthesis = block Exactly the same but now we want P(B|N(B)) Much faster: synthesize all pixels in a block at once Not the same as multi-scale! Synthesizing a block Slide 8 Computer Vision 8 Input texture B1B2 Random placement of blocks block B1 B2 Neighboring blocks constrained by overlap B1B2 Minimal error boundary cut Slide 9 Computer Vision 9 min. error boundary Minimal error boundary overlapping blocksvertical boundary _ = 2 overlap error Slide 10 Computer Vision 10 Slide 11 Computer Vision 11 Slide 12 Computer Vision 12 Slide 13 Computer Vision 13 Why do we see more flowers in the distance? Perpendicular textures [Leung & Malik CVPR97] Slide 14 Computer Vision 14 Shape-from-texture Slide 15 Computer Vision 15 Jan 16/18-Introduction Jan 23/25CamerasRadiometry Jan 30/Feb1Sources & ShadowsColor Feb 6/8Linear filters & edgesTexture Feb 13/15Multi-View GeometryStereo Feb 20/22Optical flowProject proposals Feb27/Mar1Affine SfMProjective SfM Mar 6/8Camera CalibrationSilhouettes and Photoconsistency Mar 13/15Springbreak Mar 20/22SegmentationFitting Mar 27/29Prob. SegmentationProject Update Apr 3/5Tracking Apr 10/12Object Recognition Apr 17/19Range data Apr 24/26Final project Tentative class schedule Slide 16 Computer Vision 16 THE GEOMETRY OF MULTIPLE VIEWS Reading: Chapter 10. Epipolar Geometry The Essential Matrix The Fundamental Matrix The Trifocal Tensor The Quadrifocal Tensor Slide 17 Computer Vision 17 Epipolar Geometry Epipolar Plane Epipoles Epipolar Lines Baseline Slide 18 Computer Vision 18 Epipolar Constraint Potential matches for p have to lie on the corresponding epipolar line l. Potential matches for p have to lie on the corresponding epipolar line l. Slide 19 Computer Vision 19 Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981) Slide 20 Computer Vision 20 Properties of the Essential Matrix E p is the epipolar line associated with p. E T p is the epipolar line associated with p. E e=0 and E T e=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). T T Slide 21 Computer Vision 21 Epipolar Constraint: Small Motions To First-Order: Pure translation: Focus of Expansion Slide 22 Computer Vision 22 Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992) Slide 23 Computer Vision 23 Properties of the Fundamental Matrix F p is the epipolar line associated with p. F T p is the epipolar line associated with p. F e=0 and F T e=0. F is singular. T T Slide 24 Computer Vision 24 The Eight-Point Algorithm (Longuet-Higgins, 1981) | F | =1. Minimize: under the constraint 2 Slide 25 Computer Vision 25 Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F, using an appropriate rank-2 parameterization. Slide 26 Computer Vision 26 Problem with eight-point algorithm linear least-squares: unit norm vector F yielding smallest residual What happens when there is noise? Slide 27 Computer Vision 27 The Normalized Eight-Point Algorithm (Hartley, 1995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q = T p, q = T p. Use the eight-point algorithm to compute F from the points q and q. Enforce the rank-2 constraint. Output T F T. T iiii ii Slide 28 Computer Vision 28 Epipolar geometry example Slide 29 Computer Vision 29 Example: converging cameras courtesy of Andrew Zisserman Slide 30 Computer Vision 30 Example: motion parallel with image plane (simple for stereo rectification) courtesy of Andrew Zisserman Slide 31 Computer Vision 31 Example: forward motion e e courtesy of Andrew Zisserman Slide 32 Computer Vision 32 Fundamental matrix for pure translation courtesy of Andrew Zisserman auto-epipolar Slide 33 Computer Vision 33 Fundamental matrix for pure translation courtesy of Andrew Zisserman Slide 34 Computer Vision 34 Trinocular Epipolar Constraints These constraints are not independent! Slide 35 Computer Vision 35 Trinocular Epipolar Constraints: Transfer Given p and p, p can be computed as the solution of linear equations. 123 Slide 36 Computer Vision 36 problem for epipolar transfer in trifocal plane! image from Hartley and Zisserman Trinocular Epipolar Constraints: Transfer There must be more to trifocal geometry Slide 37 Computer Vision 37 Trifocal Constraints Slide 38 Computer Vision 38 Trifocal Constraints All 3x3 minors must be zero! Calibrated Case Trifocal Tensor Slide 39 Computer Vision 39 Trifocal Constraints Uncalibrated Case Trifocal Tensor Slide 40 Computer Vision 40 Trifocal Constraints: 3 Points Pick any two lines l and l through p and p. Do it again. 2323 T( p, p, p )=0 1 2 3 Slide 41 Computer Vision 41 Properties of the Trifocal Tensor Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares Impose the constraints a posteriori. For any matching epipolar lines, l G l = 0. The matrices G are singular. They satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995). 213 Ti 1 i Slide 42 Computer Vision 42 For any matching epipolar lines, l G l = 0. 213 Ti The backprojections of the two lines do not define a line! Slide 43 Computer Vision 43 108 putative matches 18 outliers 88 inliers 95 final inliers (26 samples) (0.43) (0.23) (0.19) Trifocal Tensor Example courtesy of Andrew Zisserman Slide 44 Computer Vision 44 additional line matches Trifocal Tensor Example images courtesy of Andrew Zisserman Slide 45 Computer Vision 45 Transfer: trifocal transfer doesnt work if l=epipolar line image courtesy of Hartley and Zisserman (using tensor notation) Slide 46 Computer Vision 46 Image warping using T(1,2,N) (Avidan and Shashua `97) Slide 47 Computer Vision 47 Multiple Views (Faugeras and Mourrain, 1995) Slide 48 Computer Vision 48 Two Views Epipolar Constraint Slide 49 Computer Vision 49 Three Views Trifocal Constraint Slide 50 Computer Vision 50 Four Views Quadrifocal Constraint (Triggs, 1995) Slide 51 Computer Vision 51 Geometrically, the four rays must intersect in P.. Slide 52 Computer Vision 52 Quadrifocal Tensor and Lines Slide 53 Computer Vision 53 Quadrifocal tensor determinant is multilinear thus linear in coefficients of lines ! There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor Slide 54 Computer Vision 54 Scale-Restraint Condition from Photogrammetry Slide 55 Computer Vision 55 from perspective to omnidirectional cameras perspective camera (2 constraints / feature) radial camera (uncalibrated) (1 constraints / feature) 3 constraints allow to reconstruct 3D point more constraints also tell something about cameras multilinear constraints known as epipolar, trifocal and quadrifocal constraints (0,0) l =(y,-x) (x,y) Slide 56 Computer Vision 56 Radial quadrifocal tensor Linearly compute radial quadrifocal tensor Q ijkl from 15 pts in 4 views Reconstruct 3D scene and use it for calibration (2x2x2x2 tensor) (2x2x2 tensor) Not easy for real data, hard to avoid degenerate cases (e.g. 3 optical axes intersect in single point). However, degenerate case leads to simpler 3 view algorithm for pure rotation Radial trifocal tensor T ijk from 7 points in 3 views Reconstruct 2D panorama and use it for calibration (x,y) Slide 57 Computer Vision 57 Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV05) normalized radius angle Models fish-eye lenses, cata-dioptric systems, etc. Slide 58 Computer Vision 58 Non-parametric distortion calibration (Thirthala and Pollefeys, ICCV05) normalized radius angle 90 o Models fish-eye lenses, cata-dioptric systems, etc. Slide 59 Computer Vision 59 Next class: Stereo F&P Chapter 11 image I(x,y) image I(x,y) Disparity map D(x,y) (x,y)=(x+D(x,y),y)