Multiple View Geometry in Computer Vision Marc Pollefeys Comp 290-089

Embed Size (px)

Text of Multiple View Geometry in Computer Vision Marc Pollefeys Comp 290-089

  • Multiple View Geometryin Computer VisionMarc Pollefeys

    Comp 290-089

  • Multiple View GeometryabcA(a,b) A(a,b) cf(a,b,c)=0abc(a,b,c) (a,b,c)(reconstruction)(calibration)(transfer)

  • Course objectivesTo understand the geometric relations between multiple views of scenes.

    To understand the general principles of parameter estimation.

    To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.

  • Relation to other vision/image coursesFocuses on geometric aspectsNo image processing

    Comp 254: Image Processing an AnalysisMostly orthogonal to this course, complementary

    Comp 256: Computer Vision (fall 2003)Will be much broader, based on new book:Computer Vision: a modern approachDavid Forsyth and Jean Ponce

  • MaterialTextbook:Multiple View Geometry in Computer Visionby Richard Hartley and Andrew ZissermanCambridge University Press

    Alternative book:The Geometry from Multiple Imagesby Olivier Faugeras and Quan-Tuan LuongMIT PressOn-line tutorial:

  • Learning approach read the relevant chapters of the books and/or reading assignements before the course. In the course the material will then be covered in detail and motivated with real world examples and applications. Small hands-on assignements will be provided to give students a "feel" of the practical aspects. Students will also read and present some seminal papers to provide a complementary view on some of the covered topics.Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course.

    Grade distribution Class participation: 20% Hands-on assignments: 10% Paper presentation: 10% Implementation assignment/project: 40% Final: 20%

  • ApplicationsMatchMovingCompute camera motion from video (to register real an virtual object motion)

  • Applications3D modeling

  • ContentBackground: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.Single View: Camera model, Calibration, Single View Geometry.Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.Three Views: Trifocal Tensor, Computing T.More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality

  • Multiple View Geometry course schedule(tentative)

  • Fast Forward!Quick overview of what is coming

  • BackgroundLa reproduction interdite (Reproduction Prohibited), 1937, Ren Magritte.

  • Projective 2D GeometryPoints, lines & conicsTransformations

    Cross-ratio and invariants

  • Projective 3D GeometryPoints, lines, planes and quadrics


    , and

  • EstimationHow to compute a geometric relation from correspondences, e.g. 2D trafoLinear (normalized), non-linear and Maximum Likelihood EstimationRobust (RANSAC)

  • Evaluation and error analysisHow good are the results we getBounds on performance

    Covariance propagation & Monte-Carlo estimation


  • Single-View GeometryThe Cyclops, c. 1914, Odilon Redon

  • Camera ModelsMostly pinhole camera modelbut also affine cameras, pushbroom camera,

  • Camera CalibrationCompute P given (m,M)(normalized) linear, MLE,

    Radial distortion

  • More Single-View GeometryProjective cameras and planes, lines, conics and quadrics.Camera center and camera rotation

    Camera calibration and vanishing points, calibrating conic and the IAC

  • Single View MetrologyAntonio Criminisi

  • Two-View GeometryThe Birth of Venus (detail), c. 1485, Sandro Botticelli

  • Epipolar GeometryFundamental matrix Essential matrix

  • Two-View Reconstruction

  • Epipolar Geometry Computation(normalized) linear:



    RANSAC and automated two view matching

  • RectificationWarp images to simplify epipolar geometry

  • Structure ComputationPoints: Linear, optimal, direct optimal

    Also lines and vanishing points

  • Planes and HomographiesRelation between plane and H given P and PRelation between H and F, H from F, F from HThe infinity homography H

  • Three-View GeometryThe Birth of Venus (detail), c. 1485, Sandro Botticelli

  • Trifocal Tensor

  • Three View Reconstruction(normalized) linearminimal (6 points)MLE (Gold Standard)

  • Multiple-View GeometryThe Birth of Venus (detail), c. 1485, Sandro Botticelli

  • Multiple View GeometryQuadrifocal tensor81 parameters, but only 29 DOF!

  • Multiple View ReconstructionAffine factorizationProjective factorization

  • Multiple View ReconstructionSequential reconstruction

  • Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion

  • Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion

  • Bundle adjustmentNo bundle adjustmentBundle adjustment needed to avoid drift of virtual objectthroughout sequenceBundle adjustment (including radial distortion)

  • Auto-calibration

  • Dynamic Structure from Motion

  • CheiralityOriented projective geometryAllows to use fact that points are in front of camera to recover quasi-affine reconstruction to determine order for image warping to determine orientation for rectification with epipoles in images etc.

  • DualityGives possibility to interchange role of P and X in algorithms

  • Contact informationMarc Pollefeys, Room 205marc@cs.unc.eduTel. 962 1845