55:148 Digital Image ProcessingChapter 11 3D Vision, Geometry

Topics:Basics of projective geometry

Points and hyperplanes in projective spaceHomographyEstimating homography from point correspondence

The single perspective cameraAn overview of single camera calibrationCalibration of one camera from the known scene

Scene reconstruction from multiple viewsTriangulationProjective reconstructionMatching constraintsBundle adjustment

Two cameras, stereopsisThe geometry of two cameras. The fundamental matrixRelative motion of the camera; the essential matrixEstimation of a fundamental matrix from image point correspondencesCamera Image rectificationApplications of the epipolar geometry in vision

Three and more camerasStereo correspondence algorithms

Image epipolar rectification

Illustrative description

• Both image planes being coplanar• Baseline along the horizontal axis (x-axis)

of the image plane• Two epipoles at infinity along the

horizontal direction• Horizontal epipolar lines

Before

After

Image epipolar rectification

Illustrative description

• Both image planes being coplanar• Baseline along the horizontal axis (x-axis)

of the image plane• Two epipoles at infinity along the

horizontal direction• Horizontal epipolar lines

• Method: Apply suitable homographic transformation on each image

Before

After

Image rectification (before)

Image rectification (after)

Image rectificationAnalytic description

• Suppose that we know camera matrices and for the two cameras• The objective is to rotate each image planes around their optical centers until their

focal planes becomes coplanar • Thereby, containing the baseline• More specifically, the new x-axis is parallel to the baseline

• Both images have y-axis x-axisThus, new camera matrices

Image rectificationAnalytic description

• Suppose that we know camera matrices and for the two cameras• The objective is to rotate each image planes around their optical centers until their

focal planes becomes coplanar • Thereby, containing the baseline• More specifically, the new x-axis is parallel to the baseline

• Both images have y-axis x-axisThus, new camera matrices

where,

Image rectificationAnalytic description

• Suppose that we know camera matrices and for the two cameras• The objective is to rotate each image planes around their optical centers until their

focal planes becomes coplanar • Thereby, containing the baseline• More specifically, the new x-axis is parallel to the baseline

• Both images have y-axis x-axisThus, new camera matrices

where,

1) The new x-axis is parallel to the baseline

Image rectificationAnalytic description

• Suppose that we know camera matrices and for the two cameras• The objective is to rotate each image planes around their optical centers until their

focal planes becomes coplanar • Thereby, containing the baseline• More specifically, the new x-axis is parallel to the baseline

• Both images have y-axis x-axisThus, new camera matrices

where,

1) The new x-axis is parallel to the baseline 2) The new y-axis is orthogonal to the new x-axis , for some and

Image rectificationAnalytic description

• Suppose that we know camera matrices and for the two cameras• The objective is to rotate each image planes around their optical centers until their

focal planes becomes coplanar • Thereby, containing the baseline• More specifically, the new x-axis is parallel to the baseline

• Both images have y-axis x-axisThus, new camera matrices

where,

1) The new x-axis is parallel to the baseline 2) The new y-axis is orthogonal to the new x-axis , for some and 3) The new z-axis is orthogonal to the new xy-plane

A proof that Image rectification a suitable homographic transformation

A proof that Image rectification a suitable homographic transformation

Let

For any 3D scene point , projected image points using camera matrices are:

A proof that Image rectification a suitable homographic transformation

Let

For any 3D scene point , projected image points using camera matrices are:

Using the relation in homogeneous coordinate system,

A proof that Image rectification a suitable homographic transformation

Let

For any 3D scene point , projected image points using camera matrices are:

Using the relation in homogeneous coordinate system,

A proof that Image rectification a suitable homographic transformation

Let

For any 3D scene point , projected image points using camera matrices are:

Using the relation in homogeneous coordinate system,

i.e.,

A proof that Image rectification a suitable homographic transformation

Let

For any 3D scene point , projected image points using camera matrices are:

Using the relation in homogeneous coordinate system,

i.e.,

: scale factor: 3-by-3 matrix ≈ perspective transformation ≈ homography equivalent

HENCE PROVED

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system:

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get Put together all correspondences

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get Put together all correspondences

Compute and apply singular value decomposition; choose along the eigenvector corresponding to the smallest eigenvalue

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get Put together all correspondences

Compute and apply singular value decomposition; choose along the eigenvector corresponding to the smallest eigenvalueFor to be a valid fundamental matrix, its rank must be 2; but it may not satisfy in reality (why?); thus needs correction

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get Put together all correspondences

Compute and apply singular value decomposition; choose along the eigenvector corresponding to the smallest eigenvalueFor to be a valid fundamental matrix, its rank must be 2; but it may not satisfy in reality (why?); thus needs correctionSVD decomposition: ; set the smallest eigenvalue of to zero giving a new diagonal matrix ; use the new matrix

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Using Kronecker product identity: , we get Put together all correspondences

Compute and apply singular value decomposition; choose along the eigenvector corresponding to the smallest eigenvalueFor to be a valid fundamental matrix, its rank must be 2; but it may not satisfy in reality (why?); thus needs correctionSVD decomposition: ; set the smallest eigenvalue of to zero giving a new diagonal matrix ; use the new matrix

Note that ML estimation method incorporate the validity condition of F into the optimization process

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Output: Fundamental matrix

Task 2: Decompose the fundamental to matrix camera matrices

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Output: Fundamental matrix

Task 2: Decompose the fundamental to matrix camera matrices

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Output: Fundamental matrix

Task 2: Decompose the fundamental to matrix camera matrices

Input of the algorithm : A linear system: Output of the algorithm: camera matrices and

Image rectificationAlgorithmic description

• As we have seen from analytic discussionGiven the knowledge of and , image rectification process is precisely defined So, the first task is to determine camera matrices and

Algorithm Compute camera matrices and Task 1: Solve using corresponding point pairs in two images and algebraic error minimizationInput: A linear system: Output: Fundamental matrix

Task 2: Decompose the fundamental to matrix camera matrices

Input of the algorithm : A linear system: Output of the algorithm: camera matrices and

Now what?

Image rectificationAlgorithmic description

Algorithm Compute image rectification

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositions

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and Determine new intrinsic parameters by averaging and and then annulling skew component

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and Determine new intrinsic parameters by averaging and and then annulling skew componentCompute new projection matrices:

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and Determine new intrinsic parameters by averaging and and then annulling skew componentCompute new projection matrices: and

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and Determine new intrinsic parameters by averaging and and then annulling skew componentCompute new projection matrices: and Compute homography transformations

Image rectificationAlgorithmic description

Algorithm Compute image rectificationInput: and Output: and and two homography transformations: and

Decompose each camera matrices: and Determine two optical centers and using and and their decompositionsDetermine new rotation matrix common to and Determine new intrinsic parameters by averaging and and then annulling skew componentCompute new projection matrices: and Compute homography transformations and

Image rectification (before)

Image rectification (after)

Image rectification: application

3D reconstruction becomes easier

a

b

c

Panoramic view

Panoramic view: challenges

Elementary stereo geometry in the rectified image configuration

The depth of a 3D scene point can be calculated using the disparity measure

Elementary stereo geometry in the rectified image configuration

The depth of a 3D scene point can be calculated using the disparity measure

Elementary stereo geometry in the rectified image configuration

The depth of a 3D scene point can be calculated using the disparity measure

Elementary stereo geometry in the rectified image configuration

The depth of a 3D scene point can be calculated using the disparity measure