17
Computing the Fundamental matrix Peter Praženica FMFI UK May 5, 2008

Computing the Fundamental matrix Peter Praženica FMFI UK May 5, 2008

Embed Size (px)

Citation preview

Computing the Fundamental matrix

Peter PraženicaFMFI UK

May 5, 2008

Contents

• Fundamental matrix• Linear methods

Basic algorithmNormalized

• Nonlinear methodsThe distance to epipolar linesThe Gradient criterion and an interpretation as a distanceThe “optimal” (reprojected) method

• M-Estimator with the Tukey function• Monte Carlo - Least-median-of-squares method• Example of estimation with comparison

Fundamental matrix

-The basic description of the projective geometry of two views

- The computation of the Fundamental matrix is of utmost theoretical and practical importance (self-calibration method, recovering the 3D projective geometry)Stage for this chapter- Two views of a scene taken by two (perspective projective) cameras- n (respectively n’) points of interest have been detected in image 1 (respectively in image 2)- Goal: to establish a set of correspondences and to compute the estimate of the Fundamental matrix F that best explains those correspondences

The key idea: Each point correspondence (m, m') yields one equation , therefore with a sufficient number of point correspondences in general position we can determine F. No knowledge about the cameras or scene structure is necessary.

Principle: Let and be two corresponding points. If the Fundamental matrix is , then the epipolar constraint can be rearranged as

Where

Equation 6.1 is linear and homogeneous in the 9 unknown coefficients of matrix F. Thus if we are given 8 matches, then we will be able, in general, to determine a unique solution for F, defined up to a scale factor. This approach is known as the eight-point algorithm.

Tvum 1,, ''' Tvum 1,,

3,1

jiijFF 0' FmmT

0FU T

T

T

FFFFFFFFFf

vuvvvuvuvuuuU

333231232221131211

''''''

,,,,,,,,

1,,,,,,,,

Linear methods

0' Fmm T

(6.1)

In this case, we have more linear equations than unknowns, and, in the presence of noise, we look for the minimum of the error criterion

which is equivalent to finding the minimum of

(6.3)

Where

2 solutions1) closed-form solution via the linear equations2) solution of subject to

2

1

)'( i

p

i

Ti FmmFC

2~)( fUfC

Tp

T

U

U

U 1

~

LM: Least-squares solution

2min Axx 1

2 x

LM: Least-squares solutionClosed-form solution via the linear equations: One of the coefficients of F is set to 1, yielding a parameterization of F by eight values, which are the ratios of the eight other coefficients to the normalizing one. Let us assume for the sake ofdiscussion that the normalizing coefficient is F33. We can rewrite (6.3) as

where is formed with the first 8 columns of , and b is the ninth column of . The solution is known to be

if the matrix is invertible.

The second method solves the classical problem

with

The solution is the eigenvector associated with the smallest eigenvalue of .

2~bgVnim

g

U~

V~

TFFFFFFFFg 3231232221131211 ,,,,,,,U~

bVVVg TT ~~~ 1

fUf

~min 1f

UU T ~~

VV T ~~

Linear methods: DiscussionThe advantage of the eight-point algorithm is that it leads to a noniterativecomputation method which is easy to implement using any linear algebranumerical package; however, we have found that it is quite sensitive to noise, even with numerous data points.

• The constraint det(F) = 0 is not satisfied

• The quantity that is minimized in (6.2) does not have a good geometric interpretation, one could say that one might be trying to minimize an irrelevant quantity

• In all of the linear estimation techniques that will be described in this section, wedeal with projective coordinates of the pixels. In a typical image of dimensions512 x 512, a typical image point will have projective coordinates whose orders ofmagnitude are given componentwise by the triple (250,250,1). The fact that thefirst two coordinates are more than one order of magnitude larger than the thirdone has the effect of causing poor numerical conditioning. Solution: normalizing the points in the two images so that their three projective coordinates are of roughly the same order.

LM: Enforcing the rank constraint

Two-step approximate linear determination of parameters: The idea is tonote that in Equation 5.10 of Proposition 5.12, if the epipoles are known, then theFundamental matrix is a linear function of the coefficients (a, b, c, d) of the epipolarhomography.

1. We determine the epipole e by solving the constrained minimization problem subject to ,

which yields e as the unit norm eigenvector of matrix corresponding to thesmallest eigenvalue

2. Once the epipoles are known, we can use Equation 5.10 or, if the epipoles are notat infinity, Equation 5.14, and write the expression as a linear expressionof the coefficients of the epipolar homography:

where (respectively ) are the affine coordinates of (respectively )This equation can be used to solve for the coefficients of the epipolar homographyin a least-squares formulation, thereby completing the determination of theparameters of F.

2min Fee

12 e

FF T

iT Fmm'

0

'

deevueuvecvveeevve

beeueeuuuaevuvueeeFmm

yxiiyiixiiyyyiiy

xxixxiiixiiiiyxyiTi

ii vu , ii vu , im im

LM: Enforcing the rank constraint

Using the closest rank 2 matrix: The matrix F found by the eight-point algorithmis replaced by the singular matrix that minimizes the Frobenius norm llF - F'll.This step can be done using a singular value decomposition. If ,with , then .

Solving a cubic equation: One selects two components of F, say, withoutloss of generality, F32 and F33 and rewrites (6.3) as

The vector h is made of the first 7 components of the vector f, the matrix ismade of the first 7 columns of matrix , and the vectors W8 and W9 are the eighthand ninth column vectors of that matrix, respectively. The solution h is

We have a linear family of solutions parametrized by F32 and F33. In that familyof solutions, there are some that satisfy the condition det(F) = 0. They are obtainedby expressing the determinant of the solutions as a function of F32 and F33.One obtains a homogeneous polynomial of degree 3 which must be zero. Solving forone of the two ratios F32 : F33 or F33 : F32 one obtains at least one real solutionfor the ratio and therefore at least one solution for the Fundamental matrix.

TVUdiagF 321 ,, 0321 TVUdiagF 0,, 21

2

933832

~min WFWFhWh

W~

U~

933832

1 ~~~ˆ WFWFWWWh TT

Nonlinear methods

The distance to epipolar lines

The idea is to use a non-quadratic error function and to minimize

However, two images should play the symmetric roles

It can be written

using the fact that . It can be observed that this criteriondoes not depend on the scale factor used to compute F.

The Gradient criterion and an interpretation as a distance

Uses criterion

i

ii Fmmd ,2

i

iT

iii mFmdFmmd ,, 22

22

2

2

1

2

2

2

1

11i

Ti

i iT

iT

ii

FmmmFmFFmFm

iTT

iiTi mFmFmm

i iT

iT

ii

iTi

mFmFFmFm

Fmm2

2

2

1

2

2

2

1

2

Nonlinear methodsThe “optimal” methodThe nonlinear method based on the minimization of distances betweenobserved points and reprojected ones

where f is a vector of parameters for F, {Mi} are the 3D coordinates of the points inspace, and P(f), P’(f) are the perspective projection matrices in the first and secondimages, given the Fundamental matrix F(f).

That is, we estimate not only F, but also the most likely true positions .Because of the epipolar constraint, there are 4 - 1 degrees of freedom for each point correspondence, so the total number of parameters is 3p + 7.

The estimation of the Fundamental matrix parameters f can be decoupled from the estimation of the parameters Mi : Since the optimal parameters can be determined from the parameters f, we have to minimize only over f. At each iteration, we have to perform the projective reconstruction of p points. The problem can be rewritten as

i

iiiiMMf

fmdfmdp

,2

,2

, ,,1

min

ii mm ˆ,ˆ

i

iiiiMf

fmdfmdi

,,minmin 22

M-EstimatorLet be the residual of the i-th datum. For example, in the case of the linear methods , in the case of the nonlinear methods

The standard least-squares method tries to minimize , which is unstable ifthere are outliers present in the data. The M-estimators try to reduce the effect of outliers by replacing the squared residuals by another function of the residuals, yielding the error function where p is a symmetric, positive function witha unique minimum at zero and is chosen to be growing slower than the square function.

The M-estimator of f based on the function is a vector f which is a solutionof the system of q equations (6.16)

where the derivative is called the influence function.If we now define define a weight function

then Equation 6.16 becomes

ir

iTii Fmmr iT

iiii mFmdFmmdr ,, 222

i

ir2

2ir

i

if

r )(min

)( ir

qjforf

rr

i j

ii ,,1,0)(

dxxdx /)()(

x

xx

)()(

qjforf

rrr

j

ii

ii ,,1,0)(

M-Estimator with the Tukey function

crif

c

c

rcr i

i

i

2

322

116

otherwise

where is some estimated standard deviation of errors and c = 4.6851 is the tuningconstant. The corresponding weight function is

otherwise

crifc

r

i

i

i

22

0

1

Monte Carlo - Least-median-of-squares method

It estimates the parameters by solving the nonlinear minimization problem

Given p point correspondences ,we proceed through the following steps

1) A Monte Carlo type technique is used to draw m random subsamples of q = 7different point correspondences

2) For each subsample, indexed by J, we use the technique described in Proposition5.14 to compute the Fundamental matrix FJ. We may have at most 3 solutions

3) Compute the median, denoted by MJ, of the squared residuals

4) Retain the estimate FJ for which MJ is minimal among all m MJ’s

2min iif

rmedian

pimm ii ,,1,

iTJiiJi

piJ mFmdmFmdmedianM

,, 22

,,1

Example of estimation with comparison

- Pair of calibrated stereo images- There are 241 point matches which are established automatically- Outliers have been discarded- The calibrated parameters of the cameras are not used, but the Fundamentalmatrix computed from these parameters serves as a ground truthMethod e e’ RMS CPUCalibrated 5138.18 -8875.85 1642.02 -

2528.910.99

Linear 5.85% 304.018 124.039 256.219 230.306 3.40 0.13s

Normalization

7.20% -3920.6 7678.71 8489.07 -15393.5

0.89 0.15s

Nonlinear 0.92% 8135.03 -14048.3 1896.19 -2917.11

0.87 0.38s

Gradient 0.92% 8166.05 -14104.1 1897.80 -2920.12

0.87 0.40s

M-estimator 0.12% 4528.94 -7516.3 1581.19 -2313.72

0.87 1.05s

Reprojected 0.92% 8165.05 -14102.3 1897.74 -2920.01

0.87 19.1s

LMedS 0.13% 3919.12 6413.1 1500.21 -2159.65

0.75 2.40s

F

%100 CalibFFF (+ each F is normalized by its Frobenius norm)

e, e’ positions of 2 epipolesRMS = squared distances between points and their epipolar linesCPU = approximate CPU time in seconds when the program is run on a Sparc 20 workstation

Better comparison method

1) Choose randomly a point m in the first image2) Draw epipolar line of m in the second image using F1

3) If the epipolar line does not intersect the second image, then go to Step 1

4) Choose randomly a point m’ on the epipolar line. Note that m and m’ correspond to each other exactly with respect to F1

5) Draw the epipolar line of m in the second image using F2 , and compute the distance, denoted by d’1 , between point m’ and line F2m

6) Draw the epipolar line of m’ in the first image using F2 , and compute the distance, denoted by d1 , between point m and line F2Tm’

7) Conduct the same procedure from Step 2 through Step 6, but reversing the roles of F1 and F2, and compute d2 and d’2

8) Repeat Step 1 through Step 7 N times9) Compute the average distance of d’s, which is the measure of

difference between the two Fundamental matrices

New comparason

Linear

Normalized

Nonlinear

Gradient

M-Estimator

Reprojected

LMeDS

Calibrated

116.4

5.97 2.66 2.66 2.27 2.66 1.33

Linear 117.29 115.97 116.40 115.51 116.25 115.91

Normalized

4.13 4.12 5.27 4.11 5.89

Nonlinear 0.01 1.19 0.01 1.86

Gradient 1.19 0.00 1.86

M-Estimator

1.20 1.03

Reprojected

1.88

• Linear method is very bad• The linear method with prior data normalization gives quite a reasonable result• The nonlinear method based on point-line distances and that based on gradient-weighted epipolar errors give very similar result to those obtained based on minimization of distances between observed points and reprojected ones. The latter should be avoided because it is too time consuming.• M-Estimators or the LMedS method give still better results because they try to limit or eliminate the effect of poorly localized points. The epipolar geometry estimated by LMedS is closer to the one computed through stereo calibration.