Lecture4:3DReconstruction

CSE152:ComputerVisionManmohanChandraker

Quiz

Gradients1(a) ForimageI,areIx andIy smallorlargeatpointsA,B,CandD?

x

y

A B

C

D

Corners2(a) Isthissecondmomentmatrixlikelytorepresentacorner?Why?

2(b) Isthissecondmomentmatrixlikelytorepresentacorner?Why?

2(c) Isthissecondmomentmatrixlikelytorepresentacorner?Why?

SIFT3(a)WhatareinvariancesencodedbytheSIFTfeature?

3(b) Youobtained10truepositives,5falsepositivesand5falsenegativesataparticularSIFTmatchingthreshold.

Howwouldyouchangethethresholdto:(i) Increasetruepositives?(ii) Decreasefalsepositives?(iii) Decreasefalsenegatives?

Recap

Basicidea:• Take16x16squarewindowarounddetectedfeature• Computeedgeorientation (angleofthegradient- 90°)foreachpixel• Throwoutweakedges(threshold gradientmagnitude)• Createhistogramofsurviving edgeorientations

ScaleInvariantFeatureTransform

Adaptedfromslide byDavidLowe

0 2p

angle histogram

Modelingprojection

• Thecoordinatesystem– Wewillusethepinholemodelasanapproximation– Puttheopticalcenter(CenterOfProjection)attheorigin– Puttheimageplane(ProjectionPlane)infrontoftheCOP– Thecameralooksdownthenegativezaxis.

Modelingprojection

• Projectionequations– ComputeintersectionwithPPofrayfrom(x,y,z)toCOP– Derivedusingsimilartriangles

– Wegettheprojectionbythrowingoutthelastcoordinate:

Homogeneous coordinatesIs this a linear transformation?

Trick: add one more coordinate:

homogeneous image coordinates

homogeneous scene coordinates

Converting from homogeneous coordinates

• No, since division by z is nonlinear

Modelingprojection

• Amatrixmultiplicationusinghomogeneouscoordinates

Perspectiveprojection

(intrinsics) (convertsfrom3Draystopointsonimageplane)

Projection

dd

Fromimageplanetopixelcoordinates

(intrinsics)Projection

(0,0)

(-cx,-cy)

dd

d

(convertsfrom3Draystopointsonimageplane)

Fromimageplanetopixelcoordinates

(intrinsics)

Ingeneral,

:aspectratio(1unlesspixelsarenotsquare)

:skew (0unlesspixelsareshapedlikerhombi/parallelograms)

:principalpoint ((0,0) unlessopticalaxisdoesn’t intersectprojectionplaneatorigin)

(upper triangularmatrix)

(convertsfrom3Draysincameracoordinatesystemtopixelcoordinates)

Projection

dd

dd

Cameraparameters

• Howcanwemathematicallydescribeacamera?

• Weneedtodescribeitsinternalparameters• Weneedtodescribeitsposeintheworld

ATaleofTwoCoordinateSystems

“TheWorld”

Camera

x

y

z

v

w

u

o

COP

Twoimportantcoordinatesystems:1.Worldcoordinatesystem2.Cameracoordinatesystem

Cameraparameters• Toprojectapoint(x,y,z)inworld coordinatesintoacamera

• Firsttransform(x,y,z)intocamera coordinates• Needtoknow

– Cameraposition(inworldcoordinates)– Cameraorientation(inworldcoordinates)

• Thenprojectintotheimageplane– Needtoknowcameraintrinsics

• Thesecanallbedescribedwithmatrices.

Extrinsics• Howdowegetthecamerato“canonicalform”?

– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)

0

Step1:Translateby-c

Extrinsics• Howdowegetthecamerato“canonicalform”?

– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)

0

Step1:Translateby-c

Howdowerepresenttranslationasamatrixmultiplication?

Extrinsics• Howdowegetthecamerato“canonicalform”?

– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)

0

Step1:Translateby-cStep2:RotatebyR

3x3rotationmatrix

Extrinsics• Howdowegetthecamerato“canonicalform”?

– (Centerofprojectionattheorigin,x-axispointsright,y-axispointsup,z-axispointsbackwards)

0

Step1:Translateby-cStep2:RotatebyR

Projectionmatrix

translationrotationprojectionintrinsics

Denotethisbyt.

Projectionmatrix

0

=

(inhomogeneous imagecoordinates)

StructurefromMotion(SFM)VisualSLAM

FeaturedetectionSeveralimagesobserveascenefromdifferentviewpoints

FeaturedetectionDetectfeaturesusing,forexample,SIFT[Lowe,IJCV2004]

FeaturematchingMatchfeaturesbetweeneachpairofimages

Structurefrommotion

Camera1

Camera2

Camera3R1,t1

R2,t2R3,t3

X1

X4X3

X2

X5X6

X7

minimizeg(R,T,X)

p1,1p1,2

p1,3

non-linearleastsquares

Bundleadjustment• Minimizesumofsquaredreprojection errors:

– Optimizedwithnon-linearleastsquares– Levenberg-Marquardtisapopularchoice

• Practicalchallenges?– Initialization– Outliers

predictedimagelocation

observedimagelocationindicatorvariable:

whetherpoint i visibleinimagej

Featurematching

Robustness• Let’sconsiderasimplerexample…linefitting

• Howcanwefixthis?Problem:Fitalinetothesedatapoints Leastsquaresfit

Idea

• Givenahypothesizedline• Countthenumberofpointsthat“agree” withtheline– “Agree” =withinasmalldistanceoftheline– Thesearetheinlierstothatline

• Forallpossiblelines,selecttheonewiththelargestnumberofinliers

Countinginliers

Countinginliers

Inliers:3

Countinginliers

Inliers:20

Howdowefindthebestline?

• Unlikeleast-squares,nosimpleclosed-formsolution

• Hypothesize-and-test– Tryoutmanylines,keepthebestone– RANSAC:RandomSampleConsensus

• Numberofsamplesdependson– Outlierratio– Probabilityofcorrectanswer– Modelsize

RANSAC

• Generalversion:1. Randomlychooses samples

• Typicallys =minimumsamplesizetofitamodel

2. Fitamodel(say,line)tothosesamples3. Countthenumberofinliersthatapproximatelyfit

themodel4. RepeatN times5. Choosethemodelwiththelargestsetofinliers

Two-ViewReconstruction

Image1

Image2

R1,t1

R2,t2

X

FundamentalMatrix

x1x2

xT1Fx2 = 0

x1↔ x2

RANSACtoEstimateFundamentalMatrix

• Formanytimes– Pick8points– Computeasolutionforusingthese8points– Countnumberofinlierswithcloseto0

• Picktheonewiththelargestnumberofinliers

FxT1Fx2