Motion and Optical FlowMotion and Optical Flow

Moving to Multiple ImagesMoving to Multiple Images

• So far, we’ve looked at processing aSo far, we’ve looked at processing asingle imagesingle image

• Multiple imagesMultiple images– Multiple cameras at one time: stereoMultiple cameras at one time: stereo– Single camera at many times: videoSingle camera at many times: video– (Multiple cameras at multiple times)(Multiple cameras at multiple times)

Applications of Multiple Applications of Multiple ImagesImages

• 2D2D– Feature / object trackingFeature / object tracking– Segmentation based on motionSegmentation based on motion

• 3D3D– Shape extractionShape extraction– Motion captureMotion capture

Applications of Multiple Applications of Multiple ImagesImages

in Graphicsin Graphics• Stitching images into panoramasStitching images into panoramas

• Automatic image morphingAutomatic image morphing

• Reconstruction of 3D models for Reconstruction of 3D models for renderingrendering

• Capturing articulated motion for Capturing articulated motion for animation animation

Applications of Multiple Applications of Multiple ImagesImages

in Biological Systemsin Biological Systems• Shape inferenceShape inference

• Peripheral sensitivity to motionPeripheral sensitivity to motion

• Looming field – obstacle avoidanceLooming field – obstacle avoidance

• Very similar applications in roboticsVery similar applications in robotics

Looming FieldLooming Field

• Pure Pure translation: translation: motion looks motion looks like it like it originates at originates at a point – a point – focus of focus of expansionexpansion

Key ProblemKey Problem

• Main problem in most multiple-image Main problem in most multiple-image methods: methods: correspondencecorrespondence

CorrespondenceCorrespondence

• Small displacementsSmall displacements– Differential algorithmsDifferential algorithms– Based on gradients in space and timeBased on gradients in space and time– Dense correspondence estimatesDense correspondence estimates– Most common with videoMost common with video

• Large displacementsLarge displacements– Matching algorithmsMatching algorithms– Based on correlation or featuresBased on correlation or features– Sparse correspondence estimatesSparse correspondence estimates– Most common with multiple cameras / Most common with multiple cameras /

stereostereo

Result of CorrespondenceResult of Correspondence

• For points in image For points in image ii displacements displacements to corresponding locations in image to corresponding locations in image jj

• In stereo, usually called In stereo, usually called disparitydisparity

• In video, usually called In video, usually called motion fieldmotion field

Computing Motion FieldComputing Motion Field

• Basic idea: a small portion of the Basic idea: a small portion of the imageimage(“local neighborhood”) shifts position(“local neighborhood”) shifts position

• AssumptionsAssumptions– No / small changes in reflected lightNo / small changes in reflected light– No / small changes in scaleNo / small changes in scale– No occlusion or disocclusionNo occlusion or disocclusion– Neighborhood is correct size: Neighborhood is correct size: apertureaperture

problemproblem

Actual and Apparent MotionActual and Apparent Motion

• If these assumptions violated, can still If these assumptions violated, can still use the same methods – use the same methods – apparent apparent motionmotion

• Result of algorithm is Result of algorithm is optical flowoptical flow (vs. (vs. ideal motion field)ideal motion field)

• Most obvious effects:Most obvious effects:– Aperture problem: can only get motion Aperture problem: can only get motion

perpendicular to edgesperpendicular to edges– Errors near discontinuities (occlusions)Errors near discontinuities (occlusions)

Computing Optical Flow:Computing Optical Flow:PreliminariesPreliminaries

• Image sequence I(x,y,t)Image sequence I(x,y,t)

• Uniform discretization along x,y,t –Uniform discretization along x,y,t –“cube” of data“cube” of data

• Differential framework: compute Differential framework: compute partial derivatives along x,y,t by partial derivatives along x,y,t by convolving with derivative of convolving with derivative of GaussianGaussian

Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy

• Basic idea: a small portion of the Basic idea: a small portion of the imageimage(“local neighborhood”) shifts position(“local neighborhood”) shifts position

• Brightness constancy assumptionBrightness constancy assumption

0dt

dI0

dt

dI

Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy

• This does not say that the image This does not say that the image remainsremainsthe same brightness!the same brightness!

• vs. : total vs. partial derivativevs. : total vs. partial derivative

• Use chain ruleUse chain rule t

I

dt

dy

y

I

dt

dx

x

I

dt

ttytxdI

),(),(

t

I

dt

dy

y

I

dt

dx

x

I

dt

ttytxdI

),(),(

dt

dIdt

dI

t

I

t

I

Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy

• Given optical flow Given optical flow vv(x,y)(x,y)

0)(

0

0),(),(

T

tII

t

I

dt

dy

y

I

dt

dx

x

Idt

ttytxdI

v

0)(

0

0),(),(

T

tII

t

I

dt

dy

y

I

dt

dx

x

Idt

ttytxdI

v

Image brightness constancy equationImage brightness constancy equation

Computing Optical Flow:Computing Optical Flow:DiscretizationDiscretization

• Look at some neighborhood N:Look at some neighborhood N:

0

0),(),(

want

want

N),(

T

bAv

v jiIjiIji

t

0

0),(),(

want

want

N),(

T

bAv

v jiIjiIji

t

),(

),(

),(

),(

),(

),(

22

11

22

11

nnt

t

t

nn jiI

jiI

jiI

jiI

jiI

jiI

bA

),(

),(

),(

),(

),(

),(

22

11

22

11

nnt

t

t

nn jiI

jiI

jiI

jiI

jiI

jiI

bA

Computing Optical Flow:Computing Optical Flow:Least SquaresLeast Squares

• In general, overconstrained linear In general, overconstrained linear systemsystem

• Solve by least squaresSolve by least squares

bAAAv

bAvAA

bAv

T1T

TT

want

)(

)(

0

bAAAv

bAvAA

bAv

T1T

TT

want

)(

)(

0

Computing Optical Flow:Computing Optical Flow:StabilityStability

• Has a solution unless Has a solution unless CC = = AATTAA is is singularsingular

Ny

Nyx

Nyx

Nx

nn

nn

III

III

jiI

jiI

jiI

jiIjiIjiI

2

2

22

11

2211

T

),(

),(

),(

),(),(),(

C

C

AAC

Ny

Nyx

Nyx

Nx

nn

nn

III

III

jiI

jiI

jiI

jiIjiIjiI

2

2

22

11

2211

T

),(

),(

),(

),(),(),(

C

C

AAC

Computing Optical Flow:Computing Optical Flow:StabilityStability

• Where have we encountered Where have we encountered CC before? before?

• Corner detector!Corner detector!

• C C is singular if intensity is constant is singular if intensity is constant oror if if there’s an edgethere’s an edge

• Use eigenvalues of Use eigenvalues of CC::– to evaluate stability of optical flow to evaluate stability of optical flow

computationcomputation– to find good places to compute optical flowto find good places to compute optical flow

(finding good features to track)(finding good features to track)

Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements

• Assumption that optical flow is Assumption that optical flow is constant over neighborhood not constant over neighborhood not always goodalways good

• Decreasing size of neighborhood Decreasing size of neighborhood CC more likely to be singular more likely to be singular

• Alternative: weighted least-squaresAlternative: weighted least-squares– Points near center = higher weightPoints near center = higher weight– Still use larger neighborhoodStill use larger neighborhood

Computing Optical Flow:Computing Optical Flow:Weighted Least SquaresWeighted Least Squares

• Let Let WW be a matrix of weights be a matrix of weights

Wbb

WAA

Wbb

WAA

bWAAWAv

bAAAv2T12T

T1T

)(

)(

w bWAAWAv

bAAAv2T12T

T1T

)(

)(

w

Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements

• What if windows are still bigger?What if windows are still bigger?

• Adjust motion model: no longer Adjust motion model: no longer constant within a windowconstant within a window

• Popular choice: affine modelPopular choice: affine model

Computing Optical Flow:Computing Optical Flow:Affine Motion ModelAffine Motion Model

• Translational modelTranslational model

• Affine modelAffine model

y

x

v

v

y

x

y

x

1

1

2

2

y

x

v

v

y

x

y

x

1

1

2

2

y

x

v

v

y

x

dc

ba

y

x

1

1

2

2

y

x

v

v

y

x

dc

ba

y

x

1

1

2

2

Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements

• Larger motion: how to maintain Larger motion: how to maintain “differential” approximation?“differential” approximation?

• Solution: iterateSolution: iterate

• Even better: adjust window / smoothingEven better: adjust window / smoothing– Early iterations: use larger Gaussians to Early iterations: use larger Gaussians to

allow more motion allow more motion – Late iterations: use less blur to find exact Late iterations: use less blur to find exact

solution, lock on to high-frequency detailsolution, lock on to high-frequency detail

Computing Optical Flow:Computing Optical Flow:Lucas-KanadeLucas-Kanade

• Iterative algorithm:Iterative algorithm:1.1. Set Set = large (e.g. 3 pixels)= large (e.g. 3 pixels)

2.2. Set I’ Set I’ I I11

3.3. Set Set vv 0 0

4.4. Repeat while SSD(I’, IRepeat while SSD(I’, I22) > ) > 1.1. vv += Optical flow(I’ += Optical flow(I’ I I22))

2.2. I’ I’ Warp(I Warp(I11, , vv))

5.5. After After nn iterations, iterations,set set = small (e.g. 1.5 pixels) = small (e.g. 1.5 pixels)

Computing Optical Flow:Computing Optical Flow:Lucas-KanadeLucas-Kanade

• I’ always holds warped version of II’ always holds warped version of I11

– Best estimate of IBest estimate of I22

• Gradually reduce thresholdsGradually reduce thresholds

• Stop when difference between I’ and Stop when difference between I’ and II22 small small– Simplest difference metric = sum of Simplest difference metric = sum of

squared differences (SSD) between squared differences (SSD) between pixelspixels

Optical Flow ApplicationsOptical Flow Applications

[Feng & Perona][Feng & Perona]

Video FramesVideo Frames

Optical Flow ApplicationsOptical Flow Applications

Optical FlowOptical Flow Depth ReconstructionDepth Reconstruction

[Feng & Perona][Feng & Perona]

Optical Flow ApplicationsOptical Flow Applications

Obstacle Detection: Unbalanced Optical FlowObstacle Detection: Unbalanced Optical Flow

TemizerTemizer

Optical Flow ApplicationsOptical Flow Applications

• Collision Collision avoidance: keep avoidance: keep optical flow optical flow balanced between balanced between sides of imagesides of image

TemizerTemizer