Motion Estimation IWhat affects the induced image motion?

• Camera motion• Object motion• Scene structure

Example Flow Fields

• This lesson – estimation of general flow-fields• Next lesson – constrained by global transformations

The Aperture Problem

So how much information is there locally…?

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Not enough info in local regions

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Not enough info in local regions

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

The Aperture Problem

Copyright, 1996 © Dale Carnegie & Associates, Inc.

Information is propagated from regions with high certainty (e.g., corners) to regions with low certainty.

Such info propagation can cause optical illusions…

Illusory corners

1. Gradient based methods (Horn &Schunk, Lucase & Kanade)

2. Region based methods (Correlation, SSD, Normalized correlation)

• Direct (intensity-based) Methods

• Feature-based Methods

Image J (taken at time t)

yx,

Brightness Constancy Assumption

yxJ , vyuxI ,

Image I(taken at time t+1)

vyux ,

Brightness Constancy Equation:

The Brightness Constancy Constraint

),(),( ),(),( yxyx vyuxIyxJ

),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):

),(),(),(),(),(),(0 yxJyxIyxvyxIyxuyxI yx

),(),(),(),(),( yxIyxvyxIyxuyxI tyx

),(),(

),(),( yxI

yxv

yxuyxI t

T

* One equation, 2 unknowns

* A line constraint in (u,v) space.

* Can recover Normal Flow.

Observations:

Need additional constraints…

Horn and Schunk (1981)

Add global smoothness term

2222

),(yxyx

yx

vvuu

Smoothness error

2

),(tyx

yx

IvIuIE

Error in brightness constancy equation

sc EE Minimize:

Solve by using calculus of variations

Horn and Schunk (1981)

Problems…

* Smoothness assumption wrong at motion/depth discontinuities over-smoothing of the flow field.

* How is Lambda determined…?

Lucas-Kanade (1984)

Windowyx

tyx IvyxIuyxIvuE),(

2),(),(),(

Assume a single displacement (u,v) for all pixels within a small window

Minimize E(u,v):

Geometrically -- Intersection of multiple line constraints

Algebraically --

Lucas-Kanade (1984)

Windowyx

tyx IvyxIuyxIvuE),(

2),(),(),(

Differentiating w.r.t u and v and equating to 0:

ty

tx

yyx

yxx

II

II

v

u

III

III2

2

tT IIUII

Solve for (u,v)[ Repeat this process for each and every pixel in the image ]

Minimize E(u,v):

Problems…

* Still smoothes motion discontinuities (but unlike Horn & Schunk, does not propagate error across the entire image)

* Singularities (partially solved by coarse-to-fine)

Lucas-Kanade (1984)

Singularites

2

2

yyx

yxx

III

III

Where in the image will this matrix be invertible and where not…?

Homework

Linearization approximation iterate & warp

xx0

Initial guess:

Estimate:

estimate update

xx0

estimate update

Initial guess:

Estimate:

Linearization approximation iterate & warp

xx0

Initial guess:

Estimate:

Initial guess:

Estimate:

estimate update

Linearization approximation iterate & warp

xx0

Linearization approximation iterate & warp

Revisiting the small motion assumption

Is this motion small enough?Probably not—it’s much larger than one pixel (2nd order terms

dominate)How might we solve this problem?

0 tyx IvIuI ==> small u and v ...

u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

image Iimage J

u

iterate refine

u

uΔ

+

Pyramid of image J Pyramid of image I

image Iimage J

Coarse-to-Fine EstimationAdvantages: (i) Larger displacements. (ii) Speedup. (iii) Information from multiple window sizes.

Optical Flow Results

Optical Flow Results

1. Gradient based methods (Horn &Schunk, Lucase & Kanade, …)

2. Region based methods (SSD, Normalized correlation, etc.)

Copyright, 1996 © Dale Carnegie & Associates, Inc.

But… (despite coarse-to-fine estimation)

• rely on B.C.

• cannot handle very large motions (no more than 10%-15% of image width/height)

• small object moving fast…?

Region-Based Methods

* Define a small area around a pixel as the region.

* Match the region against each pixel within a search area in next image.

* Use a match measure (e.g., sum of-squares difference, normalized correlation, etc.)

* Choose the maximum (or minimum) as the match.

SSD Surface – Textured area

SSD Surface -- Edge

SSD – homogeneous area

[Anandan’89 - Use coarse-to-fine SSD of local windows to find matches. - Propagate information using directional confidence measures extracted from each local SSD surface]

B.C. + Additional constraints:

Copyright, 1996 © Dale Carnegie & Associates, Inc.

• Increase aperture: [e.g., Lucas & Kanade]

Local singularities at degenerate image regions.

• Increase analysis window to large image regions => Global model constraints: (NEXT LESSON)

Numerically stable, but requires prior model selection:

• Planar (2D) world model

[e.g., Bergen-et-al:92, Irani-et-al:92+94, Black-et-al]

• 3D world model[e.g., Hanna-et-al:91+93, Stein & Shashua:97, Irani-et-al:1999]

• Spatial smoothness: [e.g., Horn & Schunk:81, Anandan:89]

Violated at depth/motion discontinuities