Optical Flow

Optical Flow

Digital PhotographyCSE558, Spring 2003

Richard Szeliski

(notes cribbed from P. Anandan)

3/9/2003 Optical Flow 2

Classes of Techniques

Feature-based methods• Extract salient visual features (corners, textured areas) and track

them over multiple frames• Analyze the global pattern of motion vectors of these features• Sparse motion fields, but possibly robust tracking• Suitable especially when image motion is large (10-s of pixels)

Direct-methods• Directly recover image motion from spatio-temporal image

brightness variations• Global motion parameters directly recovered without an

intermediate feature motion calculation• Dense motion fields, but more sensitive to appearance variations• Suitable for video and when image motion is small (< 10 pixels)


Brightness Constancy Equation:

The Brightness Constraint

),(),( ),(),( yxyx vyuxIyxJ

Or, better still, Minimize :2)),(),((),( vyuxIyxJvuE

),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):


Local Patch Analysis


Patch Translation [Lucas-Kanade]

yx

tyx IvyxIuyxIvuE,

2),(),(),(

Minimizing

Assume a single velocity for all pixels within an image patch

ty

tx

yyx

yxx

II

II

v

u

III

III2

2

tT IIUII

LHS: sum of the 2x2 outer product tensor of the gradient vector


The Aperture Problem

TIIMLet

• Algorithm: At each pixel compute by solving

• M is singular if all gradient vectors point in the same direction• e.g., along an edge• of course, trivially singular if the summation is over a single pixel or there is no texture• i.e., only normal flow is available (aperture problem)

• Corners and textured areas are OK

and

ty

tx

II

IIb

U bMU


Aperture Problem and Normal Flow


Local Patch Analysis


Iterative Refinement

Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation

Warp one image toward the other using the estimated flow field(easier said than done)

Refine estimate by repeating the process


Limits of the gradient method

Fails when intensity structure in window is poor

Fails when the displacement is large (typical operating range is motion of 1 pixel)

Linearization of brightness is suitable only for small displacements

Also, brightness is not strictly constant in imagesactually less problematic than it appears, since we can

pre-filter images to make them look similar


Pyramids

Pyramids were introduced as a multi-resolution image computation paradigm in the early 80s.

The most popular pyramid is the Burt pyramid, which foreshadows wavelets

Two kinds of pyramids:

Low pass or “Gaussian pyramid”

Band-pass or “Laplacian pyramid”


Gaussian Pyramid

Convolve image with a small Gaussian kernel• Typically 5x5

Subsample (decimate by 2) to get lower resolution imageRepeat for more levelsA sequence of low-pass filtered images

)2,2(),(

),(),(),(

),(),(

^

1

^

0

yxGyxG

yxgyxGyxG

yxIyxG

ll

ll


Laplacian pyramid

Laplacian as Difference of Gaussian

Band-pass filtered images

Highlights edges at different spatial scales

For matching, this is less sensitive to image illumination changes

But more noisy than using Gaussians

),(),(),(^

yxGyxGyxL lll



image Iimage J

aJwwarp refine

a

aΔ+

Pyramid of image J Pyramid of image I

image Iimage J

Coarse-to-Fine Estimation

u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels


J Jw Iwarp refine

ina

a

+

J Jw Iwarp refine

a

a+

J

pyramid construction

J Jw Iwarp refine

a+

I

pyramid construction

outa

Coarse-to-Fine Estimation


Global Motion Models

2D Models:AffineQuadraticPlanar projective transform (Homography)

3D Models:Instantaneous camera motion models Homography+epipolePlane+Parallax


0)()( 654321 tyx IyaxaaIyaxaaI

Example: Affine Motion

Substituting into the B.C. Equation:yaxaayxv

yaxaayxu

654

321

),(

),(

Each pixel provides 1 linear constraint in 6 global unknowns

0 tyx IvIuI

2 tyx IyaxaaIyaxaaIaErr )()()( 654321

Least Square Minimization (over all pixels):


Quadratic – instantaneous approximation to planar motion

Other 2D Motion Models

287654

82

7321

yqxyqyqxqqv

xyqxqyqxqqu

yyvxxu

yhxhh

yhxhhy

yhxhh

yhxhhx

','

and

'

'

987

654

987

321

Projective – exact planar motion


Correlation and SSD

For larger displacements, do template matching• Define a small area around a pixel as the template• Match the template against each pixel within a

search area in next image.• Use a match measure such as correlation,

normalized correlation, or sum-of-squares difference

• Choose the maximum (or minimum) as the match• Sub-pixel interpolation also possible


SSD Surface – Textured area


SSD Surface -- Edge


SSD – homogeneous area


Discrete Search vs. Gradient Based Estimation

Consider image I translated by

21

,00

2

,1

)),(),(),((

)),(),((),(

yxvvyuuxIyxI

vyuxIyxIvuE

yx

yx

00 ,vu

),(),(),(

),(),(

1001

0

yxyxIvyuxI

yxIyxI

The discrete search method simply searches for the best estimate.The gradient method linearizes the intensity function and solves for the estimate


Correlation Window Size

Small windows lead to more false matchesLarge windows are better this way, but…

• Neighboring flow vectors will be more correlated (since the template windows have more in common)

• Flow resolution also lower (same reason)• More expensive to compute

Another way to look at this:Small windows are good for local search but more precise

and less smoothLarge windows good for global search but less precise and

more smooth method


Robust Estimation

Standard Least Squares Estimation allows too much influence for outlying points

)()

)()(

)()(

2

mxx

x

mxx

xmE

i

ii

ii

( Influence


Robust Estimation

tsysxssd IvIuIvuE ),( Robust gradient constraint

),(),(),( ssssd vyuxJyxIvuE Robust SSD


References

J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May 1992.

M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, 1996.

H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug. 1996.

Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997.

Documents

Optical Flow