Image Motion

COMPSCI 527 — Computer Vision

COMPSCI 527 — Computer Vision Image Motion 1 / 12

Outline

1 Image Motion

2 Occlusion, Correspondence, Motion Boundaries

3 Constancy of Appearance

4 Motion Field and Optical Flow

5 The Aperture Problem

6 Estimating the Motion Field

COMPSCI 527 — Computer Vision Image Motion 2 / 12

Image Motion

Sensor Irradiance→ Pixel Values

sensor

lensaperture

t

TTe

PsP

yx

• Irradiance is the patterns of colors on the image sensor,compressed as an (r ,g,b) triple e(x, t)• A pixel value is a noisy and quantized version of the integral

of irradiance over a volume of size Ps × Ps × Te:f(i,n) = Q

(´ nT +Te/2nT−Te/2

[˜ iP+Ps/2iP−Ps/2 e(x, t) dx

]dt + ν(i,n)

)COMPSCI 527 — Computer Vision Image Motion 3 / 12

Image Motion

Motion Field and Displacement• We are interested in real-valued positions x and times t• We have to infer those from integer-valued positions i and

times n• Motion field at x and at time t is the instantaneous velocity

of the image point that is visible at x at time t• The true image velocity of a world point• Displacement at x between times s and t is the difference

between the position of a point at time t and the position ofthe same point at time s < t• Displacement is the integral of image velocity between two

times• We typically approximate image velocity with displacement

between consecutive image frames

COMPSCI 527 — Computer Vision Image Motion 4 / 12

Occlusion, Correspondence, Motion Boundaries

The Displacement Field is not a 1− 1 Map

• Point visible at x at time sthat becomes hidden at time t(with s < t) forms an occlusion• When s > t , this is called a disocclusion• If points x at time s and y at time t do not form an occlusion

and are projections of the same point in the world, theycorrespond to each other• The displacement field is generally not integer-valued, so

we cannot compute a 1− 1 map between image pixels evenif no occlusions or disocclusions exist• A displacement field is typically given as a map Z2 → R2,

undefined at occlusions

COMPSCI 527 — Computer Vision Image Motion 5 / 12

Constancy of Appearance

Constancy of Appearance

• What is assumed to remain constant across images?• Motion estimation is impossible without such an assumption• Most generic assumption: The appearance of a point does

not change with time or viewpoint• If two image points in two images correspond, they look the

same• If x at time s and y at time t correspond, then

e(x, s) = e(y, t) (finite-displacement formulation)• Equivalently, de(x(t),t)

dt = 0 (differential formulation)• This is the key constraint for motion estimation

COMPSCI 527 — Computer Vision Image Motion 6 / 12

Motion Field and Optical Flow

Motion Field and Optical Flow• Extreme violations of constancy of appearance:

B. K. P. Horn, Robot Vision, MIT Press, 1986

• Ill-defined distinction:• Motion field ≈ true motion• Optical flow ≈ locally observed motion

COMPSCI 527 — Computer Vision Image Motion 7 / 12

Motion Field and Optical Flow

The Optical Flow Constraint Equation

• The appearance of a point does not change with time orviewpoint: de(x(t),t)

dt = 0• Total derivative, not partial:

de(x(t), t)dt

def= lim∆t→0

e(x(t+∆t), t+∆t)−e(x(t), t)∆t

• Use chain rule on de(x(t),t)dt = 0 to obtain the

Optical Flow Constraint Equation (OFCE)

∂e∂xT

dxdt

+∂e∂t

= 0

• v def= dx

dt is the unknown motion field• This is the key constraint for motion estimation

COMPSCI 527 — Computer Vision Image Motion 8 / 12

The Aperture Problem

The Aperture Problem• Issues arise even when the appearance is constant

OFCE:∂e∂xT v +

∂e∂t

= 0

• Three equations in two unknowns• However, changes in irradiance are often caused by

shading or shadows, which affects r ,g,b similarly

• The Jacobian ∂e∂xT

def=

∂r∂x1

∂r∂x2

∂g∂x1

∂g∂x2

∂b∂x1

∂b∂x2

has often rank close to 1

and the components of ∂e∂t are close to each other

• This degeneracy is called the aperture problemCOMPSCI 527 — Computer Vision Image Motion 9 / 12

The Aperture Problem

The Aperture Problem for Black-and-White Video

• The aperture problem is extreme for black-and-whiteimages, for which e ∈ R:

∂e∂xT v +

∂e∂t

= 0

(OFCE is one scalar equation in the two unknowns in v)• We cannot recover motion based on local measurements

alone• Only recover the normal component along the gradient∇e(x) = ∂e

∂xT (if the gradient is nonzero):

v(x) def= ‖∇e(x)‖−1 [∇e(x)]T v(x)

• In practice, this is very often the case also with color video

COMPSCI 527 — Computer Vision Image Motion 10 / 12

Estimating the Motion Field

Smoothness and Motion Boundaries

• The assumption of constancy of appearance yields aboutone equation in two unknowns at every point in the image• To solve for v, we need further assumptions• The motion field v : R2 → R2 is usually modeled as

piecewise smooth (in space)• OFCE is solved in the LSE sense, and an additional

regularization term is added to penalize deviations fromsmoothness• Smoothness holds almost everywhere, but not everywhere• Motion discontinuities are smooth image curves called

motion boundaries

COMPSCI 527 — Computer Vision Image Motion 11 / 12

Estimating the Motion Field

Estimating the Motion Field• Because of the aperture problem, we can only estimate

several displacement vectors d or motion field vectors vsimultaneously, not each individually• Local methods

• The image displacement d in a small window around a pixelx is assumed to be constant (extreme local smoothness)

• Write one constancy of appearance equation for every pixelin the window

• Solve for the one displacement that satisfies all theseequations as much as possible (in the LSE sense)

• Global methods• A data term measures deviations from constancy of

appearance at every pixel in the image• A smoothness term measures deviations of the motion field

v(x) from smoothness• Minimize a linear combination of the two types of terms

COMPSCI 527 — Computer Vision Image Motion 12 / 12