View
2
Download
0
Category
Preview:
Citation preview
Outline
1 Image Motion
2 Occlusion, Correspondence, Motion Boundaries
3 Constancy of Appearance
4 Motion Field and Optical Flow
5 The Aperture Problem
6 Estimating the Motion Field
COMPSCI 527 — Computer Vision Image Motion 2 / 12
Image Motion
Sensor Irradiance→ Pixel Values
sensor
lensaperture
t
TTe
PsP
yx
• Irradiance is the patterns of colors on the image sensor,compressed as an (r ,g,b) triple e(x, t)• A pixel value is a noisy and quantized version of the integral
of irradiance over a volume of size Ps × Ps × Te:f(i,n) = Q
(´ nT +Te/2nT−Te/2
[˜ iP+Ps/2iP−Ps/2 e(x, t) dx
]dt + ν(i,n)
)COMPSCI 527 — Computer Vision Image Motion 3 / 12
Image Motion
Motion Field and Displacement• We are interested in real-valued positions x and times t• We have to infer those from integer-valued positions i and
times n• Motion field at x and at time t is the instantaneous velocity
of the image point that is visible at x at time t• The true image velocity of a world point• Displacement at x between times s and t is the difference
between the position of a point at time t and the position ofthe same point at time s < t• Displacement is the integral of image velocity between two
times• We typically approximate image velocity with displacement
between consecutive image frames
COMPSCI 527 — Computer Vision Image Motion 4 / 12
Occlusion, Correspondence, Motion Boundaries
The Displacement Field is not a 1− 1 Map
• Point visible at x at time sthat becomes hidden at time t(with s < t) forms an occlusion• When s > t , this is called a disocclusion• If points x at time s and y at time t do not form an occlusion
and are projections of the same point in the world, theycorrespond to each other• The displacement field is generally not integer-valued, so
we cannot compute a 1− 1 map between image pixels evenif no occlusions or disocclusions exist• A displacement field is typically given as a map Z2 → R2,
undefined at occlusions
COMPSCI 527 — Computer Vision Image Motion 5 / 12
Constancy of Appearance
Constancy of Appearance
• What is assumed to remain constant across images?• Motion estimation is impossible without such an assumption• Most generic assumption: The appearance of a point does
not change with time or viewpoint• If two image points in two images correspond, they look the
same• If x at time s and y at time t correspond, then
e(x, s) = e(y, t) (finite-displacement formulation)• Equivalently, de(x(t),t)
dt = 0 (differential formulation)• This is the key constraint for motion estimation
COMPSCI 527 — Computer Vision Image Motion 6 / 12
Motion Field and Optical Flow
Motion Field and Optical Flow• Extreme violations of constancy of appearance:
B. K. P. Horn, Robot Vision, MIT Press, 1986
• Ill-defined distinction:• Motion field ≈ true motion• Optical flow ≈ locally observed motion
COMPSCI 527 — Computer Vision Image Motion 7 / 12
Motion Field and Optical Flow
The Optical Flow Constraint Equation
• The appearance of a point does not change with time orviewpoint: de(x(t),t)
dt = 0• Total derivative, not partial:
de(x(t), t)dt
def= lim∆t→0
e(x(t+∆t), t+∆t)−e(x(t), t)∆t
• Use chain rule on de(x(t),t)dt = 0 to obtain the
Optical Flow Constraint Equation (OFCE)
∂e∂xT
dxdt
+∂e∂t
= 0
• v def= dx
dt is the unknown motion field• This is the key constraint for motion estimation
COMPSCI 527 — Computer Vision Image Motion 8 / 12
The Aperture Problem
The Aperture Problem• Issues arise even when the appearance is constant
OFCE:∂e∂xT v +
∂e∂t
= 0
• Three equations in two unknowns• However, changes in irradiance are often caused by
shading or shadows, which affects r ,g,b similarly
• The Jacobian ∂e∂xT
def=
∂r∂x1
∂r∂x2
∂g∂x1
∂g∂x2
∂b∂x1
∂b∂x2
has often rank close to 1
and the components of ∂e∂t are close to each other
• This degeneracy is called the aperture problemCOMPSCI 527 — Computer Vision Image Motion 9 / 12
The Aperture Problem
The Aperture Problem for Black-and-White Video
• The aperture problem is extreme for black-and-whiteimages, for which e ∈ R:
∂e∂xT v +
∂e∂t
= 0
(OFCE is one scalar equation in the two unknowns in v)• We cannot recover motion based on local measurements
alone• Only recover the normal component along the gradient∇e(x) = ∂e
∂xT (if the gradient is nonzero):
v(x) def= ‖∇e(x)‖−1 [∇e(x)]T v(x)
• In practice, this is very often the case also with color video
COMPSCI 527 — Computer Vision Image Motion 10 / 12
Estimating the Motion Field
Smoothness and Motion Boundaries
• The assumption of constancy of appearance yields aboutone equation in two unknowns at every point in the image• To solve for v, we need further assumptions• The motion field v : R2 → R2 is usually modeled as
piecewise smooth (in space)• OFCE is solved in the LSE sense, and an additional
regularization term is added to penalize deviations fromsmoothness• Smoothness holds almost everywhere, but not everywhere• Motion discontinuities are smooth image curves called
motion boundaries
COMPSCI 527 — Computer Vision Image Motion 11 / 12
Estimating the Motion Field
Estimating the Motion Field• Because of the aperture problem, we can only estimate
several displacement vectors d or motion field vectors vsimultaneously, not each individually• Local methods
• The image displacement d in a small window around a pixelx is assumed to be constant (extreme local smoothness)
• Write one constancy of appearance equation for every pixelin the window
• Solve for the one displacement that satisfies all theseequations as much as possible (in the LSE sense)
• Global methods• A data term measures deviations from constancy of
appearance at every pixel in the image• A smoothness term measures deviations of the motion field
v(x) from smoothness• Minimize a linear combination of the two types of terms
COMPSCI 527 — Computer Vision Image Motion 12 / 12
Recommended