11

Motion Analysis using Optical flow

CIS601

Longin Jan Latecki

Fall 2003CIS Dept of Temple University

22

Usual input of a motion analysis Usual input of a motion analysis system is a temporal image system is a temporal image sequencesequence

Motion analysis is often connected Motion analysis is often connected with real-time analysiswith real-time analysis

Part 1: Motion Analysis

33

Three main groups of motion analysis problem

• Motion detection: - register any detected motion - single static camera

• Moving object detection and location:

- moving object detection only : motion based segmentation methods

- detection of a moving object, detection of the trajectory of its motion, prediction of its future trajectory: image object matching techniques are often used to solve these tasks (direct matching of image data, matching of object features, specific representative object points (corner etc.),represent moving object as graphs and matching these graphs); another useful method is optical flow

• Derivation of 3D object propertiesDerivation of 3D object properties:: from a set of 2D from a set of 2D projections of acquired at different time instants of object motionprojections of acquired at different time instants of object motion

44

Reflects the image changes due to motion during a time interval Reflects the image changes due to motion during a time interval dt, dt, which is short enough to guarantee small inter-frame motion which is short enough to guarantee small inter-frame motion changeschanges

The immediate objective of optical flow is to determine a The immediate objective of optical flow is to determine a velocity velocity field:field:A 2D representation of a (generally) 3D motion is called a motion A 2D representation of a (generally) 3D motion is called a motion field (velocity field). Whereas each point is assigned a velocity field (velocity field). Whereas each point is assigned a velocity vector corresponding the motion direction, velocity and distance vector corresponding the motion direction, velocity and distance from an observer at an appropriate image locationfrom an observer at an appropriate image location

Based on 2 assumptions:Based on 2 assumptions:

- The observed brightness of any object point is constant over - The observed brightness of any object point is constant over timetime

- Nearby points in the image plane move in a similar - Nearby points in the image plane move in a similar manner(velocity smoothness constraint)manner(velocity smoothness constraint)

Optical flow

55

Optical flow

Motion field is a projection of ‘real’ motion vectors of 3D objects to the image plane.We can only compute an optical flow from time-varying brightnessin the image sequence, which is an approximation of the motion field.

66The program can be obtained here.

77

• Let us suppose we have a continuous image, the image intensity is given by I(x,y,t), where the intensity is now a function of time t, as well as of x and y.

• If this point (x,y) moves to a point (x+dx,y+dy) at time t+dt, the following equation holds, which we transform by Taylor expansion:

Computation Rationale for Optical Flow

0),,(),,(),,(

0

),,(),,(),,(),,(),,(

),,(),,(

dttyxIdytyxIdxtyxI

error

errordttyxIdytyxIdxtyxItyxItyxI

dttdyydxxItyxI

tyx

tyx

88

tT

tT

Idt

dy

dt

dxI

Idt

dy

dt

dxI

t

I

dt

dy

y

I

dt

dx

x

Idt

ttytxdIdt

dI

],[)(

0],[)(

0

0)),(),((

0

Thus, the displacement vector [dx/dt, dy/dt] at pixel (x,y) at frame tis the one for which the spatial derivative of the image intensity Iat point (x,y) is equal to the minus of the temporal derivative of I at (x,y).

99

The aperture problemThe aperture problem

tT I

dt

dy

dt

dxI ],[)(

Only a component of the motion field in the direction of the spatial image gradientcan be computed using this equation.

This component is called the normal component, because the spatial image gradientis normal to the spatial direction along which the image intensity remains constant.See this link for a nice demo movie.

1010

t

I

dt

dy

dt

dx

y

I

x

IT

,,

To compute the displacement vector [dx/dt, dy/dt] at pixel (x,y) at frame twe need at least two equations, but so far we have only one.

We will consider two methods to obtain additional equations.The first method works for color videos, where we have red R, green G, and blue color intensity values:

t

B

dt

dy

dt

dx

y

B

x

B

t

G

dt

dy

dt

dx

y

G

x

G

t

R

dt

dy

dt

dx

y

R

x

R

T

T

T

,,

,,

,,Since this is an

overdetermined system of equations, we have two ways to solve it:

1. Use least squares. 2. Disregard, say blue

color, and use Gauss elimination. See an example Matlab program.

1111

The second method ( from E. Trucco and A. Verri: Introductory Techniques for 3-D Computer Vision)

Based on 2 assumptions:Based on 2 assumptions:

1.1. The observed brightness of any object point is constant over The observed brightness of any object point is constant over timetime

2.2. Nearby points in the image plane move in a similar manner Nearby points in the image plane move in a similar manner (velocity smoothness constraint) implies a constant motion (velocity smoothness constraint) implies a constant motion vector field within vector field within a small patch of the image planea small patch of the image plane

1212

Computation Method

1313

Computation Method

Homework (optional):Implement this algorithm and test on a synthetic and real image sequences.

1414

Optical flow in motion analysis

Motion, as it appears in dynamic images, is usually some Motion, as it appears in dynamic images, is usually some combination of 4 basic elements:combination of 4 basic elements:

(a)Translation at constant distance from the observer.(a)Translation at constant distance from the observer.

---parallel motion vectors---parallel motion vectors

(b)Translation in depth relative to the observer.(b)Translation in depth relative to the observer.

---Vectors having common focus of expansion.---Vectors having common focus of expansion.

(c) Rotation at constant distance from view axis.(c) Rotation at constant distance from view axis.

------concentric motion vectors.concentric motion vectors.

(d) Rotation of planar object perpendicular to the view axis.(d) Rotation of planar object perpendicular to the view axis.

---- ---- vectors starting from straight line segments.vectors starting from straight line segments.