3D Rigid/Nonrigid Registration

1) Known features, correspondences, transformation model – feature based

2) Specific motion type, unknown correspondences – feature based

3) Known transformation model, unknown correspondences – region based

4) Specific motion model – feature based

5) Unknown motion model, unknown correspondences – region based

Visual Motion

Jim Rehg

(G.Tech)

Motion (Displacement) of Environment

Imageplane dt

tdt

)()(

rv

SceneFlowMotion

Field

)(tr

)()( tt vPw

Visual motion results from the displacement of the scene with respect to a fixed camera (or vice-versa).Motion field is the 2-D velocity field that results from a projection of the 3-D scene velocities

Examples of Visual Motion

Examples of Visual Motion

Examples of Visual Motion

Applications of Motion Analysis

Visual tracking

Structure recovery

Robot (vehicle) navigation

Applications of Motion Analysis

Visual tracking

Structure recovery

Robot (vehicle) navigation

Motion Segmentation

Where are the independently moving objects (and how many are there)?

Optical Flow

2-D velocity field describing the apparent motion in an image sequence

A vector at each pixel indicates its motion (between a pair of frames).

Ground truthHorn and Schunk

Optical Flow and Motion Field

In general the optical flow is an approximation to the motion field.

When the scene can be segmented into rigidly moving objects (for example) the relationship between the two can be made precise.

We can always think of the optical flow as summarizing the temporal change in an image sequence.

Computing Optical Flow

Courtesy of Michael Black

Cost Function for Optical Flow

Ryx

SSD tyxItvyuxIvuE,

2)],,()1,,([),(

Courtesy of Michael Black

Lucas-Kanade Method

Brute-force minimization of SSD error can be inefficient and inaccurateMany redundant window evaluations

Answer is limited to discrete u, v pairs

Lucas-Kanade Method

Problems with brute-force minimization of SSD errorMany redundant window evaluations

Answer is limited to discrete u, v pairs

Related to Horn-Schunk optical flow equations

Several key innovationsEarly, successful use of patch-based model in low-level vision. Today

these models are used everywhere.

Formulation of vision problem as non-linear least squares optimization, a trend which continues to this day.

Optical Flow Estimation

Optical Flow Estimation

Optical Flow Constraint

I_t is one-to-one in the first

Iteration and changes as u,v changes

Optimization

Optimization

Optimization

Quality of Image Patch

Eigenvalues of the matrix contain information about local image structureBoth eigenvalues (close to) zero: Uniform area

One eigenvalue (close to) zero: Edge

No eigenvalues (close to) zero: Corner

Contributions of Lucas-Kanade

Basic idea of patch or template is very old (goes back at least to Widrow)

But in practice patch models have worked much better than the alternatives:Point-wise differential equations with smoothnessEdge-based descriptions

Patches provide a simple compact enforcement of spatial continuity and support (robust) least-squares estimators.

Apparent Motion

Apparent motion of objects on the image planeCaution required!!

Consider a perfectly uniform sphere that is rotating but no change in the light directionOptic flow is zero

Perfectly uniform sphere that is stationary but the light is changingOptic flow exists

Hope – apparent motion is very close to the actual motion

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Aperture Problem

1) The aperture problem arises due to

uniformly colored surfaces in the scene. In the

absence of strong lighting effects, a uniform

surface in the scene appears nearly uniform in the

projection. It is then impossible to determine

correspondences within these regions.

2) We are able to only measure the component

Of the optic flow that is in the direction of the

Intensity gradient. (Unable to measure component

In the tangential direction, edge).

Aperture Problem

We can measure Terms that can be measuredTerms to be computedNumber of equations - 1

The component of the motion field that is orthogonal to the spatial image gradient is not constrained by the image brightness constancy assumption

IntuitivelyThe component of the flow in the gradient direction is determinedThe component of the flow parallel to an edge is unknown

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

tEyExE ,,

dtdydtdx ,

Different physical motion but same measurable motion within a fixed window