Applications of Image Motion Estimation I

Mosaicing

Harpreet S. Sawhneyhsawhney@sarnoff.com

Princeton UniversityCOS 429 Lecture

Feb. 26, 2004

Visual Motion Estimation : Recapitulation

Plan

• Explain optical flow equations

• Show inclusion of multiple constraints for solution

• Another way to solve is to use global parametric models

Brightness Constancy Assumption

)p(I1 )p(I2

p)p(up

Model image transformation as :

)p(I))p(up(I)p(I 112 ′== - BrightnessConstancy

ReferenceCoordinate

CorrespondingCoordinate

OpticalFlow

How do we solve for the flow ?

)p(I))p(up(I)p(I 112 ′== -

Use Taylor Series Expansion

)2(O)p(uI)p(I)p(I T112 +∇= -

Image Gradient

Convert constraint into an objective function

∑∈

∇Rp

2T1SSD ))p(I)p(uI()u(E δ+=

)p(I)p(I 12 -

Optical Flow Constraint Equation

)2(O)p(uI)p(I)p(I T112 +∇= -

0)p(I)p(uIT1 ≈+ δ∇

At a Single Pixel

Leads to

0IuIuI ty

yx

x =++

Normal FlowI

I- t

∇

Multiple Constraints in a Region

∑∈

∇Rp

2T1SSD ))p(I)p(uI()u(E δ+=

X

Solution

0u

)u(ESSD =∂

∑∈

∇Rp

2T1SSD ))p(I)p(uI()u(E δ+=

∑∈

∇∇Rp

T1 0))p(I)p(uI(I =+ δ

∑ ∑∈ ∈

∇-∇∇Rp Rp

T1 IIu]II[ δ=

bAu =

Solution

bAu =

]III

III[A

R

2y

Ryx

Ryx

R

2x

= ∑∑

∑∑

δ

δ= ]

II

II[b

Ry

Rx

∑

∑

-

-

Observations:

• A is a sum of outer products of the gradient vector• A is positive semi-definite• A is non-singular if two or more linearly independent gradients are available• Singular value decomposition of A can be used to compute a solution for u

Another way to provide unique solution

Global Parametric Models

∑∈

∇Rp

2T1SSD ))p(I)p(uI()u(E δ+=

• u(p) is described using an affine transformation valid within the whole region R

tp)p(u +Η= ]hh

hh[H

2221

1211

= = ]t

t[t

2

1

T2122211211 ]tthhhh][

10yx00

0100yx[)p(u = βΒ= )p()p(u

∑∈

∇Rp

2T1SSD ))p(I)p(BI()u(E δ+β=

0u

)u(ESSD =∂ ∑ ∑

∈ ∈

∇-∇∇Rp Rp

TT1

T II)p(B)]p(BII)p(B[ δ=β

bA =β

Affine Motion

Good approximation for :– Small motions– Small Camera rotations– Narrow field of view camera– When depth variation in the scene is small

compared to the average depth and small motion

– Affine camera images a planar scene

Affine Motion

• Affine camera: ]Y

X[sp = ]

Y

X[sp

′

′′=′ • 3D Motion: TRPP +=′

xy23

13T22

yT2

xT1

TsZ]r

r[spRs]

TPr

TPr[sp ′+′+′=

+

+′=′

• A 3D Plane: η+βα=η+β+α= p][s

1YXZ

xy23

13T22 Tsp][

s

1]

r

r[spRsp ′+η+βα′+′=′

tHpp-p)p(u +=′=

Two More Ingredients for Success

• Iterative solution through image warping– Linearization of the BCE is valid only when u(p) is small– Warping brings the second image “closer” to the

reference

• Coarse-to-fine motion estimation for estimating a wider range of image displacements– Coarse levels provide a convex function with unique local

minima– Finer levels track the minima for a globally optimum

solution

Image Warping

))p(up(I)p(I 12 -=

• Express u(p) as: )p(u)p(u)p(u )k( δ+=

)p(I))p(up(I w1

)k(1 ≈-

u(p))-p(I)u(p)-)p(up(I)p(I w1

)k(12 δδ= ≈-

Target Image : EmptySource Image : Filled

p)p(up )k(-

Coarse-to-fine Image Alignment : A Primer

p

2

ΘΘ))u(p;(pI(p)I( 21min )

{ R, T, d(p) }{ H, e, k(p) }

{ dx(p), dy(p) }

d(p)

Warper -

Mosaics In Art

…combine individual chips to create a big picture...

Part of the Byzantine mosaic floor that has been preserved in the Church of Multiplication in Tabkha (near the Sea of Galilee).

www.rtlsoft.com/mmmosaic

Image Mosaics

• Chips are images.

• May or may not be captured from known locations of the camera.

OUTPUT IS A SEAMLESS MOSAIC

VIDEOBRUSH IN ACTION

WHAT MAKES MOSAICING POSSIBBLE…the simplest geometry...

Single Center of Projection for all Images

Images

Projection Surfaces

COP

Planar Mosaic Construction

• Align Pairwise: 1:2, 2:3, 3:4, ...• Select a Reference Frame• Align all Images to the Reference Frame• Combine into a Single Mosaic

1 2 3 4

Virtual Camera (Pan)Image Surface - PlaneProjection - Perspective

Key Problem What Is the Mapping From Image Rays to the Mosaic

Coordinates ?

ImagesCOP

P,P'

Rotations/HomographiesPlane Projective Transformations

RPP' cc Rpp '

RKppK ''

RKpKp 1''

pHp'

IMAGE ALIGNMENT IS A BASIC REQUIREMENT

PYRAMID BASED COARSE-TO-FINE ALIGNMENT… a core technology ...

• Coarse levels reduce search.

• Models of image motion reduce modeling complexity.

• Image warping allows model estimation without discrete feature extraction.

• Model parameters are estimated using iterative non-linear optimization.

• Coarse level parameters guide optimization at finer levels.

Assume that at the kth iteration, kΡ, is available

)p(ΡI)p(I)(pI wkww

model the residual transformation between the coordinate systems, wpand p, as:

D]p[Ipw

I(p)D|D

pI(p;0))(pID))(p;(pI 0D

wwwwww T

0D

w

|D

p

1ydxd

d)yd(1xd1ydxd

dyd)xd(1

p

3231

232221

3231

131211

w

2

2

0D

w

yxy1yx000

xyx0001yx|

D

p

D][IΡΡ (k)1)(k

ITERATIVE SOLUTION OF THE ALIGNMENT MODEL

ITERATIVE REWEIGHTED SUM OF SQUARES

• Given a solution )m(Θ at the mth iteration, find Θδ by solving :

k

ii

i

i

l il

l

i

k

i

i

i

θ

rr

r

)r(ρθ

θ

r

θ

r

r

)r(ρ

k

iw

• iw acts as a soft outlier rejecter :

2SS

σ

1

r

)r(ρ

222

2GM

)rσ(

σ2

r

)r(ρ

PROGRESSIVE MODEL COMPLEXITY…combining real-time capture with accurate alignment...

• Provide user feedback by coarsely aligning incoming frames with a low order model– robust matching that covers a wide search range– achieve about 6-8 frames a sec. on a Pentium 200

• Use the coarse alignment parameters to seed the fine alignment– increase model complexity from similarity, to affine, to

projective parameters– coarse-to-fine alignment for wide range of motions and

managing computational complexity

delay

estimateglobal shift

warp

)(tI

)1(ˆ tI

)(td

)(td

I (t 1)

COARSE-TO-FINE ALIGNMENT

VIDEO MOSAIC EXAMPLE

Princeton Chapel Video Sequence54 frames

UNBLENDED CHAPEL MOSAIC

Image Merging withLaplacian Pyramids

Image 1 Image 2

1 2

Combined Seamless Image

VORONOI TESSELATIONS W/ L1 NORM

BLENDED CHAPEL MOSAIC

1D vs. 2D SCANNING

• 1D : The topology of frames is a ribbon or a string.

Frames overlap only with their temporal neighbors.

• 2D : The topology of frames is a 2D graph Frames overlap with neighbors on many sides

(A 300x332 mosaic captured by mosaicing a 1D sequence of 6 frames)

1D vs. 2D SCANNING

The 1D scan scaled by 2 to 600x692 A 2D scanned mosaic of size 600x692

CHOICE OF 1D/2D MANIFOLD

Plane Cylinder Cone

Sphere Arbitrary

THE “DESMILEY” ALGORITHM

• Compute 2D rotation and translation between successive frames

• Compute L by intersecting central lines of each frame• Fill each pixel [l y] in the rectified planar mosaic by

mapping it to the appropriate video frame

2D MOSAICING THROUGH TOPOLOGY INFERENCE&

LOCAL TO GLOBAL ALIGNMENT

… automatic solution to two key problems ...

• Inference of 2D neighborhood relations (topology) between frames

– Input video just provides a temporal 1D ordering of frames

– Need to infer 2D neighborhood relations so that local constraints may be setup between pairs of frames

• Globally consistent alignment and mosaic creation

– Choose appropriate alignment model

– Local constraints incorporated in a global optimization

PROBLEM FORMULATION

Given an arbitrary scan of a scene

Create a globally aligned mosaic by minimizing

Gij i

iij EEE mosaic) theofArea (min 2

}{

iP

Like an MDL measure : Create a compact appearance while being geometrically consistent

ERROR MEASURE

Gij i

iij EEE mosaic) theofArea (min 2

}{

iP

where

ation transformdistortionleast like criterion prioria

for allow toerror term reference to Frame:

relations odneighborho therepresents that Graph :

and neighbors betweenerror alignment of measureAny :

mapping, image-to- Reference:

i

ij

E

G

jiE

XPuP iii

ALGORITHMIC APPROACH

From a 1D ordered collection of framesto

A Globally consistent set of alignment parameters

Iterate through

1. Graph Topology Determination

Given: pose of all framesEstablish neighborhood relations min(Area of Mosaic)

Graph G2. Local Pairwise Alignment

Given: GQuality measure validates hypothesized arcsProvides pairwise constraints

3. Globally Consistent Alignment

Given: pairwise constraintsCompute reference-to-frame pose parameters min ijE

GRAPH TOPOLOGY DETERMINATION

• Given: Current estimate of pose*

• Lay out each frame on the 2D manifold (plane, sphere, etc.)

• Hypothesize new neighbors based on– proximity

– predictability of relative pose

– non redundancy w.r.t. current G

• Specifically, try arc (i,j) ifNormalized Euclidean dist dij << Path distance Dij

• Validate hypothesis by local registration

• Add arc to G if good quality registration

* Initialize using low order frame-to-frame mosaic algorithm on a plane

dij

Dij

LOCAL COARSE & FINE ALIGNMENT

• Given: a frame pair to be registered

• Coarse alignment– Low order parametric model e.g. shift, or 2D R & T

– Majority consensus among subimage estimates

• Fine alignment [Bergen, ECCV 92]– Coarse to fine over Laplacian pyramid

– Progressive model complexity, up to projective

– Incrementally adjust motion parameters to minimize SSD

• Quality measure– Normalized correlation helps reject invalid registrations

GLOBALLY CONSISTENT ALIGNMENT

• Given: arcs ij in graph G of neighbors

• The local alignment parameters, Qij, help establish feature correspondence between i and j

• If uil and ujl are corresponding

points in frames i,j, then

211 |)()(| jljili PP uuEij

• Incrementally adjust poses Pi to minimize

ui uj

Eij

Gij i

iij EEE}{

miniP

SPECIFIC EXAMPLES : 1. PLANAR MOSAICS

• Mosaic to frame transformation model:

• Local Registration– Coarse 2D translation & fine 2D projective alignment

• Topology : Neighborhood graph defined over a plane– Initial graph topology computed with the 2D T estimates– Iterative refinement using arcs based on projective alignment

• Global Alignment

k

ijE2|)()(| jkjiki uAuA

2

1

2|)())()((|k

iE kkkjki AA

Pair Wise Alignment Error

Minimum Distortion Error

XPu i

PLANAR TOPOLOGY EVOLUTION

Whiteboard Video Sequence75 frames

PLANAR TOPOLOGY EVOLUTION

FINAL MOSAIC

SPECIFIC EXAMPLES : 2. SPHERICAL MOSAICS

• Frame to mosaic transformation model:

• Local Registration– Coarse 2D translation & fine 2D projective alignment

• Parameter Initialization– Compute F and R’s from the 2D projective matrices

• Topology : – Initial graph topology computed with the 2D R & T estimates on a plane– Subsequently the topology defined on a sphere– Iterative refinement using arcs based on alignment with F and R’s

• Global Alignment

k

ijE2|| jk

1jik

1i uFRuFR

XFRu iT

SPHERICAL MOSAICS

Sarnoff Library VideoCaptures almost the complete sphere

with 380 frames

SPHERICAL TOPOLOGY EVOLUTION

SPHERICAL MOSAICSarnoff Library

SPHERICAL MOSAICSarnoff Library

NEW SYNTHESIZED VIEWS

FINAL MOSAICPrinceton University Courtyard

Mosaicing from Strips

Mosaicfrom center strips

Image 1 Image 2 Image 3 Image 4 Image 5

Problem: Forward Translation

General Camera Motion

Parallel Flow: Straight Strip

Radial Flow (FOE): Circular Strip

• Strip Perpendicular to Optical Flow• Cut/Paste Strip (warp to make Optical Flow parallel)

Mosaic Construction

Mosaic

Simple Cases

0Max

0)(2

22 Myxb

0

a

v

u

by

bx

v

u

Horizontal Translation

Zoom

(M determines displacement)

(M determines radius)

Manifold for Forward Motion

• Stationary (but rotating) Camera – Viewing Sphere

• Translating Camera– Sphere carves a “Pipe” in space

Pipe Projection

One Image Sequence

Concatenation of Pipes

Forward Motion Mosaicing

Example: Forward Motion

Side View of Mosaic

Forward Mosaicing II

Mosaic Construction

OmniStereo: Stereo in Full 360o Two Panoramas: One for Each Eye

Each panorama can be mapped on a cylinder

Paradigm: A Rotating Stereo Pair of Slit Cameras

•Rays are tangent to viewing circle (Gives 360o stereo)

•Image planes are

radial

(Makes mosaicing difficult)

Panoramic Projections of Slit Cameras

Image Surface

Left Eye panorama (viewing circle)Projection rays tilted right

Right Eye panorama (viewing circle)Projection rays tilted left

Directional Viewpoints

Regular panorama (single viewpoint):Projection rays orthogonal to cylinder

Image Plane

PinholeAperture

(a) (b)Center Strip

(c)Side Strip

Vertical Slit

Slit Camera Model

•Center Strip: Rays perpendicular to image plane

•Side Strip: Rays tilted from image plane

Stereo Panorama from Strips

Left-eye-Panoramafrom right strips

Right-eye-Panoramafrom left strips

Image 1 Image 2 Image 3 Image 4 Image 5

MultiView Panoramas

0

180

360

Regular Panorama

Left-EyePanorama

Right-Eye Panorama

Stereo Panorama from VideoStereo Panorama from Video

Stereo viewing with Red/Blue Glasses

Viewing Panoramic Stereo Printed Cylindrical Surfaces

• Print panorama on a cylinder

• No computation needed!!!