2/12/2001 Vision for Graphics 3
Block-based motion prediction
Break image up into square blocks
Estimate translation for each block
Use this to predict next frame, code difference (MPEG-2)
2/12/2001 Vision for Graphics 4
Layered motion
Break image sequence up into “layers”:
=
Describe each layer’s motion
2/12/2001 Vision for Graphics 5
Outline
• Why layers?• 2-D layers [Wang & Adelson 94; Weiss 97]• 3-D layers [Baker et al. 98]• Layered Depth Images [Shade et al. 98]• Transparency [Szeliski et al. 00]• Bayesian estimation [Torr et al. 99]
2/12/2001 Vision for Graphics 6
Layered motion
Advantages:• can represent occlusions / disocclusions• each layer’s motion can be smooth• video segmentation for semantic processing
Difficulties:• how do we determine the correct number?• how do we assign pixels?• how do we model the motion?
2/12/2001 Vision for Graphics 8
Background modeling (MPEG-4)
Convert masked images into a background sprite for layered video coding
+ + +
=
2/12/2001 Vision for Graphics 9
What are layers?
[Wang & Adelson, 1994]
• intensities• alphas• velocities
2/12/2001 Vision for Graphics 13
How do we estimate the layers?
1. compute coarse-to-fine flow
2. estimate affine motion in blocks (regression)
3. cluster with k-means
4. assign pixels to best fitting affine region
5. re-estimate affine motions in each region…
2/12/2001 Vision for Graphics 14
Layer synthesis
For each layer:• stabilize the sequence with the affine motion• compute median value at each pixel
Determine occlusion relationships
2/12/2001 Vision for Graphics 16
What if the motion is not affine?
Use a “regularized” (smooth) motion field
[Weiss, CVPR’97]
2/12/2001 Vision for Graphics 19
Examples:• Disparity-Spaces [Intille and Bobick, ‘94] [Scharstein and Szeliski, ‘96] • Space-Coloring [Seitz and Dyer, ‘97]• Maximum-Flow Stereo [Roy and Cox, ‘98]
Advantages:• Modeling occlusions [Intille and Bobick, ‘94] • Mixed pixels + transparency [Szeliski and Golland, ‘98]• Equal treatment of many images [Collins, ‘96]
Volumetric Approaches to Stereo
Camera 1Camera 2
x
z
y
2/12/2001 Vision for Graphics 20
2.5-D Layered Approach
Additional advantages over volumetric approaches:• Fewer degrees of freedom• Less resampling artifacts• Robustness of global model + local correction
– c.f. “Plane + Parallax” and “Model-Based Stereo”
• Output particularly suitable for certain applications– e.g. Image-based rendering and interactive editing
Layer 2
Camera 1
Layer 1
Layer 3Camera 2
2/12/2001 Vision for Graphics 21
Layered Stereo
Use arbitrarily oriented sprites
Estimate 3D plane equation for each sprite
layers (“sprites”)layers (“sprites”)
2/12/2001 Vision for Graphics 23
Layer Representation
Coordinate frame defined by (u, v, 1)Tu = = Q l x
Layer sprite L = (a . r , a . g , a . b , a)l
Residual depth Zl
World origin
x = (x, y, z, 1)TWorld point
u
v
Plane vectorn = (n , n , n , n )
Tx zy
Plane equationn . x = 0
l
l
d
2/12/2001 Vision for Graphics 24
Image Formation
Scene
Image I k
Camera Pk
v
uMasked image M k l
v
u
Boolean mask B k l
v
u
Layer l
2/12/2001 Vision for Graphics 25
Overview
Output: n , L , & Zll l
Refine Layer Sprites L l
Input: Images I & Cameras Pkk
Re-assign pixelslayers Bkl
Estimate residual depth Z l
Estimate plane vectors n l
Estimate sprite images Ll
Initialize layer assignment B kl
2/12/2001 Vision for Graphics 26
Layer Initialization Alternatives
Iterate dominant motion estimation• e.g. [Irani et al., ‘95]
Apply simple stereo algorithm + fit planesColor segmentation
• e.g. [Sawhney and Ayer, ‘94]
Human initialization• e.g. [Debevec et al., ‘96]
2/12/2001 Vision for Graphics 28
H lik
M klH lik o
Estimation of Plane Equations
Camera Pk
M klCamera P j M jl
H lij
M jlH lij o
Camera Pi
M il
Layer l
Warped images , , … functions of n only M jlH lij o M klH l
ik ol
Minimize image variance using hierarchical gradient descent
2/12/2001 Vision for Graphics 30
Estimation of Layer Sprites
Camera Pk
M klCamera P j M jl
Camera Pi
M il
Plane nl
“Blend” the masked images, warped onto the layer plane
2/12/2001 Vision for Graphics 31
Estimation of Residual Depth
Per-pixel residual depth estimation• plane plus parallax [Anandan et al.]• model-based stereo [Debevec et al.]
• better accuracy / fidelity• makes forward warping more difficult
2/12/2001 Vision for Graphics 32
Estimation of Residual Depth
Camera Pk
M klCamera P j
M jl
Camera P i
M il
Perturbed Plane n + (0,0,0,d)lT
Warp masked images onto perturbed plane
Compute variance image
For each pixel, choose d that minimizes variance
Smooth, incorporating confidence weighting [Szeliski & Golland, ‘98]
Recompute sprite using “Plane + Parallax” warp
2/12/2001 Vision for Graphics 33
Pixel Assignment
Camera Pi
M il
Plane nlSprite L l
• Warp masked image onto each layer plane • Compute difference images • Un-warp difference images • For each pixel, choose the best difference across layers • Smooth pixel assignment
Un-warpeddifference
image
2/12/2001 Vision for Graphics 34
Flower Garden Results
Initial Segmentation
Image 1 Image 9
Grey coded planar depth
2/12/2001 Vision for Graphics 35
Recovered Sprite: Without residual depth estimation
Recovered Sprite: With residual depth estimation
Flower Garden Results
2/12/2001 Vision for Graphics 36
Graphics Symposium Results
Image 1 of 5 Initial segmentation
Grey coded planar depth Residual depth
2/12/2001 Vision for Graphics 38
Graphics Symposium Results
Original image 3 Re-synthesized image 3
Novel view without residual depth Novel view with residual depth
2/12/2001 Vision for Graphics 40
Discussion
Layer initialization:• Can tolerate bad initial plane estimates• Residual depth estimation:
– Plane sweep algorithm, similar to
[Szeliski and Golland, ‘98]
Pixel assignment:• Combine color and residual depth estimates• Currently under investigation
2/12/2001 Vision for Graphics 41
Summary
New approach to stereo matching:• represent scene as collection of layers• each layer has a 3-D plane equation, an alpha-matted color
image, and an optional residual depth• generalizes layered motion to 3-D
Computation:• plane eqns. by warping mosaics of masked images• residual depth by perturbing planes• iteratively refine color values and pixel assignments
2/12/2001 Vision for Graphics 43
How to render a layer + parallax?
Can’t use inverse warping [Laveau 94]
2/12/2001 Vision for Graphics 44
3D Sprites with Depth
3D sprite consists of:• alpha-matted image I1(x1,y1)
• 4×4 camera matrix C1 [ w1x1 w1y1 w1d1 w1]T = C1 [X Y Z
1]T
• plane equation AX + BY + CZ + D = 0(forms third row of C1 )
• optional per-pixel depth d1 (x1,y1)
2/12/2001 Vision for Graphics 45
Sprites with Depth
Store d1(x1,y1) (scaled displacement) along with each sprite image I1(x1,y1)
I1 d1 I1 d1
2/12/2001 Vision for Graphics 46
3D Sprites — Reprojection
sprites new view
use standard texture mapping (projective warp)
2/12/2001 Vision for Graphics 47
Forward Mapping
Mapping equation with per-pixel depth d1:[ w2x2 w2y2 w2 ]T = H1,2 [ x1 y1 1 ]T + d1 e1,2
I1 d1 (I2 ) I2
Problems: gaps and aliasing
2/12/2001 Vision for Graphics 48
Inverse Mapping
Reverse order of images 1 & 2:[ w1x1 w1y1 w1 ]T = H2,1 [ x2 y2 1 ]T + d2 e2,1
I1 (I2) d2 I2
Problem: we don’t know d2!
2/12/2001 Vision for Graphics 49
Crude perspective map
How to map d1 d2?
Simple idea: use perspective transform H2,1
I1 d1 d2 I2
Works well for small amounts of motion
2/12/2001 Vision for Graphics 50
Better forward map
How to map d1 d2?
Better idea: use full H1,2x1+d1e1,2 fwd. map
I1 d1 d2 I2
Works better for moderate amounts of motion
2/12/2001 Vision for Graphics 51
2-pass Mapping
Why is 2-pass mapping (d1 d2 forward followed by I1 I2 backward) a good idea?
• can tolerate bigger errors in d1 mapping (since d1 is typically smooth)
• can store/process d1 at lower resolution
• can use better filtering on color image
2/12/2001 Vision for Graphics 53
Refinements
Only forward map d1 with parallax component
Use affine approximation to parallax flow
Better gap filling
Forward map (u,v) flow instead of d1 depth
2/12/2001 Vision for Graphics 54
Layered Depth Images (LDIs)
Store multiple (color,z) values at each pixel
Similar to [sparse] volumetric representation
Render with forward warp (splat)
Layer extraction from multiple images containing reflections and transparency
Richard SzeliskiShai AvidanP. Anandan
CVPR’2000
2/12/2001 Vision for Graphics 57
Previous work
Physics-based vision and polarization[Shafer et al.; Wolff; Nayar et al.]
Perception of transparency [Adelson…]
Transparent motion estimation[Shizawa & Mase; Bergen et al.; Irani et al.; Darrell & Simoncelli]
3-frame layer recovery [Bergen et al.]
2/12/2001 Vision for Graphics 58
Problem formulation
X
Y
MotionMotionX,iX,i( )( )
MotionMotionY,iY,i( )( )++
2/12/2001 Vision for Graphics 59
Image formation model
Pure additive mixing of positive signals
mk(x) = l Wkl fl(x)
or
mk = l Wkl fl
Assume motion is planar (perspective transform, aka homography)
2/12/2001 Vision for Graphics 60
Two processing stages
Estimate the motions and initial layer
estimates
Compute optimal layer estimates (for
known motion)
2/12/2001 Vision for Graphics 61
Dominant motion estimation
Stabilize sequence by dominant motion
robust affine [Bergen et al. 92; Szeliski & Shum]
2/12/2001 Vision for Graphics 62
Dominant layer estimate
How do we form composite (estimate)?
TimeTime
Inte
nsit
yIn
tens
ity
2/12/2001 Vision for Graphics 64
Median?
Hint: all layers are non-negative
TimeTime
Inte
nsit
yIn
tens
ity
2/12/2001 Vision for Graphics 65
Min-composite
Smallest value is over-estimate of layer
TimeTime
Inte
nsit
yIn
tens
ity
2/12/2001 Vision for Graphics 66
Difference sequence
Subtract min-composite from original image
=
original - min composite = difference imageoriginal - min composite = difference image
2/12/2001 Vision for Graphics 67
Min composite
TimeTime
Inte
nsit
yIn
tens
ity
(overestimate of background layer)(overestimate of background layer)
2/12/2001 Vision for Graphics 68
Difference sequence
TimeTime
Inte
nsit
yIn
tens
ity
(underestimate of foreground layer)(underestimate of foreground layer)
2/12/2001 Vision for Graphics 69
Stabilizing secondary motion
TimeTime
Inte
nsit
yIn
tens
ity
How do we form composite (estimate)?How do we form composite (estimate)?
2/12/2001 Vision for Graphics 70
Max-composite
TimeTime
Inte
nsit
yIn
tens
ity
Largest value is Largest value is under-estimateunder-estimate of layer of layer
2/12/2001 Vision for Graphics 71
Min-max alternation
Subtract secondary layer (under-estimate) from original sequence
Re-compute dominant motion and better min-composite
Iterate …
Does this process converge?
2/12/2001 Vision for Graphics 72
Min-max alternation
Does this process converge?• in theory: yes• each iteration reduces number of mis-estimated
pixels (tightens the bounds) — proof in paper
2/12/2001 Vision for Graphics 73
Min-max alternation
Does this process converge?• in practice: no• resampling errors and noise both lead to
divergence — discussion in paper
resampling error noisy
2/12/2001 Vision for Graphics 74
Two processing stages
Estimate the motions and initial layer
estimates
Compute optimal layer estimates (for
known motion)
2/12/2001 Vision for Graphics 75
Optimal estimation
Recall: additive mixing of positive signals
mk = l Wkl fl
Use constrained least squares(quadratic programming)
min k | l Wkl fl – mk |2 s.t. fl 0
2/12/2001 Vision for Graphics 76
Least squares example
background foregroundbackground foreground
blue: least squaresblue: least squares
red: constrained LSred: constrained LS
2/12/2001 Vision for Graphics 77
Uniqueness of solution
If any layer does not have a “black” region, i.e., if fl c, then can add this offset to another layer (and subtract it from fl)
background background foreground foreground
2/12/2001 Vision for Graphics 78
Degeneracies in solution
If motion is degenerate (e.g., horizontal), regions (scanlines) decouple (w/o MRF)
mixedmixed
scaled scaled errors errors
recovered recovered
2/12/2001 Vision for Graphics 79
Noise sensitivity
In general, low-frequency components hard to recover for small motions
mixedmixed
recovered recovered
scaled scaled errors errors
2/12/2001 Vision for Graphics 81
Complete algorithm
Dominant motion with min-composites
Difference (residual) images
Non-dominant motion on differences
Improve the motion estimates
Unconstrained least-squares problem
Constrained least-squares problem
2/12/2001 Vision for Graphics 82
Complete example
originaloriginal
stabilizedstabilizedmin-compositemin-composite
2/12/2001 Vision for Graphics 83
Complete example
differencedifference
stabilizedstabilizedmax-compositemax-composite
2/12/2001 Vision for Graphics 86
Results: Anne and books
= += +
original background foreground (photo)original background foreground (photo)
2/12/2001 Vision for Graphics 87
Transparent layer recovery
Pure (additive) mixing of intensities• simple constrained least squares problem• degeneracies for simple or small motions
Processing stages• dominant motion estimation• min- and max-composites to initialize• optimization of motion and layers
2/12/2001 Vision for Graphics 88
Future work
Mitigating degeneracies (regularization)Opaque layers ( estimation)
Non-planar geometry (parallax)
2/12/2001 Vision for Graphics 89
BibliographyJ. Y. A. Wang and E. H. Adelson. Representing moving images with
layers. IEEE Transactions on Image Processing, 3(5):625--638, September 1994.
Y. Weiss and E. H. Adelson. A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'96), pages 321--326, San Francisco, California, June 1996.
Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), pages 520--526, San Juan, Puerto Rico, June 1997.
P. R. Hsu, P. Anandan, and S. Peleg. Accurate computation of optical flow by using layered motion representations. In Twelfth International Conference on Pattern Recognition (ICPR'94), pages 743--746, Jerusalem, Israel, October 1994. IEEE Computer Society Press
2/12/2001 Vision for Graphics 90
BibliographyT. Darrell and A. Pentland. Cooperative robust estimation using layers of
support. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(5):474--487, May 1995.
S. X. Ju, M. J. Black, and A. D. Jepson. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'96), pages 307--314, San Francisco, California, June 1996.
M. Irani, B. Rousso, and S. Peleg. Computing occluding and transparent motions. International Journal of Computer Vision, 12(1):5--16, January 1994.
H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):814--830, August 1996.
M.-C. Lee et al. A layered video object coding system using sprite and affine motion model. IEEE Transactions on Circuits and Systems for Video Technology, 7(1):130--145, February 1997.
2/12/2001 Vision for Graphics 91
BibliographyS. Baker, R. Szeliski, and P. Anandan. A layered approach to stereo
reconstruction. In IEEE CVPR'98, pages 434--441, Santa Barbara, June 1998.
R. Szeliski, S. Avidan, and P. Anandan. Layer extraction from multiple images containing reflections and transparency. In IEEE CVPR'2000, volume 1, pages 246--253, Hilton Head Island, June 2000.
J. Shade, S. Gortler, L.-W. He, and R. Szeliski. Layered depth images. In Computer Graphics (SIGGRAPH'98) Proceedings, pages 231--242, Orlando, July 1998. ACM SIGGRAPH.
S. Laveau and O. D. Faugeras. 3-d scene representation as a collection of images. In Twelfth International Conference on Pattern Recognition (ICPR'94), volume A, pages 689--691, Jerusalem, Israel, October 1994. IEEE Computer Society Press.
P. H. S. Torr, R. Szeliski, and P. Anandan. An integrated Bayesian approach to layer extraction from image sequences. In Seventh ICCV'98, pages 983--990, Kerkyra, Greece, September 1999.