39
Matting and Compositing Digital Visual Effects, Spring 2008 Yung-Yu Chuang 2008/4/29 Outline • Traditional matting and compositing • The matting problem • Bayesian matting and extensions • Matting with less user inputs • Matting with multiple observations • Beyond the compositing equation* • Conclusions Outline • Traditional matting and compositing • The matting problem • Bayesian matting and extensions • Matting with less user inputs • Matting with multiple observations • Beyond the compositing equation* • Conclusions Photomontage The Two Ways of Life, 1857, Oscar Gustav Rejlander Printed from the original 32 wet collodion negatives.

Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Embed Size (px)

Citation preview

Page 1: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Matting and Compositing

Digital Visual Effects, Spring 2008Yung-Yu Chuang2008/4/29

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Photomontage

The Two Ways of Life, 1857, Oscar Gustav RejlanderPrinted from the original 32 wet collodion negatives.

Page 2: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Photographic compositions

Lang Ching-shan

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Optical compositing

King Kong (1933) Stop-motion + optical compositing

Page 3: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Digital matting and compositing

The lost world (1925) The lost world (1997)

Miniature, stop-motion Computer-generated images

Digital matting and composting

King Kong (1933) Jurassic Park III (2001)

Optical compositingBlue-screen matting, digital composition,

digital matte painting

Titanic

Matting and Compositing Matting and Compositing

backgroundreplacement

backgroundediting

Page 4: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Digital matting: bluescreen matting

Forrest Gump (1994)

• The most common approach for films.• Expensive, studio setup.• Not a simple one-step process.

Color difference method (Ultimatte)

Blue-screenphotograph

C=F+αB

Spill suppressionif B>G then B=G

F

Matte creationα=B-max(G,R)

α

demo with Paint Shop Pro (B=min(B,G))

Problems with color difference

Background color is usually not perfect! (lighting, shadowing…)

Chroma-keying (Primatte)

Page 5: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Chroma-keying (Primatte)

demo

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Compositing

BFC )α(α −+= 1

αF B

C

foreground color alpha matte background plate

composite

compositingequationB

F

C

α=0

Compositing

BFC )α(α −+= 1

αF B

Ccomposite

compositingequationB

F

Cα=1

Page 6: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Compositing

BFC )α(α −+= 1

αF B

Ccomposite

compositingequationB

FC

α=0.6

Matting

CobservationBFC )α(α −+= 1

compositingequation

αF B

Matting

CBFC )α(α −+= 1

compositingequation

αF B

Three approaches:

1 reduce #unknowns

2 add observations

3 add priors

Matting (reduce #unknowns)

C

F BB

BFC )α(α −+= 1

differencematting

α

Page 7: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Matting (reduce #unknowns)

C

F

BFC )α(α −+= 1B

blue screenmatting

α

Matting (add observations)

F

BFC )α(α −+= 1

triangulation

α

BFC )α(α −+= 1

BC

Natural image matting

BC

Matting (add priors)

F

BFC )α(α −+= 1

α B

rotoscopingRuzon-Tomasi

FG

BG

unknown

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Page 8: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Bayesian framework

f(z)+εz ypara-meters

observedsignal

)|(max* yzPzz

=

)()()|(max

yPzPzyP

z=

)()|(max zLzyLz

+=

Example: super-resolutionde-blurringde-blocking…

Bayesian framework

)()|(max* zLzyLzz

+=

2

2)(σ

zfy −data evidence

a-prioriknowledge

f(z)+εz ypara-meters

observedsignal

Bayesian framework

likelihood priors

posterior probability

Priors

Page 9: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Bayesian matting Optimization

repeat

until converge

1. fix alpha

2. fix F and B

Page 10: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)
Page 11: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)
Page 12: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)
Page 13: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Demo

Page 14: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

inputtrimapalpha

Results

input composite

Results

Comparisons

input imagetrimap

Comparisons

Bayesian Ruzon-Tomasi

Page 15: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Comparisons

Bayesian Ruzon-Tomasi

Comparisons

Mishima

Comparisons

Bayesian

Comparisons

input image

Page 16: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Comparisons

Bayesian Mishima

Comparisons

Bayesian Mishima

inputvideo

Video matting

inputkeytrimaps

inputvideo

Video matting

Page 17: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

interpo-latedtrimaps

inputvideo

Video matting

interpo-latedtrimaps

inputvideo

outputalpha

Video matting

Compo-site

interpo-latedtrimaps

inputvideo

outputalpha

Video matting

Page 18: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

optical flow optical flow

Page 19: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)
Page 20: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Sample composite Garbage mattes

Page 21: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Garbage mattes Background estimation

Background estimation Alpha matte

Page 22: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Comparison

without background

withbackground

Page 23: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

C

B

P(F)

Page 24: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Problems with Bayesian matting

• It requires fine trimaps for good results• It is tedious to generate fine trimaps• Its performance rapidly degrades when

foreground and background patterns become complex

• There is no direct and local control to the resulted mattes

Page 25: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Motivation

LazySnapping

LazySnapping

Page 26: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

LazySnapping LazySnapping

Matting approaches• Sampling approaches: solve for each

alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian matting.

• Propagation approaches: solve the whole matte together by optimizing, e.g. Poisson, BP, random walker, closed-form and robust matting.

Poisson matting

Page 27: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Poisson matting Robust matting• Jue Wang and Michael Cohen, CVPR

2007

Robust matting• Instead of fitting models, a non-

parametric approach is used

Bayesian Robust

Robust matting• We must evaluate hypothesized

foreground/background pairs

Bj

Fi

C

distance ratio

Page 28: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Robust matting• To encourage pure fg/bg pixels, add

weights

B F1

C

F2

Robust matting• Combine them together. Pick up the

best 3 pairs and average themconfidence

Robust matting Robust matting

matte confidence

Page 29: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Matte optimization

Solved by Random Walk Algorithm

Matte optimizationdata constraints

neighborhood constraints

Demo (EZ Mask) Evaluation• 8 images collected in 3 different

ways• Each has a “ground truth” matte

Page 30: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Evaluation• Mean square error is used as the

accuracy metric• Try 8 trimaps with different accuracy

for testing robustness • 7 methods are tested: Bayesian,

Belief propagation, Poisson, Random Walk, KnockOut2, Closed-Form and Robust matting

Quantitative evaluation Subjective evaluation

Page 31: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Subjective evaluation Ranks of these algorithms

PoissonRandom walkKnockout2BayesianBelief PropagationClose-formRobust matting

accuracy6.96.04.53.93.32.61.0

robustness6.84.44.56.03.12.01.3

Summary• Propagation-based methods are more

robust• Sampling-based methods often

generate more accurate mattes than propagation-based ones with fine trimaps

• Robust matting combines strengths of both

Soft scissor• Jue Wang et. al., SIGGRAPH 2007• Users interact in a similar way to

intelligent scissors

Page 32: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Flowchart Flowchart

Soft scissor Demo (Power Mask)

Page 33: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Matting with multiple observations• Invisible lights

– Polarized lights– Infrared

• Thermo-key• Depth Keying (ZCam)• Flash matting

Invisible lights (Infared) Invisible lights (Infared)

Page 34: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Invisible lights (Infared) Invisible lights (Infared)

Invisible lights (Infared) Invisible lights (Infared)

Page 35: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Invisible lights (Polarized) Invisible lights (Polarized)

Thermo-Key Thermo-Key

Page 36: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

ZCam Defocus matting

videoMatting with camera arrays

video

Page 37: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Flash matting

flash no flash matte

Flash matting

Background is much further than foreground and receives almost no flash light

Flash matting

Foreground flash matting equation

Generate a trimap and directly apply Bayesian matting.

Foreground flash matting

Page 38: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Joint Bayesian flash matting Joint Bayesian flash matting

Comparison

flash no flash

Comparison

foreground flash matting

ioint Bayesian flash matting

Page 39: Matting and Compositing - csie.ntu.edu.twcyy/courses/vfx/08spring/lectures/... · Matting and Compositing Digital Visual Effects, ... Digital matting and composting King Kong (1933)

Flash matting

Outline• Traditional matting and compositing• The matting problem• Bayesian matting and extensions• Matting with less user inputs• Matting with multiple observations• Beyond the compositing equation*• Conclusions

Conclusions• Matting algorithms improves a lot in

these 10 years• In production, it is still always

preferable to shoot against uniform backgrounds

• Algorithms for more complex backgrounds

• Devices or algorithms for automatic matting