Image Matting and Its Applications
Chen-Yu TsengAdvisor: Sheng-Jyh Wang
2012-10-29
Image Matting
• A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color)
Input Image Alpha Matte Extracted Foreground
Two Approaches of Image Matting
• Supervised Matting• With User’s Guidance
• Unsupervised Matting• Without User’s Guidance
Input Image User’s Guidance
e.g. Trimap:White ForegroundBlack BackgroundUnknown Gray
Two Schemes of Supervised Matting
Propagation-based Scheme• Infer Alpha Matte with
Propagation through a Graphical Model
• A Global-based Approach
Sampling-based Scheme• Infer Alpha Matte with
Some Color Samples• A Local-based Approach
Foreground Pixel
Background Pixel
Unknown Pixel
Foreground Color Set
Background Color Set
Unknown Pixel
Propagation-based scheme -Matting Laplacian Approach
• A Graphical Model with Connectivity between Pixels• The Connectivity Is Learned from the Image Structure
• Capability for Dealing with Both • Supervised Matting (Inference Problem)• Unsupervised Matting (Decomposition Problem)
Foreground Pixel
Background Pixel
Unknown Pixel
Reference of Matting Laplacian Approach
• First proposed by Levin et al. for supervised matting (closed-form matting)• A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural
Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.• Extended to unsupervised matting (spectral matting)
• A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008.
• Extended to learning-based matting• Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In
ICCV, pages 889–896, 2009.• Extended to multi-layer matting
• D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.
Matting Laplacian
Input Image
EstimatingPair-wise Affinity
Graphical ModelNode: Image PixelsEdge: Affinity
Supervised Matting
Background
Foreground
Matting Laplacian Matrix:Recording the Connectivity between Pair of Pixels
Introduction of Graph Laplacian
2
3
1
4
5
A Graph with Five Vertexes
: Adjacency Matrix
𝐿=𝐷−𝑊: Laplacian Matrix
𝑊=(𝑤𝑖𝑗)𝑖 , 𝑗=1 ,… ,𝑛
: Degree Matrix
𝑑𝑖𝑖=∑𝑗=1
𝑛
𝑤𝑖𝑗
0 1 1 0 01 0 1 0 01 1 0 0 00 0 0 0 10 0 0 1 0
12345
1 2 3 4 5
: Adjacency Matrix
Vertex Index
Cutting Cost Function with Graph Laplacian
𝜶𝑇 𝐿𝜶=12 ∑𝑖 , 𝑗=1
𝑛
𝑤𝑖𝑗 (𝛼 𝑖−𝛼 𝑗 )2
Cost Function for Cutting Criterion
Low-costAssignment
High-costAssignment2
3
1
4
5
2
3
1
4
5
Construction of Matting Laplacian
• Color-model-based Approach (Original)• Estimating Affinity Based on Relative Color Distance
• Learning-based Approach (Extended)• Learning Affinity Based on Image Structure
Construction of Matting LaplacianColor-model-based Approach
Color Distribution
𝐼 𝑖
𝐼 𝑗
𝜇𝑘
Input Image
A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.
gr
b
Construction of Matting LaplacianLearning-based Approach
• Learning Affinity among Local Pixels
¿ [𝐱 𝑖𝑇 1 ] [ 𝜷𝛽0 ]
Linear Alpha-color Model for Single Pixel:
: Alpha Value for Pixel i: Feature Vector (): Linear Coefficient
Extending to a Local Patch qAssuming all Pixels Sharing the Same Linear Coefficient 𝛼 𝑖=𝐱 𝑖
𝑇 𝜷+𝛽0�⃗�𝑞=𝐗𝒒
𝑇 [ 𝜷𝛽0]: Alpha Vector for Patch q: Feature Matrix: Linear Coefficient
Construction of Matting LaplacianLearning-based Approach
[ 𝜷𝛽0]=argmin𝜷 ,𝛽0‖�⃗�𝑞−𝐗𝒒𝑇 [ 𝜷𝛽0]‖
2
+𝜆𝑟 𝜷𝑇 𝜷
¿ (𝐗𝒒𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
�⃗�𝑞=𝐗𝒒𝑇 [ 𝜷𝛽0]
¿𝐗𝒒𝑇 (𝐗𝒒
𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
Derived Linear Coefficient
Rewritten Linear Model
Construction of Matting LaplacianLocal Cost Function
�⃗�𝑞=𝐗𝒒𝑇 (𝐗𝒒
𝑇𝐗𝒒+𝜆𝑟 𝐈 )−𝟏𝐗𝒒 �⃗�𝑞
Local Cost Function
¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞
: Local Laplacian Matrix for Patch qInput Image
Patch q
Local Linear Model
Construction of Matting LaplacianLocal Global
Local Cost Function
¿ �⃗�𝑞𝑇 𝑳𝑞 �⃗�𝑞
: Local Laplacian Matrix for Patch q
Input Image
Patch q
Global Cost Function
¿ �⃗�𝑇 𝑳 �⃗�
Supervised Matting (Closed-form Matting)
Foreground Pixel
Background Pixel
Unknown Pixel
Input Image User’s Guidance,
𝐸 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�+(�⃗�− �⃗�)𝑇𝚲 (�⃗�− �⃗�)
Foreground
Background
Unknown
1 0 -1 1 0
Cost Function for Supervised Matting
Affinity Cost Data Cost
�⃗�∗=(𝑳+𝚲 )−1𝚲 �⃗�Optimal Solution
Experimental Results
Input Image Alpha Matte Synthesized Result
Unsupervised Matting (Spectral Matting)
• Solving Alpha Matte without User’s Guidance• Procedures
• Decomposing Image into Several Matting Components• Combining Matting Components into Alpha Matte
Spectral Clustering
s.t. =1 𝐿 𝒇 =λ 𝒇1. L is symmetric and positive semi-definite.2. The smallest eigenvalue of L is 0, the
corresponding eigenvector is the constant one vector 1.
3. L has n non-negative, real-valued eigenvalues
0= λ 1 ≦ λ 2 ≦ . . . ≦ λ n.
: Eigenvector: Eigenvalue
2
3
1
4
5
A Graph Example
2 -1 -1 0 0-1 2 -1 0 0-1 -1 2 0 00 0 0 1 -10 0 0 -1 1
12345
1 2 3 4 5
: Laplacian Matrix
0.0470.0470.0470.0470.047
0.5770.5770.57700𝒇 1 𝒇 2
Spectral Clustering & Matting Components
2 -1
-1
0 0 0 0
-1
2 -1
0 0 0 0
-1
-1
2 0 0 0 0
0 0 0 1 1 0 00 0 0 -
1-1
0 0
0 0 0 0 0 1 10 0 0 0 0 -
1-1
: Laplacian Matrix
1110000
0001100
0000011
Zero-Eigenvectors Binary Indicating Vectors
×𝑹3×3Linear
Transformation
Overview of Spectral Matting
Input Image
Smallest Eigenvectors
Matting Components
K-means Clustering
&Linear
TransformationMatting
Laplacian
Spectral Clustering & K-means
Input Image
s-smallest Eigenvectors
…
Pixel i
s-dimensional
Space
K-means Clustering
Generating Matting Components
Smallest Eigenvectors
Projection into Eigen Space
..K-means .… … …
𝑬=[𝒆𝟏 … 𝒆𝒔 ] 𝒎𝒌 𝜶𝒌=𝑬 𝑬𝑻𝒎𝒌
Reconstructing Alpha Matte from Matting Components
=+ +
Input Image
Matting Components
Selected Matting Components
Alpha Matte
Reconstructing Alpha Matte by Grouping Matting Components
Matting cost function
𝐽 ( �⃗� )=�⃗�𝑇 𝑳 �⃗�
�⃗�=[ �⃗�1 … �⃗�𝑘 ] �⃗�Alpha Matte Generation
: Combination Vector
¿ �⃗�𝑇 [ �⃗�1 … �⃗�𝑘 ]𝑇 𝑳 [�⃗�1 … �⃗�𝑘 ] �⃗�¿ �⃗�𝑇𝜱�⃗�
Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte
Results by Levin et al.
Summary
• Constructing Matting Laplacian• Solving Supervised Matting Problem• Solving Unsupervised Matting Problem
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)
Input Image Reconstructed Depth
Depth Reconstruction from Single Image
Depth Reconstruction in Shape From Focus (SFF)
Input Image Reconstructed Depth
Cell-based Framework
Image
Pixel-wise Data Distribution
Cell-wise Data Distribution
ConventionalMatting Laplacian
Cell-basedMatting Laplacian
Pixel-wise Affinity
Cell-wise Affinity
Multi-scale Affinity Learning
Image & Computation Patterns
Pixel-based Approach
Cell-based Approach
Multi-scale Affinity Learning
…Finest Level Coarsest
Level …
Cell-based Graph
Results of Reconstructed Alpha Matte
1st Rank 2nd Rank
(a) Grouping Results by Levin et al.
(b) Grouping Results by Levin et al. with Coarse-to-fine Scheme.
(c) Ours
Input
Results
(a) Input images
(b) Levin’s result (c) Our result
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations• Multi-scale Analysis• Extended Applications (Depth Image Reconstruction)
Input Image Reconstructed Depth
Depth Reconstruction from Single Image
Depth Reconstruction in Shape From Focus (SFF)
Input Image Reconstructed Depth
Depth Reconstruction in Shape From Focus (SFF)
Optical Direction
Multi-focus Image Sequence
Optical Direction
FocusValueW1
W2
W2
W1
Low-SNR Problem
• Spatially Varying Precision• Low-texture Low-SNR• Leading Noisy Result
Input Image Observation
High-precision
Low-precision
Proposed Maximum-a-posteriori Estimation
Multi-focus Image Sequence
Learning-based Graph
Local Learning
Inference
Reconstructed Depth
Proposed Maximum-a-posteriori Estimation
𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) ): Optimal Result: Depth Image: Observation: Input Image
Posterior Likelihood Prior
Local Observation with Spatial-varying Precision
Learned from Image
Likelihood Model
Input Observation
Precision Result
High-precision
Low-precision
𝑰 𝒀
𝚲 𝐷∗
Posterior Likelihood Prior
Local Observation with Spatial-varying Precision
Prior Model Posterior Likelihood Prior
Learning from Input Image
Learning-based Graph
Local Learning
Multi-focus Image Sequence
Maximum-a-posteriori Estimation for Depth Reconstruction
𝐷∗=max (𝑝 (𝐷|𝑌 , 𝐼 ) )
− log𝑝 (𝐷|𝑌 , 𝐼 )∝ ( �⃗�−𝒚 )𝑇𝚲 ( �⃗�− �⃗� )+ �⃗�𝑇 𝑳 �⃗�
𝐷∗= (𝑳+𝚲 )−1𝚲 �⃗�
Input Image Observation Reconstructed Depth
Results of Shape from Focus
Input Image M. Mahmood, 2012 T. Aydin, 2008 OursS. Nayar, 1994
Conclusions
• Construction of Matting Laplacian• Conventional Approach• Multi-scale Cell-based Approach
• Supervised Matting• Spectral Matting• Depth Reconstruction