Upload
adcaine
View
2.276
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Shows how Gaussian Mixture Models can be used in image segmentation
Citation preview
ALLAN [email protected]
HTTP: / /WWW.CS.UWATERLOO.CA/~ADCAINE/
MAY 28 , 2009
Chroma Keying using Gaussian Mixture Models (GMMs)
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
2
Expectations
You should take notes in case this material appears on an exam or assignment, possibly in a different form
It is expected that you will use the Matlab software found at http://www.cs.uwaterloo.ca/~adcaine/ to generate a toy problem and solve it
These lecture notes are on-line at the above address
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
3
Inspiration
Have you ever wondered how is this done? Can we emulate this process on a computer?
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
4
Co-ordination
By viewing an off-camera monitor, theweather announcer can co-ordinate herhand movements with the background.
The result is a seemingly realistic televisionbroadcast, in spite of the fact that theannouncer is not actually in front of anyweather map.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
5
Teaching Objectives
1. Describe how chroma keying (blue screening) works
2. Theorize how classification might solve the chroma keying problem on a computer
3. Enumerate the axioms of probability 4. Explain the concept behind digital images5. Express the intuition behind the Expectation-
Maximization algorithm in terms of a Gaussian Mixture Model (GMM), using the definition of a Gaussian.
6. Generate a toy GMM problem, and solve the toy problem, recovering the hidden variables
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
6
Outline
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
7
Chroma Keying and our Assumptions
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
8
Basic Idea behind Chroma Keying
If the background is of one color, saygreen, and the foreground person is notof that color, then the person can besegmented from the background.
The person can then be superimposed on abackground of our choosing.
Normally, the chroma key is assumed to beknown. To make this problem moreinteresting, we will assume that the chromakey is unknown, and must be learned. So,it’s an AI problem.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
9
Assumptions for the Segmentation Problem
1. We want to segment the foreground from the background
2. There is a background of a single color3. However, the background color is unknown;
the background color must be learned4. No part of the person or object in the
foreground has the background color5. The foreground person or object can be in
any position in the image; we cannot deduce the background color simply by examining a pixel in a certain fixed position
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
10
RBG Color Model
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
11
Explanation of RGB Color Model
Every color is expressed as an ordered triple
Each value can be between 0 and 255; i.e. 256 possibilities 8 bits can store 256 possibilities
The numbers represent the amounts of red, green, and blue in that order (R,G,B)
If R = G = B, then the pixel is a shade of gray
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
12
200 222 32 12
88 22 45 23
30 31 56 22
11 98 99 100
89 120 56 23
89 32 90 111
45 67 23 29
49 111 121 122
RGB as a Data Structure in Matlab
45 188 64 90
111 255 0 11
88 44 21 211
45 32 45 88
)12,23,90()32,56,64(
)222,120,188()200,89,45(
43
21
xx
xx
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
13
Scatter Diagrams
If only we could determine which class a pixel belonged to: foreground orbackground; the purple sphere or the green background, respectively.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
14
Make your own Scatter Plot
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
15
Actual Scatter Plot
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
16
Axioms of Probability
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
17
(Some) Axioms of Probability
)(1)(
1)(
0)(
1)(
1)(0
aPaP
dDP
falseP
trueP
aP
ii
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
18
Bayesian Philosophy of Choosing
Suppose two events, A and B, occur with probability P(A) and P(B) such that P(A) + P(B) = 1.
If you had to place a bet on which event would occur and if P(A) ≠ P(B), what event would you bet on?
Suppose P(A) = P(B). What event would you bet on?
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
19
Gaussian Distributions
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
20
Gaussian Distributions
)()(2
1 1
)2(
1)(
μxμxx
T
ePn
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
21
One-D Gaussian
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
22
Variations in μ
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
23
Variations in σ
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
24
2D Gaussian
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
25
Comparison of two 2D Gaussians
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
26
2D Gaussian with σx ≠ σy
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
27
3D Gaussian
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
28
Gaussian Mixture Model (GMM)
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
29
GMM Model
The number of classes is knownEach class has a particular mean and
varianceEach class has a mixing (or membership)
weight. All mixing weights add up to one
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
30
Toy Problem Parameters
ClassMixingWeights μ Σ
1 .2 (0.0 -0.1) 1.0 0.20.2 0.8
2 .4 (10.1 10.1)
2.6 1.81.8 1.9
3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
31
Generating the Toy Problem’s DataSet
Start
8.02.0
2.00.1)1.0,0.0(
9.18.1
8.16.2)1.0,0.10(
1.26.1
6.13.3)1.2,9.9(
20%
40%
40%
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
32
The GMM Problem
Start
????
??????)(??,
??%
????
??????)(??,
????
??????)(??,
??%
??%
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
33
Bayes’ Rule and the Expectation Step
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
34
Bayes’ Rule
)(
)()|()|(
AP
BPBAPABP
The likelihood of A given B
A normalization constantThe likelihood of B given A
A prior probability on B
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
35
The Main of Idea of GMMs: E-Step
)(
)()|()|(
j
jj P
iCPiCPiCP
x
xx
The probability that position j is generatedby class 1 or 2: foreground or background
The probability of position j havinga pixel with values x given class i. Basically, evaluate a Gaussian at x using μi and Σi as parameters
The prior probability ofclass i; i.e. the mixingweights
Essentially, a normalization constantto ensure that P(C=1|xj) +P(C=2|xj) = 1for j fixed.
)(
)()|()|(
AP
BPBAPABP
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
36
Update Rules and the Maximization Step
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
37
Update Rules: M-Step
records ofnumber
:
)(
)|(
:
ii
i
Tijjij
ji
i
jiji
i
ijj
i
jij
pw
p
p
p
p
Update
iCPw
pp
iCPp
Define
μxμxΣ
xμ
x
iWhat do you think is the intuitionbehind these three update rules?
In words, define what each ofthese mathematical statementsmean.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
38
Solving a Toy Problem
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
39
Toy Problem Visually
ClassMixingWeights μ Σ
1 .2 (0.0 -0.1) 1.0 0.20.2 0.8
2 .4 (10.1 10.1)
2.6 1.81.8 1.9
3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
40
Typical Output
Class 1Class 3
Class 2
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
41
Solution to the Toy Problem
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
42
Comparison of Statistics
ClassToy Problem GMM Solver
Mix μ Σ Mix μ Σ
1 .2 (0.0 -0.1) 1.0 0.20.2 0.8 .2 (0.0 -0.1) 1.0 0.2
0.2 0.8
2 .4 (10.1 10.1)
2.6 1.81.8 1.9 .4 (10.1
10.1)2.6 1.81.8 1.9
3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1 .4 (9.9 -2.1) 3.0 1.6
1.6 2.1
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
43
Recovery of the Model
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
44
3D GMM Problem
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
45
Segmentation using a GMM and Masks
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
46
Segmentation Example
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
47
Segmentation
The upshot of all of this is that the foreground has a parameterization of:
ClassMixingWeights μ Σ
Background 0.9695 (62 195
52)
137 42 159
42 149 49159 49
188
Foreground 0.0305 (122 35
140)
167 26 191
26 134 30191 30
220
The novelty of this result is that we have a complete and numericalcharacterization of the image without knowing beforehand thecolor of the background. These parameters have been learned.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
48
Producing the Mask
)2()2|()1()1|(
)1()1|()|1(
CPCPCPCP
CPCPCP
jj
jj xx
xx
The mixing weight 0.9695
The mixing weight 0.0305
A Gaussian computed using themean and variance of the foregroundclass.
A Gaussian computed using themean and variance of the backgroundclass.
How would we use a Bayesian philosophy to decide ifa pixel represents the foreground or background?
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
49
Masks
This mask permits the sphereto show through while excludingthe background. It’s used tosegment the sphere; that is,segment out the foreground.
This mask is the logical complementof the first mask. It indicates wherethe background pixels can be found.This mask is used to prepare the newbackground so that it may becombined with the foreground.
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
50
Using the Masks for the Foreground and Background
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
51
Adding the Images Together
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
52
Before and After
Before After
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
53
Further Work
Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work
Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/
54
Further Work
Complex foregrounds and backgroundsTracking objectsSpeeding up processingSamplingSmoothing the transition from background to
foreground to avoid ragged edgesUsing alternative color spaces such as Hue-
Saturation-Value (HSV) for better classification