Real-time on-line learning of transformed hidden Markov models
Nemanja Petrovic, Nebojsa Jojic, Brendan Frey and Thomas Huang
Microsoft, University of Toronto, University of Illinois
2
Six break points vs. six things in video
• Traditional video segmentation: Find breakpoints Example: MovieMaker (cut and paste)
• Our goal: Find possibly recurring scenes or objects
1 32 42 1 4 3 2 3 2 3 5 6
timeline
REPRESENTATIVE FRAMES
3
Transformed hidden Markov model
zT
x
c Class with prior P(c=k) = πk
P(z|c) = N(z;μc,Φc)
x = Tz
Ex = Tμ
Var x = TΦTT
p(x|c,T) = N(x; Tμc, TΦcTT)
Generation is repeated for each frame of the sequence, with the pair (T,c) being the state of a Markov chain.
Translation T with uniform prior
Latent image z
Observed frame x
4
Goal: maximize total likelihood of a dataset
log p(X) = log Σ{T,c} Σz p(X,{T,c},Z)
= log Σ{T,c} Σz q({T,c},Z)p(X,{T,c},Z)/q({T,c},Z)
≥ Σ{T,c} Σz q({T,c},Z)log p(X,{T,c},Z)
- Σ{T,c} Σz q({T,c},Z)log q({T,c},Z) = B
We express q(T,c,z) = q({T,c}) * q(Z|{T,c})
{T,c} represents values of transformation and class for all frames, i.e., the path that the video sequence takes through the state space of the model.
Instead of the likelihood, we optimize the bound B, which is tight for q=p({T,c},Z|X)
5
Posterior approximation
We allow q({T,c}) to have a non-zero probability only on M most probable paths:
q({T,c}) = Σm=1:M rmδ({T,c} - {T,c}*m)
(Viterbi 1982)
This reduces a number of problems with adaptive scaling in the exact forward-backward inference.
6
Expensive part of the E stepFind quick way to calculatelog p(x|c,T) = -N/2 log(2π) – ½ log|TΦcTT| - ½ (x-Tμc)T(TΦcTT)-1(x-Tμc) for all possible shifts T in the E step of EM algorithm
Shifted cluster mean Tμ
Frame x
T
log p
7
Computing Mahalanobis distance using FFTs
= Σ .* T
= sum x.*Tμ
Φ
= IFFT FFT(x) .* conj( FFT )
xT(TΦTT)-1Tμ = xTT(Φ-1μ) = xTT(diag Φ-1 .* μ)
All terms that have to be evaluated for all T can be expressed as correlations, e.g. :
μ
Φ
(where summation is over pixels)
N log N versus N2 !
8
Parameter optimization
F = ΣT Σc Σz q(T,c,z)log p(X,T,c,z)
= ΣT Σc Σz q({T,c}) * q(z|{T,c}) x
(logπ{Tc} + Σtimelog p(xt,zt|Tt,ct)
+ Σtimelog p(ct+1|ct) log p(Tt+1|Tt,ct))
Solve ∂F/∂()=0 for an estimated q.
9
On-line vs. batch EM
Example:Update equation for the class mean
Σt ΣT q(Tt,ct)E[z|xt,ct,Tt] = Σtq(ct) μct
Batch EM:– solve for μ using all frames.– Inference and parameter optimization iterated.
On-line EM:– rewrite the equation for one extra frame– establish the relationsip between μ(t+1) and μ(t). – Parameters updated after each frame. No need for iteration.
10
Reducing the complexity of the M step
ΣT q(Tt,ct)E[z|xt,ct,Tt] can be expressed as a sum of convolutions.
For example, when there is no observation noise, E[z|xt,ct,Tt ]= Tt
Txt, and
ΣT q(Tt,ct)E[z|xt,ct,Tt] = IFFT (FFT(q).* FFT(x))
(similar trick applies to variance estiamates)
11
Represent pixels on a polar grid!
Shifts in the log-polar
coordinates correspond
to scale and rotation
changes in the Cartesian
coordiante system
How to deal with scale and rotation?
rotation
scale
12
Estimating the number of classes
• The algorithm is initialized with a single class
• A new class is introduced whenever the frame likelihood drops bellow a threshold
• The classes can be merged in the end to achieve a more compact representation
13
Clustering a 20-minute whale watching video
Clustering a 20-minute beach video
15
0 min 9 min
Shots from the first class
16
Discovering objects using motion priors
Different motion prior predefined for each of the classes
Three characteristic frames from 240x320 input sequence
Learned means and variances
17
Tracking results
18
Summary
Before - CVPR 99/00
28x44 images
Grayscale images
1 day of computation for 15 sec video
Batch EM
Exact inference
Fixed number of clusters
Limited number of translations
Memory inefficient
Now
120x160 images
Full color images
5-10 frames/sec
On-line EM
Approximate inference
Variable number of clusters
All possible translations
Memory efficient
19
Sneak preview: Panoramic THMMs
zT
x
c P(c=k) = πk
P(z|c) = N(z;μc,Φc)
x = WTz
Ex = WTμ
Var x = WTΦTTWT
p(x|c,T) = N(x; WTμc, WTΦcTTWT)
WT
20
Video clustering - model
• Appearance meanvariance
• Camera/object motion
• Temporal constraints
• Unsupervised learning – the only input is the video
21
Current implementation
• DirectShow filter for frame clustering (5-15 frames/sec!)
• Translation invariance• On-line learning algorithm• Classes repeating across video• Potential applications:
– Video segmentation– Content based search/retrieval– Short video summary creation– DVD chapter creation
22
Comparing with layered sprites
“Perfect” segmentation
Layered sprites. Jojic, CVPR 01
But, THMM is hundreds/thousands of time faster!
Example with more content