Upload
elmer-fowler
View
216
Download
0
Embed Size (px)
Citation preview
Separating Style and Content with Bilinear Models
Joshua B. Tenenbaum, William T. Freeman
Computer Examples
Barun Singh25 Feb, 2002
PHILOSOPHY & REPRESENTATIONData contains two components: style and contentWant to represent them separately
1 1
T
I J
ij i ji j
a b
y w
a Wb
Symmetric Bilinear Model:y : observed dataa : style vectorb : content vectorI, j : components of style and contentW : matrix of basis vectors (e.g., “eigenfaces”)
Y : (SK) x C
A : (SK) x J
b : J x C
Ab
Asymmetric Bilinear Model:A : matrix of style-specific basis vectors
More flexible modelEasier to deal with
PROBLEMS TO BE SOLVED
Given a labeled training set of observations in multiple styles and content classes,
Fit asymmetric model (find A and b for known styles and contents) using SVD
1
1
2 2
Csc s c s OLC
c
SOLC s
ss
E
y A b A A
A A
Find style matrix that best explains data for incomplete style (i.e., minimizes E given below)
Extrapolate using the estimated style matrix
OLC used to solve overfitting problem Parameters involved:
= 0 : Purely asymmetric model = : Purely symmetric model
extrapolate a new style to unobserved content classes
PROBLEMS TO BE SOLVEDGiven a labeled training set of observations in multiple styles and
content classes,
Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style
Parameters: model dimensionality J, model variance 2, max number of EM iterations tmax
2
,
2Pr( | , ) exp / 2σ
Pr( ) Pr( | , ) Pr( , )
Pr( , | ) Pr( | , ) Pr( , ) / Pr( )
s c
c s
s c
s c s c
s c s c s c
y y A b
y y
y y y
*
1
1 1
choose to maximize log-likelihood
L = log Pr( )
Pr( , | )
Pr( , | )
s
C CT Ts sc c sc c c
c c
sc
sc
n
s c
n s c
y
y
y
A
y
A m b b b
m y y
y
classify content observed in a new style Fit asymmetric model
Select content class c that maximizes Pr(s’,c|y)
PROBLEMS TO BE SOLVED
Given a labeled training set of observations in multiple styles and content classes,
translate from new content observed only in new styles into known styles or content classes
Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure
1VTs c
a Wb y
1VTc VT s
b W a y
Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):
TOY EXAMPLE - introImage made of 4 pixels, each of which are either white or red. Style represents if the top or bottom rows are red or whiteContent represents if the left or right columns are red or white.
SYMMETRIC MODEL
Basis Images ( W )
Content Vectors ( b )
Sty
le V
ecto
rs (
a ) O
utp
ut Im
ag
es ( y
)
TOY EXAMPLE - intro
ASYMMETRIC MODEL
*Note: Images drawn as blocks, but represented as vectors, not matrices
Content Vectors ( b )
Sty
le-s
pecifi
c B
asis
Im
ag
es
( A
)
Ou
tpu
t Imag
es ( y
)
TOY EXAMPLE - extrapolation
?
Fitting the asymmetric model
Content Vectors ( b )
Sty
le-s
pecifi
c B
asis
Im
ag
es
( A
)
Extrapolate
FONTS EXAMPLE - extrapolation
Training Set Incomplete Style
model dim= 60
Con
ten
t (L
ett
er)
Style (Font)No prior, OLC only, best result, true letters.
FONTS EXAMPLE - extrapolationNo prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.Asymmetric ModelNo prior, OLC only, best result, true letters.
10 20 30 40 50 60
model dimension
No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.
10 20 30 40 50 60
model dimension
Symmetric Model
No prior, OLC only, best result, true letters.No prior, OLC only, best result, true letters.
Sym. W/ Asym. Prior
(dim = 60) vs. Actual
TOY EXAMPLE - classification
1: Fit asymmetric model to training set
Content Vectors ( b )
Sty
le-s
pecifi
c
Basis
Im
ag
es
( A
)
TOY EXAMPLE - classification
2: Use Separable Mixture Model w/ EM to classify
2 = 0.5
2 = 0.6
2 = 0.35
Content Vectors ( b )
Sty
le-s
pecifi
c
Basis
Im
ag
es
( A
)
Actual
Resu
lting
Im
ag
es