View
217
Download
1
Category
Preview:
Citation preview
1
Multilinear Representation of Image Ensemblesfor Recognition and Compression
Multilinear Representation of Image Ensemblesfor Recognition and Compression
TensorFaces:
The Problem with Linear (PCA) Appearance Based Recognition
Methods
The Problem with Linear (PCA) Appearance Based Recognition
Methods• Eigenimages work best for recognition when only a single
factor – e.g., object identity – is allowed to vary
• Natural images are the composite consequences of multiple factors (or modes) related to scene structure, illumination and imaging
Decomposition from Pixel Space to Factor Spaces
Decomposition from Pixel Space to Factor Spaces Multilinear Model ApproachMultilinear Model Approach
• Non- linear appearance based technique
• Appearance based model that explicitly accounts for each of the multiple factors inherent in image formation
• Multilinear algebra, the algebra of higher order tensors
• Applied to facial images, we call our tensor technique “TensorFaces” [ Vasilescu & Terzopoulos, ECCV 02, ICPR 02 ]
TensorFaces vs Eigenfaces(PCA)
TensorFaces vs Eigenfaces(PCA)
88%27%
Training: 23 people, 5 viewpoints (0,+17, -17,+34,-34), 3 illuminations
Testing: 23 people, 5 viewpoints (0,+17, -17,+34,-34), 4th illumination
80%61%
Training: 23 people, 3 viewpoints (0,+34,-34), 4 illuminations
Testing: 23 people, 2 viewpoints (+17,-17), 4 illuminations (center,left,right,left+right)
TensorFacesPCAPIE Recognition Experiment
PIE Database (Weizmann)PIE Database (Weizmann)
2
Data OrganizationData Organization
• Linear/PCA: Data Matrix– Rpixels x images
– a matrix of image vectors
• Multilinear: Data Tensor– Rpeople x views x illums x express x pixels
– N-dimensional array– 28 people, 45 images/person– 5 views, 3 illuminations,
3 expressions per person
exilvpp ,,,i
People
Illu
min
atio
nsEx
pres
sions
Views
D
D
Pixe
ls
ImagesD Tensor Decomposition
Complete Dataset
}Learning Stage
Tensor DecompositionTensor Decomposition
Views
Illu
ms.
People
Matrix Decomposition - SVDMatrix Decomposition - SVD
• A matrix has a column and row space
• SVD orthogonalizes these spaces and decomposes
• Rewrite in terms of mode-n products
2xI1IIR∈D
T21 UUD ∑=
2UUD x1x 2 1 ∑=
Pixe
ls
ImagesD
( contains the eigenfaces )1U
D
Tensor DecompositionTensor Decomposition• D is a n-dimensional matrix, comprising N-spaces
• N-mode SVD is the natural generalization of SVD
• N-mode SVD orthogonalizes these spaces & decomposes
D as the mode-n product of N-orthogonal spaces
• Z core tensor; governs interaction between mode matrices
• , mode-n matrix, is the column space of nU )(nD
nn UUUU ... x x x xx 3 21 1 432 ZD = D 321 x xx 32 1 UUU Z=
Multilinear (Tensor) DecompositionMultilinear (Tensor) Decomposition
( ) ( ) ( )ZD vecvec 123 UUU ⊗⊗=
323
32121
3211
3
1
2
1
1
1rrr
rrrr
rr,,, uuu oo∑∑∑=
===
RRR
σ
U1
U2
U3
D Z=
3
N-Mode SVD AlgorithmN-Mode SVD Algorithm
1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD.
2. Solve for the core tensor as follows
)( nDnU
nU
TNN
Tnn
TT UUUU ××××= LL 2211 DZ
Multilinear (Tensor) AlgebraMultilinear (Tensor) Algebra
mode - n tensor flattening
NIII L××ℜ∈ 21A
( )( )Nnnn IIIIII
nLL 1121 +−×ℜ∈A
flattening=
“matricize”
N = 3
pixelsxexpressxillums.xviews xpeoplex 51 UUUUU . ZD 432=
Facial Data Tensor DecompositionFacial Data Tensor DecompositionPeople
Illu
min
atio
nsEx
pres
sions
ViewsD
People
Illu
min
atio
nsEx
pres
sions
ViewsD(views)D
Computing UviewsComputing Uviews
Images
• D(views)- flatten D along the view point dimension• Uviews – orthogonalize the column space of D(views)
Pixe
ls
Images
Computing UpixelsComputing Upixels
D(pixels)
• D(pixels)- flatten D along the pixel dimension
• Upixels – orthogonalize D(pixels)
– eigenimages
N-Mode SVD AlgorithmN-Mode SVD Algorithm1. For n=1,…,N, compute matrix by computing the
SVD of the flattened matrix and setting to be the left matrix of the SVD.
2. Solve for the core tensor as follows
)( nDnU
nU
TNN
Tnn
TT UUUU ××××= LL 2211 DZ
4
I1
J1
x1
• Mode-n product is a generalization of the product of two matrices• It is the product of a tensor with a matrix
• Mode-n product of andNn III x...xx...x1ℜ∈A
I1
I2
I3
I2
J2
J2
I2
J2
I2
= x2
Nnnn IIJII x..xxx.x...x 111 +−ℜ∈B
nn IJ xℜ=M
Mn×= AB
Mode-n ProductMode-n Product
AB M x3
J3
I3
I3
J3
nnNnnn ijiiiii
niNininjniin ma ......
......A
111
111+−∑=×
+−
M
Multilinear (Tensor) AlgebraMultilinear (Tensor) Algebra
N-th order tensor NIII L××ℜ∈ 21A
nn IJ ×ℜ∈Mmatrix (2nd order tensor)
mode - n product:
Mn×= AB ( ) ( )nn AMB =where
( )44444444 344444444 21321321
matrixt coefficien
matrix basis
matrix data
expressillums.viewspeople(pixels)pixels(pixels)
T.UUUUZUD ⊗⊗⊗=
Eigenfaces vs TensorFaces Eigenfaces vs TensorFaces • Multilinear Analysis / TensorFaces:
pixelsxexpressxillums.xviews xpeoplex 51 UUUUU . ZD 432=
( ) T express U(pixels) Z viewsU⊗illums.U⊗ peopleU⊗=
321matrix data
(pixels)D321
matrix basis pixelsU
• Linear Analysis / Eigenfaces:
• TensorFaces subsumes Eigenfaces
TensorFaces:TensorFaces:
Views
Illums.
People
B = Z x5 Upixels
TensorFaces:explicitly representcovariance acrossfactors
PCA:
TensorFaces:
TensorFaces
Mean Sq. Err. = 409.153 illum + 11 people param.
33 basis vectors
PCA
Mean Sq. Err. = 85.7533 parameters
33 basis vectors
Strategic Data Compression = Perceptual Quality
Strategic Data Compression = Perceptual Quality
Original
176 basis vectors
TensorFaces
6 illum + 11 people param.66 basis vectors
• TensorFaces data reduction in illumination space primarily degrades illumination effects (cast shadows, highlights)
• PCA has lower mean square error but higher perceptual error
5
Dimensionality Reduction -Truncation
Dimensionality Reduction -Truncation
∑+∑+∑≤===
− N
NNN
I
Rii
I
Rii
I
Rii σσσ 2222 2
222
1
111
LDD
Iterative Multilinear Model -Dimensionality Reduction
Iterative Multilinear Model -Dimensionality Reduction
• Iterative data reduction approach:– Optimize mode per mode in an iterative way– Alternating Least Squares [Golub & Van Loan] improves data fit
1 - IUUU ... U ... U −Σ+×××××==
Tnnn
N
nNn Nne λ11 ZD
1. Initialize U10, U2
0, …, UN0:
– Compute U1, U2, …, UN using N-Mode SVD and – Truncate each mode matrix
2. Iterate:– U 1
t = D x2 U2t-1T
x3… xN UN
t-1T
U1t = svd( U 1
t(1) )
– U 2t = D x1 U1
tT x3 U3
t-1T … xN UNt-1T
U2t = svd( U 2
t(2) )
3. Z = D x1 U1tT
x2 U2tT
x3… xN UN
tT
Iterative Multilinear ModelIterative Multilinear Model Construction of Projection Basis
Construction of Projection Basis
• Multilinear decomposition allows for the construction of different basis depending on the application needs
• Object Specific TensorFaces: person appearance model; eigenvectors span an individuals set of images
• View Based TensorFaces
• Recognition basis – basis maps an image into people parameter space, Upeople
Person Specific TensorFacesB =Z x1 Upeop. x5 Upixels
Person Specific TensorFacesB =Z x1 Upeop. x5 Upixels
• Basis spanning one person’s set of images
Views
Illumi
natio
ns
Higher -Order Statistics
2nd -Order Statistics(covariance)
Our Nonlinear (Multilinear) Models
Linear Models
Perspective on Our Face Recognition Approach
Perspective on Our Face Recognition Approach
Multilinear PCATensorFaces
PCAEigenfaces
ICA Multilinear ICAIndependent TensorFaces
Vasilescu & Terzopoulos, Learning 2004
6
. ...
... .. .
. .....
... ..
..
.. ......... ..
pixel 1
pixe
l kl
2550
255
. ...
... .. .
. .....
... ..
..
.. ......... ..
PCA ICA
TUSVD =coefficient matrix
basis matrix
UD = TSVCK=
independent components coefficient matrix
pixel 1
pixe
l kl
2550
255
TSV TT −WW
Toy Example: Observed DataToy Example: Observed Data
Toy Example: Hidden VariablesToy Example: Hidden Variables Basis VectorsBasis Vectors
ICA
Hidden Variable RepresentationHidden Variable Representation
• MICA
• MPCA
1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD. Compute using ICA. Our new mode matrix is
2. Solve for the core tensor as follows
Tnn
Tnn VZWK )(−=
N-Mode ICAN-Mode ICA
)(nDnU
nU
11122
111 −−−− ××××××= NNnn KKKK LL DS
Tnnnn VZUD )()( = ( ) T
n(n)
Tn
Tnn VZWWU −=
TnW
TNN
Tnn
TT −−−− ××××××= WWWW 2211 LL ZS
nK
Recommended