Probabilistic Visual Learning for Object Representation - … · 696 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 Probabilistic Visual

696 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997

Probabilistic Visual Learningfor Object Representation

Baback Moghaddam, Student Member, IEEE, and Alex Pentland, Member, IEEE

Abstract —We present an unsupervised technique for visual learning, which is based on density estimation in high-dimensionalspaces using an eigenspace decomposition. Two types of density estimates are derived for modeling the training data: amultivariate Gaussian (for unimodal distributions) and a Mixture-of-Gaussians model (for multimodal distributions). These probabilitydensities are then used to formulate a maximum-likelihood estimation framework for visual search and target detection for automaticobject recognition and coding. Our learning technique is applied to the probabilistic visual modeling, detection, recognition, andcoding of human faces and nonrigid objects, such as hands.

Index Terms —Face recognition, gesture recognition, target detection, subspace methods, maximum-likelihood, density estimation,principal component analysis, Eigenfaces.

—————————— ✦ ——————————

1 INTRODUCTION

ISUAL attention is the process of restricting higher-levelprocessing to a subset of the visual field, referred to as

the focus-of-attention (FOA). The critical component of visualattention is the selection of the FOA. In humans, this processis not based purely on bottom-up processing and is, in fact,goal-driven. The measure of interest or saliency is modu-lated by the behavioral state and the demands of the par-ticular visual task that is currently active.

Palmer [25] has suggested that visual attention is theprocess of locating the object of interest and placing it in acanonical (or object-centered) reference frame suitable forrecognition (or template matching). We have developed acomputational technique for automatic object recognition,which is in accordance with Palmer’s model of visual at-tention (see Section 4.1). The system uses a probabilisticformulation for the estimation of the position and scale ofthe object in the visual field and remaps the FOA to an ob-ject-centered reference frame, which is subsequently usedfor recognition and verification.

At a simple level, the underlying mechanism of attentionduring a visual search task can be based on a spatiotopicsaliency map S(i, j), which is a function of the image infor-mation in a local region R

S(i, j) = f[{I(i + r, j + c) : (r, c) Œ R}] (1)

For example, saliency maps have been constructed whichemploy spatio-temporal changes as cues for foveation [1] orother low-level image features, such as local symmetry fordetection of interest points [32]. However, bottom-up tech-niques based on low-level features lack context with respectto high-level visual tasks, such as object recognition. In arecognition task, the selection of the FOA is driven by

higher-level goals and, therefore, requires internal repre-sentations of an object’s appearance and a means of com-paring candidate objects in the FOA to the stored objectmodels.

In view-based recognition (as opposed to 3D geometricor invariant-based recognition), the saliency can be formu-lated in terms of visual similarity, using a variety of metricsranging from simple template matching scores to more so-phisticated measures using, for example, robust statisticsfor image correlation [5]. In this paper, however, we areprimarily interested in saliency maps which have a prob-abilistic interpretation as object-class membership functionsor likelihoods. These likelihood functions are learned by ap-plying density estimation techniques in complementarysubspaces obtained by an eigenvector decomposition. Ourapproach to this learning problem is view-based—i.e., thelearning and modeling of the visual appearance of the ob-ject from a (suitably normalized and preprocessed) set oftraining imagery. Fig. 1 shows examples of the automaticselection of FOA for detection of faces and hands. In eachcase, the target object’s probability distribution was learnedfrom training views and then subsequently used in com-puting likelihoods for detection. The face representation isbased on appearance (normalized grayscale image),whereas the hand’s representation is based on the shape ofits contour. The maximum likelihood (ML) estimates ofposition and scale are shown in the figure by the cross-hairsand bounding box, respectively.

1.1 Object DetectionThe standard detection paradigm in image processing isthat of normalized correlation or template matching. How-ever, this approach is only optimal in the simplistic case ofa deterministic signal embedded in additive white Gaussiannoise. When we begin to consider a target class detectionproblem—e.g., finding a generic human face or a humanhand in a scene—we must incorporate the underlyingprobability distribution of the object. Subspace methods

0162-8828/97/$10.00 © 1997 IEEE

————————————————

• The authors are with the Perceptual Computing Section, The Media Labo-ratory, Massachusetts Institute of Technology, 20 Ames Street, Cambridge,MA 02139. E-mail: {baback, sandy}@media.mit.edu.

Manuscript received 3 Nov. 1995. Recommended for acceptance by J. Daugman.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 105026.

V

MOGHADDAM AND PENTLAND: PROBABILISTIC VISUAL LEARNING FOR OBJECT REPRESENTATION 697

and eigenspace decompositions are particularly well-suitedto such a task since they provide a compact and parametricdescription of the object’s appearance and also automati-cally identify the degrees-of-freedom of the underlying statis-tical variability.

In particular, the eigenspace formulation leads to a pow-erful alternative to standard detection techniques such astemplate matching or normalized correlation. The recon-struction error (or residual) of the eigenspace decomposi-tion (referred to as the “distance-from-face-space” in thecontext of the work with “eigenfaces” [36]) is an effectiveindicator of similarity. The residual error is easily com-puted using the projection coefficients and the original sig-nal energy. This detection strategy is equivalent to match-ing with a linear combination of eigentemplates and allowsfor a greater range of distortions in the input signal(including lighting, and moderate rotation and scale). In astatistical signal detection framework, the use of eigentem-plates has been shown to yield superior performance incomparison with standard matched filtering [18], [27].

In [27], we used this formulation for a modular eigenspacerepresentation of facial features where the correspondingresidual—referred to as “distance-from-feature-space” orDFFS—was used for localization and detection. Given aninput image, a saliency map was constructed by computingthe DFFS at each pixel. When using M eigenvectors, this re-quires M convolutions (which can be efficiently computedusing an FFT) plus an additional local energy computation.The global minimum of this distance map was then selectedas the best estimate of the location of the target.

In this paper, we will show that the DFFS can be inter-preted as an estimate of a marginal component of the prob-ability density of the object and that a complete estimatemust also incorporate a second marginal density based on acomplementary “distance-in-feature-space” (DIFS). Usingour estimates of the object densities, we formulate the

problem of target detection from the point of view of amaximum likelihood (ML) estimation problem. Specifically,given the visual field, we estimate the position (and scale)of the image region which is most representative of the tar-get of interest. Computationally, this is achieved by slidingan m-by-n observation window throughout the image andat each location computing the likelihood that the local su-bimage x is an instance of the target class W—i.e., P(x|W).After this probability map is computed, we select the loca-tion corresponding to the highest likelihood as our ML es-timate of the target location. Note that the likelihood mapcan be evaluated over the entire parameter space affectingthe object’s appearance, which can include transformationssuch as scale and rotation.

1.2 Relationship to Previous ResearchIn recent years, computer vision research has witnessed agrowing interest in eigenvector analysis and subspace de-composition methods. In particular, eigenvector decompo-sition has been shown to be an effective tool for solvingproblems which use high-dimensional representations ofphenomena which are intrinsically low-dimensional. Thisgeneral analysis framework lends itself to several closelyrelated formulations in object modeling and recognition,which employ the principal modes or characteristic degrees-of-freedom for description. The identification and parametricrepresentation of a system in terms of these principalmodes is at the core of recent advances in physically-basedmodeling [26], correspondence and matching [34], andparametric descriptions of shape [7].

Eigenvector-based methods also form the basis for dataanalysis techniques in pattern recognition and statisticswhere they are used to extract low-dimensional subspacescomprised of statistically uncorrelated variables which tendto simplify tasks such as classification. The Karhunen-Loeve Transform (KLT) [19] and Principal ComponentsAnalysis (PCA) [14] are examples of eigenvector-basedtechniques which are commonly used for dimensionalityreduction and feature extraction in pattern recognition.

In computer vision, eigenvector analysis of imagery hasbeen used for characterization of human faces [17] andautomatic face recognition using “eigenfaces” [36], [27].More recently, principal component analysis of imagery hasalso been applied for robust target detection [27], [6], non-linear image interpolation [3], visual learning for object rec-ognition [23], [38], as well as visual servoing for robotics [24].

Specifically, Murase and Nayar [23] used a low-dimensional parametric eigenspace for recovering object iden-tity and pose by matching views to a spline-based hypersur-face. Nayar et al. [24] have extended this technique to visualfeedback control and servoing for a robotic arm in “peg-in-the-hole” insertion tasks. Pentland et al. [27] proposed a view-based multiple-eigenspace technique for face recognition un-der varying pose as well as for the detection and description offacial features. Similarly, Burl et al. [6] used Bayesian classifi-cation for object detection using a feature vector derived fromprincipal component images. Weng [38] has proposed a visuallearning framework based on the KLT in conjunction with anoptimal linear discriminant transform for learning and recog-nition of objects from 2D views.

(a) (b)

(c) (d)

Fig. 1. (a) input image, (b) face detection, (c) input image, (d) handdetection.


However, these authors (with the exception of [27]) haveused eigenvector analysis primarily as a dimensionalityreduction technique for subsequent modeling, interpola-tion, or classification. In contrast, our method uses an ei-genspace decomposition as an integral part of an efficienttechnique for probability density estimation of high-dimensional data.

2 DENSITY ESTIMATION IN EIGENSPACE

In this section, we present our recent work using ei-genspace decompositions for object representation andmodeling. Our learning method estimates the completeprobability distribution of the object’s appearance using aneigenvector decomposition of the image space. The desiredtarget density is decomposed into two components: thedensity in the principal subspace (containing the tradition-ally-defined principal components) and its orthogonalcomplement (which is usually discarded in standard PCA).We derive the form for an optimal density estimate for thecase of Gaussian data and a near-optimal estimator for ar-bitrarily complex distributions in terms of a Mixture-of-Gaussians density model.

We note that this learning method differs from supervisedvisual learning with function approximation networks [30]in which a hypersurface representation of an input/outputmap is automatically learned from a set of training exam-ples. Instead, we use a probabilistic formulation whichcombines the two standard paradigms of unsupervisedlearning—PCA and density estimation—to arrive at a com-putationally feasible estimate of the class conditional den-sity function.

Specifically, given a set of training images { }xttNT=1 , from

an object class W, we wish to estimate the class membershipor likelihood function for this data—i.e., P(x|W). In this sec-tion, we examine two density estimation techniques forvisual learning of high-dimensional data. The first methodis based on the assumption of a Gaussian distribution whilethe second method generalizes to arbitrarily complex dis-tributions using a Mixture-of-Gaussians density model.Before introducing these estimators, we briefly review ei-genvector decomposition as commonly used in PCA.

2.2 Principal Component Imagery

Given a training set of m-by-n images { }I ttNT=1 , we can form a

training set of vectors {xt}, where x Œ 5N=mn, by lexico-

graphic ordering of the pixel elements of each image It. Thebasis functions for the KLT [19] are obtained by solving theeigenvalue problem

L = FTSF (2)

where S is the covariance matrix, F is the eigenvector ma-trix of S, and L is the corresponding diagonal matrix of ei-genvalues. The unitary matrix F defines a coordinate trans-form (rotation) which decorrelates the data and makes ex-plicit the invariant subspaces of the matrix operator S. InPCA, a partial KLT is performed to identify the largest-eigenvalue eigenvectors and obtain a principal component

feature vector y x= FMT ~, where ~x x x= - is the mean-

normalized image vector and FM is a submatrix of F con-taining the principal eigenvectors. PCA can be seen as alinear transformation y = 7(x) : 5N Æ 5M which extracts alower-dimensional subspace of the KL basis correspondingto the maximal eigenvalues. These principal componentspreserve the major linear correlations in the data and dis-card the minor ones.1

By ranking the eigenvectors of the KL expansion with re-spect to their eigenvalues and selecting the first M principalcomponents, we form an orthogonal decomposition of thevector space 5N into two mutually exclusive and comple-mentary subspaces: the principal subspace (or featurespace) F i i

M= ={ }F 1 containing the principal components and

its orthogonal complement F i i MN= = +{ }F 1. This orthogonal

decomposition is illustrated in Fig. 2a, where we have aprototypical example of a distribution which is embeddedentirely in F. In practice there is always a signal componentin F due to the minor statistical variabilities in the data orsimply due to the observation noise which affects everyelement of x.

(a)

(b)

Fig. 2. (a) Decomposition into the principal subspace F and its or-thogonal complement F for a Gaussian density. (b) A typical eigen-value spectrum and its division into the two orthogonal subspaces.

1. In practice, the number of training images NT is far less than the di-mensionality of the imagery N, consequently, the covariance matrix S issingular. However, the first M < NT eigenvectors can always be computed(estimated) from Nt samples using, for example, a Singular Value Decom-position [12].


In a partial KL expansion, the residual reconstruction er-ror is defined as

e2 2

1

2 2

1

x xa f = = -= + =Â Ây yi

i M

N

ii

M~ (3)

and can be easily computed from the first M principal com-ponents and the L2 norm of the mean-normalized image ~x .Consequently the L2 norm of every element x Œ 5N can bedecomposed in terms of its projections in these two sub-spaces. We refer to the component in the orthogonal sub-space F as the “distance-from-feature-space” (DFFS) whichis a simple Euclidean distance and is equivalent to the re-sidual error e2(x) in (3). The component of x which lies inthe feature space F is referred to as the “distance-in-feature-space” (DIFS) but is generally not a distance-based norm,but can be interpreted in terms of the probability distribu-tion of y in F.

2.2 Gaussian DensitiesWe begin by considering an optimal approach for estimat-ing high-dimensional Gaussian densities. We assume thatwe have (robustly) estimated the mean x and covariance Sof the distribution from the given training set {xt}.2 Underthis assumption, the likelihood of an input pattern x isgiven by

P

T

Nxx x x x

WS

Sd i

a f a fa f=

- - -LNM

OQP

-exp12

2

1

2 1 2p (4)

The sufficient statistic for characterizing this likelihood isthe Mahalanobis distance

d Tx x xa f = -~ ~S 1 (5)

where ~x x x= - . However, instead of evaluating this quad-ratic product explicitly, a much more efficient and robustcomputation can be performed, especially with regard to

the matrix inverse S-1. Using the eigenvectors and eigen-

values of S we can rewrite S-1 in the diagonalized form

d T

T T

x x x

x x

a f =

=

-

-

~ ~

~ ~S

FL F

1

1

= -y yTL 1 (6)

where y x= FT~ are the new variables obtained by thechange of coordinates in a KLT. Because of the diagonal-ized form, the Mahalanobis distance can also be expressed interms of the sum

dyi

ii

N

xa f ==Â

2

1l (7)

In the KLT basis, the Mahalanobis distance in (5) is conven-iently decoupled into a weighted sum of uncorrelated com-ponent energies. Furthermore, the likelihood becomes a

2. In practice, a full rank N-dimensional covariance S can not be esti-mated from NT independent observations when NT < N. But, as we shallsee, our estimator does not require the full covariance, but only its first Mprincipal eigenvectors where M < NT.

product of independent separable Gaussian densities. De-spite its simpler form, evaluation of (7) is still computation-ally infeasible due to the high-dimensionality. We thereforeseek to estimate d(x) using only M projections. Intuitively, anobvious choice for a lower-dimensional representation isthe principal subspace indicated by PCA, which capturesthe major degrees of statistical variability in the data.3

Therefore, we divide the summation into two independentparts corresponding to the principal subspace F i i

M= ={ }F 1

and its orthogonal complement F i i MN= = +{ }F 1

dy yi

ii

Mi

ii M

N

xa f = += = +Â Â

2

1

2

1l l (8)

We note that the terms in the first summation can be com-puted by projecting x onto the M-dimensional principalsubspace F. The remaining terms in the second sum{ }yi i M

N= +1, however, cannot be computed explicitly in prac-

tice because of the high-dimensionality. However, the sumof these terms is available and is in fact the DFFS quantitye

2(x) which can be computed from (3). Therefore, based onthe available terms, we can formulate an estimator for d(x)as follows

$dy

yi

ii

M

ii M

N

xa f = +LNMM

OQPP= = +

Â Â2

1

2

1

1l r

= +=Â yi

ii

M 2

1

2

l re xa f

(9)

where the term in the brackets is e2(x), which as we haveseen can be computed using the first M principal compo-nents. We can therefore write the form of the likelihoodestimate based on $( )d x as the product of two marginal andindependent Gaussian densities

$

exp exp

P

yi

ii

M

Mi

i

M N Mx

x

Wd ia f

a f

b ga f=

-FHG

IKJ

L

N

MMMMMM

O

Q

PPPPPP

◊

-FHG

IKJ

L

N

MMMMMM

O

Q

PPPPPP

=

=

-

Â

’

12

2

2

2

2

1

2 1 2

1

2

2

l

p l

r

pr

e

= P PF Fx xW Wd i d i$ (10)

where PF(x|W) is the true marginal density in F-space and$PF x Wd i is the estimated marginal density in the orthogo-

nal complement F -space. The optimal value of r can nowbe determined by minimizing a suitable cost function J(r).From an information-theoretic point of view, this cost func-tion should be the Kullback-Leibler divergence or relative

entropy [9] between the true density P(x|W) and its estimate$P x Wd i

3. We will see shortly that, given the typical eigenvalue spectra observedin practice (e.g., Fig. 2b), this choice is optimal for a different reason: Itminimizes the information-theoretic divergence between the true densityand our estimate of it.


J PP

Pd

P

Prb g d i d i

d id id i

= =L

NMM

O

QPPz x

x

xx

x

xW

W

W

W

Wlog

$log

$E (11)

Using the diagonalized forms of the Mahalanobis distanced(x) and its estimate $( )d x and the fact that E[ ]yi i

2 = l , it canbe easily shown that

J i

ii M

N

rlr

rlb g = - +

LNM

OQP= +

Â12 1

1

log (12)

The optimal weight r* can be then found by minimizingthis cost function with respect to r. Solving the equation∂∂r

J = 0 yields

r l* = -= +Â1

1N M i

i M

N

(13)

which is simply the arithmetic average of the eigenvalues inthe orthogonal subspace F .4 In addition to its optimality, r*also results in an unbiased estimate of the Mahalanobis dis-tance—i.e., E E[ $( ; *)] [ ( )]d dx xr = . This derivation shows thatonce we select the M-dimensional principal subspace F (asindicated, for example, by PCA), the optimal estimate of thesufficient statistic $( )d x will have the form of (9) with rgiven by (13). In actual practice of course, we only have thefirst NT - 1 of the eigenvalues in Fig. 2b and, consequently,the remainder of the spectrum must be estimated by fittinga (typically 1/f) function to the available eigenvalues, usingthe fact that the very last eigenvalue is simply the estimatedpixel noise variance. The value of r* can then be estimatedfrom the extrapolated portion of the spectrum.

It is interesting to consider the minimal cost J(r*)

Jii M

N

rrl* log

*b g == +Â1

21

(14)

from the point of view of the F -space eigenvalues {li : i =

M + 1, �, N}. It is easy to show that J(r*) is minimizedwhen the the F -space eigenvalues have the least spreadabout their mean r*. This suggests a strategy for selectingthe principal subspace: choose F such that the eigenvaluesassociated with its orthogonal complement F have the leastabsolute deviation about their mean. In practice, the higher-order eigenvalues typically decay and stabilize near theobservation noise variance. Therefore, this strategy is usu-ally consistent with the standard PCA practice of discard-ing the higher-order components since these tend to corre-spond to the “flattest” portion of the eigenvalue spectrum(see Fig. 2b). In the limit, as the F -space eigenvalues be-come exactly equal, the divergence J(r*) will be zero andour density estimate $P x Wd i approaches the true density

P(x|W).We note that, in most applications, it is customary to

4. Cootes et al. [8] have used a similar decomposition of the Mahalanobis

distance but instead use an ad-hoc parameter value of r l= +12 1M as an

approximation.

simply discard the F -space component and simply workwith PF(x|W). However, the use of the DFFS metric or

equivalently the marginal density PF x Wd i is critically im-

portant in formulating the likelihood of an observation x—especially in an object detection task—since there are aninfinity of vectors which are not members of W which canhave likely F-space projections. Without PF x Wd i a detec-

tion system can result in a significant number of falsealarms.

2.3 Multimodal DensitiesIn the previous section we assumed that probability densityof the training images was Gaussian. This lead to a likeli-hood estimate in the form of a product of two independentmultivariate Gaussian distributions (or equivalently thesum of two Mahalanobis distances: DIFS + DFFS). In ourexperience, the distribution of samples in the feature spaceis often accurately modeled by a single Gaussian distribu-tion. This is especially true in cases where the training im-ages are accurately aligned views of similar objects seenfrom a standard view (e.g., aligned frontal views of humanfaces at the same scale and orientation). However, when thetraining set represents multiple views or multiple objectsunder varying illumination conditions, the distribution oftraining views in F-space is no longer unimodal. In fact thetraining data tends to lie on complex and nonseparablelow-dimensional manifolds in image space. One way totackle this multimodality is to build a view-based (or ob-ject-based) formulation where separate eigenspaces areused for each view [27]. Another approach is to capture thecomplexity of these manifolds in a universal or parametriceigenspace using splines [23], or local basis functions [3].

If we assume that the F -space components are Gaussianand independent of the principal features in F (this wouldbe true in the case of pure observation noise in F ) we canstill use the separable form of the density estimate $P x Wd iin (10) where PF(x|W) is now an arbitrary density P(y) in theprincipal component vector y. Fig. 3 illustrates the decom-position, where the DFFS is the residual e2(x) as before. TheDIFS, however, is no longer a simple Mahalanobis distancebut can nevertheless be interpreted as a “distance” by re-lating it to P(y)—e.g., as DIFS = -log P(y).

Fig. 3. Decomposition into the principal subspace F and its orthogonalcomplement F for an arbitrary density.


The density P(y) can be estimated using a parametricmixture model. Specifically, we can model arbitrarily com-plex densities using a Mixture-of-Gaussians

P gi i ii

Nc

y yQ Sd i c h==Âp m; ,

1

(15)

where g(y; m, S) is an M-dimensional Gaussian density withmean vector m and covariance S, and the pi are the mixing pa-

rameters of the components, satisfying Âpi = 1. The mixture is

completely specified by the parameter Q S= ={ , , }p mi i i iNc

1 .

Given a training set { }yttNT=1, the mixture parameters can be

estimated using the ML principle

Q Q* arg max=LNMM

OQPP=

’ P t

t

NT

ye j1

(16)

This estimation problem is best solved using the Expecta-tion-Maximization (EM) algorithm [11], which consists ofthe following two-step iterative procedure:

• E-step:

h tg

gik i

k tik

ik

jk t

jk

ik

j

Nca f e j

e j=

=Â

p m

p m

y

y

; ,

; ,

S

S1

(17)

• M-step:

p ik

ik

t

N

ik

t

N

i

N

h t

h t

T

Tc

+ =

==

=Â

ÂÂ1 1

11

a f

a f(18)

m ik

ik t

t

N

ik

t

N

h t

h t

T

T

+ =

=

=Â

Â1 1

1

a f

a f

y

(19)

Sik

ik t

ik t

ik T

t

N

ik

t

N

h t

h t

T

T

+

+ +

=

=

=- -Â

Â1

1 1

1

1

a fe je j

a f

y ym m (20)

The E-step computes the a posteriori probabilities hi(t)which are the expectations of “missing” component labelszi(t) = {0, 1}, which denote the membership of yt in the ithcomponent. Once these expectations have been computed,the M-step maximizes the joint likelihood of the data andthe “missing” variables zi(t). The EM algorithm is mono-tonically convergent in likelihood and is thus guaranteed tofind a local maximum in the total likelihood of the trainingset. Further details of the EM algorithm for estimation ofmixture densities can be found in [31].

Given our operating assumptions—that the training datais M-dimensional (at most) and resides solely in the princi-pal subspace F with the exception of perturbations due towhite Gaussian measurement noise, or, equivalently, that

the F -space component of the data is itself a separableGaussian density—the estimate of the complete likelihoodfunction P(x|W) is given by

$ * $P P PFx y xW Q Wd i d i d i= (21)

where $PF x Wd i is a Gaussian component density based on

the DFFS, as before.

3 MAXIMUM LIKELIHOOD DETECTION

The density estimate $P x Wd i can be used to compute a local

measure of target saliency at each spatial position (i, j) in aninput image based on the vector x obtained by the lexico-graphic ordering of the pixel values in a local neighborhood R

S i j P, ; $W Wb g d i= x (22)

for x = Ø [{I(i + r, j + c) : (r, c) Œ R}], where Ø[•] is the op-erator which converts a subimage into a vector. The MLestimate of position of the target W is then given by

(i, j)ML = argmax S(i, j; W) (23)

This ML formulation can be extended to estimate objectscale with multiscale saliency maps. The likelihood compu-tation is performed (in parallel) on linearly scaled versionsof the input image Ik for a predetermined set of scales. TheML estimate of the spatial and scale indices is then definedby

(i, j, k)ML = argmax S(i, j, k ; W) (24)

4 APPLICATIONS

The above ML detection technique has been tested in thedetection of complex natural objects including human faces,facial features (e.g., eyes), and nonrigid articulated objectssuch as human hands. In this section, we will present sev-eral examples from these application domains.

4.1 Face Detection, Coding, and RecognitionOver the years, various strategies for facial feature detec-tion have been proposed, ranging from edge map projec-tions [15], to more recent techniques using generalizedsymmetry operators [32] and multilayer perceptrons [37].In any robust face processing system, this task is criticallyimportant, since a face must first be geometrically normal-ized by aligning its features with those of a stored modelbefore recognition can be attempted.

The eigentemplate approach to the detection of facialfeatures in “mugshots” was proposed in [27], where theDFFS metric was shown to be superior to standard tem-plate matching for target detection. The detection task wasthe estimation of the position of facial features (the left andright eyes, the tip of the nose and the center of the mouth)in frontal view photographs of faces at fixed scale. Fig. 4shows examples of facial feature training templates and theresulting detections on the MIT Media Laboratory’s data-base of 7,562 “mugshots.”


(a) (b)

Fig. 4. (a) Examples of facial feature training templates. (b) The result-ing typical detections.

We have compared the detection performance of threedifferent detectors on approximately 7,000 test images fromthis database: a sum-of-square-differences (SSD) detectorbased on the average facial feature (in this case the left eye),an eigentemplate or DFFS detector and a ML detector basedon S(i, j; W) as defined in Section 2.2. Fig. 5a shows the re-ceiver operating characteristic (ROC) curves for these de-tectors, obtained by varying the detection threshold inde-pendently for each detector. The DFFS and ML detectorswere computed based on a five-dimensional principal sub-space. Since the projection coefficients were unimodal, aGaussian distribution was used in modeling the true distri-bution for the ML detector as in Section 2.2. Note that theML detector exhibits the best detection vs. false-alarmtradeoff and yields the highest detection rate (95 percent).Indeed, at the same detection rate, the ML detector has afalse-alarm rate which is nearly two orders of magnitudelower than the SSD.

Fig. 5b provides the geometric intuition regarding theoperation of these detectors. The SSD detector’s threshold isbased on the radial distance between the average template(the origin of this space) and the input pattern. This leads tohyperspherical detection regions about the origin. In con-trast, the DFFS detector measures the orthogonal distanceto F, thus forming planar acceptance regions about F. Con-sequently, to accept valid object patterns in W, which arevery different from the mean, the SSD detector must oper-ate with high thresholds which result in many false alarms.However, the DFFS detector can not discriminate betweenthe object class W and non-W patterns in F. The solution isprovided by the ML detector which incorporates both theF -space component (DFFS) and the F-space likelihood(DIFS). The probabilistic interpretation of Fig. 5b is as fol-lows: SSD assumes a single prototype (the mean) in addi-tive white Gaussian noise, whereas the DFFS assumes auniform density in F. The ML detector, on the other hand,uses the complete probability density for detection.

We have incorporated and tested the multiscale version ofthe ML detection technique in a face detection task. Thismultiscale head finder was tested on the FERET databasewhere 97 percent of 2,000 faces were correctly detected. Fig. 6shows examples of the ML estimate of the position and scaleon these images. The multiscale saliency maps S(i, j, k; W)were computed based on the likelihood estimate $P x Wd i in

a 10-dimensional principal subspace using a Gaussianmodel (Section 2.2). Note that this detector is able to local-ize the position and scale of the head despite variations inhair style and hair color, as well as presence of sunglasses.Illumination invariance was obtained by normalizing theinput subimage x to a zero-mean unit-norm vector.

We have also used the multiscale version of the ML de-tector as the attentional component of an automatic systemfor recognition and model-based coding of faces. The blockdiagram of this system is shown in Fig. 7, which consists ofa two-stage object detection and alignment stage, a contrastnormalization stage, and a feature extraction stage whoseoutput is used for both recognition and coding. Fig. 8 illus-trates the operation of the detection and alignment stage ona natural test image containing a human face. The functionof the face finder is to locate regions in the image whichhave a high likelihood of containing a face.

(a)

(b)

Fig. 5. (a) Detection performance of an SSD, DFFS, and an ML detec-tor. (b) Geometric interpretation of the detectors.

Fig. 6. Examples of multiscale face detection.


(a) (b)

(c) (d)

Fig. 8. (a) Original image. (b) Position and scale estimate. (c) Normal-ized head image. (d) Position of facial feature.

The first step in this process is illustrated in Fig. 8b,where the ML estimate of the position and scale of the faceare indicated by the cross-hairs and bounding box. Oncethese regions have been identified, the estimated scale andposition are used to normalize for translation and scale,yielding a standard “head-in-the-box” format image(Fig. 8c). A second feature detection stage operates at thisfixed scale to estimate the position of four facial features:the left and right eyes, the tip of the nose, and the center ofthe mouth (Fig. 8d). Once the facial features have been de-tected, the face image is warped to align the geometry and

shape of the face with that of a canonical model. Then thefacial region is extracted (by applying a fixed mask) andsubsequently normalized for contrast. The geometricallyaligned and normalized image (shown in Fig. 9a) is thenprojected onto a custom set of eigenfaces to obtain a featurevector which is then used for recognition purposes as wellas facial image coding.

Fig. 9 shows the normalized facial image extracted fromFig. 8d, its reconstruction using a 100-dimensional ei-genspace representation (requiring only 85 bytes to encode)and a comparable nonparametric reconstruction obtainedusing a standard transform-coding approach for imagecompression (requiring 530 bytes to encode). This exampleillustrates that the eigenface representation used for recog-nition is also an effective model-based representation for datacompression. The first eight eigenfaces used for this repre-sentation are shown in Fig. 10.

Fig. 10. The first eight eigenfaces.

Fig. 11 shows the results of a similarity search in an im-age database tool called Photobook [28]. Each face in thedatabase was automatically detected and aligned by theface processing system in Fig. 7. The normalized faces werethen projected onto a 100-dimensional eigenspace. The im-age in the upper left is the one searched on and the remain-der are the ranked nearest neighbors in the FERET data-base. The top three matches in this case are images of thesame person taken a month apart and at different scales.The recognition accuracy (defined as the percent correctrank-one matches) on a database of 155 individuals is 99percent [21]. More recently, in the September 1996 FERETface recognition competition, an improved Bayesianmatching version of our system [22] achieved a 95 percentrecognition rate with a database of approximately 1,200individuals, the highest recognition rate obtained in thecompetition [29].

In order to have an estimate of the recognition perform-ance on much larger databases, we have conducted ourown tests on a database of 7,562 images of approximately3,000 people. The images were collected in a small booth ata Boston photography show, and include men, women, and

Fig. 7. The face processing system.

(a) (b) (c)

Fig. 9. (a) Aligned face. (b) Eigenspace reconstruction (85 bytes).(c) JPEG, reconstruction (530 bytes).


children of all ages and races. Head position was controlledby asking people to take their own picture when they werelined up with the camera. Two LEDs placed at the bottomof holes adjacent to the camera allowed them to judge theiralignment; when they could see both LEDs, then they werecorrectly aligned.

The eigenfaces for this database were approximated us-ing a principal components analysis on a representativesample of 128 faces. Recognition and matching was subse-quently performed using the first 20 eigenvectors.

To assess the average recognition rate, 200 faces wereselected at random, and a nearest-neighbor rule was usedto find the most-similar face from the entire database. If themost-similar face was of the same person, then a correctrecognition was scored. In this experiment, the eigenvector-based recognition system produced a recognition accuracyof 95 percent. This performance is somewhat surprising,because the database contains wide variations in expres-sion, and has relatively weak control of head position andillumination. In a verification task, our system yielded afalse rejection rate of 1.5 percent at a false acceptance rate of0.01 percent.

4.2 View-Based RecognitionThe problem of face recognition under general viewingconditions (change in pose) can also be approached usingan eigenspace formulation. There are essentially two waysof approaching this problem using an eigenspace frame-work. Given N individuals under M different views, onecan do recognition and pose estimation in a universal ei-genspace computed from the combination of NM images.In this way, a single “parametric eigenspace” will encodeboth identity as well as pose. Such an approach, for exam-ple, has recently been used by Murase and Nayar [23] forgeneral 3D object recognition.

Alternatively, given N individuals under M differentviews, we can build a “view-based” set of M distinct ei-genspaces, each capturing the variation of the N individualsin a common view. The view-based eigenspace is essen-tially an extension of the eigenface technique to multiple

sets of eigenvectors, one for each combination of scale andorientation. One can view this architecture as a set of par-allel “observers,” each trying to explain the image data withtheir set of eigenvectors (see also Darrell and Pentland [10]).In this view-based, multiple-observer approach, the firststep is to determine the location and orientation of the tar-get object by selecting the eigenspace which best describesthe input image. This can be accomplished by calculatingthe likelihood estimate using each viewspace’s eigenvectorsand then selecting the maximum.

The key difference between the view-based andparametric representations can be understood by consid-ering the geometry of facespace. In the high-dimensionalvector space of an input image, multiple-orientation train-ing images are represented by a set of M distinct regions,each defined by the scatter of N individuals. Multiple viewsof a face form nonconvex (yet connected) regions in imagespace [2]. Therefore, the resulting ensemble is a highlycomplex and nonseparable manifold.

The parametric eigenspace attempts to describe this en-semble with a projection onto a single low-dimensionallinear subspace (corresponding to the first n eigenvectors ofthe NM training images). In contrast, the view-based ap-proach corresponds to M independent subspaces, each de-scribing a particular region of the facespace (correspondingto a particular view of a face). The relevant analogy here isthat of modeling a complex distribution by a single clustermodel or by the union of several component clusters. Natu-rally, the latter (view-based) representation can yield amore accurate representation of the underlying geometry.

This difference in representation becomes evident whenconsidering the quality of reconstructed images using thetwo different methods. Fig. 13 compares reconstructionsobtained with the two methods when trained on images offaces at multiple orientations. In Fig. 13a, we see first animage in the training set, followed by reconstructions ofthis image using, first, the parametric eigenspace, and then,the view-based eigenspace. Note that in the parametric re-construction, neither the pose nor the identity of the indi-vidual is adequately captured. The view-based reconstruc-tion, on the other hand, provides a much better characteri-zation of the object. Similarly, in Fig. 13b, we see a novelview (+68°) with respect to the training set (-90° to +45°).Here, both reconstructions correspond to the nearest viewin the training set (+45°), but the view-based reconstructionis seen to be more representative of the individual’s iden-tity. Although the quality of the reconstruction is not a di-rect indicator of the recognition power, from an informa-tion-theoretic point-of-view, the multiple eigenspace repre-sentation is a more accurate representation of the signalcontent.

We have evaluated the view-based approach with datasimilar to that shown in Fig. 12. This data consists of 189images consisting of nine views of 21 people. The nineviews of each person were evenly spaced from -90$ to +90$

along the horizontal plane. In the first series of experiments,the interpolation performance was tested by training on asubset of the available views {±90°, ±45°, 0°} and testing onthe intermediate views {±68°, ±23°}. A 90 percent averagerecognition rate was obtained. A second series of experi-

Fig. 11. Photobook: FERET face database.


ments tested the extrapolation performance by training on arange of views (e.g., -90° to +45°) and testing on novelviews outside the training range (e.g., +68° and +90°). Fortesting views separated by ±23° from the training range, theaverage recognition rates were 83 percent. For ±45° testing

views, the average recognition rates were 50 percent (see[27] for further details).

4.3 Modular RecognitionThe eigenface recognition method is easily extended to fa-cial features as shown in Fig. 14a. This leads to an im-provement in recognition performance by incorporating anadditional layer of description in terms of facial features.This can be viewed as either a modular or layered repre-sentation of a face, where a coarse (low-resolution) descrip-tion of the whole head is augmented by additional (higher-resolution) details in terms of salient facial features.

The utility of this layered representation (eigenface pluseigenfeatures) was tested on a small subset of our large facedatabase. We selected a representative sample of 45 indi-viduals with two views per person, corresponding to dif-ferent facial expressions (neutral vs. smiling). This set ofimages was partitioned into a training set (neutral) and atesting set (smiling). Since the difference between theseparticular facial expressions is primarily articulated in themouth, this feature was discarded for recognition purposes.

Fig. 14b shows the recognition rates as a function of thenumber of eigenvectors for eigenface-only, eigenfeature-only, and the combined representation. What is surprisingis that (for this small dataset at least) the eigenfeaturesalone were sufficient in achieving an (asymptotic) recogni-tion rate of 95 percent (equal to that of the eigenfaces). More

Fig. 12. Some of the images used to test accuracy at face recognitiondespite wide variations in head orientation. Average recognition accuracywas 92 percent, the orientation error had a standard deviation of 15$.

Training View

Testing View

(a)

(b)

Fig. 13. (a) Parametric vs. view-based eigenspace reconstructions fora training view and a novel testing view. The input image is shown inthe left column. The middle and right columns correspond to theparametric and view-based reconstructions, respectively. All recon-structions were computed using the first 10 eigenvectors. (b) A sche-matic representation of the two approaches.

(a)

(b)

Fig. 14. (a) Facial eigenfeature regions. (b) Recognition rates for ei-genfaces, eigenfeatures, and the combined modular representation.


surprising, perhaps, is the observation that in the lowerdimensions of eigenspace, eigenfeatures outperformed theeigenface recognition. Finally, by using the combined rep-resentation, we gain a slight improvement in the asymp-totic recognition rate (98 percent). A similar effect was re-ported by Brunelli and Poggio [4], where the cumulativenormalized correlation scores of templates for the face,eyes, nose, and mouth showed improved performance overthe face-only templates.

A potential advantage of the eigenfeature layer is theability to overcome the shortcomings of the standard eigen-face method. A pure eigenface recognition system can befooled by gross variations in the input image (hats, beards,etc.). Fig. 15a shows additional testing views of three indi-viduals in the above dataset of 45. These test images areindicative of the type of variations which can lead to falsematches: a hand near the face, a painted face, and a beard.Fig. 15b shows the nearest matches found based on stan-dard eigenface matching. None of the three matches corre-spond to the correct individual. On the other hand, Fig. 15cshows the nearest matches based on the eyes and nose, andresults in correct identification in each case. This simpleexample illustrates the potential advantage of a modularrepresentation in disambiguating low-confidence eigenfacematches.

We have also extended the normalized eigenface repre-sentation into an edge-based domain for facial description.We simply run the normalized facial image through aCanny edge detector to yield an edge map as shown in

Fig. 16a. Such an edge map is simply an alternative repre-sentation which imparts mostly shape (as opposed to tex-ture) information and has the advantage of being less sus-ceptible to illumination changes. The recognition rate of apure edge-based normalized eigenface representation (on aFERET database of 155 individuals) was found to be 95 per-cent, which is surprising considering that it utilizes whatappears to be (to humans at least) a rather impoverishedrepresentation. The slight drop in recognition rate is mostlikely due to the increased dimensionality of this represen-tation space and its greater sensitivity to expressionchanges, etc.

Interestingly, we can combine both texture and edge-based representations of the object by simply performing aKL expansion on the augmented images shown in Fig. 16.The resulting eigenvectors conveniently decorrelate thejoint representation and provide a basis set which optimallyspans both domains simultaneously. With this bimodalrepresentation, the recognition rate was found to be 97 per-cent. Though still less than a normalized grayscale repre-sentation, we believe a bimodal representation can havedistinct advantages for tasks other than recognition, such asdetection and image interpolation.

4.4 Hand Modeling and RecognitionWe have also applied our eigenspace density estimationtechnique to articulated and nonrigid objects, such ashands. In this particular domain, however, the original in-tensity image is an unsuitable representation since, unlikefaces, hands are essentially textureless objects. Their iden-tity is characterized by the variety of shapes they can as-sume. For this reason, we have chosen an edge-based rep-resentation of hand shapes which is invariant with respectto illumination, contrast, and scene background. A training

(a)

(b)

(c)

Fig. 15. (a) Test views. (b) Eigenface matches. (c) Eigenfeaturematches.

(a)

(b)

Fig. 16. (a) Examples of combined texture/edge-based face represen-tations. (b) Few of the resulting eigenvectors.


set of hand gestures was obtained against a black back-ground. The 2D contour of the hand was then extractedusing a Canny edge-operator. These binary edge maps,however, are highly uncorrelated with each other due totheir sparse nature. This leads to a very high-dimensionalprincipal subspace. Therefore, to reduce the intrinsic di-mensionality, we induced spatial correlation via a diffusionprocess on the binary edge map, which effectively broadensand “smears” the edges, yielding a continuous-valuedcontour image which represents the object shape in terms ofthe spatial distribution of edges. Fig. 17 shows examples oftraining images and their diffused edge map representa-tions. We note that this spatiotopic representation of shapeis biologically motivated and therefore differs from meth-ods based purely on computational considerations (e.g.,moments [13], Fourier descriptors [20], “snakes” [16], PointDistribution Models [7], and modal descriptions [34]).

Fig. 17. Top: Examples of hand gestures and bottom: their diffusededge-based representation.

It is important to verify whether such a representation isadequate for discriminating between different hand shapes.Therefore, we tested the diffused contour image represen-tation in a recognition experiment which yielded a 100 per-cent rank-one accuracy on 375 frames from an image se-quence containing seven hand gestures. The matchingtechnique was a nearest-neighbor classification rule in a 16-dimensional principal subspace. Fig. 18a shows some ex-amples of the various hand gestures used in this experi-ment. Fig. 18b shows the 15 images that are most similar tothe “two” gesture appearing in the top left. Note that thehand gestures judged most similar are all objectively thesame shape.

Naturally, the success of such a recognition system iscritically dependent on the ability to find the hand (in anyof its articulated states) in a cluttered scene, to account forits scale, and to align it with respect to an object-centeredreference frame prior to recognition. This localization wasachieved with the same multiscale ML detection paradigmused with faces, with the exception that the underlying im-age representation of the hands was the diffused edge maprather the grayscale image.

The probability distribution of hand shapes in this repre-sentation was automatically learned using our eigenspacedensity estimation technique. In this case, however, thedistribution of training data is multimodal due to the differ-ent hand shapes. Therefore, the multimodal density esti-mation technique in Section 2.3 was used. Fig. 19a shows aprojection of the training data on the first two dimensions

of the principal subspace F (defined in this case by M = 16)which exhibit the underlying multimodality of the data.Fig. 19b shows a 10-component Mixture-of-Gaussians den-sity estimate for the training data. The parameters of thisestimate were obtained with 20 iterations of the EM algo-rithm. The orthogonal F -space component of the densitywas modeled with a Gaussian distribution as in Section 2.3.

The resulting complete density estimate $P x Wd i was

then used in a detection experiment on test imagery ofhand gestures against a cluttered background scene. In ac-cordance with our representation, the input imagery wasfirst preprocessed to generate a diffused edge map and thenscaled accordingly for a multiscale saliency computation.Fig. 20 shows two examples from the test sequence, wherewe have shown the original image, the negative log-likelihood saliency map, and the ML estimates of positionand scale (superimposed on the diffused edge map). Note

(a)

(b)

Fig. 18. (a) A random collection of hand gestures. (b) Images orderedby similarity (left-to-right, top-to-bottom) to the image at the upper left.


that these examples represent two different hand gesturesat slightly different scales.

To better quantify the performance of the ML detectoron hands, we carried out the following experiment. Theoriginal 375-frame video sequence of training hand gestureswas divided into two parts. The first (training) half of thissequence was used for learning, including computation ofthe KL basis and the subsequent EM clustering. For thisexperiment we used a five-component mixture in a 10-dimensional principal subspace. The second (testing) half ofthe sequence was then embedded in the background scene,which contains a variety of shapes. In addition, severe noiseconditions were simulated as shown in Fig. 21a.

We then compared the detection performance of an SSDdetector (based on the mean edge-based hand representa-tion) and a probabilistic detector based on the completeestimated density. The resulting negative-log-likelihooddetection maps were passed through a valley-detector toisolate local minimum candidates, which were then sub-jected to an ROC analysis. A correct detection was definedas a below-threshold local minimum within a five-pixelradius of the ground truth target location. Fig. 21b showsthe performance curves obtained for the two detectors. Wenote, for example, that, at an 85 percent detection probabil-ity, the ML detector yields (on the average) one false alarmper frame, whereas the SSD detector yields an order ofmagnitude more false alarms.

5 DISCUSSION

In this paper, we have described an eigenspace density es-timation technique for unsupervised visual learning whichexploits the intrinsic low dimensionality of the training im-agery to form a computationally simple estimator for thecomplete likelihood function of the object. Our estimator isbased on a subspace decomposition and can be evaluatedusing only the M-dimensional principal component vector.We derived the form for an optimal estimator and its asso-ciated expected cost for the case of a Gaussian density. Incontrast to previous work on learning and characteriza-tion—which uses PCA primarily for dimensionality reduc-tion and/or feature extraction—our method uses the ei-genspace decomposition as an integral part of estimatingcomplete density functions in high-dimensional imagespaces. These density estimates were then used in a maxi-mum likelihood formulation for target detection. The mul-tiscale version of this detection strategy was demonstratedin applications in which it functioned as an attentional sub-system for object recognition. The performance was foundto be superior to existing detection techniques in experi-ments with large numbers of test data.

We note that, from a probabilistic perspective, the classconditional density P(x|W) is the most important object rep-resentation to be learned. This density is the critical compo-

(a) (b)

Fig. 19. (a) Distribution of training hand shapes (shown in the first twodimensions of the principal subspace). (b) Mixture-of-Gaussians fitusing 10 components.

(a) (b) (c)

Fig. 20. (a) Original grayscale image. (b) Negative log-likelihood map(at most likely scale). (c) ML estimate of position and scale superim-posed on edge map.

(a)

(b)

Fig. 21. (a) Example of test frame containing a hand gesture amidstsevere background clutter. (b) ROC curve performance contrastingSSD and ML detectors.


nent in detection, recognition, prediction, interpolation, andgeneral inference. For example, having learned these densi-ties for several object classes {W1, W2, �, Wn}, one can invoke aBayesian framework for classification and recognition:

PP P

P Pi

i i

i ij

nWW W

W Wx

x

xd i d i c h

d i c h=

=Â

1

(25)

where now a maximum a posteriori (MAP) classificationrule can be used for object/pose identification.

Such a framework is also important in detection. In fact,the ML detection framework can be extended using thenotion of a “not-class” W , resulting in a posteriori saliencymaps of the form

S(i, j, k; W) = P(W|x) (26)

where now a maximum a posteriori (MAP) rule can be usedto estimate the position and scale of the object. One diffi-culty with such a formulation is that the “not-class” W is, inpractice, too broad a category and is therefore multimodaland very high-dimensional. One possible approach to thisproblem is to use ML detection to identify the particularsubclass of W which has high likelihoods (e.g., false alarms)and then to estimate this distribution and use it in the MAPframework. This can be viewed as a probabilistic approachto learning using positive as well as negative examples.

In fact, such a MAP framework can be viewed as a Baye-sian formulation of some neural network approaches totarget detection. Perhaps the most closely related is the re-cent neural network face detector of Sung and Poggio [35]which is essentially a trainable nonlinear binary patternclassifier. The,y too, learn the distribution of the object classwith a Mixture-of-Gaussians model (using an elliptical k-means algorithm instead of EM). Instead of likelihoods,however, input patterns are represented by a set of dis-tances to each mixture component (similar to a combinationof the DIFS and DFFS), thus forming a feature vector in-dicative of the overall class membership. Another recentexample of a neural network technique for object detectionwhich also utilizes negative examples is the face-findersystem of Rowley et al. [33]. The experimental results ob-tained with these methods clearly demonstrate the need forincorporating negative examples in building robust detec-tion systems.

We have recently used a Bayesian/MAP technique forfacial recognition, whereby an a posteriori similarity meas-ure is computed using Bayes rule for the two classes corre-sponding to interapersonal facial variation and extraper-sonal facial variation [22]. This approach to recognition hasproduced a significant improvement over the accuracy weobtained using a standard eigenface nearest-neighbormatching rule. This new probabilistic similarity measurewas used in the September 1996 FERET competition, andwas found to be the top-performing system, by a typicalmargin of 10 percent to the next-best competitor [29].

ACKNOWLEDGMENTSThis research was funded by the U.S. Army FERET pro-gram and by British Telecom. The FERET database wasprovided by the U.S. Army Research Laboratory. The mul-tiple-view face database was provided by WestinghouseElectric Systems. We would also like to thank WasiuddinWahid for his hard work in helping us participate in theFERET testing.

REFERENCES

[1] C. Anderson, P. Burt, and G. Van der Wall, “Change Detectionand Tracking Using Pyramid Transform Techniques,” Proc. SPIEConf. Intelligence, Robots, and Computer Vision, vol. 579, pp. 72-78,1985.

[2] M. Bichsel, and A. Pentland, “Human Face Recognition and theFace Image Set’s Topology,” CVGIP: Image Understanding, vol. 59,no. 2, pp. 254-261, 1994.

[3] C. Bregler, and S. Omohundro, “Surface Learning with Applica-tions to Lip Reading,” Advances in Neural Information ProcessingSystems 6, J.D. Cowan, G. Tesauro, and J. Alspector, eds. , pp. 43-50. San Mateo, Calif.: Morgan Kaufman, 1994.

[4] R. Brunelli, and T. Poggio, “Face Recognition: Features vs. Tem-plates,,” IEEE Trans. Pattern Analysis and Machine Intelligence,, vol. 15,no. 10, Oct. 1993.

[5] R. Brunelli, and S. Messelodi, “Robust Estimation of Correlation:An Application to Computer Vision,” IRST Technical Report no.9310-015, Oct. 1993.

[6] M. Burl et al., “Automating the Hunt for Volcanos on Venus,”Proc. IEEE Conf. Computer Vision and Pattern Recognition, Seattle,June 21-23, 1994.

[7] T. Cootes and C. Taylor, “Active Shape Models: Smart Snakes,”Proc. British Machine Vision Conf. , pp. 9-18. Springer-Verlag, 1992.

[8] T. Cootes, A. Hill, C. Taylor,. and J. Haslam, “Use of Active ShapeModels for Locacting Structures in Medical Images,” Image andVision Computing, vol. 12, no. 6, pp. 355-365, July/Aug. 1994.

[9] M. Cover and J. Thomas, Elements of Information Theory. NewYork: John Wiley & Sons, 1994.

[10] T. Darrell, and A. Pentland, “Space-Time Gestures,” Proc. IEEEConf. Computer Vision and Pattern Recognition, New York, June1993.

[11] A. Dempster, N. Laird, and D. Rubin, “Maximum Likelihoodfrom Incomplete Data via the EM Algorithm,” J. Royal StatisticalSoc. B, vol. 39, 1977.

[12] G. Golub and C. Van Loan, Matrix Computations. Johns HopkinsPress, 1989.

[13] M. Hu, “Visual Pattern Recognition by Moment Invariants,” IEEETrans. Information Theory, vol. 8, pp. 179-187, 1962.

[14] I. Jolliffe, Principal Component Analysis. New York: Springer-Verlag, 1986.

[15] T. Kanade, “Picture Processing by Computer Complex and Rec-ognition of Human Faces,” technical report, Kyoto Univ., Dept. ofInformation Science, 1973.

[16] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active ContourModels,” Int’l J. Computer Vision, vol. 1, no. 4, pp. 321-331, 1987.

[17] M. Kirby, and L. Sirovich, “Application of the Karhunen-LoeveProcedure for the Characterization of Human Faces,” IEEE Trans.Pattern Analysis and Machine Intelligence, vol. 12, no. 1, Jan. 1990.

[18] B. Kumar, D. Casasent, and H. Murakami, “Principal ComponentImagery for Statistical Pattern Recognition Correlators,” OpticalEng., vol. 21, no. 1, Jan./Feb. 1982.

[19] M. Loeve, Probability Theory. Princeton, N.J.: Van Nostrand, 1955.[20] T. McElroy, E. Wilson, and G. Anspach, “Fourier Descriptors and

Neural Networks for Shape Classification,” Proc. Int’l Conf. Acous-tics, Speech, and Signal Processing, Detroit, May 1995.

[21] B. Moghaddam and A. Pentland, “Face Recognition Using View-Based and Modular Eigenspaces,” Automatic Systems for the Identi-fication and Inspection of Humans, SPIE vol. 2277, 1994.

[22] B. Moghaddam, C. Nastar, and A. Pentland, “A Bayesian Similar-ity Measure for Direct Image Matching,” Proc. Int’l Conf. PatternRecognition, Vienna, 1996.

[23] H. Murase and S. Nayar, “Visual Learning and Recognition of 3DObjects from Appearance,” Int’l J. Computer Vision, vol. 14, no. 1,1995.


[24] S. Nayar, H. Murase, and S. Nene, “General Learning Algorithmfor Robot Vision,” Neural & Stochastic Methods in Image & SignalProcessing, SPIE vol. 2304, July 1994.

[25] S. Palmer, “The Psychology of Perceptual Organization: A Trans-formational Approach,” Human and Machine Vision, J. Beck, B.Hope, and A. Rosenfeld, eds. Academic Press, 1983.

[26] A. Pentland and S. Sclaroff, “Closed-Form Solutions for Physi-cally Based Shape Modeling and Recovery,” IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 13, no. 7, pp. 715–729, July1991.

[27] A. Pentland, B. Moghaddam, and T. Starner, “View-Based andModular Eigenspaces for Face Recognition,” Proc. IEEE Conf.Computer Vision and Pattern Recognition, Seattle, June 1994.

[28] A. Pentland, R. Picard, and S. Sclaroff, “Photobook: Tools forContent-Based Manipulation of Image Databases,” Int’l J. Com-puter Vision, vol. 18, no. 3. ,pp. 233-254

[29] P.J. Phillips, H. Moon, P. Rauss, and S.A. Rizvi, “The FERET Sep-tember 1996 Database and Evaluation Procedure,” Proc. First Int’lConf. Audio and Video-Based Biometric Person Authentication, Crans-Montana, Switzerland, Mar. 1997.

[30] T. Poggio and F. Girosi, “Networks for Approximation andLearning,” Proc. IEEE, vol. 78, no. 9, pp. 1,481-1,497, 1990.

[31] R. Redner, and H. Walker, “Mixture Densities, Maximum Likeli-hood and the EM Algorithm,” SIAM Rev., vol. 26, no. 2, pp. 195-239, 1984.

[32] D. Reisfeld, H. Wolfson, and Y. Yeshurun, “Detection of InterestPoints Using Symmetry,” Proc. Int’l Conf. Computer Vision, Osaka,Japan, Dec. 1990.

[33] H. Rowley, S. Baluja, and T. Kanade, “Human Face Detection inVisual Scenes,” Technical Report CMU-CS-95-158, Carnegie Mel-lon Univ., July 1995.

[34] S. Sclaroff and A. Pentland, “Modal Matching for Correspondenceand Recognition,” IEEE Trans. Pattern Analysis and Machine Intelli-gence, vol. 17, no. 6, pp. 545-561, June 1995.

[35] K. Sung and T. Poggio, “Example-Based Learning for View-BasedHuman Face Detection,” Proc. Image Understanding Workshop,Monterey, Calif., Nov. 1994.

[36] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cogni-tive Neuroscience, vol. 3, no. 1, 1991.

[37] J. Vincent, J. Waite, and D. Myers, “Automatic Location of Vis-ual Features by a System of Multilayered Perceptrons,” IEEProc., vol. 139, no. 6, Dec. 1992.

[38] J. Weng, “On Comprehensive Visual Learning,” Proc. NSF/ARPAWorkshop Performance vs. Methodology in Computer Vision, Seattle,June 24-25, 1994.

Baback Moghaddam received the BS (magnacum laude) and MS (honors) degrees in electri-cal and computer engineering from George Ma-son University in 1989 and 1992, respectively.He is currently finishing his PhD degree in theDepartment of Electrical Engineering and Com-puter Science at the Massachusetts Institute ofTechnology. Since 1993, he has been a re-search assistant in the Vision and Modelinggroup at the MIT Media Laboratory, where hehas developed an automatic face recognition

system which competed in the US Army’s “FERET” face recognitioncompetition. His research interests include computer vision, imageprocessing, computational learning theory and statistical pattern rec-ognition. Moghaddam is a member of Eta Kappa Nu and the ACM anda student member of the IEEE.

Alex Pentland received his PhD from the Mas-sachusetts Institute of Technology in 1982. He isthe academic head of the MIT Media Laboratory.He is also the Toshiba Professor of Media Artsand Sciences, an endowed chair last held byMarvin Minsky. After receiving his PhD, heworked at SRI’s AI Center and as a lecturer atStanford University, winning the DistinguishedLecturer award in 1986. In 1987, he returned toMIT to found the Perceptual Computing Sectionof the Media Laboratory, a group that now in-

cludes more than 50 researchers in computer vision, graphics, speech,music, and human-machine interaction. He has done research in hu-man-machine interface, computer graphics, artificial intelligence, ma-chine and human vision, and has published more than 180 scientificarticles in these areas. His most recent research focus is understand-ing human behavior in video, including face, expression, gesture, andintention recognition, as described in the April 1996 issue of ScientificAmerican. He has won awards from the AAAI for his research intofractals; the IEEE for his research into face recognition; and from ArsElectronica for his work in computer vision interfaces to virtual envi-ronments. He is a member of the IEEE.

Documents

Probabilistic Visual Learning for Object Representation - … · 696 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 Probabilistic Visual