Int J Comput Vis (2008) 79: 1–12DOI 10.1007/s11263-007-0085-5

Keypoint Detection and Local Feature Matching for Textured 3DFace Recognition

Ajmal S. Mian · Mohammed Bennamoun ·Robyn Owens

Received: 6 March 2007 / Accepted: 20 August 2007 / Published online: 25 September 2007© Springer Science+Business Media, LLC 2007

Abstract Holistic face recognition algorithms are sensitiveto expressions, illumination, pose, occlusions and makeup.On the other hand, feature-based algorithms are robust tosuch variations. In this paper, we present a feature-based al-gorithm for the recognition of textured 3D faces. A novelkeypoint detection technique is proposed which can repeat-ably identify keypoints at locations where shape variation ishigh in 3D faces. Moreover, a unique 3D coordinate basiscan be defined locally at each keypoint facilitating the ex-traction of highly descriptive pose invariant features. A 3Dfeature is extracted by fitting a surface to the neighborhoodof a keypoint and sampling it on a uniform grid. Featuresfrom a probe and gallery face are projected to the PCA sub-space and matched. The set of matching features are usedto construct two graphs. The similarity between two faces ismeasured as the similarity between their graphs. In the 2Ddomain, we employed the SIFT features and performed fu-sion of the 2D and 3D features at the feature and score-level.The proposed algorithm achieved 96.1% identification rateand 98.6% verification rate on the complete FRGC v2 dataset.

Keywords Feature-based face recognition · Keypointdetection · Invariant features · Feature-level fusion

This work is supported by ARC grant number DP0664228.

A.S. Mian (�) · M. Bennamoun · R. OwensSchool of Computer Science and Software Engineering,The University of Western Australia, 35 Stirling Highway,Crawley, WA 6009, Australiae-mail: ajmal@csse.uwa.edu.au

M. Bennamoune-mail: bennamou@csse.uwa.edu.au

R. Owense-mail: robyn.owens@uwa.edu.au

1 Introduction

The human face has emerged as one of the most promis-ing biometrics due to the social acceptability and non-intrusiveness of its measurement through imaging. It re-quires minimal or no cooperation from the subject making itideal for surveillance and applications where customer satis-faction is important. However, machine recognition of facesis very challenging because the distinctiveness of facial bio-metrics is quite low compared to other biometrics (e.g. fin-gerprints and iris) (Jain et al. 2004). Moreover, changescaused by expressions, illumination, pose, occlusions andfacial makeup (e.g. beard) impose further challenges on ac-curate face recognition.

A comprehensive survey of basic face recognition algo-rithms is given by Zhao et al. (2003) who categorize facerecognition into holistic, feature-based and hybrid match-ing algorithms. Holistic matching algorithms basically ex-tract global features from the entire face. Eigenfaces (Turkand Pentland 1991) and Fisherfaces (Belhumeur et al. 1997)are well known examples of holistic face recognition algo-rithms. Feature-based matching algorithms extract local fea-tures or regions such as the eyes and nose and then matchthese features or their local statistics for recognition. Oneexample of this category is the region-based 3D matchingalgorithm (Mian et al. 2007) which matches the 3D point-clouds of the eyes-forehead and the nose regions separatelyand fuse the results at the score-level. Another example isface recognition using local boosted features (Jones and Vi-ola 2003) which matches rectangular regions from facial im-ages at different locations, scales and orientations. Hybridmatching methods use a combination of global and local-features for face recognition e.g. Huang et al. 2003).

One limitation of holistic matching is that it requires ac-curate normalization of the faces according to pose, illu-

2 Int J Comput Vis (2008) 79: 1–12

mination and scale. Variations in these factors can affectthe global features extracted from the faces leading to in-accuracies in the final recognition. Normalization is usu-ally performed by manually identifying landmarks on thefaces which makes the whole process semi-automatic. Man-ual normalization is also imperfect as it is commonly per-formed on the basis of only a few landmarks (3 to 5) andhumans cannot identify landmarks with subpixel accuracy.Replacing this manual process with an automatic landmarkidentification algorithm usually deteriorates the final recog-nition results. Moreover, global features are also sensitive tofacial expressions and occlusions. Feature-based matchingalgorithms have an advantage over holistic matching algo-rithms because they are robust to variations in pose, illumi-nation, scale, expressions and occlusions.

Multimodal 2D-3D face recognition provides more ac-curate results than either of the individual modalities alone(Bowyer et al. 2006). An up to date survey of 3D and multi-modal face recognition is given by Bowyer et al. (2006) whoargue that 3D face recognition has the potential to overcomethe limitations of its 2D counterpart. 3D data can assist inthe normalization as well as recognition of faces. For exam-ple, 3D face data can be used for illumination normaliza-tion (Al-Osaimi et al. 2006) and pose correction (Mian et al.2006c) of 2D faces. In the recognition phase, additional fea-tures can be extracted from the 3D faces and fused with the2D features at the score or feature-level for greater accuracy.Bowyer et al. (2006) stress upon the need for better 3D facerecognition algorithms which are more robust to variations.However, they also agree that multiple algorithms or classi-fiers can increase performance. For example, Gokberk et al.(2005) reported an increase in recognition performance bycombining the results of multiple 3D face recognition algo-rithms.

Many 3D face recognition approaches are based on theIterative Closest Point (ICP) algorithm (Besl and McKay1992) or its modified versions (Mian et al. 2007). There aretwo major advantages of ICP based approaches. Firstly, per-fect normalization of the faces is not required as the algo-rithm iteratively corrects registration errors while matching.Secondly, a partial region of a face can be matched witha complete face. The latter advantage has been exploitedto avoid facial expressions (Mian et al. 2006a, 2007) andto handle pose variations by matching 2.5D scans to com-plete face models (Lu et al. 2006). On the downside, themajor disadvantage of ICP is that it is an iterative algorithmand is therefore computationally very expensive. Moreover,ICP does not extract any feature from the face which rulesout the possibilities of feature-level fusion and indexing tospeed up the matching process. Unless another classifierand/or modality is used to perform indexing or prior rejec-tion of unlikely faces from the gallery (Mian et al. 2006c),ICP based algorithms must perform a brute force matching

thereby making the recognition time linear to the gallerysize. Matching expression insensitive regions of the face isa potentially useful approach to overcome the sensitivity ofICP to expressions. However, deciding upon such regionsis a problem worth exploring as such regions may not onlyvary between different persons but between different expres-sions as well.

Our literature review leads us to the following conclu-sions: (1) Local features are more robust compared to globalfeatures; (2) Multimodal 2D-3D face recognition is more ac-curate than either of the individual modalities; (3) The facerecognition literature can always benefit from not only bet-ter but more number of face recognition algorithms becausetheir combination is likely to improve performance; (4) Un-like ICP, it is more advantageous to extract features from the3D face in order to facilitate feature-level fusion and index-ing.

Based on the above motivations, we propose an algorithmfor textured 3D face recognition using local features ex-tracted separately from the 3D shape and texture of the face.In the 3D domain, a novel approach for keypoint detection isproposed. The identification of keypoints is repeatable andallows for the extraction of highly descriptive 3D features atthe keypoints. Features are extracted with reference to localobject-centered 3D coordinates which are uniquely definedat each keypoint using its neighboring surface. The repeata-bility of the keypoint locations and reference frames pro-vides pose invariance to the 3D features. Each feature is ex-tracted by fitting a surface to the neighborhood of a keypointand sampling it on a uniform grid. In order to achieve robust-ness to noise, approximation is used for surface fitting ratherthan interpolation. Multiple features are extracted from eachgallery face and projected to the PCA subspace. The dimen-sionality of the feature vectors is reduced by consideringonly the most significant components. During recognition,features are extracted at keypoints on the probe. These fea-tures are also projected to the PCA subspace and matchedwith those in the gallery. The set of matching features froma probe and gallery face are individually meshed to formgraphs. A spatial constraint is used to remove false matchesand the remaining ones are used to calculate the similaritymeasure between the two faces. The similarity measure isnot only based on the average error between the correspond-ing features but also takes into account the error between thetwo graphs and the total number of correct matches betweenthe faces.

We also employed the SIFT features in the 2D domainand fused the two modalities at the score and feature-level.In the former case, the two feature sets (2D and 3D) areextracted and matched independently and their similarityscores are fused. In the latter case, both the 2D and 3D fea-tures are extracted at the same keypoints on the face andprojected to their respective PCA subspaces. The dimen-sionality of each feature is reduced and each feature vector

Int J Comput Vis (2008) 79: 1–12 3

is normalized so that its variance along every dimension isequal. Finally, the feature vectors are normalized to a unitmagnitude and concatenated to form a multimodal (2D-3D)feature vector. The same algorithm as described above isalso used for matching the multimodal features. The pro-posed algorithm was tested on the FRGC v2 (Phillips et al.2005) data set and achieved identification rates of 99.38%and 92.11% for probes with neutral and non-neutral expres-sions, respectively. The verification rates at 0.001 FAR forthe same were 99.85% and 96.62%.

Preliminary results of our algorithm have been publishedin (Mian et al. 2006d). However, a number of extensionshave been made since then which resulted in a signifi-cant improvement in the accuracy (from 89.5% to 99.38%3D face identification rate) and efficiency of the algorithm.These extensions include a keypoint detection algorithm, 3Dcoordinate basis derivation from single keypoints, projectionof the features to PCA subspace, feature-level fusion, usinga more sophisticated graph matching approach and perform-ing experiments on the complete FRGC v2 data set.

The rest of this paper is organized as follows. Section 2gives details of the keypoint identification algorithm. Sec-tion 3 explains the extraction of local 3D features. Sec-tion 4.1 gives a brief overview of the SIFT features. Feature-level fusion is described in Sect. 4.2. Details of the matchingalgorithm are given in Sect. 5. Results are reported in Sect. 6.Finally, discussion and conclusions are given in Sects. 7and 8, respectively.

2 Keypoint Detection

The aim of keypoint detection is to determine points on asurface (a 3D face in our case) which can be identified withhigh repeatability in different range images of the same sur-face in the presence of noise and pose variations. In additionto repeatability, the features extracted from these keypointsshould be sufficiently distinctive in order to facilitate ac-curate matching. The keypoint identification technique pro-posed in this paper is simple yet robust due to its repeatabil-ity and the descriptiveness of the features extracted at thesekeypoints.

The input to our algorithm is a point cloud of a faceF = [xi yi zi]T , where i = 1, . . . , n. The input face is sam-pled at uniform (x, y) intervals (4 mm in our case) and ateach sample point p, a local surface is cropped from theface using a sphere of radius r1 centered at p. The valueof r1 decides the degree of locality of the extracted featureand offers a trade off between descriptiveness and sensitivityto variations e.g. due to expressions. The smaller the valueof r1, the less the sensitivity of the local feature to variationsand vice versa. However, on the downside a small value of r1

will also decrease the descriptiveness of the feature.

Let Lj = [xj yj zj ]T (where j = 1, . . . , nl) be the pointsin the region cropped by the sphere of radius r1 centeredat p. The mean vector m and the covariance matrix C of Lare given by

m = 1

nl

nl∑

j=1

Lj , and (1)

C = 1

nl

nl∑

j=1

Lj LTj − mmT , (2)

where Lj is the j th column of L. Performing PrincipalComponent Analysis on the covariance matrix C gives thematrix V of eigenvectors

CV = DV, (3)

where D is the diagonal matrix of the eigenvalues of C. Thematrix L can be aligned with its principal axes using (4),known as the Hotelling transform (Gonzalez and Woods1992):

L′j = V(Lj − m) {j = 1, . . . , nl}. (4)

Let L′x and L′

y represent the x and y components of the pointcloud L′ i.e.

L′x = xj and L′

y = yj where {j = 1, . . . , nl}, (5)

δ = max(L′x) − min(L′

x) − (max(L′y) − min(L′

y)). (6)

In (6), δ is the difference between the first two principalaxes of the local region. The value of δ will be zero if L′ isplanar or spherical. However, if there is unsymmetrical vari-ation in the depth of L′, then δ will have a non-zero valueproportional to the variation. This depth variation is an indi-cation that L′ contains descriptive information.

The value of δ is always non-negative and if it is greaterthan a threshold (i.e. δ ≥ t1), p is taken as a keypoint, other-wise it is rejected. The threshold t1 governs the total numberof keypoints. Taking t1 = 0 will result in every point end-ing up as a keypoint and as the value of t1 increases thetotal number of keypoints will decrease. There are only twothresholds in the keypoint detection namely r1 and t1 whichare empirically chosen as r1 = 20 mm and t1 = 2 mm. How-ever, the algorithm is not sensitive to these parameters andsmall changes in these values will not have a significant ef-fect on its performance. The values of r1 and t1 depend onthe scale of human faces. This keypoint detection techniqueis generic and can be extended to other 3D shapes and ob-jects. However, the values of r1 and t1 will change accord-ingly relative to the scale of the objects to be recognized.

Figure 1 shows some examples of keypoints identified onthree different range images each of four individuals. We can

4 Int J Comput Vis (2008) 79: 1–12

Fig. 1 Illustration of keypointrepeatability. Each columncontains three range images ofthe same individual. Notice thatthe keypoints are repeatablyidentified at the sameneighborhoods for the sameindividual. The height of rangeimages is 160 mm which is whythe keypoints appear very close

see that the keypoints are repeatably identified at the samelocations for a given individual. Moreover, the keypointsvary between individuals because they have different facialshapes. For example, in the first column, the keypoints clus-ter mostly on the nose. In the second column, the keypointscluster on the sides of the nose as well. In the third column,some keypoints are also detected on the cheek bones. Thedifference between keypoint locations on the faces of dif-ferent individuals enhances the accuracy of the recognitionphase.

Figure 2 shows the results of our keypoint repeatabilityexperiment performed on the 3D faces of the FRGC v2 data(Phillips et al. 2005) i.e. 4007 three-dimensional scans of466 individuals. The data was preprocessed for spike re-moval using neighborhood thresholding. Holes in the facedata where then filled using cubic interpolation. Note thatthe FRGC consists of real data and therefore ground truthdense correspondence between the 3D faces is unavailable.Ground truth correspondence is necessary to calculate theerror between the keypoints of different range images of thesame person. Two approaches can be taken to overcome thisproblem. In the first approach which was also taken by Lowe(2004), data with ground truth is synthetically generated.This is done by adding synthetic changes like noise, rota-tion, scaling etc. which makes it possible to precisely cal-culate where each keypoint in an original image should ap-pear in the transformed one. In the second approach, groundtruth is approximated by an accurate registration algorithm.

In this paper, we have adopted the latter approach becauseit is more realistic i.e. repeatability is calculated using realdata. The 3D faces belonging to the same individual are au-tomatically registered using a modified ICP algorithm (Mianet al. 2007) and the errors between the nearest neighbors oftheir keypoints (one from each face) are recorded. Figure 2shows the cumulative percentage repeatability as a functionof increasing distance. The repeatability reaches 86% forfaces with neutral expression at an error of 4 mm. This iscomparable to the repeatability of SIFT (Lowe 2004). Asexpected, the repeatability drops rapidly below 4 mm be-cause this corresponds to the sampling distance (minimumdistance) between the keypoints. For non-neutral expressionfaces, the repeatability at 4 mm drops to 75.6% because the3D shape of the face changes with expressions. Note thatthese repeatability values are sufficient as the matching al-gorithm makes its decision on a subset of the total features(see Sect. 5).

The strengths of our keypoint detection algorithm can besummarized as follows. One, keypoint locations are repeat-ably identified in the range images of the same individual.Two, keypoints vary between different individuals. Three,keypoints are identified at locations where the shape varia-tion is high (i.e. nonplanar and nonspherical regions). Four,the keypoint detection process also provides stable and re-peatable local 3D coordinate frames for the computation oflocal features.

Int J Comput Vis (2008) 79: 1–12 5

Fig. 2 Repeatability of keypoints

3 3D Feature Extraction

Once a keypoint has been detected, a local feature is ex-tracted from its neighborhood L′. The proposed local 3Dfeature is an extension of the tensor representation (Mian etal. 2006b, 2006e) which quantizes local surface patches ofa 3D object into three-dimensional grids defined in locallyderived coordinate bases. In Mian et al. (2006b), the localcoordinate bases were derived from two points and theircorresponding normals. However, in a later work (Mian etal. 2006d), the local coordinates were derived from a singlepoint and some further invariant information i.e. the SIFTorientations or the location of the nose tip, in order to avoidthe Cn

2 combinatorial problem (Mian et al. 2006b). In thispaper, we use the principal directions of the local surfacepatch L′ as the 3D coordinates to calculate the feature. Thisavoids the Cn

2 combinatorial problem (Mian et al. 2006b)without the knowledge of the nose tip (Mian et al. 2006d).Since the keypoints are detected such that there is no ambi-guity in the principal directions of the neighboring surfacepatch, the derived 3D coordinate bases are stable and so arethe features.

A surface is fitted to the points in L′ using approximationas opposed to interpolation. This way the surface fitting isnot sensitive to noise and outliers in the data. Each point inL′ pulls the surface towards itself and a stiffness factor con-trols the flexibility of the surface. The surface is sampled ona uniform lattice. We use a 20×20 lattice in our experimentsto extract the feature. Figure 3b shows a surface fitted to theneighborhood of a keypoint using a 20 × 20 lattice. In orderto avoid the effects of boundaries that appear on the flanksof L′, we crop a larger region first using r2 (where r2 > r1)and fit a surface to it. This surface is then sampled on a big-ger lattice and only the central 20 ×20 samples covering ther1 region are concatenated to form a feature vector of di-mension 400. For surface fitting, we used publicly availablecode (D’Erico 2006). However, any surface fitting algorithmthat uses approximation can be used for this purpose.

Keeping a constant value for the threshold t1 will resultin different numbers of keypoints identified for different in-dividuals. However, we have put an upper limit on the totalnumber of local features that are calculated for a face in thegallery. This is important in order to avoid the recognitionresults being biased in favor of the gallery faces that havemore local features. For every face in the gallery, a total of200 feature vectors are calculated. The 200 keypoints are se-lected using a uniform random distribution. A dimension of400 is quite large for a feature vector that describes a localsurface. Fortunately, it is possible to compress these vectorsby projecting them into a subspace defined by the eigen-vectors of their largest eigenvalues using Principal Compo-nent Analysis (PCA). Let � = [f1 . . . f200N ] (where N is thegallery size) be the 400 × 200N matrix of all the featurevectors of all the faces in the gallery. Each column of �

contains a feature vector of dimension 400. The mean of �

or the mean feature vector is given by

f = 1

200N

200N∑

i

fi . (7)

The mean feature vector is subtracted from all features

f ′i = fi − f. (8)

The mean subtracted feature matrix becomes

�′ = [f ′

1 . . . f ′200N ]. (9)

The covariance matrix of the mean subtracted feature vec-tors is given by

C = �′(�′)T , (10)

where C is a 400 × 400 matrix. The eigenvalues and eigen-vectors of C are calculated using Singular Value Decompo-sition (SVD) as

USVT = C, (11)

where U is a 400 × 400 matrix of the eigenvectors sortedin decreasing order i.e. the eigenvector corresponding to thehighest eigenvalue is in the first column. S is a diagonal ma-trix of the eigenvalues, also sorted in decreasing order. Thedimension of the PCA subspace is governed by the amountof required accuracy (fidelity) in the projected space. Thiscan be easily understood by plotting the ratio of the first k

eigenvalues to the total eigenvalues given by

ψ =∑k

i=1 λi∑400i=1 λi

, (12)

where λi is the ith eigenvalue. Figure 4 shows a plot of theratio ψ as a function of the number of eigenvalues k. The

6 Int J Comput Vis (2008) 79: 1–12

Fig. 3 a A keypoint displayed(in white colour) on a 3D face(top) and its correspondingtexture map (bottom). b A localsurface fitted to theneighborhood of the keypoint(top) using a 20 × 20 lattice.The corresponding texture mapis also rotated (bottom) foralignment with the coordinatesof the 3D surface. c The texturemapped on the local surface

curve reaches a value of 0.99 very quickly at only k = 11which means that taking eigenvectors corresponding to onlythe highest 11 eigenvalues gives us 99% accuracy and acompression ratio of (400−11)

400 = 97.3%. This is not surpris-ing given that all human faces have a similar topologicalstructure and are roughly symmetric on either side of thenose. Repeating the keypoint detection with different val-ues of parameters (r1, t1) as well as randomly selecting key-points showed that the value of k = 11 is repeatable. Thevalue of k = 11 has certain significance here. It shows thatlocal shape variation in 3D faces can be represented with99% accuracy by a vector of fairly small dimension. This isin contrast with 2D faces where local appearance variationis represented with only 60% accuracy by a vector of equaldimension (see Sect. 4.2). The first k eigenvectors are takenas

Uk = Ui , (13)

where i = 1, . . . , k and Uk is a 400 × k matrix of the firstk eigenvectors. The mean subtracted feature matrix is pro-jected to the eigenspace as

�λ = (Uk)

T�

′, (14)

where �λ is a k × 200N matrix of the 3D feature vectors of

the gallery faces. �λ is normalized so that its variance along

each of the k dimensions is equal

�λrc = �

λrc

λr

where r = 1, . . . , k and c = 1, . . . ,200N.

(15)

In (15), r stands for the dimension or row number andc stands for the feature or column number. Finally, the fea-ture vectors in �

λ (i.e. the columns) are normalized to unitmagnitude and saved in a database along with f and Uk

for online feature-based recognition of faces. Note that the

Fig. 4 A plot of the ratio ψ as a function of the number of eigen-values k shows that the eigenvectors corresponding to the highest 11eigenvalues give 99% accuracy

representation of faces in the gallery is quite compact. Eachface is represented by only 200 feature vectors each of di-mension 11.

4 2D Feature Extraction and Fusion

4.1 2D Feature Extraction

In the 2D domain we used an off the shelf feature extractionalgorithm. We chose SIFT (Scale Invariant Feature Trans-form) (Lowe 2004) for this purpose as our keypoint detec-tion algorithm was inspired by the SIFT keypoint detectioneven though the two algorithms take completely differentapproaches to keypoint detection. In SIFT (Lowe 2004), acascaded filtering approach is used to locate the keypointswhich are stable over scale space. First, keypoint locationsare detected as the scale space extrema in the Difference-of-Gaussian function convolved with the image. A thresh-

Int J Comput Vis (2008) 79: 1–12 7

old is then applied to eliminate keypoints with low con-trast and those which are poorly localized along an edge.Finally, a threshold on the ratio of principal curvatures isused to select the final set of stable keypoints. For each key-point, the gradient orientations in its local neighborhood areweighted by their corresponding gradient magnitudes andby a Gaussian-weighted circular window and put in a his-togram. Dominant gradient directions corresponding to thepeaks in the histogram are used to assign one or more orien-tations to the keypoint.

For every orientation of a keypoint, the gradients of its lo-cal neighborhood are used to extract a feature (SIFT). SIFTis invariant to orientations because it is extracted after ro-tating the gradients relative to the keypoint orientation. Thegradient magnitudes are weighted by a Gaussian functiongiving more weight to closer points. Next, 4 × 4 sample re-gions are used to create orientation histograms, each witheight orientation bins forming a 4 × 4 × 8 = 128 dimen-sional feature vector. To achieve robustness to illumination,the feature vector is normalized to unit magnitude and largegradient magnitudes are thresholded to a ceiling of 0.2 eachand the vector is renormalized to unit magnitude once again.A detailed explanation of the SIFT keypoint detection andfeature extraction is given by Lowe (2004).

4.2 Feature-level Fusion

It is believed that feature-level fusion can provide better re-sults than score-level fusion (Jain et al. 2004). However, cur-rent research in the area of feature-level fusion is still inits infancy. Some of the challenges in feature-level fusioninclude the relative incompatibility of features and dimen-sionality problems. Incompatibilities can be in the dimen-sionality of the features, the variance of the features alongeach dimension and the location from which these featuresare extracted. In this section, we address these problems oneby one. First, we standardize the keypoint locations so thatboth 2D and 3D features can be extracted from the samepoints. Second, we reduce the dimensionality of the 2D fea-ture and normalize it so that both 2D and 3D features con-tribute equally to the resultant fused features.

The process described in Sect. 4.1 generally identifies 2Dkeypoints at locations that are different from the 3D key-points described in Sect. 2. This is not a problem when bothfeatures are matched independently and fusion is performedat the score-level. However, for feature-level fusion, boththe 2D and 3D features must be extracted at the same lo-cations. There are two ways to do this. The first is to extract3D features at points on the 3D face that correspond to the2D keypoints on the 2D image. This approach was testedin our earlier work (Mian et al. 2006d) but the results indi-cated that the keypoint locations or orientations provided bythe 2D features are not suitable for extracting 3D features.

Fig. 5 A plot of the ratio ψ as a function of the number of eigenvaluesk for the 128 dimensional SIFT features shows that the eigenvectorscorresponding to the highest 64 eigenvalues give 95% accuracy

The second approach is tested in this paper whereby SIFTfeatures are extracted at the 2D image locations which cor-respond to the keypoint locations and orientations on the 3Dface. The 2D pixel values corresponding to the local regionL are mapped on the point cloud and rotated using (4) inorder to align them with the coordinates of the 3D feature.These pixels are then sampled on a uniform 20 × 20 grid sothat the pixel-to-point correspondence is maintained to thelocal surface. Figure 3b shows a local 2D image patch af-ter rotation and sampling. The same image patch is mappedonto its corresponding 3D surface in Fig. 3c.

Note that using the 3D keypoints for extracting SIFTshas an added advantage that the image scale can be normal-ized with respect to the absolute 3D scale. In other words,SIFTs can be extracted from the local 2D image patches atthe same scale as that of the local surface. Once the 2D SIFTfeatures are extracted from the same keypoints as the 3Dones, they are projected to the PCA subspace in a similarway as described in Sect. 3. To decide on the number ofSIFT dimensions (eigenvectors) that must be used, the ra-tio ψ is plotted versus k in Fig. 5. Ideally, it is desirable tokeep the dimensions of the 3D and 2D features equal so thatthe resultant feature and the matching process is not biased.However, this is not feasible as the two features have sig-nificantly different reconstruction accuracies at equal valuesof k (see Fig. 4 and Fig. 5). For example, at k = 11 the ac-curacy of SIFT is only 0.6 whereas that of the 3D featuresis 0.99. To strike a balance between accuracy and compres-sion, we selected eigenvectors of SIFTs that correspond tothe 64 highest eigenvalues (i.e. k = 64). The variance of the64 dimensional vectors along each dimension is then nor-malized using (15) and the resultant vector is normalized tounit magnitude. Corresponding 2D and 3D feature vectorsare concatenated and normalized to unit magnitude to forma multimodal feature vector.

8 Int J Comput Vis (2008) 79: 1–12

Fig. 6 a Correct match.b Incorrect match

(a) (b)

5 Feature Matching

It is possible to speed up the matching process with the helpof indexing or hashing (Mian et al. 2006e). However, thesetechniques are not included here because we are interestedin matching a probe to every gallery face in order to gen-erate a large number of impostor scores which are usefulfor drawing the Receiver Operating Characteristic (ROC)curves. Currently, an unoptimized implementation of our al-gorithm in MATLAB can perform 23 matches per secondusing a 3.2 GHz Pentium IV machine with 1 GB RAM.

During online recognition, features are extracted from theprobe face using the same parameters as the gallery. Let fpbe a feature extracted from a probe face. This feature is firstprojected to the PCA subspace

fλp = (Uk)T (fp − f). (16)

To calculate the similarity between a probe and a galleryface, their local features are matched using the followingequation

e = cos−1(fλp(fλg)T ), (17)

where fλp and fλg correspond to the probe and gallery featuresin the PCA subspace, respectively. These features could be2D, 3D or multimodal 2D-3D. The same matching algo-rithm (to the code level) is used for matching different typesof features. If the two features are exactly equal, the value ofe will be zero indicating a perfect match. However, in realitya finite error exists between the features extracted from theexact same locations on different images of the same face.For a given probe feature, the feature from the gallery facethat has the minimum error with it is taken as its match.Once all the features are matched, the list of matching fea-tures is sorted according to e. If a gallery feature matchesmore than one probe feature, only the one with the minimumvalue of e is considered and the rest are removed from thelist of matches. This allows for only one-to-one matches andthe total number of matches m is different for every match-ing of probe-gallery faces.

The keypoints corresponding to the matching features onthe probe face are projected on the xy-plane, meshed usingDelaunay triangulation and projected back to the 3D space.This results in a 3D graph. The edges of this graph are usedto construct a graph from the corresponding nodes (key-points) of the gallery face using the list of matches. If thelist of matches is correct i.e. the matching pairs of featurescorrespond to the same location on the probe and galleryface, the two graphs are deemed to be similar (see Fig. 6).The similarity measure between the two graphs is

γ = 1

nε

nε∑

i

|εpi − εgi | (18)

where εpi and εgi are the lengths of the corresponding edgesof the probe and gallery graphs, respectively. The value nε isthe total number of edges. Equation (18) is an efficient wayof measuring the spatial error between the matching pairsof probe and gallery features. The similarity γ is invariantto the facial pose because the edge lengths of the graphswill not vary if the graph is rotated or translated. Anothersimilarity measure between the two faces is calculated asthe mean Euclidean distance d between the nodes of the twographs after least squared error minimization. Outlier nodeswhich have an error above a threshold are removed beforecalculating the mean error. This threshold is determined bythe sampling distance of the faces used to find the keypointsin Sect. 2.

The matching algorithm described above results in fourmeasures of similarity between the two faces i.e. the averageerror e between the features, the total number of matches m

between the faces, the graph edge error γ and the graph nodeerror d between the two graphs. Except for m, all other simi-larity measures have a negative polarity (i.e. a smaller valuemeans a better similarity). A probe is matched with everyface in the gallery resulting in four vectors sq of similaritymeasures (where q corresponds to one of the four similaritymeasures namely e, m, γ and d). The nth element of eachvector corresponds to the similarity between the probe with

Int J Comput Vis (2008) 79: 1–12 9

Fig. 7 Individual identificationperformance of the localfeatures and their combinedperformance when fusion isperformed at the score-level.The combined rank oneidentification rate for neutralversus all faces is 96.1%

(a) (b)

(c)

the nth gallery face. Each vector is normalized on the scaleof 0 to 1 using the min-max rule

s′q = sq − min(sq)

max(sq − min(sq)) − min(sq − min(sq)), (19)

where s′q contains the normalized similarity measures. The

operators min(sq) and max(sq) produce the minimum andmaximum values of the vectors, respectively. The elementsof s′

m (i.e. the number of matches) are subtracted from 1in order to reverse their polarity. The overall similarity ofthe probe with the gallery faces is then calculated using aconfidence weighted sum rule:

s = κes′e + κm(1 − s′

m) + κγ s′γ + κds′

d , (20)

where κq is the confidence in each individual similarity mea-sure. Confidence measures can be calculated offline from theresults obtained on training data or dynamically during on-line recognition as:

κq = sq − min(sq)

sq − min2(sq), (21)

where sq is the mean value of sq and the operator min2(sq)

produces the second minimum value of the vector sq . Notethat κm is calculated from 1−s′

m. The gallery face which hasthe minimum value in the vector s is declared as the identity

of the probe when the decision is to be made on the basis ofindividual 2D or 3D features or the fused multimodal 2D-3D features. In the case of score-level fusion, the similarityvectors resulting from individual 2D and 3D feature match-ing are normalized and fused using a weighted sum rule.The weights are calculated in a similar manner to (21). Theresultant vector is then used to make the decision.

6 Results and Analysis

The FRGC v2 data consist of a training and a validation set.The validation set comprises 4007 three-dimensional scansof faces along with their texture maps. There are 466 sub-jects in the validation set. Minor pose variations and ma-jor expression and illumination variations exist in the data-base. A detailed description of the database can be foundin Phillips et al. (2005). We selected one textured 3D faceper individual under neutral expression to make a gallery of466. The remaining faces (4007 − 466 = 3541) were treatedas probes and were divided into two sets i.e. one with neutralexpressions (1944 probes) and the other with non-neutral ex-pressions (1597 probes).

Figure 7 shows our identification results. Note that it isnot the aim of this paper to provide a true and unbiasedcomparison of the 3D and 2D features but to demonstratetheir use for face recognition in the presence of expression

10 Int J Comput Vis (2008) 79: 1–12

Fig. 8 Individual verificationperformance of the localfeatures and their combinedperformance when fusion isperformed at the score-level.The combined verification ratefor neutral versus all faces is98.6%

(a) (b)

(c)

and illumination variations. The 3D local features achievedrank one identification rates of 99.0% and 86.7% for probeswith neutral and non-neutral expressions, respectively. In theneutral expressions case, only two probes (out of a total of1944) are above rank 11 and only one is above rank 17 i.e.a 100% recognition rate is achieved at rank 17. In the caseof non-neutral expressions, the identification rate drops sig-nificantly however it should be kept in mind that 3D facerecognition algorithms are generally more sensitive to ex-pressions compared to 2D face recognition. For example,the face recognition rate in Lu et al. (2006) (using surfaceonly) drops by 30% i.e. from 98% to 68%. However in ourcase, due to the use of local-features, the recognition ratedrops by only 12.3%. Another point to note in the case ofnon-neutral expressions is that there is a steep rise in therecognition rate (i.e. 86.7% to 95%) from rank 1 to rank 5.This is an indication that it is possible to significantly im-prove the rank one recognition rate by fusing other featurese.g. global. The combined (using 2D and 3D features) rankone identification rates are 99.4% and 92.1% for probes withneutral and non-neutral expressions, respectively.

Figure 8 shows the ROC curves of our algorithm.At 0.001 FAR (False Acceptance Rate), the 3D featuresachieved verification rates of 99.9% and 92.7%, respec-tively for probes with neutral and non-neutral expressions.In the neutral expression case, a 100% verification rate wasachieved at 0.01 FAR. The combined verification rates are

99.9% and 96.6% for probes with neutral and non-neutral

expressions, respectively. The combined results reported in

Fig. 7 and Fig. 8 are for score-level fusion of the 3D and 2D

features. Table 1 compares score and feature level fusion and

recognition based on 3D features alone. Score-level fusion

performs better than feature-level fusion which is contra-

dictory to the claims of Jain et al. (2004) who argue that

feature-level fusion is more powerful. Possible reasons for

this anomaly are given in Sect. 7.

It is not the aim of this paper to report the most accu-

rate results on the FRGC v2 data and we believe that better

results can be obtained by combining our local 3D features

with other features (e.g. global) in a multi-algorithm set up.

However, to give some idea on the performance of our al-

gorithm, we compare our results in Table 2 to others who

used the FRGC v2 data set (Passalis et al. 2005; Maurer et

al. 2005; Husken et al. 2005). The FRGC benchmark (veri-

fication rate at 0.001 FAR) is used for comparison. Table 2

shows that our algorithm has the highest 3D and multimodal

verification rates in general. The performance of our local

3D feature-based face recognition especially stands out with

8% higher verification rate than its nearest competitor on the

complete FRGC v2 data set (i.e. neutral versus all).

Int J Comput Vis (2008) 79: 1–12 11

Table 1 Comparison of 3Dfeature-based face recognitionwith multimodal feature-basedface recognition using score andfeature-level fusion techniques

Neutral vs. All Neutral vs. Neutral Neutral vs. Non-neutral

Identification Verification Identification Verification Identification Verification

Rate Rate Rate Rate Rate Rate

Score level fusion 96.1% 98.6% 99.4% 99.9% 92.1% 96.6%

of 2D & 3D features

3D features alone 93.5% 97.4% 99.0% 99.9% 86.7% 92.7%

Feature level fusion 93.1% 97.2% 97.8% 99.5% 87.4% 93.5%

of 2D & 3D features

Table 2 Comparison ofverification rates at 0.001 FARon the FRGC v2 data set

Neutral vs. All Neutral vs. Neutral Neutral vs. Non-neutral

3D Multimodal 3D Multimodal 3D Multimodal

This paper 97.4% 98.6% 99.9% 99.9% 92.7% 96.6%

Maurer et al. 2005 86.5% 95.8% 97.8% 99.2% NA NA

Husken et al. 2005 89.5% 97.3% NA NA NA NA

Passalis et al. 2005 85.1% NA 94.9% NA 79.4% NA

FRGC baseline 45% 54% NA 82% 40% 43%

7 Discussion

Not undermining the potential of feature-level fusion, thereare four possible explanations for why this fusion techniquedid not even perform as well as the 3D features alone (seeTable 1). The foremost reason is related to errors in theFRGC v2 data itself (Mian et al. 2007) as the 3D facesand their corresponding texture maps are not perfectly regis-tered. Consequently, a keypoint detected on a 3D face some-times corresponds to a different location on the texture mapand hence the fused multimodal feature ends up distorted.The second reason is that given two sets of local features(2D + 3D), some features will be deteriorated in each setby variations due to noise, expressions and illumination etc.It is more likely that the deteriorated features from each setwill belong to different keypoints and when the correspond-ing features of the two sets are fused, the number of dete-riorated fused features will be greater than the those in ei-ther set. The third possible reason is that the same keypointsmay not be suitable for extracting both 2D and 3D features.The fourth reason is that in score-level fusion, the two setsof features contain more information as they are extractedfrom different keypoints and therefore lead to better results.Ross and Govindarajan (2005) also report similar findingswhere score-level fusion performs better than feature-levelfusion. Many other researchers have reported feature-levelfusion techniques but did not compare their results to score-level fusion and one is left to speculate how their results willcompare to score-level fusion.

Intuitively, one tends to agree with Jain et al. (2004)on the potential of feature-level fusion. However, there aremany problems that need to be addressed before one could

expect any improvement over score-level fusion. Our resultsshow that score-level fusion currently remains the more ro-bust (if not accurate) choice as it does not impose any re-striction on the features in terms of size, variance, locationand registration. Feature-level fusion can be considered as aglobal representation but in the feature domain as opposedto the spatial domain. Global features (in the spatial domain)are more sensitive to variations (Zhao et al. 2003) and bycorollary one could imagine that feature-level fusion (globalfeatures in the feature domain) will also be more sensitiveto variations. These arguments by no means rule out the ad-vantages of interaction between multimodal features at thefeature-level. Such interactions are beneficial as one featurecould assist in the normalization or selection of another fea-ture.

8 Conclusion

We presented a novel feature-based algorithm for the recog-nition of textured 3D faces. We proposed a keypoint detec-tion algorithm which can repeatably identify locations ona face where shape variation is high. Moreover, a uniquelocal 3D coordinate basis can be defined at each keypointwhich allows for the extraction of highly descriptive 3D fea-tures. The 3D features were fused with existing 2D SIFTfeatures at the score and feature-level and the performanceof both fusion techniques was compared. We also proposed agraph-based feature matching algorithm and tested it on thelargest publicly available corpus of textured 3D faces. Ouralgorithm has an Equal Error Rate (EER) of 0.45% (neutral

12 Int J Comput Vis (2008) 79: 1–12

versus all case) and has the potential for further improve-ments if integrated with other features and/or classifiers in amulti-algorithm setup.

Acknowledgements We would like to thank the FRGC (Phillips etal. 2005) organizers for the face data, D’Erico for the surface fittingcode, Lowe and El-Maraghi for the SIFT code. This research is spon-sored by an ARC Discovery Grant DP0664228.

References

Al-Osaimi, F., Bennamoun, M., & Mian, A. S. (2006). Illuminationnormalization for color face images. In International symposiumon visual computing (pp. 90–101).

Belhumeur, P., Hespanha, J., & Kriegman, D. (1997). Eigenfacesvs. fisherfaces: recognition using class specific linear projection.IEEE Transactions on Pattern Analysis and Machine Intelligence,19, 711–720.

Besl, P. J., & McKay, N. D. (1992). Reconstruction of real-world ob-jects via simultaneous registration and robust combination of mul-tiple range images. IEEE Transactions on Pattern Analysis andMachine Intelligence, 14(2), 239–256.

Bowyer, K. W., Chang, K., & Flynn, P. (2006). A survey of approachesand challenges in 3D and Multi-modal 3D + 2D face recognition.Computer Vision and Image Understanding, 101(1), 1–15.

D’Erico, J. (2006). Surface fitting using gridfit. MATLAB Central FileExchange Select.

Gokberk, G., Salah, A., & Akarun, L. (2005). Rank-based decision fu-sion for 3d shape-based face recognition. In International confer-ence on audio and video-based biometric person authentication(pp. 1019–1028).

Gonzalez, R. C., & Woods, R. E. (1992). Digital image processing.Reading: Addison–Wesley.

Huang, J., Heisele, B., & Blanz, V. (2003). Component-based facerecognition with 3d morphable models. In International confer-ence on audio and video-based biometric person authentication.

Husken, M., Brauckmann, M., Gehlen, S., & Malsburg, C. (2005),Strategies and benefits of fusion of 2d and 3d face recognition.In IEEE workshop on FRGC experiments.

Jain, A. K., Ross, A., & Prabhakar, S. (2004). An introduction to bio-metric recognition. IEEE Transactions on Circuits and Systemsfor Video Technology, 14(1), 4–20.

Jones, M., & Viola, P. (2003). Face recognition using boosted localfeatures. In IEEE international conference on computer vision.

Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

Lu, X., Jain, A. K., & Colbry, D. (2006). Matching 2.5D scans to 3Dmodels. IEEE Transactions on Pattern Analysis and Machine In-telligence, 28(1), 31–43.

Maurer, T., Guigonis, D., Maslov, I., Pesenti, B., Tsaregorodtsev,A., West, D., & Medioni, G. (2005). Performance of geometrixActiveIDTM 3D face recognition engine on the FRGC data. InIEEE workshop on FRGC experiments.

Mian, A. S., Bennamoun, M., & Owens, R. A. (2006a). 2D and 3Dmultimodal hybrid face recognition. In European conference oncomputer vision (pp. 344–355).

Mian, A. S., Bennamoun, M., & Owens, R. A. (2006b). A novel repre-sentation and feature matching algorithm for automatic pairwiseregistration of range images. International Journal of ComputerVision, 66(1), 19–40.

Mian, A. S., Bennamoun, M., & Owens, R. A. (2006c). Automatic 3Dface detection, normalization and recognition. In 3D data process-ing, visualization and transmission.

Mian, A. S., Bennamoun, M., & Owens, R. A. (2006d). Face recogni-tion using 2D and 3D multimodal local features. In Internationalsymposium on visual computing (pp. 860–870).

Mian, A. S., Bennamoun, M., & Owens, R. A. (2006e). Three-dimensional model-based object recognition and segmentation incluttered scenes. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 28(10), 1584–1601.

Mian, A. S., Bennamoun, M., & Owens, R. A. (2007). An efficientmultimodal 2D–3D hybrid approach to automatic face recogni-tion. IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 29(11), 1927–1943.

Passalis, G., Kakadiaris, I., Theoharis, T., Tederici, G., & Murtaza, N.(2005). Evaluation of 3D face recognition in the presence of facialexpressions: an annotated deformable model approach. In IEEEworkshop on FRGC experiments.

Phillips, P. J., Flynn, P. J., Scruggs, T., Bowyer, K. W., Chang, J., Hoff-man, K., Marques, J., Min, J., & Worek, W. (2005). Overviewof the face recognition grand challenge. In IEEE computer visionand pattern recognition (pp. 947–954).

Ross, A., & Govindarajan, R. (2005) Feature Level Fusion Using Handand Face Biometrics. In Biometric technology for human identifi-cation. Bellingham: SPIE.

Turk, M., & Pentland, A. (1991). Eigenfaces for recognition. Journalof Cognitive Neuroscience, 3, 71–86.

Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003).Face recognition: a literature survey. In ACM computing survey(pp. 399–458).