ABSTRACT

The face recognition is a fairly controversial subject right now. A system such as this can recognize and track dangerous criminals and terrorists in a crowd, but some contend that it is an extreme invasion of privacy. The proponents of large-scale face recognition feel that it is a necessary evil to make our country safer. It could benefit the visually impaired and allow them to interact more easily with the environment. Also, a computer vision-based authentication system could be put in place to allow computer access or access to a specific room using face recognition. Another possible application would be to integrate this technology into an artificial intelligence system for more realistic interaction with humans. We propose a face recognition method called the PCA (Principal Component Analysis). Principal Component Analysis (PCA) is a statistical method under the broad title of factor analysis. The purpose of PCA is to reduce the large dimensionality of the data space (observed variables) to the smaller intrinsic dimensionality of feature space (independent variables), which are needed to describe the data economically. This is the case when there is a strong correlation between observed variables. The jobs which PCA can do are prediction, redundancy removal, feature extraction, data compression, etc. Because PCA is a known powerful technique which can do something in the linear domain, applications having linear models are suitable, such as signal processing, image processing, system and control theory, communications, etc. The main idea of using PCA for face recognition is to express the large 1-D vector of pixels constructed from 2-D face image into the compact principal components of the feature space. This is called eigenspace projection. Eigenspace is calculated by identifying the eigenvectors of the covariance matrix derived from a set of fingerprint images (vectors).

1 | P a g e

1. INTRODUCTION

1.1. PCA

The Principal Component Analysis (PCA) is one of the most successful techniques that have been used in image recognition and compression. PCA is a statistical method under the broad title of factor analysis. The purpose of PCA is to reduce the large dimensionality of the data space (observed variables) to the smaller intrinsic dimensionality of feature space (independent variables), which are needed to describe the data economically. This is the case when there is a strong correlation between observed variables. The jobs which PCA can do are prediction, redundancy removal, feature extraction, data compression, etc. Because PCA is a classical technique which can do something in the linear domain, applications having linear models are suitable, such as signal processing, image processing, system and control theory, communications, etc. Face recognition has many applicable areas. Moreover, it can be categorized into face identification, face classification, or sex determination. The most useful applications contain crowd surveillance, video content indexing, personal identification (ex. driver’s licence), mug shots matching, entrance security, etc. The main idea of using PCA for face recognition is to express the large 1-D vector of pixels constructed from 2-D facial image into the compact principal components of the feature space. This can be called eigenspace projection. Eigenspace is calculated by identifying the eigenvectors of the covariance matrix derived from a set of facial images (vectors).

1.2. MATHEMATICS OF PCA

A 2-D facial image can be represented as 1-D vector by concatenating each row (or column) into a long thin vector. Let’s suppose we have M vectors of size N (= rows of image £ columns of image) representing a set of sampled images. pj’s represent the pixel values.

x i= [ p1 …… p N ] T ;i=1 ;……; M (1)

The images are mean centred by subtracting the mean image from each image vector. Let m represent the mean image.

m=1 /M∑i=1

M

x i (2)

And let w i be defined as mean centred image

w i= x i – m (3)

Our goal is to find a set of e i which have the largest possible projection onto each of the w i. We wish to find a set of M orthonormal vectorse i for which the quantity

λ i=1M

∑i=1

M

(e iT wn)

2 (4)

2 | P a g e

is maximized with the orthonormality constraint

e lT ek=δ lk (5)

It has been shown that the e i and λ i are given by the eigenvectors and eigenvalues of the covariance matrix

C = W W T (6)

where W is a matrix composed of the column vectors w i placed side by side. The size of C is N × N which could be enormous. For example, images of size 64 × 64 create the covariance matrix of size 4096×4096. It is not practical to solve for the eigenvectors of C directly. A common theorem in linear algebra states that the vectors e i and scalars λ i can be obtained by solving for the eigenvectors and eigenvalues of the M ×M matrixW T W . Let d i and μi be the eigenvectors and eigenvalues of W T W , respectively. W T W d i=μi d i (7)

By multiplying left to both sides by W

WW T (W d i) = μi(Wdi) (8)

which means that the first M - 1 eigenvectors e i and eigenvalues λ i of W W T

are given by Wdi and μirespectively. Wdineeds to be normalized in order to be equal to e i Since we only sum up a finite number of image vectors, M, the rank of the covariance matrix cannot exceed M - 1(The -1 come from the subtraction of the mean vector m).

The eigenvectors corresponding to nonzero eigenvalues of the covariance matrix produce an orthonormal basis for the subspace within which most image data can be represented with a small amount of error. The eigenvectors are sorted from high to low according to their corresponding eigenvalues. The eigenvector associated with the largest eigenvalue is one that reflects the greatest variance in the image. That is, the smallest eigenvalue is associated with the eigenvector that finds the least variance. They decrease in exponential fashion, meaning that the roughly 90% of the total variance is contained in the first 5% to 10% of the dimensions.A facial image can be projected onto M ı(≪M ) dimensions by computing

Ω=[v1 v2 v3 …… ..vM 1]T (9)

where v i=e iT wi . v i

is the it h coordinate of the facial image in the new space, which came to be the principal component. The vectors e iare also images, so called, eigenimages, or eigenfaces in our case. They can be viewed as images and indeed look like faces. So,Ω describes the contribution of each eigenface in representing the facial image by treating the eigenfaces as a basis set for facial images. The simplest method for determining which face class provides the best description of an input facial image is to find the face class k that minimizes the Euclidean distance

ϵ k=‖Ω−Ωk‖ (10)

3 | P a g e

where Ωkis a vector describing the k t h face class. If ϵ k is less than some predefined threshold θϵ, a face is classified as belonging to the class k.

2. FACE RECOGNITION

Once the eigenfaces have been computed, several types of decision can be made depending on the application. What we call face recognition is a broad term which may be further specified to one of following tasks:

identification where the labels of individuals must be obtained, recognition of a person, where it must be decided if the individual has already been

seen, Categorization where the face must be assigned to a certain class.

PCA computes the basis of a space which is represented by its training vectors. These basis vectors, actually eigenvectors, computed by PCA are in the direction of the largest variance of the training vectors. As it has been said earlier, we call them eigenfaces. Each eigenface can be viewed a feature. When a particular face is projected onto the face space, its vector into the face space describes the importance of each of those features in the face. The face is expressed in the face space by its eigenface coefficients (or weights). We can handle a large input vector,facial image, only by taking its small weight vector in the face space. This means that we can reconstruct the original face with some error, since the dimensionality of the image space is much larger than that of face space. In this report, let’s consider face identification only. Each face in the training set is transformed into the face space and its components are stored in memory. The face space has to be populated with these known faces. An input face is given to the system, and then it is projected onto the face space. The system computes its distance from all the stored faces. However, two issues should be carefully considered:

What if the image presented to the system is not a face? What if the face presented to the system has not already learned, i.e., not stored as a

known face?

The first defect is easily avoided since the first eigenface is a good face filter which can test whether each image is highly correlated with itself. The images with a low correlation can be rejected. Or these two issues are altogether addressed by categorizing following four different regions:

1. Near face space and near stored face⇒ known faces2. Near face space but not near a known face ⇒ unknown faces3. Distant from face space and near a face class ⇒ non-faces4. Distant from face space and not near a known class ⇒ non-faces

Since a face is well represented by the face space, its reconstruction should be similar to the original; hence the reconstruction error will be small. Non-face images will have a large reconstruction error which is larger than some threshold θr. The distance ϵ kdetermines whether the input face is near a known face.

4 | P a g e

3. REQUIREMENTS

3.1. HARDWARE REQUIREMENTS

o Camera integrated system

3.2. SOFTWARE REQUIREMENTS

o Windows OS

o Visual Studio

o Open-CV version 2.10

5 | P a g e

4. UNDERLYING PRINCIPLE

4.1. Main idea behind eigenfaces

- Suppose G is an N2 ×1vector, corresponding to an N ×N face image I .- The idea is to represent Γ ¿ mean face) into a low-dimensional space: Φ̂ - mean = w1u1+w2u2+………….+w k uk(K<<N2)

4.2. Computation of the Eigenfaces

Step 1: obtain face images I 1, I 2 , I 3 , …. , I M (training faces)(very important: the face images must be cantered and of the same size)

Step 2: represent every image I i as a vector Γ i

Step 3: compute the average face vector Ψ :

Ψ = 1M

∑i=1

M

Γ i

Step 4: subtract the mean face:

Φ i=Γ i−Ψ

Step 5: compute the covariance matrix C:

C =1M∑n=1

M

Φn ΦnT=A AT ¿

where A =[Φ1Φ2……ΦM ] (N2 ×M matrix)

Step 6: compute the eigenvectors uiof AAT

The matrix AAT is very large --> not practical !! Step 6.1: consider the matrix AT A (MxM matrix) Step 6.2: compute the eigenvectors v iof AT A AT A v i =μi v i

What is the relationship between usiand v i? AT A v i=μi v i⇒ AAT A v i=μ i A v i⇒ CA v i=μ iAv i or μi uiwhere ui Av i

Thus, AAT and ATA have the same eigenvalues and their eigenvectors are related as follows: μi =Av i !! Note 1: AAT can have up to N2 eigenvalues and eigenvectors. Note 2: AT A can have up to M eigenvalues and eigenvectors.

6 | P a g e

Note 3: The M eigenvalues of AT A (along with their corresponding eigenvectors) correspond to the M largest eigenvalues of A AT (along with their corresponding eigenvectors). Step 6.3: compute the M best eigenvectors of A AT : ui=¿Av i

(important: normalize uisuch that ||ui|| =1)Step 7: keep only K eigenvectors (corresponding to the K largest eigenvalues)

7 | P a g e

4.3. REPRESENTING FACES ONTO THIS BASIS

- Each face (minus the mean) Φ i in the training set can be represented as a linearcombination of the best K eigenvectors:

Φ i−mean=∑j=1

K

w ju j ,(w j=u jT Φ i)

(we call the u j ’s eigenfaces)

- Each normalized training face Φ i is represented in this basis by a vector:

Ωi=w1

i

….wK

i , i=1,,2.......,M

8 | P a g e

4.4. FACE RECOGNITION USING EIGEN FACES

- Given an unknown face image Γ(centered and of the same size like the trainingfaces) follow these steps:

Step 1: normalize Γ :Φ=Γ−Ψ

Step 2: project on the eigenspace

Φ̂=∑i=1

K

w iu i(wi=uiT Φ)

Step 3: represent Φ as:

Ω=w1

…w k

Step 4: find er=minl‖Ω−Ωl‖

Step 5: ifer,< T r then Γ is recognized as face l from the training set.- The distance er, is called distance within the face space (difs)

Comment: we can use the common Euclidean distance to compute er, however, ithas been reported that the Mahalanobis distance performs better:

‖Ω−Ωk‖=∑i=1

k1λi

¿

(variations along all axes are treated as equally significant)

9 | P a g e

4.5. FACE DETECTION USING EIGENFACES

- Given an unknown image ΓStep 1: compute Φ=Γ−ΨStep 2: compute

Φ̂=∑i=1

K

w iu i(wi=uiT Φ)

Step 3: compute ed=‖Φ−Φ̂‖Step 4: if ed< T d, then Γ is a face.- The distance ed is called distance from face space (dffs)

10 | P a g e

5. TECHONOLOGY USED

5.1. MICROSOFT VISUAL C++

Visual C++ is IDE of Microsoft to develop application using C++ & its STL (Standard Template Library). Application developed by visual C++ can be of different types such as MFC application, win console application etc. In our case, we will be developing win console application using OpenCV.

5.2. OPEN CV

OpenCV [OpenCV] is an open source (see http://opensource.org) computer vision library available from http://SourceForge.net/projects/opencvlibrary. Th e library is written in C and C++ and runs under Linux, Windows and Mac OS X. Th ere is active development on interfaces for Python, Ruby, Matlab, and other languages. OpenCV was designed for computational efficiency and with a strong focus on realtime applications. OpenCV is written in optimized C and can take advantage of multicore processors. If you desire further automatic optimization on Intel architectures [Intel], you can buy Intel’s Integrated Performance Primitives (IPP) libraries [IPP], which consist of low-level optimized routines in many different algorithmic areas. OpenCV automatically uses the appropriate IPP library at runtime if that library is installed. One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that helps people build fairly sophisticated vision applications quickly. The OpenCV library contains over 500 functions that span many areas in vision, including factory product inspection, medical imaging, security, user interface, camera calibration, stereo vision, and robotics. Because computer vision and machine learning often go hand-in hand, OpenCV also contains a full, general-purpose Machine Learning Library (MLL). This sub library is focused on statistical pattern recognition and clustering. The MLL is highly useful for the vision tasks that are at the core of OpenCV’s mission, but it is general enough to be used for any machine learning problem.

11 | P a g e

5.2.1 OpenCV Structure and Content

OpenCV is broadly structured into fi ve main components, four of which are shown in Figure. The CV component contains the basic image processing and higher-level computer vision algorithms; ML is the machine learning library, which includes many statistical classifiers and clustering tools. HighGUI contains I/O routines and functions for storing and loading video and images, and CXCore contains the basic data structures and content.

12 | P a g e

6. USER DEFINED FUNCTIONS AND THEIR PURPOSE

6.1. printusage This function is used to control the program from the command line user interface. Depending upon the user input it transfers the control of program to either training phase or testing phase or online recognition of a face from camera.

6.2. learn

This function is used for offline recognition mode where it trains from the data given in text file and store the trained data into the file ‘facedata.xml ‘.

6.3. doPCA

This function is used to perform PCA operation on the training faces.

6.4. StoreTrainingdata This function is used to store the recognition data as an xml file.

6.5. loadTrainingdata

This function is used to load the training data from the file 'facedata.xml'.

6.6. Findnearestneighbour

This function is used to find the most likely person based on detection. It returns the index of image, and stores the confidence value into variable pConfidence.

6.7. loadfaceImgArray

This function is used to read the names & image of people from a text file, and load all those images listed.

6.8. Recognizefilelist It is used to recognize the face in each of the test images given, and compare the results with the truth.

6.9. Getcameraframe It is used to grab the next camera frame. It waits until the next frame is ready, and provides direct access to it, as a result it do NOT modify the returned image or free it. It automatically initializes the camera on the first frame.

13 | P a g e

6.10. Convertimageto grayscale

It is used to return a new image that is always greyscale, irrespective of the input type of image. It free the returned image using cvReleaseImage() when finished.

6.11. cropImage It returns a new image that is a cropped version of the original image.

6.12. resizeImage

It creates a new image copy that is of a desired size. Remember to free the new image later.

6.13. convertFloatImagetoUcharImage

It gets an 8-bit equivalent of the 32-bit Float image. It returns a new image, so remember to call cvReleaseImage() on the result.

6.14. SaveFloatImage

It stores a greyscale floating-point CvMat image into a BMP/JPG/GIF/PNG image,since cvSaveImage() can only handle 8bit images (not 32bit float images).

6.15. detectFaceInImage

Perform face detection on the input image, using the given Haar cascade classifier. It returns a rectangle for the detected region in the given image.

6.16. Retrainonline It is used to re-train the new face recorded in a database without shutting down.Depending on the number of images in the training set and number of people, it might take 30 seconds or so.

14 | P a g e

7. Snapshots of project on running mode

Figure : It represents start of the project where user have to enter a command, according to which it wil run further.

Figure : It represents status of the program when user has entered recognize as its command.

15 | P a g e

Figure : It represents status of the program where the program is trying to recognize a face

Figure: It represents a camera frame which has been captured by camera. After detecting a face in the frane the program identifies it and returns name of the function and its confidence level.

16 | P a g e

8. Future Work

There are many interesting problems that remain in the area of face recognition. One problem is image preprocessing prior to the application of the eigenface method. It may be possible to gain better accuracy in classification if one segments the spectrum of people into different spaces. For example, if one was able to determine if an image was of a man or a woman, one could use this categorization to send an image to one of two classifiers, each specifically trained with that type of individual in mind. This would mean that there would be a set of eigenfaces specifically for males and one specifically for females (face spaces with gender, so to speak). Work in this area has been done by Lizama, Waldoestl and Nickolay [4], however it would be interesting to extend it to use eigenfaces to act as the gender classifier as well. A general face-space would be created in addition to the male and female face-spaces, with the sole purpose of being used to classify an image as male or female.Another area of future work is improving our neural network classifier. As mentioned in thepreviously, it is possible to construct the network to take its input directly from the image data rather from the vector that results from an images projection into face-space. Perhaps learning the face projection function could increase the accuracy of the neural network classifier. Additionally, more experiments are needed to see if there are other ways to tweak the network configuration to produce better results.

17 | P a g e

9. Reference

Learning OpenCV –Computer Vision with the openCV Library O’Reilly Publication Gary Bradsk1 and Andrian Kehlar

M.A. Turk and A.P. Pentland, “Face Recognition Using Eigenfaces”, IEEE Conf. on Computer Vision and Pattern Recognition, pp. 586-591, 1991.

Kyungnam Kim ” Face Recognition using Principle Component Analysis”

18 | P a g e