Btp viewmorph

Face Recognition Using View Morphing

Motivation

• View morphing has tremendously been used in special effects.

• It allows us to combine the salient features of different images into one.

• In surveillance and security systems, where cameras are used to capture the footage of people coming in and out. It is tough to capture the perfect frontal face of any person.

Motivation

• We often get faces from certain angle.• In such scenarios it becomes very difficult to

perform face recognition on all the faces seen in the footage and get the correct results.

Motivation• Face recognition technology is

rapidly growing as evidenced by the new surveillance system.

• Here is an example of ‘surveillance system that can recognize a face from 36 million other faces in one second’, from Hitachi.

• It processes faces as it records and keep a pre indexed gallery of faces rather than a messy footage.

• Super speed is achieved as it compares the faces in real time by pre grouping faces with similar features, so that it can narrow down the field of search quickly.

Source : Hitachi Face Recognition System in Surveillance Systems http://mashable.com/2012/03/23/hitachi-face-recognition/

Motivation

• Limitations – The software assumes that people are looking

right into the camera or facing not more than 30 degrees of the centre.

– Also it requires the people to be close to the camera. Anything smaller than 40 x 40 pixel square face is not recognizable at all.

Research Problem

Can we use View Morphing in Recognition systems ?

• As we see, for any surveillance system we need a frontal face to perform recognition.

• We investigate through our study that whether we can generate a face given the footage images which can be used for face recognition.

Research Problem

• Primary focus of the research was to be able to perform view morphing on face images with different poses and expressions to generate intermediate pose and expression images which capture all the salient features of the face.

Datasets Used

• For pose variant images we used MULTI PIE database.

Sample images from the database

Dataset Used• For expression variant images we used JAFFE database.

Sample images from the database

Introduction

• View morphing is a technique used for transforming one image into another.

• Hollywood films and Disney uses morphing for effects and animation.

• Algorithms has been used to generate morphing between images of face of different people as well as between images of face of individuals.

• Warping and Cross Fading are the two techniques involved in any kind of morphing.

Introduction

• Cross Fading is simple but does not produce smooth transitions, major problem is to warp.

• Cross Fading generate a series of frames that are linear combinations of the original images.

Example of cross fading,Source : http://odonnell- wiki.cs.uchicago.edu/index.php/TenStepCrossfade

Introduction

• For smoother transitions it is necessary to use warping.

• Warping transforms shape where as cross dissolve transform color only color.

• We will now demonstrate the two different approaches that were taken during the project involving different kinds of warping and cross fade techniques.

Approach 1

For Pose Variant Face Images

Feature Based Field MorphingOutline of the process

Input Image 1 & image 2Input Image1

& image 2

Feature Based Field Morphing

• Pre Processing– Estimating Fundamental Matrix (F) using 8 Point

Algorithm.– Finding Eigen vectors of F , the epipoles.– Rotate the epipolar vectors and compute the

transform.– Apply the transform to get the prewarped image.

Estimating F using 8-point algorithm

• To calculate F, the user first inputs corresponding feature points from both the images.

• These points are used to perform a mapping between the two images.

Estimating F using 8-point algorithm

• The fundamental matrix F is defined by

0Fxx'

for any pair of matches x and x’ in two images.

• Let x=(u,v,1)T and x’=(u’,v’,1)T,

333231

232221

131211

fff

fff

fff

F

each match gives a linear equation

0'''''' 333231232221131211 fvfuffvfvvfuvfufvufuu

8-point algorithm

0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

vuvvvvuuuvuu

vuvvvvuuuvuu

vuvvvvuuuvuu

nnnnnnnnnnnn

• In reality, instead of solving , we seek f to minimize , least eigenvector of .

0Af

Af AA

8-point algorithm – Problem?• F should have rank 2• To enforce that F is of rank 2, F is replaced by F’ that

minimizes subject to the rank constraint. 'FF

• This is achieved by SVD. Let , where

, let

then is the solution.

VUF Σ

3

2

1

00

00

00

Σ

000

00

00

Σ' 2

1

VUF Σ''

Image Transformation

• After Computing F we now have variables that can be used to transform the image and compute the pre-warped image.

• Following is an example from the experiment.

The first two images are the original images with features marked for estimating F. The last two images are the resultant pre-warped images.

Feature Morphing Algorithm

• Step 1– Input corresponding feature LINES. – For every line PQ in source image there is a

matching line P’Q’ in the destination image.

Feature Morphing Algorithm

• Step 2

Multiple Pair of Lines: Problem?

Step 3: Algorithm

Feature Morphing

• Step 4– Cross dissolve.– First image fade out while second image fade in.

Blend = (1-t) * Source + t * Destination , 0 <= t <= 1

Results

Results

Results

Approach 2

For Expression Variant Face Images

Mesh warping

• Mesh warping algorithms relates features with non-uniform mesh in the source and destination images.

• The original images are first broken down into small regions then these regions are mapped onto each other to produce a morph.

Mesh Warping

Source : Davis.wpi.edu/~matt/courses/morph

Mesh Warping Algorithm

• Two step process– In the first step, we generate the mapping by the

process of scattered data interpolation. The interpolation technique that we have attempted to use is the biharmonic spline interpolation, built in the Matlab Programming Environment.

– In the second step, the mapping is applied incrementally to shift the features from the first image to the second and vice versa.

– The two sequences are combined via a linear cross fade to form the final morph.

Process of Generating Mapping Arrays

1. Input 2 arrays of size n x 2 containing the coordinates of n corresponding points in the initial and final images: Ri and Rf .

2. Compute the difference array Subtract Ri from Rf and split the resultant array into column vectors [Hx, Hy] = Rf - Ri.

3. Now create 2 arrays equal in size to the original image: Mx and My.

4. Create data points in the arrays Mx and My using Hx and Hy

1. Mx(Rx,i, Ry,i) = Hxi

2. My(Rx,i, Ry,i) = Hyi

3. Now Apply the interpolation (cubic or biharmonic spline) to assign values in Mx and My.

4. Save Mx and My.

The Interpolation Process 1. Input number of frames over, which morph should occur, N+1. Also,

mapping arrays Mx and My and initial and final images Ii and If.

2. Create array Fn of equal size to images.

3. Define ‘frame weighting’ W = (n−1)/(N-1) W’= (1-W)

4. For all values of (x,y) within Fn.

1. Define a = Mx(x,y)*W ,

b = My(x,y)*W

2. Set Fn(x,y) = W*I(x+a,y+b) + W’*I(x-a,y-b)

5. Compile all the frames from F1 to Fn.

Interface

• The Matlab program asks the user to first select the two input images.

Interface

• The user then selects the critical points between the two images and saves them into respective .mat files.

Resultant imagesThe images on the top left corner and bottom right corner are nearly the original images.

30 intermediate frames were generated between the two faces with different expressions.

The images shows 12 of the 30 intermediate frames.







Experiments – Face Recognition

• 4 experiments were performed on the resultant images. • For face recognition we used Local Binary Pattern and

Chi Square Distance measure. • LBP is a feature vector used for classification in

computer feature. • It divides the image into cells and computes a 8 digit

binary number for each cell using the neighboring cells. • A histogram over the cells are computed using the

frequency of the each number occurring.

Experiment 1 Field Morphing Case 1

Scenario Before Morphing

Data Set Multi Pie

Number of Subjects in gallery 30

Number of images per subject 2

Difference of Pose

Angle difference 30 degrees

Number of intermediate images generated

0

Total number of images in gallery 60

Number of test images 30

Number of correct matches 22

Percentage Accuracy 73.3

Experiment 2 Field Morphing Case 2

Scenario After Morphing

Data Set Multi Pie



Difference of Pose

Angle difference 30 degrees


20




Percentage Accuracy 66.6

Result of Face RecognitionAfter morphingTop row shows an example of mismatchBottom row shows an example of correct match.

Experiment 3 Mesh Warping Case 1

Scenario Before Morphing

Data Set JAFFE



Difference of Expression

Angle difference 0


0




Percentage Accuracy 75

Experiment 4 Mesh Warping Case 2

Scenario After Morphing

Data Set JAFFE



Difference of Expression

Angle difference 0


30




Percentage Accuracy 60

Result of Face RecognitionAfter MorphingTop row shows an example of mismatchBottom row shows an example of correct match.

Analysis – Feature Morphing

1. All the salient features of the face were restored in all the frames.

2. There were few distortions but those were not important if we see from a face recognition point of view.

3. While performing warping, the shape and size of the images change, due to which the LBP features of the similar frames change, leading to low accuracy.

4. Also the test images have to warped differently using one of the gallery image so that the feature vector can be compared, which can lead to another variant of shape.

5. LBP uses unwanted information from the images to compute the feature vector.

Analysis – Mesh Warping

• For JAFFE database and mesh warping algorithm there was no such problem of shape and size of the images.

• The only anticipated reason for reduced accuracy was the not so perfect test images.

Conclusion

• From the above discussion and analysis we conclude that view morphing which was never used for face recognition purpose before is a valid tool, with few shortcomings which can be removed.

• Morphing is a common technique in special effects and is now becoming and important part of computer vision and pattern recognition.

• Face recognition can be seen as a new application of the existing view morphing algorithms.

References

• View Morphing - Seitz and Dyer, 1996• View Interpolation - Chen and Williams, 1993• Image Morphing - Beier and Neely, 1992• Image Morphing - Mullens and Notley, 2006• Image Morphing: A survey - G. Wolberg, 1998

Education

Btp viewmorph