Soft Biometrics at CUBS
Venu GovindarajuCUBS, University at Buffalo
Background
Traits of biometrics Universality Distinctiveness Permanence Collectability Acceptability
Present perfect? No biometric is truly universal. It is estimated that 2-
4% of the population have unusable fingerprints Each biometric has a lower bound for errors
(constraint of algorithm + individuality) Individual biometrics need to be augmented by other
biometrics (multi-modal) or traits (soft biometrics)
Soft Biometrics
Soft Biometrics Not very distinctive Can be used to augment
regular biometrics Not typically used during
verification/identification More intuitive than strong
biometrics
Definition[1] Soft biometric traits are those characteristics that provide some
information about the individual but are not distinctive enough to sufficiently differentiate any two individuals
[1]
[1] A. K. Jain, S. Dass, K. Nandakumar, “Soft Biometrics for Personal Identification”, SPIE Defense and Security Symposium 2003
Soft Biometrics : Examples
Other classification Continuous: Age, Height, Weight etc. Discrete: Gender, Eye Color, Ethnicity etc.
Motivation Heckathorn[3] have shown that a combination of
personal attributes can be used to identify the individual reliably
Binning and Indexing Hardening primary biometric Speech Recognition Can be used to tune individual biometrics Socially aware computing (call centers)?
Extracting Soft Biometric Traits
Devices Color video Stereo images
Challenges Controlled vs Uncontrolled environment
Pose variations Illumination variation Complex backgrounds
Feature selection and extraction Features used in traditional biometrics do not encode soft
biometric traits
Decision systems (soft thresholds)
Problems in Representation
Purely statistical features
Fuzzy class boundaries
Soft Biometrics Research at CUBS
Speech Gender Identification Accent Identification
Face Face Catalog: Semantic Face Retrieval Gender Classification
Skin Skin spectroscopy
Soft Biometric Traits in Speech
Gender There exists a difference in the pitch period between genders This difference is fundamental in the discrimination between
males and females Accent[1]
Temporal features: onset time, closure/voicing/word duration Prosodic/Intonation slope patterns Formant frequencies
Age The average power measurement and speech rate are used
as indicators for measurement of agedness in a speaker
[1]A Study of Temporal Features and Frequency characteristics in American English Foreign Accent
L.M. Arslan, J.H.L. Hansen , Journal of the Acoustical society of America, July 1997
Uses of Soft Biometrics in Speech
Soft Biometrics for binning
PrimaryBiometric
Soft Biometric(s)P(w|x1) P(w|x1y)
Soft Biometrics for improving accuracy
Loose Gender Classification (PITCH)
3 Methods Fast Fourier Transform Linear Predictive Analysis Cepstral Analysis
Data 75 files Males -41, Females -34
Male Low Male Medium Male High Female Low Female Medium Female High
132Hz 156Hz 171Hz 205Hz 230Hz 287Hz
Results
Definition of Accent (linguistics)
An accent is the perceived peculiarities of pronunciation and intonation of a speaker or group of speakers
A foreign accent is defined in a way that the phonology of the spoken language is modified by the phonology of another language, more familiar to the speaker
3 major language groups American Chinese Indian
Proposed Approach for Accent
First identify the accent markers Determine the effect of gender and co-articulation Initially develop a text dependent model Accumulate evidence over time Features:
formants phoneme duration instantaneous (mel)cepstral slopes
HMMs
Accent Markers A look at various non-native pronunciations of English
CHINESE ‘r’ read sometimes as ‘l’ or ‘w’ ‘v’ read as ‘w’ ‘th’ read as ‘d’ ‘n’ and ‘l’ often confused Often drop articles like ‘the’ and ‘a’
INDIAN SUBCONTINENT Use of the rhotic ‘r’ Use of rolling ‘l’ Fast speech tempo with choppy syllables Rhythmic variation of pitch
Webster’s Revised Unabridged Dictionary
Definition of non-native pronunciations of English – wordIQ.com
F2
F2 F2
F2
F3
F3
F3
F3
PLEASE STELLA
SLABS PLASTIC
MALES – PHONEME CONTAINING ‘L’ American - Indian -
F3
F2
BRING
F3
F2
RED
F3
F2
FRESH
MALES – PHONEMES CONTAINING ‘R’ AND ‘AA’
F2
F3
ASK
American - Indian -
FEMALES – SEGMENTED PHONEMES ‘L’, ‘R’, ‘AA’
F3
F2
PLEASE
F3
F2
STELLA
F3
F2
RED
F3
F2
ASK
American - Indian -
Soft Biometrics for Law Enforcement
Novel Forensic System
Law Enforcement Application: Face Catalog
User can select some facial feature to describe.System will prompt the user after each query with the best feature for the next query.
Related Work
Identikit [1] composes faces by putting together transparencies of facial features.
Evofit [2], automate the process of identikits. Phanthomas [3] face composition using elastic
graph matching. CAFIIRIS [4] and Photobook [5] use PCA for face
composition and matching. But general description of users are semantic!
1. V. Bruce, “Recognizing Faces”, Faces as Patterns, pp. 37-58, Lawrence Earlbaum Associates, 19882. Frowd, C.D., Hancock, P.J.B., & Carson, D. (2004). “EvoFIT: A Holistic, Evolutionary Facial Imaging
Technique for Creating Composites”, ACM TAP, Vol. 1 (1)3. “Phantomas: Elaborate Face Recognition “.Product description: http://www.global-security-
solutions.com/FaceRecognition.htm 4. J. K. Wu, Y. H. Ang, P. C. Lam, S. K. Moorthy, A. D. Narasimhalu, ”Facial Image Retrieval,
Identification, and Inference System”
5. A. Pentland, R. Picard, S. Sclaroff, “Photobook: tools for content based manipulation of image
databases”, Proc. SPIE: Storage and Retrieval for Image and Video Databases II, vol. 2185
Face Catalog System Overview
Face Detection
Lip Location and parameterization
Eye Location
Parameterization of other Features
Query Sub-System
Prompting Sub-System
Sem
anti
cD
es c
r ipti
on
Face Image
Database
Meta Database
Input Image
Sorted Images
user
Semantic Face Retrieval System
Enrollment Sub-System Face Detection. Lips and eye detection. Locate and parameterize other
features.
Query Sub-System
Pruning images based on descriptions given? What if user makes a mistake in one of the
description. Ranking images based on their probability of being
the required person is a better idea. Bayesian learning can be used to update probability
of each face being the required one.
Prompting users the feature with highest entropy at each step.
Example Query
Query = []
Query = [Spectacles = Yes]
Query = [Spectacles = Yes + Mustache = Yes]
Query = [Spectacles = Yes + Mustache = Yes + Nose = Big]
Probabilities of Faces
Results Results of Enrollment Sub-system (Database of 150 images)
Results of Query (25 users, 125 test cases)
Top 5 Top 10 Top 15
Average no. of queries.
7.12 5.08 2.49
Features Number of False Accepts
Number of False Rejects
Spectacles 1 2
Mustache 2 4
Beard 4 0
Long Hair 2 8
Balding 1 0
Gender Classification in Images
Gender classification Identifying male or female from facial image
Existing approaches Geometric feature based [1]-[2]
Appearance feature based (raw data feature or PCA + classifier) [3]
Approaches using other features, e.g., wrinkle and skin color [4]
[1] A. Burton, V. Bruce and N. Dench, “What’s the difference between men and women? Evidence from facial measurements,” Perception, vol. 22, pp.153-176, 1993.[2]R. Brunelli and T. Poggio, “Hyperbf network for gender classification,” DARPA Image Understanding Workshop, pp. 311-314, 1992.[3]B.A. Golomb, D.T. Lawrence, T.J. Sejnowski, “Sexnet: A Neural Network Identifies Sex from Human Faces,” Advances in Neural Information Processing Systems3, R.P Lippmann, J.E. Moody, D.S. Touretzky, eds. Pp. 572-577, 1991.[4] J. Hayashi, M. Yasumoto, H. Ito, H. Koshimizu, “Age and gender estimation based on wrinkle texture and color of facial images,”, Proceedings of 16th International Conference on Pattern Recognition, vol. 1, pp. 405 - 408, 11-15 Aug. 2002
Gabor Feature based gender classification system
Feature Extractor Using
Gabor Wavelet
SVM Classifier
Preprocessing (Face detection,
normalization, etc.)
Raw Image
Decision
Facial image Normalization
Mapping feature points to fixed positions
Feature points Centers of two pupils Tip of the nose
Normalized image 64 by 64 Convert from color to grayscale
by averaging RGB components
Gabor feature
Gabor filter and Gabor wavelet [B.S. Manjunath, et al, PAMI, 1996]
jWx
yxyxg
yxyx
22
1exp
2
1),(
2
2
2
2
Gabor Filter:
.0 ,0
thenscales, ofnumber : ns.orientatio ofnumber :;/:
)cossin('),sincos('
integer, 1, ,),,/,,','(),(
KnSm
SKKn
yxayyxax
m, naKnWyxgayxgmm
yxm
mn
Gabor Wavelet:
Fourier Transform
of g(x, y):
2
2
2
2)(
2
1exp),(
vu
vWvuG
yv
xu
2/1
2/1 where
Redundancy reduction [B.S. Manjunath, et al, PAMI, 1996] Let and denote the lowest and highest frequencies of interest are determined by
Gabor feature (cont.)
2
1
2
222
1
1
)2ln2(2ln2
2ln2
2tan
2ln2)1(
)1(
)/(
l
u
l
ulv
lu
Slh
UUU
K
a
Ua
UUa
lU hU
vua ,,
Gabor feature (cont.)
Characteristics of Gabor wavelet A powerful tool to capture changes of signals Selective on certain frequency and orientation by setting
parameters m, n Gabor feature for gender classification
Gabor WT at 4 scalses, 4 orientations (m = 0, .., 3; n = 0, …, 3) Each output image of Gabor WT (64 by 64) is divided into non-
overlapping blocks of the size 2m+2 by 2m+2 (m: the scale number). Average of magnitudes in each block as a feature
Total number of features
)component) (imaginarycomponent) (real( 22 magnitude
13602/646443
0
22
m
m
Gabor feature (cont.)
64.0
08.0
4,4
h
l
U
U
KS
Classification
Features 1360-dimensional training and testing vectors fed into SVM
classifier Classifier
SVM with Gaussian RBF kernel [6] (B. Moghaddam, et al, PAMI 2002)
Adjust γ to minimize error rate 1360 features from Gabor WT (in 4 scales, 4 orientations) of
64×64 input image Training and testing vectors (of 1360 dimensions) normalized into
unit vectors
Experimental Results Dataset: AR face database [A.M. Martinez and R. Benavente, “The AR face
database,” CVC Tech. Report #24, 1998]
Overall: 3265 frontal facial images including 136 Caucasian people (768 by 576, color)
Training: 2246 samples including 91 individuals Testing: 1019 samples including 45 individuals
Test #1 393 regular samples. Accuracy: 96.2%
Test #2 626 irregular samples (occluded by dark sun-glasses or
masks) Accuracy: 92.7%
Method Accuracy of test #1 Accuracy of test #2
Gabor feature + SVM with Gaussian RBF kernel 96.2% 92.7%
Raw data feature + SVM with Gaussian RBF kernel 94.7% 89.8%
Skin Spectroscopy
Measures the composition of the skin using IR(Deep tissue biometric) Based on spectroscopy Fool proof against fake fingers (Can detect liveness) Can be easily integrated into solid state devices Immune to surface degradations Currently implemented by only one Vendor (Lumidigm Inc)
Skin composition
Chromophores in skin
Melanin Absorbs light at all wavelengths Absorbance decreases with increase in wavelength
Hemoglobin Strongest absorption bands in 405 – 430 nm and 540 – 580 nm. Lowest absorption beyond 620 nm Can be used for liveness testing
Collagen, Keratin, Carotene
Spectra of Melanin and Hemoglobin
Sample Skin Spectrum
Sample skin spectrum (contd.)
Sample skin spectrum (contd.)
Results so far
Soft classification based on skin color Melanin index used as indicator of skin color
Spectral difference noticed between different skin locations on the same individual
Recommended
