Upload
duane-blackburn
View
336
Download
3
Tags:
Embed Size (px)
DESCRIPTION
This presentation "reintroduces" watchlist statistics to the biometrics community at the 2004 Biometric Consortium Conference. At this time, many biometric studies and recommendations incorrectly referenced closed set identification statistics for open set applications.
Citation preview
Biometrics Technology
Duane BlackburnFederal Bureau of Investigation
Reintroducing Watchlist Statistics
Biometrics Technology
Face Recognition DebateFall 2001
Could the hijackers have been spotted?
Biometrics TechnologyTayler Persky
Wilfred Moesch
Nicolas Marocco
Allan Olliff
Clayton Weddell
Elinor Pardini
Joe Grammer
Matthew Millner
Gerald Galbraith
Dorothy Shedd
Gregory Dorsey
Nancy Leflore
Tara Kaye DOB: 4/21/74
Divorced Mother of 22002 Income:
$45,328
Dennis Ricks
Face Recognition DebateFall 2001
Can it track me?
Biometrics Technology
Face Recognition DebateFall 2001
Osama bin Laden
Or will this happen?
Biometrics Technology
Why the confusion?
• Confusing– Verification vs. Identification
• incorrect usage of statistics– Identification (open set) vs. Identification
(closed set)– False alarm rate, false accept rate, false
match rate, etc.
Biometrics Technology
Statistics Shortcomings
• ROC (Verification)– Making a claim to your identity
• CMC (Identification)– Each probe is in the database
• Are measuring closed set identification
• Neither relates to the problem!– Solution: develop statistics that measure
this problem.
Biometrics Technology
Chronologically
• Face Recognition at a Chokepoint -Scenario Evaluation Results. http://www.dodcounterdrug.com/facialrecognition
• FRVT 2002. http://www.frvt.org.
• Both available in the Biometrics Catalog, http://www.biometricscatalog.org.
Biometrics Technology
First Step
• Forget what you know!
Biometrics Technology
Baseline Performance
Face Recognition Vendor Test 2002 (FRVT 2002) - http://www.frvt.org
Biometrics Technology
Three Tasks
• Verification – Are you who you say you are?
• Watchlist – Are you in my database? If so, who are you?
• Identification – You are in my database, can I find you?
Biometrics Technology
Verification
• Determines if the claimed identity of a face is correct.
• Metrics:– Probability of Correct Verification
• 1- (False Reject Rate)– False Accept
Claim Response = 0.75
Response = 0.65Claim
Threshold = 0.7
Threshold = 0.6
Threshold = 0.8
Biometrics Technology
Verification Case #1 - Want a low false accept rate
00.10.20.30.40.50.60.70.80.9
1
1
Prob
abili
ty o
f Cor
rect
Ver
ifica
tion
False Accept Rate = 0.1%
Verification
Cognitec
Eyematic
Identix
False Accept Rate = 0.1%
Biometrics Technology
VerificationCase #2 - Want a high verification rate
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
False Accept Rate
Prob
abili
ty o
f Cor
rect
Ver
ifica
tion
= 95
%
Verification
Cognitec
Eyematic
Identix
Probability of Verification = 95%
Biometrics Technology
Verification ROC
Verification
0
0.2
0.4
0.6
0.8
1
1
False Accept Rate = 0.1%
Prob
abili
ty o
f Cor
rect
Ve
rific
atio
n
CognitecEyematicIdentix
Verification
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
Prob
abili
ty o
f Cor
rect
Ve
rific
atio
n =
95%
False Accept Rate
CognitecEyematicIdentix
Biometrics Technology
Watchlist
Dat
abas
e
0.75
0.65
0.60
0.25
0.55
Threshold = 0.7
Threshold = 0.8
Threshold = 0.62
Biometrics Technology
Watchlist
Dat
abas
e
0.75
0.65
0.70
0.25
0.55
Threshold = 0.8
Threshold = 0.7
Biometrics Technology
WatchlistCase #1 - Want a high detect and identification rate
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
False Alarm Rate
Dete
ct a
nd Id
entif
icat
ion
Rate
=
75%
Watchlist
Cognitec
Eyematic
Identix
Detect and Identify = 75%
Biometrics Technology
Watchlist Case #2 - Want a low false alarm rate
00.10.20.30.40.50.60.70.80.9
1
1
Dete
ctio
n an
d Id
entif
icat
ion
Rate
False Alarm Rate = 0.1%
Watchlist
Cognitec
Eyematic
Identix
False Alarm Rate = 0.1%
Biometrics Technology
Watchlist
Watchlist
0
0.2
0.4
0.6
0.8
1
1
False Alarm Rate = 0.1%
Dete
ctio
n an
d Id
entif
icat
ion
Rate
CognitecEyematicIdentix
Watchlist
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
Dete
ct a
nd Id
entif
icat
ion
Rate
= 7
5%
False Alarm Rate
CognitecEyematicIdentix
Biometrics Technology
Identification
Dat
abas
e
0.75
0.65
0.70
0.25
0.55
0.80
Biometrics Technology
Identification
Dat
abas
eThat’s watchlist, not identification!
Biometrics Technology
Identification
• I know you are in the database, but who are you?– Closed universe test.
• Compares your picture to all pictures in the database and lists them according to similarity.
• Metrics:– How often is the top answer correct?– How often is the answer in the top 2?– How often is the answer in the top 3?– How often is the answer in the top n?
Biometrics Technology
Identification
Biometrics Technology
Three Tasks
• Verification – Are you who you say you are?
• Watchlist – Are you in my database? If so, who are you?
• Identification – You are in my database, can I find you?
Make sure you use the correct terms and the correct statistics!
Biometrics Technology
• Biometrics Catalog -http://www.biometricscatalog.org
• FRVT - http://www.frvt.org• Biometric Consortium – http://www.biometrics.org
Government Biometric References