Upload
phungtu
View
229
Download
0
Embed Size (px)
Citation preview
BiasinFaceRecognition:Whatdoesthatevenmean?Andisitserious?
PatrickGrotherInformationTechnologyLaboratory
NationalInstituteofStandardsandTechnologyUnitedStatesDepartmentofCommerce
BiometricsCongress,LondonNovember2,2017
QuotingGeorgetown’sReport:“ThePerpetualLine-up”
• Themostprominentstudy[Klareetal.]foundthatseveralleadingalgorithmsperformedworseonAfricanAmericans,women,andyoungadultsthanonCaucasians,men,andolderpeople,respectively.216
• IfthesuspectisAfricanAmericanratherthanCaucasian,thesystemismorelikelytoerroneouslyfailtoidentifytherightperson,potentiallycausinginnocentpeopletobebumpedupthelist—andpossiblyeveninvestigated
• “Q:IstheBookingPhotoComparisonSystembiasedagainstminorities[?]”• “A:No…itdoesnotseerace,sex,orientationorage.Thesoftwareismatchingdistanceandpatternsonly,
notskincolor,ageorsexofanindividual.”- FrequentlyAskedQuestions,SeattlePoliceDepartment
• Thereisnoindependenttestingregimeforraciallybiasederrorrates… twomajorfacerecognitioncompaniesadmittedthattheydidnotrunthesetests
• Racialbiasintrinsictoanalgorithmmaybecompoundedbyoutsidefactors.AfricanAmericansaredisproportionatelylikelytocomeintocontactwith—andbearrestedby—lawenforcement.218
[Bias]
[Priors]
[NoTestsForBias]
ClareGarvie,AlvaroM.Bedoya,JonathanFrankleThePerpetualLine-upUnregulatedPoliceFaceRecognitionInAmerica
GeorgetownLawCenteronPrivacyandTechnologyOctober18,2016https://www.perpetuallineup.org/
[Awareness]
[Conseq-uence]
RelevantQuantities
PPriorProbability
Demographicsinlawenforcementdatabases≠generalpopulation.
InUSA:• Moremale• Moreblack• Younger
FailuretoEnrol(ImageQuality)
+Exposure
-Exposure
FNMR1:1FalseRejection
FMR1:1FalseAccept
Accuracy
⟶ Inconvenience
⟶ Securityhole
FNIR1:N“MissRate”
FPIR1:N“FalseAlarm”
⟶Missedleadininvestigation
⟶ Falselead:wasteoftime
⟶Wastedeffortonothers
⟶ Displacesactuallead
http://www.telegraph.co.uk/technology/2016/12/07/robot-passport-checker-rejects-asian-mans-photo-having-eyes/
+
+
ErrorTradeoffCharacteristic:Interpretationdifficulty
3divi_000 dermalog_001 dermalog_002
neurotechnology_000 ntechlab_000 rankone_000
rankone_001 tongyitrans_001 vigilantsolutions_000
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01False match rate (FMR)
Fals
e no
n−m
atch
rate
(FN
MR
) Sex
F
M
Race
B
W
3divi_000 dermalog_001 dermalog_002
neurotechnology_000 ntechlab_000 rankone_000
rankone_001 tongyitrans_001 vigilantsolutions_000
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01False match rate (FMR)
Fals
e no
n−m
atch
rate
(FN
MR
) Sex
F
M
Race
B
W
FalseNon-match
FalsematchrateSource:FRVT2017using~600Kmugshots. https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt-ongoing
But1:1systemsoperateatfixedT,notfixedFMR.
3divi_000 dermalog_001 dermalog_002
neurotechnology_000 ntechlab_000 rankone_000
rankone_001 tongyitrans_001 vigilantsolutions_000
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01False match rate (FMR)
Fals
e no
n−m
atch
rate
(FN
MR
) Sex
F
M
Race
B
W
3divi_000 dermalog_001 dermalog_002
neurotechnology_000 ntechlab_000 rankone_000
rankone_001 tongyitrans_001 vigilantsolutions_000
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
0.01
0.03
0.05
0.10
0.20
0.30
1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01 1e−05 1e−04 1e−03 1e−02 1e−01False match rate (FMR)
Fals
e no
n−m
atch
rate
(FN
MR
) Sex
F
M
Race
B
W
Falsenon-matchrate
Falsematchrate
Conclusions:1:1Accuracyvariesbysex,race
• Womenlessaccuratelyverified,bothFMRandFNMRhigherthanmen
• AfricanAmericansgiveslightlylowerFNMRthanWhites
• AfricanAmericansgivemuchhigherFMRthanWhites
Source:FRVT2017using~600Kmugshots. https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt-ongoing
Falserejectioninanoperationalsystem:ByUserAge
1. Thetableshowsfalsenon-matchratesforpassport-to-liveauthentication.
2. ThetimeelapsedbetweenpassportissuanceandABDtransactionisignoredinthisanalysis
3. Themostpopulousagegroupis40-somethings.
TAKEAWAYS:• Atthecurrentoperatingthreshold,false
rejectionsdeclinesteadilywithageoftheuser
• Youngadultsfailtoverifytwiceasoftenas50-somethings.
# Ageat timeofABDtransaction
FNMR Numberoftransactions
1 (0, 6] 0.13 38
2 (6,12] 0.11 386
3 (12,18] 0.07 1430
4 (18,24] 0.06 1036
5 (24,30] 0.05 1055
6 (30,36] 0.06 1060
7 (36,42] 0.04 1129
8 (42,48] 0.04 1456
9 (48,54] 0.03 1423
10 (54,60] 0.03 1138
11 (60,66] 0.04 829
12 (66,72] 0.03 555
13 (72,90] 0.02 358
ChildrenaredifficulttorecognizeLifelongreductioninfalserejection
FalseNon-MatchRate+/- 99%BootstrapCI
AgeGrou
p
Source:NISTFRVTMay2017
FRV
T-
FAC
ER
EC
OG
NIT
ION
VE
ND
OR
TE
ST-
VE
RIFIC
AT
ION
38
yisheng_000 yisheng_001 yitu_000
vcog_002 vigilantsolutions_001 vigilantsolutions_002 visionlabs_001 visionlabs_002 vocord_001 vocord_002
ntechlab_002 rankone_000 rankone_002 samtech_000 tongyitrans_001 tongyitrans_002 vcog_001
itmo_001 itmo_002 morpho_000 neurotechnology_000 neurotechnology_001 noblis_000 ntechlab_001
digitalbarriers_000 digitalbarriers_001 id3_001 id3_002 innovatrics_000 innovatrics_001 isityou_000
3divi_000 3divi_001 ayonix_000 camvi_001 cyberextruder_001 dermalog_002 dermalog_003
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
False non−match rate (FNMR) +/− 99% bootstrap CI
reor
der(a
gebi
n, a
ge_o
rder
, col
or =
fmr_
nom
inal
)
fmr_nominal
0.0001
0.001
Figure 21: For the visa images, the dots show FNMR by age group for two operating thresholds corresponding to FMR = {0.001, 0.0001} computed over all O(1010)impostor scores. Given a pair of face images taken at different times, we assign a false non-match to the bin that is the arithmetic average of the subject’s ages. This plotshows only the effect of age, not ageing. The number of comparisons in each bin is generally in the thousands. However the FNMR for the first and last bins are eachcomputed over fewer than 150 comparisons.
2017/10/
1208:35:12
FNM
R(T)
“Falsenon-m
atchrate”
FMR
(T)“False
match
rate”
FRV
T-
FAC
ER
EC
OG
NIT
ION
VE
ND
OR
TE
ST-
VE
RIFIC
AT
ION
38
yisheng_000 yisheng_001 yitu_000
vcog_002 vigilantsolutions_001 vigilantsolutions_002 visionlabs_001 visionlabs_002 vocord_001 vocord_002
ntechlab_002 rankone_000 rankone_002 samtech_000 tongyitrans_001 tongyitrans_002 vcog_001
itmo_001 itmo_002 morpho_000 neurotechnology_000 neurotechnology_001 noblis_000 ntechlab_001
digitalbarriers_000 digitalbarriers_001 id3_001 id3_002 innovatrics_000 innovatrics_001 isityou_000
3divi_000 3divi_001 ayonix_000 camvi_001 cyberextruder_001 dermalog_002 dermalog_003
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
False non−match rate (FNMR) +/− 99% bootstrap CI
reor
der(a
gebi
n, a
ge_o
rder
, col
or =
fmr_
nom
inal
)
fmr_nominal
0.0001
0.001
Figure 21: For the visa images, the dots show FNMR by age group for two operating thresholds corresponding to FMR = {0.001, 0.0001} computed over all O(1010)impostor scores. Given a pair of face images taken at different times, we assign a false non-match to the bin that is the arithmetic average of the subject’s ages. This plotshows only the effect of age, not ageing. The number of comparisons in each bin is generally in the thousands. However the FNMR for the first and last bins are eachcomputed over fewer than 150 comparisons.
2017/10/
1208:35:12
FNM
R(T)
“Falsenon-m
atchrate”
FMR
(T)“False
match
rate”
FRV
T-
FAC
ER
EC
OG
NIT
ION
VE
ND
OR
TE
ST-
VE
RIFIC
AT
ION
38
yisheng_000 yisheng_001 yitu_000
vcog_002 vigilantsolutions_001 vigilantsolutions_002 visionlabs_001 visionlabs_002 vocord_001 vocord_002
ntechlab_002 rankone_000 rankone_002 samtech_000 tongyitrans_001 tongyitrans_002 vcog_001
itmo_001 itmo_002 morpho_000 neurotechnology_000 neurotechnology_001 noblis_000 ntechlab_001
digitalbarriers_000 digitalbarriers_001 id3_001 id3_002 innovatrics_000 innovatrics_001 isityou_000
3divi_000 3divi_001 ayonix_000 camvi_001 cyberextruder_001 dermalog_002 dermalog_003
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
False non−match rate (FNMR) +/− 99% bootstrap CI
reor
der(a
gebi
n, a
ge_o
rder
, col
or =
fmr_
nom
inal
)
fmr_nominal
0.0001
0.001
Figure 21: For the visa images, the dots show FNMR by age group for two operating thresholds corresponding to FMR = {0.001, 0.0001} computed over all O(1010)impostor scores. Given a pair of face images taken at different times, we assign a false non-match to the bin that is the arithmetic average of the subject’s ages. This plotshows only the effect of age, not ageing. The number of comparisons in each bin is generally in the thousands. However the FNMR for the first and last bins are eachcomputed over fewer than 150 comparisons.
2017/10/
1208:35:12
FNM
R(T)
“Falsenon-m
atchrate”
FMR
(T)“False
match
rate”
FRV
T-
FAC
ER
EC
OG
NIT
ION
VE
ND
OR
TE
ST-
VE
RIFIC
AT
ION
38
yisheng_000 yisheng_001 yitu_000
vcog_002 vigilantsolutions_001 vigilantsolutions_002 visionlabs_001 visionlabs_002 vocord_001 vocord_002
ntechlab_002 rankone_000 rankone_002 samtech_000 tongyitrans_001 tongyitrans_002 vcog_001
itmo_001 itmo_002 morpho_000 neurotechnology_000 neurotechnology_001 noblis_000 ntechlab_001
digitalbarriers_000 digitalbarriers_001 id3_001 id3_002 innovatrics_000 innovatrics_001 isityou_000
3divi_000 3divi_001 ayonix_000 camvi_001 cyberextruder_001 dermalog_002 dermalog_003
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
(0,4](4,10]
(10,16](16,20](20,24](24,28](28,32](32,36](36,40](40,48](48,56](56,64](64,72]
(72,120]
False non−match rate (FNMR) +/− 99% bootstrap CI
reor
der(a
gebi
n, a
ge_o
rder
, col
or =
fmr_
nom
inal
)
fmr_nominal
0.0001
0.001
Figure 21: For the visa images, the dots show FNMR by age group for two operating thresholds corresponding to FMR = {0.001, 0.0001} computed over all O(1010)impostor scores. Given a pair of face images taken at different times, we assign a false non-match to the bin that is the arithmetic average of the subject’s ages. This plotshows only the effect of age, not ageing. The number of comparisons in each bin is generally in the thousands. However the FNMR for the first and last bins are eachcomputed over fewer than 150 comparisons.
2017/10/
1208:35:12
FNM
R(T)
“Falsenon-m
atchrate”
FMR
(T)“False
match
rate”
“Expected”FMRfromoldvs.youngimpostors
9
−2.1
−3.5
−4.7
−4.5
−4.4
−4.6
−4.8
−5.0
−2.6
−5.1
−5.3
−5.4
−6.0
−5.8
−3.2
−2.4
−3.1
−3.3
−3.4
−3.5
−3.7
−4.0
−2.5
−4.1
−4.4
−4.7
−5.1
−5.3
−4.4
−2.9
−2.9
−3.0
−3.0
−3.2
−3.3
−3.6
−3.7
−3.8
−4.2
−4.6
−5.2
−5.2
−4.2
−3.2
−3.0
−2.8
−2.6
−2.7
−2.9
−3.2
−3.9
−3.5
−3.9
−4.3
−5.0
−5.2
−4.2
−3.4
−3.1
−2.8
−2.4
−2.5
−2.7
−3.0
−3.9
−3.2
−3.6
−4.0
−4.7
−4.9
−4.5
−3.6
−3.3
−2.9
−2.6
−2.5
−2.7
−2.8
−4.2
−3.0
−3.4
−3.8
−4.3
−4.5
−4.8
−3.7
−3.5
−3.1
−2.8
−2.7
−2.7
−2.8
−4.4
−2.9
−3.1
−3.5
−4.0
−4.3
−5.0
−3.9
−3.7
−3.3
−3.0
−2.8
−2.8
−2.8
−4.5
−2.8
−3.0
−3.3
−3.7
−4.0
−2.4
−2.6
−3.7
−3.8
−3.7
−3.9
−4.1
−4.3
−2.2
−4.5
−4.8
−5.0
−5.4
−5.5
−5.2
−4.1
−4.0
−3.5
−3.2
−3.0
−2.9
−2.8
−4.6
−2.7
−2.7
−2.9
−3.2
−3.5
−5.3
−4.3
−4.3
−3.8
−3.4
−3.2
−3.1
−2.9
−4.8
−2.7
−2.6
−2.6
−2.8
−3.0
−5.2
−4.6
−4.7
−4.2
−3.8
−3.5
−3.4
−3.2
−5.0
−2.9
−2.6
−2.5
−2.5
−2.5
−5.3
−4.9
−5.2
−4.9
−4.4
−4.1
−3.8
−3.6
−5.1
−3.1
−2.7
−2.4
−2.3
−2.2
−5.3
−5.2
−5.7
−5.2
−5.0
−4.6
−4.2
−3.9
−5.1
−3.4
−2.9
−2.5
−2.2
−1.9
−1.8
−3.1
−4.2
−4.1
−3.8
−4.0
−4.2
−4.5
−2.2
−4.6
−4.8
−4.9
−5.9
−5.5
−2.8
−1.8
−2.4
−2.7
−2.6
−2.8
−3.0
−3.3
−2.0
−3.5
−3.9
−4.1
−4.6
−5.0
−4.0
−2.2
−2.1
−2.2
−2.1
−2.3
−2.5
−2.7
−3.2
−3.0
−3.4
−3.8
−4.4
−4.4
−3.5
−2.5
−2.3
−2.0
−1.7
−1.7
−2.0
−2.3
−3.3
−2.6
−3.0
−3.4
−4.2
−4.3
−3.5
−2.6
−2.3
−1.9
−1.5
−1.5
−1.7
−2.0
−3.3
−2.3
−2.6
−3.1
−3.8
−4.0
−3.8
−2.8
−2.5
−2.0
−1.6
−1.5
−1.7
−1.9
−3.6
−2.1
−2.4
−2.9
−3.5
−3.7
−4.2
−2.9
−2.7
−2.2
−1.8
−1.7
−1.8
−1.8
−3.8
−2.0
−2.2
−2.6
−3.1
−3.4
−4.5
−3.2
−2.9
−2.4
−2.0
−1.8
−1.8
−1.9
−3.9
−1.9
−2.1
−2.4
−2.8
−3.2
−2.1
−2.2
−3.2
−3.2
−3.0
−3.1
−3.4
−3.8
−1.8
−3.9
−4.2
−4.4
−4.9
−5.0
−4.8
−3.4
−3.2
−2.7
−2.2
−2.0
−1.9
−1.9
−4.1
−1.8
−1.9
−2.1
−2.4
−2.7
−4.8
−3.7
−3.5
−2.9
−2.5
−2.2
−2.1
−2.0
−4.3
−1.8
−1.7
−1.8
−1.9
−2.2
−4.8
−4.1
−4.0
−3.4
−2.8
−2.5
−2.4
−2.3
−4.5
−2.0
−1.7
−1.6
−1.7
−1.8
−5.2
−4.4
−4.5
−4.1
−3.5
−3.2
−2.9
−2.7
−4.6
−2.3
−1.9
−1.6
−1.5
−1.5
−5.2
−5.6
−5.2
−4.6
−4.2
−3.8
−3.4
−3.1
−4.7
−2.7
−2.1
−1.8
−1.5
−1.3
All impostor pairs Same sex and same region impostor pairs
(0,4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0](0,
4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0]
(0,4]
(04,10)
(10,16]
(16,20]
(20,24]
(24,28]
(28,32]
(32,36]
(36,40]
(40,48]
(48,56]
(56,64]
(64,72]
(72,120]
Age of enrollee
Age
of im
post
or
−6 −5 −4 −3 −2 −1log10 FMR
Cross age FMR at threshold T = 0.091 for algorithm ntechlab_000, giving FMR(T) = 0.001 globally.LowFMRDissimilar
HighFMRSimilar
Neutral
LowFMR HigherFMR
• FMR=0.001• TypicalinePassport Gates• Achievedbysettingathreshold,T• Determinedbya“large”empiricaltrial• ThresholdremainsfixedforALLtrials
1:1Impostors:Falsepositivesintheelderly• TfixedtogiveFMR=0.001
• But20-somethingsmatchwithFMR=0.01
• And30-somethingsmatchwithFMR=0.03
• But70-somethingsmatchwithFMR=0.05
• Nominal”1in1000”impostorchancehasx50securityvulnerability• Thisismassive
• Heterogeneitygives“large”varianceacrossages.
10
−2.1
−3.5
−4.7
−4.5
−4.4
−4.6
−4.8
−5.0
−2.6
−5.1
−5.3
−5.4
−6.0
−5.8
−3.2
−2.4
−3.1
−3.3
−3.4
−3.5
−3.7
−4.0
−2.5
−4.1
−4.4
−4.7
−5.1
−5.3
−4.4
−2.9
−2.9
−3.0
−3.0
−3.2
−3.3
−3.6
−3.7
−3.8
−4.2
−4.6
−5.2
−5.2
−4.2
−3.2
−3.0
−2.8
−2.6
−2.7
−2.9
−3.2
−3.9
−3.5
−3.9
−4.3
−5.0
−5.2
−4.2
−3.4
−3.1
−2.8
−2.4
−2.5
−2.7
−3.0
−3.9
−3.2
−3.6
−4.0
−4.7
−4.9
−4.5
−3.6
−3.3
−2.9
−2.6
−2.5
−2.7
−2.8
−4.2
−3.0
−3.4
−3.8
−4.3
−4.5
−4.8
−3.7
−3.5
−3.1
−2.8
−2.7
−2.7
−2.8
−4.4
−2.9
−3.1
−3.5
−4.0
−4.3
−5.0
−3.9
−3.7
−3.3
−3.0
−2.8
−2.8
−2.8
−4.5
−2.8
−3.0
−3.3
−3.7
−4.0
−2.4
−2.6
−3.7
−3.8
−3.7
−3.9
−4.1
−4.3
−2.2
−4.5
−4.8
−5.0
−5.4
−5.5
−5.2
−4.1
−4.0
−3.5
−3.2
−3.0
−2.9
−2.8
−4.6
−2.7
−2.7
−2.9
−3.2
−3.5
−5.3
−4.3
−4.3
−3.8
−3.4
−3.2
−3.1
−2.9
−4.8
−2.7
−2.6
−2.6
−2.8
−3.0
−5.2
−4.6
−4.7
−4.2
−3.8
−3.5
−3.4
−3.2
−5.0
−2.9
−2.6
−2.5
−2.5
−2.5
−5.3
−4.9
−5.2
−4.9
−4.4
−4.1
−3.8
−3.6
−5.1
−3.1
−2.7
−2.4
−2.3
−2.2
−5.3
−5.2
−5.7
−5.2
−5.0
−4.6
−4.2
−3.9
−5.1
−3.4
−2.9
−2.5
−2.2
−1.9
−1.8
−3.1
−4.2
−4.1
−3.8
−4.0
−4.2
−4.5
−2.2
−4.6
−4.8
−4.9
−5.9
−5.5
−2.8
−1.8
−2.4
−2.7
−2.6
−2.8
−3.0
−3.3
−2.0
−3.5
−3.9
−4.1
−4.6
−5.0
−4.0
−2.2
−2.1
−2.2
−2.1
−2.3
−2.5
−2.7
−3.2
−3.0
−3.4
−3.8
−4.4
−4.4
−3.5
−2.5
−2.3
−2.0
−1.7
−1.7
−2.0
−2.3
−3.3
−2.6
−3.0
−3.4
−4.2
−4.3
−3.5
−2.6
−2.3
−1.9
−1.5
−1.5
−1.7
−2.0
−3.3
−2.3
−2.6
−3.1
−3.8
−4.0
−3.8
−2.8
−2.5
−2.0
−1.6
−1.5
−1.7
−1.9
−3.6
−2.1
−2.4
−2.9
−3.5
−3.7
−4.2
−2.9
−2.7
−2.2
−1.8
−1.7
−1.8
−1.8
−3.8
−2.0
−2.2
−2.6
−3.1
−3.4
−4.5
−3.2
−2.9
−2.4
−2.0
−1.8
−1.8
−1.9
−3.9
−1.9
−2.1
−2.4
−2.8
−3.2
−2.1
−2.2
−3.2
−3.2
−3.0
−3.1
−3.4
−3.8
−1.8
−3.9
−4.2
−4.4
−4.9
−5.0
−4.8
−3.4
−3.2
−2.7
−2.2
−2.0
−1.9
−1.9
−4.1
−1.8
−1.9
−2.1
−2.4
−2.7
−4.8
−3.7
−3.5
−2.9
−2.5
−2.2
−2.1
−2.0
−4.3
−1.8
−1.7
−1.8
−1.9
−2.2
−4.8
−4.1
−4.0
−3.4
−2.8
−2.5
−2.4
−2.3
−4.5
−2.0
−1.7
−1.6
−1.7
−1.8
−5.2
−4.4
−4.5
−4.1
−3.5
−3.2
−2.9
−2.7
−4.6
−2.3
−1.9
−1.6
−1.5
−1.5
−5.2
−5.6
−5.2
−4.6
−4.2
−3.8
−3.4
−3.1
−4.7
−2.7
−2.1
−1.8
−1.5
−1.3
All impostor pairs Same sex and same region impostor pairs
(0,4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0](0,
4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0]
(0,4]
(04,10)
(10,16]
(16,20]
(20,24]
(24,28]
(28,32]
(32,36]
(36,40]
(40,48]
(48,56]
(56,64]
(64,72]
(72,120]
Age of enrollee
Age
of im
post
or
−6 −5 −4 −3 −2 −1log10 FMR
Cross age FMR at threshold T = 0.091 for algorithm ntechlab_000, giving FMR(T) = 0.001 globally.
−2.1
−3.5
−4.7
−4.5
−4.4
−4.6
−4.8
−5.0
−2.6
−5.1
−5.3
−5.4
−6.0
−5.8
−3.2
−2.4
−3.1
−3.3
−3.4
−3.5
−3.7
−4.0
−2.5
−4.1
−4.4
−4.7
−5.1
−5.3
−4.4
−2.9
−2.9
−3.0
−3.0
−3.2
−3.3
−3.6
−3.7
−3.8
−4.2
−4.6
−5.2
−5.2
−4.2
−3.2
−3.0
−2.8
−2.6
−2.7
−2.9
−3.2
−3.9
−3.5
−3.9
−4.3
−5.0
−5.2
−4.2
−3.4
−3.1
−2.8
−2.4
−2.5
−2.7
−3.0
−3.9
−3.2
−3.6
−4.0
−4.7
−4.9
−4.5
−3.6
−3.3
−2.9
−2.6
−2.5
−2.7
−2.8
−4.2
−3.0
−3.4
−3.8
−4.3
−4.5
−4.8
−3.7
−3.5
−3.1
−2.8
−2.7
−2.7
−2.8
−4.4
−2.9
−3.1
−3.5
−4.0
−4.3
−5.0
−3.9
−3.7
−3.3
−3.0
−2.8
−2.8
−2.8
−4.5
−2.8
−3.0
−3.3
−3.7
−4.0
−2.4
−2.6
−3.7
−3.8
−3.7
−3.9
−4.1
−4.3
−2.2
−4.5
−4.8
−5.0
−5.4
−5.5
−5.2
−4.1
−4.0
−3.5
−3.2
−3.0
−2.9
−2.8
−4.6
−2.7
−2.7
−2.9
−3.2
−3.5
−5.3
−4.3
−4.3
−3.8
−3.4
−3.2
−3.1
−2.9
−4.8
−2.7
−2.6
−2.6
−2.8
−3.0
−5.2
−4.6
−4.7
−4.2
−3.8
−3.5
−3.4
−3.2
−5.0
−2.9
−2.6
−2.5
−2.5
−2.5
−5.3
−4.9
−5.2
−4.9
−4.4
−4.1
−3.8
−3.6
−5.1
−3.1
−2.7
−2.4
−2.3
−2.2
−5.3
−5.2
−5.7
−5.2
−5.0
−4.6
−4.2
−3.9
−5.1
−3.4
−2.9
−2.5
−2.2
−1.9
−1.8
−3.1
−4.2
−4.1
−3.8
−4.0
−4.2
−4.5
−2.2
−4.6
−4.8
−4.9
−5.9
−5.5
−2.8
−1.8
−2.4
−2.7
−2.6
−2.8
−3.0
−3.3
−2.0
−3.5
−3.9
−4.1
−4.6
−5.0
−4.0
−2.2
−2.1
−2.2
−2.1
−2.3
−2.5
−2.7
−3.2
−3.0
−3.4
−3.8
−4.4
−4.4
−3.5
−2.5
−2.3
−2.0
−1.7
−1.7
−2.0
−2.3
−3.3
−2.6
−3.0
−3.4
−4.2
−4.3
−3.5
−2.6
−2.3
−1.9
−1.5
−1.5
−1.7
−2.0
−3.3
−2.3
−2.6
−3.1
−3.8
−4.0
−3.8
−2.8
−2.5
−2.0
−1.6
−1.5
−1.7
−1.9
−3.6
−2.1
−2.4
−2.9
−3.5
−3.7
−4.2
−2.9
−2.7
−2.2
−1.8
−1.7
−1.8
−1.8
−3.8
−2.0
−2.2
−2.6
−3.1
−3.4
−4.5
−3.2
−2.9
−2.4
−2.0
−1.8
−1.8
−1.9
−3.9
−1.9
−2.1
−2.4
−2.8
−3.2
−2.1
−2.2
−3.2
−3.2
−3.0
−3.1
−3.4
−3.8
−1.8
−3.9
−4.2
−4.4
−4.9
−5.0
−4.8
−3.4
−3.2
−2.7
−2.2
−2.0
−1.9
−1.9
−4.1
−1.8
−1.9
−2.1
−2.4
−2.7
−4.8
−3.7
−3.5
−2.9
−2.5
−2.2
−2.1
−2.0
−4.3
−1.8
−1.7
−1.8
−1.9
−2.2
−4.8
−4.1
−4.0
−3.4
−2.8
−2.5
−2.4
−2.3
−4.5
−2.0
−1.7
−1.6
−1.7
−1.8
−5.2
−4.4
−4.5
−4.1
−3.5
−3.2
−2.9
−2.7
−4.6
−2.3
−1.9
−1.6
−1.5
−1.5
−5.2
−5.6
−5.2
−4.6
−4.2
−3.8
−3.4
−3.1
−4.7
−2.7
−2.1
−1.8
−1.5
−1.3
All impostor pairs Same sex and same region impostor pairs
(0,4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0](0,
4]
(04,10
)
(10,16
]
(16,20
]
(20,24
]
(24,28
]
(28,32
]
(32,36
]
(36,40
]
(40,48
]
(48,56
]
(56,64
]
(64,72
]
(72,12
0]
(0,4]
(04,10)
(10,16]
(16,20]
(20,24]
(24,28]
(28,32]
(32,36]
(36,40]
(40,48]
(48,56]
(56,64]
(64,72]
(72,120]
Age of enrollee
Age
of im
post
or−6 −5 −4 −3 −2 −1
log10 FMR
Cross age FMR at threshold T = 0.091 for algorithm ntechlab_000, giving FMR(T) = 0.001 globally.
Source:FRVT2017using~200Kvisaimages.https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt-ongoing
Crosscountry-of-birtheffectsonFMR
0.01566
0.00006
0.00003
0.01085
0.00010
0.00004
0.01326
0.01141
0.00003
0.00056
0.00001
0.00897
0.02434
0.00000
0.00021
0.01311
0.00002
0.00001
0.02126
0.00008
0.00003
0.00015
0.00007
0.00001
0.00175
HAT
JPN
KOR
NRA
POL
HAT JPN KOR NRA POLCountry of birth of enrollee
Coun
try o
f birt
h of
impo
stor
−6 −5 −4 −3 −2 −1log10 FMR
Cross−country FMR at T = 30.260 for neurotechnology_000
• Nigeria– Korea LowFMR• Haiti– Poland LowFMR
• Poland– Poland FMR~TargetFMR
• Nigeria– Nigeria FMR=1in50• Nigeria– Haiti FMR=1in80
• Korea– Korea FMR=1in40
Impostorsaresame-sex,same-agegroupFMRnominal=0.001.
Source:FRVT2017using~200Kvisaimages.https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt-ongoing
Doesdarkskincausefalsematches?
0.02249
0.01231
0.00017
0.00009
0.01005
0.01566
0.00022
0.00015
0.00009
0.00014
0.02198
0.01335
0.00000
0.00007
0.01130
0.01367
0.08905
0.03853
0.00012
0.00011
0.04045
0.05049
0.00004
0.00001
0.00002
0.00003
0.05205
0.01748
0.00000
0.00002
0.01295
0.01556
0.00423
0.00248
0.00037
0.00013
0.00163
0.00498
0.00052
0.00015
0.00002
0.00006
0.02219
0.01039
0.00000
0.00003
0.01026
0.01472
0.00517
0.00321
0.00018
0.00014
0.00240
0.00533
0.00036
0.00020
0.00004
0.00007
0.02105
0.01260
0.00001
0.00003
0.01041
0.01483
neurotechnology_000 ntechlab_001
tongyitrans_001 yitu_000
GHAN
HAT
IND
PKST
GHAN
HAT
IND
PKST
GHAN HAT IND PKST GHAN HAT IND PKSTCountry of birth of enrollee
Cou
ntry
of b
irth
of im
post
or
−6 −5 −4 −3 −2 −1log10 FMR
• IndiaandPakistangivehighFMRwithinregion
• HaitiandGhanagivehighFMRwithinregion
• BUT
• Butcrossregiondoesnotgivefalsematches.
• NEXT:• Measureskintone.• CheckISOcompliance7bits
ofgreyonface.
Source:FRVT2017using~200Kvisaimages.https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt-ongoing
Summary
• Facerecognitionalgorithmsaresensitivetodemographics• Race>Age>Sex• Somealgorithmdependence
• Raceeffectslikelyduetotrainingdata• Opportunitiesformitigation
• Needtobepreciseaboutwhatthemetricis• Falsenegativevs.FalsePositive(vs.FailuretoCapture)
• 1:1Falsepositiveratespresentsecurityvulnerabilitiesduetodemographics• 1:NSystems
• Arelargelyuntestedfordemographics• Haveknownavenuesformitigationofdemographiceffects
• NISTwillauthorreportinlate2018ondemographics• DatafromFRVT2018– 1:NtestwithN>20million• Imageryfromimmigration:country-of-birthasproxyforrace• Imageryfromlawenforcement