Embed Size (px)
Citation preview
Improving the Quality of
Investigative Decision Making
Craig Bennell, Ph.D.
Carleton University
Ottawa, Ontario, Canada
Two-Alternative Decisions
• Is this suspect lying?• Will this negotiation end in failure?• Are these crimes linked?• Was this bite mark made by an adult?• Is this suicide note fake?• Is this allegation of rape false?
• To improve decision-making we need a valid measure of performance...
How Should We Measure Performance?
SET A DECISION
THRESHOLD
COUNT THE FREQUENCIES OF DECISION OUTCOMES
CALCULATE PERCENTAGE
CORRECT
EXAMINE THE
EVIDENCE
Relatively Inaccurate Evidence
Relatively Accurate Evidence
ROC ANALYSIS
(Bennell, 2005)
ROC ANALYSIS
(Bennell, 2005)
An AUC of .60 indicates that there is a 60% chance that a randomly selected positive case will be higher on evidence X than a randomly selected negative case
The Accuracy of Diagnostic Systems: The Polygraph CQT and GKT
• The Control Question Test– Relevant (R) and Control (C) questions– For each R-C pair a score of -3 to +3 is given– Greater than +5: truthful, Less than -5: deceitful
• The Guilty Knowledge Test– Multiple choice questions– One alternative relates to guilty knowledge (GK)– Elevated responses to GK questions: deceitful
• How accurate are these procedures?
Accuracy scores for 52 lab studies examining the validity of the polygraph (National Academy of Science, 2003)
Median AUC for GKT = .88Median AUC for CQT = .85
Identifying Accurate Predictors: The Fake Suicide Note
• What discriminates between real and fake notes?– Structure variables– Content variables– Combinations of the two
• How can we tell if some are better than others?• How can we determine how much better?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Pro
ba
biliy
of h
its
Probability of false alarms
1 - Structure model (AUC = .69)2 - Content model (AUC = .74)3 - Optimal model (AUC = .82)4 - Inclusive model (AUC = .85)
Optimal: Average sentence length (shorter in genuine)Positive affect (more in genuine)
4
3
2
1
(Jones & Bennell, 2005)
Setting Appropriate Thresholds:Linking Serial Burglaries
• Many behaviours can be used• Inter-crime distance is very useful (AUC=.80)
– The closer two crimes are to one another the more likely it is that they are linked
• When should we decide that crimes are linked?• What is an appropriate threshold?
(Bennell & Canter, 2002)
2.50km (72% H, 25% FA)
1.50km (58% H, 11% FA)
3.50km (85% H, 45% FA)
1.10km (49% H, 10% FA)
Comparing Decision Makers:Bite Mark Identification
• It is often unclear as to the source of a bite mark– Non-accidental (adult)– Accidental (child)
• How well can people perform on this task?• Is expertise required?
(Whittaker et al., 1998)
Senior forensic dentists (AUC=.69)Student dentists (AUC=.69) Junior forensic dentists (AUC=.68)General dentists (AUC=.62)Social workers (AUC=.63)Police officers (AUC=.62)
Conclusions
• ROC analysis provides a way of calculating accuracy scores that are not dependent on arbitrary thresholds
• ROC analysis also provides a way of identifying thresholds that meet the demands of specific situations
• The result for police investigations will likely be improved decision-making accuracy and utility
