Embed Size (px)
Citation preview
5/29/2020
1
Predictors and Process in Selection Decisions
Nathan R. Kuncel
4th Grade Girl’s Ratings of the Boys in Her Class (including Nathan)
5/29/2020
2
Nature and Quality of the Assessments
Process Used to Combine the Information and
Make a Decision
5/29/2020
3
We have put a lot of attention into our predictors.
Predicting Job Performance with Cognitive Ability Tests
> 21,942 primary studies
N > 5.0 x 106
Kuncel, Ones, & Sackett, 2010
Cognitive Ability Predicts Job Performance!
5/29/2020
4
Divorce
- - -
Roberts, Kuncel, Shiner, Caspi, & Goldberg (2007)
However We Know Much Less About Two Critical Processes
1. How do experts combine information to make evaluations of applicants?
2. What influences decision makers to use our carefully constructed assessments?
5/29/2020
5
Please write down the last two digits of your phone number.
5/29/2020
6
Some More Examples
1. Write down the last two digits of your SSN.2. What would you pay for this lovely toaster?
0
5
10
15
20
25
30
35
40
45
SSN < 50 SSN > 50
What Would You Pay?
$30 $41
Simonson & Drolet (2003)
5/29/2020
7
Anchoring
• 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1 = ?– 4,200
vs.
• 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 = ?– 500
Actual Answer:
1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x = 40,320
Anchoring in Hiring
• How did the interview go?
• What school did she attend?
• The first thing you look at when you read files to make a decision.
Anchors can be good or bad depending on what, how, and when.
5/29/2020
8
Expertise and Experience?
It has value but not always in the way we think.
First, let’s look at an unambiguous
example of expertise.
So a Professor, Navy Seal, and Secret Service agent walk into a firing range…
• Each fired 10 rounds with a short barreled .40 pistol at paper targets at medium-long range. – Professor of Psychology
– Lieutenant-Commander US Navy Seals
– Special Agent with the US Secret Service
5/29/2020
9
Professor of Psychology
Note: 4 complete misses
Note: Shots tend to fall to the bottom left of the target
?
Lieutenant Commander US Navy Seals
Note: All 10 shots are on target, even at this range, and 7 are in the center torso.
5/29/2020
10
Special Agent US Secret Service
Note: Wow.
Are We Hiring Sharp Shooters?Expertise will be most helpful when forecasting in a regular (consistent or predictable) environments and learning can occur practice and timely feedback.
“Wicked” environments:
– Inconsistent environment
– Low validity predictors
– Infrequent or no feedback
– Ambiguous feedback
– Distant feedback
5/29/2020
11
What Happens in Wicked Environments Like Hiring Decision Making?
.
Who is successful at Univ. of Minnesota? Admissions Officers Predicting Student GPA
High School Rank + College Admissions Test
r (accuracy) = .45
Holistic Admissions Counselor Judgment
based on full file including HS Rank and
Admissions Tests
r (accuracy) = .35
Sarbin (1943)
5/29/2020
12
C-files Study
Psychiatrists review patient files (in folders) to decide which patients are likely to be successful and who would return to the hospital after treatment?
What other method made decisions that were just as good as the ones made by the committee?
Lasky, et al. (1959)
C-files Study
A scale on which the patients’ files were weighed!!!
5/29/2020
13
Bordeaux Wine
• Grapes are harvested and made into wine which is not good to drink for years.
• Wine experts taste the raw product and make judgments about how good it will be.
• An economist used a simple equation to predict the value of wine.
• Wine experts called it all sorts of names including “absurd”.
Summer Temp and Harvest Rain for Above and Below Average Priced Bordeaux Wines
5/29/2020
14
The data points line up almost as straight as these rows of grapes.
What are we losing when predicting performance?
• At this point the magnitude of the effect should be a key consideration.
• Just what is the price paid?
• Let’s look at a meta-analysis that examined predicting performance at school and work. – Kuncel, Klieger, Connelly, & Ones (2013)
5/29/2020
15
Kuncel, Klieger, Connelly, & Ones (2013).
Expert Judgment versus Equations for Predicting Success
What is the Source of the Problem?
Human Judgment is Inconsistent
5/29/2020
16
• Leadership
• Motivation
• Judgment
• Adjustment
• Administration
• Communication
• Interpersonal
• Structured Interviews
• Personality Test
• Cognitive Tests
• In Basket
• Role Play
• Leaderless Group Discussion
Organization Sample Size
Financial Services Company 231 candidates, 26 assessors
Food Retailer Executive 195 candidates, 23 assessors
Food Retailer Line/Middle Mgmt 421 candidates, 30 assessors
Pushing the Limits of Consistency
Yu & Kuncel (in press)
Person Job Fit Rating
Candidate
Consistent Random Weights Completely Random Weights
Lead Motive … Adj. Lead Motive … Adj.
1 .02 .36
…
.19 .35 .45
…
.12
2 .02 .36 .19 .15 .11 .28
3 .02 .36 .19 .43 .06 .33
4 .02 .36 .19 .04 .41 .31
5 .02 .36 .19 .22 .17 .09
Pushing the Limits of ConsistencyYu & Kuncel (in press)
We compared Expert Judgments with Unit Weighting, Optimal Weighting, and two Random Weight Conditions
10,000X 10,000X
5/29/2020
17
Distribution of 10,000 criterion validities using completely random positive weights
Distribution of 10,000 criterion validities using consistent random positive weights
Yu & Kuncel (in press).
Expert Judgment
Unit Weights
Optimal Adj. Weights
5/29/2020
18
Sam
ple
2S
ampl
e 3
Yu & Kuncel (in press).
Inconsistency Conclusion
• Assessors are performing only slightly better than a random weight generator in predicting job performance.
• Consistent algorithmic data combination, even consistent random weights, are yielding superior prediction of job performance.
• What should we do?
• Stop using expert judgment to combine data?
5/29/2020
19
But Let’s Get Realistic
User Acceptability is Important!
How do we aid human judgment?
I think we can use theory and research from judgment and decision making to structure environments that improve decisions.
5/29/2020
20
Using the Anchoring Bias to Debias Decisions: Shu & Kuncel Experiment
“Please use the following applicant information to make 40 hiring decisions. For this job, measures of conscientiousness are generally the best predictors followed by neuroticism and agreeableness.”
Applicant: James
Neuroticism: 43%Extroversion: 72%Openness: 54%Agreeableness: 64%Conscientiousness: 68%
Applicant: AliceHireability Index: 60%
Neuroticism: 43%Extroversion: 72%Openness: 54%Agreeableness: 64%Conscientiousness: 68%
“You will also be given a algorithmic combination of the applicant data. This index works quite well but is not perfect. If you can make better decisions than your peers you will get more $$$.”
N = 1,234
“If you can make better decisions than your peers you will get more $$$.”
We used a large validation data set to present new applicant data to decision makers and then evaluated their accuracy.
Using a Bias to Debias Decisions: Shu & Kuncel Experiment
Please use the following applicant information to make 40 hiring decisions. For this job, measures of conscientiousness are generally the best predictors followed by neuroticism and agreeableness.
Applicant: James
Neuroticism: 43%Extroversion: 72%Openness: 54%Agreeableness: 64%Conscientiousness: 68%
Applicant: AliceHireability Index: 60%
Neuroticism: 43%Extroversion: 72%Openness: 54%Agreeableness: 64%Conscientiousness: 68%
You will also be given a algorithmic combination of the applicant data. This index works quite well but is not perfect. If you can make better decisions than your peers you will get more $$$.
N=1,234
If you can make better decisions than your peers you will get more $$$.
This is an AnchorSimple average of predictor scores (not optimal).
5/29/2020
21
Effects of Anchors on Weighting Policy:How are they using the information?
Effects of Anchors on Weighting Policy:How are they using the information?
5/29/2020
22
Shu & Kuncel, 2018
Conclusion
We need both the best predictors and the best process to maximize the utility of our assessments.
Although the decision making, even of experts, can suffer from inconsistency, we are optimistic that we can use decision science to enhance accuracy while retaining user acceptability.
