Embed Size (px)
Citation preview
Auditing Audit Studies:
How Name Perception and Selection Underestimates Racial Discrimination
S. Michael Gaddis
Assistant Professor of Sociology and Demography
Penn State University
August 23, 2015
S. Michael Gaddis Auditing Audit Studies:
Introduction
Our best evidence suggests racial discrimination is stillrelatively widespread in employment and housing sectors
How do we know?
Interviews and surveys? (Braddock and McPartland 1987;Kirschenman and Neckerman 1991)Observational data?Field experiments and audit studies
Introduction
Our best evidence suggests racial discrimination is stillrelatively widespread in employment and housing sectors
How do we know?
Interviews and surveys? (Braddock and McPartland 1987;Kirschenman and Neckerman 1991)Observational data?Field experiments and audit studies
Introduction
Our best evidence suggests racial discrimination is stillrelatively widespread in employment and housing sectors
How do we know?
Interviews and surveys? (Braddock and McPartland 1987;Kirschenman and Neckerman 1991)Observational data?Field experiments and audit studies
Introduction
Our best evidence suggests racial discrimination is stillrelatively widespread in employment and housing sectors
How do we know?
Interviews and surveys? (Braddock and McPartland 1987;Kirschenman and Neckerman 1991)Observational data?Field experiments and audit studies
Introduction
Our best evidence suggests racial discrimination is stillrelatively widespread in employment and housing sectors
How do we know?
Interviews and surveys? (Braddock and McPartland 1987;Kirschenman and Neckerman 1991)Observational data?Field experiments and audit studies
The Rise of Audit Studies
Beginning with HUD and the Urban Institute in 1970s
In-person audits through 80s, 90s, and early 00s; smallquantities, mostly small samples
Housing and employment applications shift to internet
Expanded to other arenas & actors (politicians - Butler andBroockman 2011; prospective roommates - Ghoshal andGaddis 2015)
The Rise of Audit Studies
Beginning with HUD and the Urban Institute in 1970s
In-person audits through 80s, 90s, and early 00s; smallquantities, mostly small samples
Housing and employment applications shift to internet
Expanded to other arenas & actors (politicians - Butler andBroockman 2011; prospective roommates - Ghoshal andGaddis 2015)
The Rise of Audit Studies
Beginning with HUD and the Urban Institute in 1970s
In-person audits through 80s, 90s, and early 00s; smallquantities, mostly small samples
Housing and employment applications shift to internet
Expanded to other arenas & actors (politicians - Butler andBroockman 2011; prospective roommates - Ghoshal andGaddis 2015)
The Rise of Audit Studies
Beginning with HUD and the Urban Institute in 1970s
In-person audits through 80s, 90s, and early 00s; smallquantities, mostly small samples
Housing and employment applications shift to internet
Expanded to other arenas & actors (politicians - Butler andBroockman 2011; prospective roommates - Ghoshal andGaddis 2015)
Online Audits: Some Pros, Some Cons
No physical contact
Addresses previous critiques (Heckman 1998; Heckman andSiegelman 1993)How do you convey race?
Online Audits: Some Pros, Some Cons
No physical contact
Addresses previous critiques (Heckman 1998; Heckman andSiegelman 1993)How do you convey race?
Online Audits: Some Pros, Some Cons
No physical contact
Addresses previous critiques (Heckman 1998; Heckman andSiegelman 1993)How do you convey race?
Online Audits: Names as Race
Capitalize on racialized patterns in naming (Cook et al. 2014;Fryer and Levitt 2004; Lieberson 2000)
SES in�uences these patterns as well
Do names con�ate race and SES and are thus a �awed proxyof race?
No (Bertrand and Mullainathan 2004)Yes (maybe?) (Figlio 2005; Gaddis 2013)
Online Audits: Names as Race
Capitalize on racialized patterns in naming (Cook et al. 2014;Fryer and Levitt 2004; Lieberson 2000)
SES in�uences these patterns as well
Do names con�ate race and SES and are thus a �awed proxyof race?
No (Bertrand and Mullainathan 2004)Yes (maybe?) (Figlio 2005; Gaddis 2013)
Online Audits: Names as Race
Capitalize on racialized patterns in naming (Cook et al. 2014;Fryer and Levitt 2004; Lieberson 2000)
SES in�uences these patterns as well
Do names con�ate race and SES and are thus a �awed proxyof race?
No (Bertrand and Mullainathan 2004)Yes (maybe?) (Figlio 2005; Gaddis 2013)
Online Audits: Names as Race
Capitalize on racialized patterns in naming (Cook et al. 2014;Fryer and Levitt 2004; Lieberson 2000)
SES in�uences these patterns as well
Do names con�ate race and SES and are thus a �awed proxyof race?
No (Bertrand and Mullainathan 2004)Yes (maybe?) (Figlio 2005; Gaddis 2013)
Online Audits: Names as Race
Capitalize on racialized patterns in naming (Cook et al. 2014;Fryer and Levitt 2004; Lieberson 2000)
SES in�uences these patterns as well
Do names con�ate race and SES and are thus a �awed proxyof race?
No (Bertrand and Mullainathan 2004)Yes (maybe?) (Figlio 2005; Gaddis 2013)
Research Questions
How do individuals perceive di�erent names?
How do these perceptions potentially in�uence our currentunderstanding of racial discrimination?
Research Questions
How do individuals perceive di�erent names?
How do these perceptions potentially in�uence our currentunderstanding of racial discrimination?
Data Sources
New York state birth record data (1994-2012)
total number of births by name, mother's race, and mother'seducation level
U.S. Census data on frequently occurring surnames (2000)
Amazon Mechanical Turk survey experiment (2014-2015)
Simulation of �eld experiments
Data Sources
New York state birth record data (1994-2012)
total number of births by name, mother's race, and mother'seducation level
U.S. Census data on frequently occurring surnames (2000)
Amazon Mechanical Turk survey experiment (2014-2015)
Simulation of �eld experiments
Data Sources
New York state birth record data (1994-2012)
total number of births by name, mother's race, and mother'seducation level
U.S. Census data on frequently occurring surnames (2000)
Amazon Mechanical Turk survey experiment (2014-2015)
Simulation of �eld experiments
Data Sources
New York state birth record data (1994-2012)
total number of births by name, mother's race, and mother'seducation level
U.S. Census data on frequently occurring surnames (2000)
Amazon Mechanical Turk survey experiment (2014-2015)
Simulation of �eld experiments
Data Sources
New York state birth record data (1994-2012)
total number of births by name, mother's race, and mother'seducation level
U.S. Census data on frequently occurring surnames (2000)
Amazon Mechanical Turk survey experiment (2014-2015)
Simulation of �eld experiments
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
~8900 respondents randomly assigned to one of six sets of 20names
120 total names; 48 black, 48 white, 24 Hispanic
Random assignment to condition:
First name onlyFirst and last name (half racially matched last name, halfmismatched)
Asked to identify the race/ethnicity they associated with eachname (open ended)
MTurk Survey Experiment
MTurk Survey Experiment
MTurk Survey Experiment
MTurk Survey Experiment
Simulation - Interpreting Inaccurate Racial Identi�cation
Names are not a perfect signal of race
Assumptions about true response rates (both white and black)
ExpectedWRR = TrueWRR � (InaccuracyWNR X[TrueBRR<=Random<=TrueWRR])
ExpectedBRR = TrueBRR � (InaccuracyBNR X[TrueBRR<=Random<=TrueWRR])
Simulation - Interpreting Inaccurate Racial Identi�cation
Names are not a perfect signal of race
Assumptions about true response rates (both white and black)
ExpectedWRR = TrueWRR � (InaccuracyWNR X[TrueBRR<=Random<=TrueWRR])
ExpectedBRR = TrueBRR � (InaccuracyBNR X[TrueBRR<=Random<=TrueWRR])
Simulation - Interpreting Inaccurate Racial Identi�cation
Names are not a perfect signal of race
Assumptions about true response rates (both white and black)
ExpectedWRR = TrueWRR � (InaccuracyWNR X[TrueBRR<=Random<=TrueWRR])
ExpectedBRR = TrueBRR � (InaccuracyBNR X[TrueBRR<=Random<=TrueWRR])
Simulation - Interpreting Inaccurate Racial Identi�cation
Names are not a perfect signal of race
Assumptions about true response rates (both white and black)
ExpectedWRR = TrueWRR � (InaccuracyWNR X[TrueBRR<=Random<=TrueWRR])
ExpectedBRR = TrueBRR � (InaccuracyBNR X[TrueBRR<=Random<=TrueWRR])
Simulation - Interpreting Inaccurate Racial Identi�cation
3 scenarios
1. med/high SES black �rst names w/ white last names;med/high SES white �rst names w/ white last names2. any SES black �rst name w/ black last names; any SESwhite �rst names w/ white last names3. low SES black �rst name w/ black last names; med/highSES white �rst names w/ white last names
10 white, 10 black names; 1,000 matched pairs; simulated1,000 �eld experiments for each scenario
Simulation - Interpreting Inaccurate Racial Identi�cation
3 scenarios
1. med/high SES black �rst names w/ white last names;med/high SES white �rst names w/ white last names2. any SES black �rst name w/ black last names; any SESwhite �rst names w/ white last names3. low SES black �rst name w/ black last names; med/highSES white �rst names w/ white last names
10 white, 10 black names; 1,000 matched pairs; simulated1,000 �eld experiments for each scenario
Simulation - Interpreting Inaccurate Racial Identi�cation
3 scenarios
1. med/high SES black �rst names w/ white last names;med/high SES white �rst names w/ white last names2. any SES black �rst name w/ black last names; any SESwhite �rst names w/ white last names3. low SES black �rst name w/ black last names; med/highSES white �rst names w/ white last names
10 white, 10 black names; 1,000 matched pairs; simulated1,000 �eld experiments for each scenario
Simulation - Interpreting Inaccurate Racial Identi�cation
3 scenarios
1. med/high SES black �rst names w/ white last names;med/high SES white �rst names w/ white last names2. any SES black �rst name w/ black last names; any SESwhite �rst names w/ white last names3. low SES black �rst name w/ black last names; med/highSES white �rst names w/ white last names
10 white, 10 black names; 1,000 matched pairs; simulated1,000 �eld experiments for each scenario
Simulation - Interpreting Inaccurate Racial Identi�cation
3 scenarios
1. med/high SES black �rst names w/ white last names;med/high SES white �rst names w/ white last names2. any SES black �rst name w/ black last names; any SESwhite �rst names w/ white last names3. low SES black �rst name w/ black last names; med/highSES white �rst names w/ white last names
10 white, 10 black names; 1,000 matched pairs; simulated1,000 �eld experiments for each scenario
Simulation Results
Simulation Results
Conclusions
Need more scienti�c inquiry into racial cues provided by auditstudies
Large variation in correct identi�cation; highly dependent onsocial class and last name information
Likely underestimating true racial discrimination
Conclusions
Need more scienti�c inquiry into racial cues provided by auditstudies
Large variation in correct identi�cation; highly dependent onsocial class and last name information
Likely underestimating true racial discrimination
Conclusions
Need more scienti�c inquiry into racial cues provided by auditstudies
Large variation in correct identi�cation; highly dependent onsocial class and last name information
Likely underestimating true racial discrimination
