Let Me Tell You What You Told Me: Dependent Interviewing ......• Panel surveys often ask for same...

Let Me Tell You What You Told Me:Dependent Interviewing in

Establishment Surveys

Heather RidolfoNational Agricultural Statistics Service

Jennifer EdgarBureau of Labor Statistics

AAPOR 2015

Background

• Panel surveys often ask for same informationacross data collection points

• Burden• Measurement error can produce false rates of

change• Dependent Interviewing one solution

AAPOR 2015

Dependent Interviewing (DI)

• ProactiveThe last time we interviewed you on January 21,2013, you had 10 employees. How many employeesdo you have now?• Reactive(after a change in status is reported) Last time youreported that there were 5 employees at thiscompany. Does knowing that change your answer inany way?

AAPOR 2015

DI in Population Surveys

• Burden– Increases & Decreases

• Rapport– Improve (PDI)– Decreases (RDI)– Little evidence of

confidentiality concerns

• Data Quality ImprovedSome of the Time– Reduces misclassification– Reduces spurious change– Question & respondent

characteristics matter– Decreases editing– Little evidence of

satisficing– Less standardization

AAPOR 2015

DI in Establishment Surveys

• Reactive is more effective than proactive inimproving data quality but still adds burden

• Establishment characteristics (size)• Construct characteristics (stability, complexity)• Satisficing is a problem

AAPOR 2015

Research Questions

• How do establishment respondents integratepreviously reported data into the questionresponse process?

• Do establishment respondents use previouslyreported data differently when it’s presentedproactively vs. reactively?

AAPOR 2015

Methods

• Occupational Employment Survey (OES)Cognitive Interviews

• Agricultural Labor Survey Behavior Coding

AAPOR 2015

• Collects data on employment, job title and wages• Considering DI to improve data quality for phone

interviews (generally small establishments)• 17 cognitive interviews conducted over the phone• Randomly assigned to DI condition

– Proactive (n=8)– Reactive (n=9)

• Employment no. captured 12 and 18 months prior

Methods: OES Cognitive Interviews

AAPOR 2015

OES Results: Response Process

• With small companies (average employment count was6), most respondents simply knew the employmentcount

• Almost all respondents reported employment withoutconsulting records or any mental calculations– “I own the company, I know who works here”

AAPOR 2015

OES Results: Proactive DIEx: Last time, you reported that there were 5employees at this company. How many employeeswork there now?• Some respondents confirmed the PRD rather than

answering with an employment count• Most respondents who had a change in

employment were able to explain it in thedebriefing

AAPOR 2015

OES Results: Reactive DIEx: (after a different employment count was provided.) Last timeyou reported that there were 5 employees at this company. Doesknowing that change your answer in any way?• Most respondents (5 out of 8) reported the same employment

so reactive DI was not done• For the three that received reactive DI, differing results

• Easily able to explain difference• Correction of answer “Oh, you must be talking about New

York”• Unwillingness to reconcile differences “maybe I made a

mistake back then”

AAPOR 2015

OES Results: Reaction to DI

• Respondents don’t recall reporting last year• No confidentiality concerns• In both conditions, respondents reported that PRD didn’t

impact their response process much

AAPOR 2015

OES Conclusions

• Little impact of DI on data quality• Some evidence of satisficing

– Proactive: “it’s the same”– Reactive: Unwilling to reconcile differences in PRD

and current reports• No discernable impact on detecting change• No evidence of burden reduction

AAPOR 2015

Methods: Ag Labor Behavior Coding

• Collects data on types and numbers of farmworkers, hours worked and wages paid

• October 2013 Ag Labor Survey• Recorded 759 interviews CATI calls• 139 contained questions of interest & quality

audio• 2,689 observations coded• 4 coders (Kappa = .62 - .66)

AAPOR 2015

Methods: Ag Labor Behavior Coding

Interviewer BehaviorCodes• Asked as worded• Major change• Used PRD• Shortcutted

Respondent BehaviorCodes• Yes• Elaborated• Corrected• Requested clarification• Answered don’t know• Gave no response

AAPOR 2015

Ag Labor Results

• PRD used 19% of the time- Quarter 1: 14%- Quarter 2: 24%

AAPOR 2015

Respondents’ Reactions to PreviouslyReported Data (n=166)

Respondent Reaction Percent

Cognitive EngagementElaborated 15Corrected 18Requested Clarification 1

Possible SatisficingYes 50Gave No Response 23

OtherAnswered Don’t know 1Inaudible 3

AAPOR 2015

Cognitive Engagement By Question

0

20

40

60

80

100

Screener WorkerType

Describe No. Workers Hours Wages

Perc

ent

Questions

Cognitive Engagement Possible Satisficing

AAPOR 2015

Ag Labor Conclusions

• Some evidence of cognitive effort acrossquestions; majority of respondents possiblesatisficing

• No evidence that question type matters• Interviewers need more training

AAPOR 2015

Recommendations

• More research is needed to understand theestablishment respondent’s response process

• Initial evidence suggest satisficing problem• There was no strong evidence promoting one DI

method over another (proactive or reactive)• For either method, interviewer training is crucial

– Interviewers need to be aware of the potential risksassociated with DI and equipped to address them

AAPOR 2015

Questions?

Heather RidolfoHeather.Ridolfo@nass.usda.gov

Jennifer EdgarEdgar.jennifer@bls.gov

AAPOR 2015