29
1 Survey Nonresponse Survey Nonresponse Survey Research Laboratory University of Illinois at Chicago www.srl.uic.edu March 16, 2010

Survey Nonresponse

  • Upload
    benjy

  • View
    69

  • Download
    0

Embed Size (px)

DESCRIPTION

Survey Nonresponse. Survey Research Laboratory University of Illinois at Chicago www.srl.uic.edu March 16, 2010. Outline. Item nonresponse (IN) Definition Causes Impact on data Strategies for minimizing Correcting for IN Reporting issues & IN. Unit nonresponse (UN) Definition - PowerPoint PPT Presentation

Citation preview

Page 1: Survey  Nonresponse

1

Survey NonresponseSurvey Nonresponse

Survey Research LaboratoryUniversity of Illinois at Chicago

www.srl.uic.eduMarch 16, 2010

Page 2: Survey  Nonresponse

2

Survey Research Laboratory

OutlineOutlineI. Unit nonresponse

(UN)A. DefinitionB. CausesC. Impact on dataD. Strategies for

minimizingE. Correcting for UNF. Reporting issues &

UN

II. Item nonresponse (IN)A. DefinitionB. CausesC. Impact on dataD. Strategies for

minimizing E. Correcting for INF. Reporting issues &

IN

Page 3: Survey  Nonresponse

3

Survey Research Laboratory

I. Unit nonresponseI. Unit nonresponse

A.Definition: “Nonresponse occurs when a sampled unit does not respond to the request to be surveyed…” (Groves et al., 2002) increasing concern, especially in

telephone surveysB. Causes of unit nonresponse

noncontact noncooperation

Page 4: Survey  Nonresponse

4

Survey Research Laboratory

Failure to contactFailure to contact

• Non-contact patterns vary by survey mode & across different subgroups

Restricted-access buildings & communities

Telephone numbers connected to answering machines, caller ID. . .

Page 5: Survey  Nonresponse

5

Survey Research Laboratory

Percent of Eligible Sample Households by Calls to Percent of Eligible Sample Households by Calls to First Contact, for Five Surveys (Groves, 2001)First Contact, for Five Surveys (Groves, 2001)

0

10

20

30

40

50

60

1 2 3 4 5 6 7-8 9 ormore

NC

Calls to First Contact

Perc

ent o

f the

Hou

seho

lds

Current Events Mental Health Election

Welfare Reform Child Health

Page 6: Survey  Nonresponse

6

Survey Research Laboratory

Failure to gain cooperation (refusals)Failure to gain cooperation (refusals)

• Survey requests commonly misclassifed

• Repeated callbacks distinguish surveys

Page 7: Survey  Nonresponse

7

Survey Research Laboratory

Evidence that survey requests are Evidence that survey requests are misclassifiedmisclassified

• Refusals made quickly (<30 seconds)

• Many refusers reconsider later• Persons apparently focus on a small

number of factors when making the decision.

Page 8: Survey  Nonresponse

8

Survey Research Laboratory

Factors typically considered by Factors typically considered by respondentsrespondents

• Burden• Saliency (topic

importance)• Incentive• Authority of sponsor

Page 9: Survey  Nonresponse

9

Survey Research Laboratory

ImplicationsImplications

• Many different reasons for refusals• No single introduction may be

suitable for all• Interviewers must tailor

introductions information from initial refusal or

contact attempt

Page 10: Survey  Nonresponse

10

Survey Research Laboratory

C. Impact of UN on dataC. Impact of UN on data

• UN will results in nonresponse bias to the extent that respondents & nonrespondents differ on variables of interest types of nonresponse bias

• representativeness• data quality• distribution of responses

strategies for estimating nonresponse bias• number of calls or refusal conversions & data• data from studies with different response rates• using census data to compare respondents &

nonrespondents

Page 11: Survey  Nonresponse

11

Survey Research Laboratory

Nonresponse bias as function of response rate (Nonresponse bias as function of response rate (ttRR))

Page 12: Survey  Nonresponse

12

Survey Research Laboratory

Empirical research in general Empirical research in general finds no response rate effectfinds no response rate effect

• Keeter et al. (2000)• Curtin, Presser and Singer (2000)• Merkle and Edelman (2002)

Page 13: Survey  Nonresponse

13

Survey Research Laboratory

Example of no relationship between Example of no relationship between nonresponse & measurement errornonresponse & measurement error

From Merkle, D.M., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In Groves, R.M., Dillman, D.A., Eltinge, J.L., & Little, R.J.A. (Eds.), Survey nonresponse (pp. 243-257). New York: Wiley.

Page 14: Survey  Nonresponse

14

Survey Research Laboratory

D. Strategies for reducing UND. Strategies for reducing UN

• Mode• Increase benefits

Incentives Convince potential

respondents of importance• introduction• publicity• trusted source or sponsor

• Reduce perceived burden

• Initial contact before interview (advance letter)

• Persistence: refusal conversions follow-ups change societal

perceptions mode changes interviewer changes

Page 15: Survey  Nonresponse

15

Survey Research Laboratory

E. Correcting for UNE. Correcting for UN

• Post data-collection weighting• Assumptions

respondents & nonrespondents in a particular category (e.g,. African Americans) are interchangeable

known population parameters of relevant characteristics

Page 16: Survey  Nonresponse

16

Survey Research Laboratory

F. Reporting issues and UNF. Reporting issues and UN

• Standards can be found at www.aapor.org• Disposition codes

Known eligible• Completes• Refusals

Known ineligible• Businesses• Disconnected numbers

Unknown eligibility• Ring, no answers• Always busy

Page 17: Survey  Nonresponse

17

Survey Research Laboratory

AAPOR AAPOR responseresponse rates rates

DenominatorNumerator

Known Eligible

Known Eligible + Unknown Eligibility

Known Eligible + e(Unknown Eligibility)

Completes LowestCompletes + Partials

Highest

Page 18: Survey  Nonresponse

18

Survey Research Laboratory

Reporting of response ratesReporting of response rates

• AAPOR standards• Examination of reporting in recent

journal articles done by Johnson and Owens (2003) AAPOR paper

Page 19: Survey  Nonresponse

19

Survey Research Laboratory

Journal article auditJournal article audit

• Sample of 95 papers selected from same set of 18 journals

• Published January 2000 or more recent

• Unit of analysis = individual• Noninstitutional population• Reporting primary survey data (no

secondary analyses)

Page 20: Survey  Nonresponse

20

Survey Research Laboratory

JournalsJournalsHealth science

journals• Amer J Psychiatry• Amer Jour Preventive

Medicine• Amer J Public Health• Annals of Epidemiology• JAMA• J Studies on Alcohol• Medical Care• Preventive Medicine• Substance Use & Misuse

Social science journals

• Amer J Community Psychology

• Amer J Political Science• Amer Political Science Review • International J Public Opinion

Research• J Community Psychology• J Health & Soc Behavior• Public Administration Review• Public Opinion Quarterly• Social Forces

Page 21: Survey  Nonresponse

21

Survey Research Laboratory

Disclosure typologyDisclosure typology

1. Report no information (5.3%)2. Report other information only (23.2%)3. Report non-AAPOR response rate (26.3%)4. Report AAPOR-consistent response rate

(24.2%)5. Report summary of sample dispositions

(21.1%)

Page 22: Survey  Nonresponse

22

Survey Research Laboratory

Disclosure typology IIDisclosure typology II

• Mail surveys are most likely to provide AAPOR response rate and/or full sample disposition information (88%).

• Telephone, face-to-face surveys are most likely to provide response rate information (44% and 43%).

• p < .001

Page 23: Survey  Nonresponse

23

Survey Research Laboratory

Other observationsOther observations

• Some argue that “no definitive formula for response rate estimates exists”

• No papers underestimate response rates

• Many appear to be overestimating response rates

Page 24: Survey  Nonresponse

24

Survey Research Laboratory

II. Item nonresponseII. Item nonresponse

A. Definition: When a given unit does not provide a substantive response to a particular question. Sometimes called “missing data”

B. Causes of IN ability to provide information willingness to provide information privacy and anonymity concerns

Page 25: Survey  Nonresponse

25

Survey Research Laboratory

C. Impact of IN on dataC. Impact of IN on data

• IN introduces nonresponse bias if it is associated with key variables

• Examine associations between IN & key variables representativeness (demographics) other substantive variables

Page 26: Survey  Nonresponse

26

Survey Research Laboratory

D. Strategies for minimizing IND. Strategies for minimizing IN

• reassurance of anonymity• follow-up question(s)• sometimes ask for less specific

information (e.g., income)

Page 27: Survey  Nonresponse

27

Survey Research Laboratory

E. Correcting for INE. Correcting for IN

• dropping variables• dropping observations• pairwise deletion• imputation of values for missing data• person mean substitution (scales)• item mean substitution• impact on central tendency

Page 28: Survey  Nonresponse

28

Survey Research Laboratory

F. Reporting issues and INF. Reporting issues and IN

• Steps to deal with IN should be reported

• If a correction strategy is used, the impact of that strategy should be reported (comparison of analysis with & without correction strategy)

Page 29: Survey  Nonresponse

29

Survey Research Laboratory

SummarySummary

• Ideally, reduce nonresponse• Correct for nonresponse• Distinction between nonresponse &

nonresponse bias• Test for nonresponse bias• Report nonresponse issues &

correction strategies