Upload
wanda
View
31
Download
2
Embed Size (px)
DESCRIPTION
Unobserved common causes of measurement and nonresponse error on the 2008 ANES Panel Survey. International Total Survey Error Workshop Stowe, VT - June 13-16, 2010 Caroline Roberts – University of Lausanne, CH Patrick Sturgis – University of Southampton, UK - PowerPoint PPT Presentation
Citation preview
1
Unobserved common causes of measurement and nonresponse error on the
2008 ANES Panel Survey
International Total Survey Error Workshop Stowe, VT - June 13-16, 2010
Caroline Roberts – University of Lausanne, CHPatrick Sturgis – University of Southampton, UK
Nick Allum – University of Essex, UK
Overview
• Background and motivation• Objectives of this study• Data and sample• Methods• Level of effort & response propensity analysis• Structural Equation Modeling• Provisional conclusions and discussion points
Background
• Several theoretical models specify ways measurement error and nonresponse bias might relate (see Olson 2007)
• ‘Common cause’ model: variables influencing response propensity also influence response accuracy
• Possibility to test the model restricted by data availability
• Focus on individual items does not address error from suboptimal response strategies
Motivation
• To what extent do common causes influence both types of error? – The role of motivation and ability
• Our approach uses:– Panel data – to investigate a range of candidate common
causes– Structural Equation Models to quantify the unobserved
component
• Theoretical and practical interest
Objectives
Two elements:1. Comparison of data quality based on respondent
‘cooperativeness’ in the panel: do the least cooperative differ from the most?
2. Analysis of common causes of response propensity and measurement error using SEM: what is the extent and magnitude of the unobserved component?
Data• 2008-2009 ANES Internet Panel Survey• Recruited by RDD telephone survey• Non-internet households got MSN Web-TV • 21 monthly Internet surveys: $10 each• Fieldwork by Knowledge Networks • Advance release data file (June 2009) include
• recruitment data (including CATI paradata)• core profile survey• plus 6 ANES waves (Jan, Feb, Jun, Sep, Oct, & Nov 2008)
DeBell, Krosnick, Lupia & Roberts, 2009
Sample and Fieldwork• Probability sample of US citizens aged 18+• Data from 1 of 2 recruitment cohorts• 12,809 landline numbers; 2,371 completed
recruitment (18.5%)• 4 month fieldwork – up to 50 call attempts• 2 protocol changes –
• Refusal conversion by NORC• Internet-only recruitment for 50+ calls
• AAPOR1 = 26% AAPOR3 = 42%• 1,738 completed recruitment + at least 1 ANES wave
Panel retentionNumber Percent of total
Completed interviews 2,371 100.0
Standard telephone 2,222 93.7
Refusal conversion 85 3.6
Internet 64 2.7
Completed 1+ ANES waves 1738 73.3
Profile 1545 65.2 (88.9)
Wave 1 (Jan 08) 1577 66.5 (90.7)
Wave 2 (Feb 08) 1438 60.6 (82.7)
Wave 6 (Jun 08) 1406 59.3 (80.9)
Wave 9 (Sep 08) 1466 61.8 (84.3)
Wave 10 (Oct 08) 1487 62.7 (85.6)
Wave 11 (Nov 08) 1482 62.5 (85.3)
Cooperativeness• 3 indicators of recruitment effort:
1. Number of calls to a complete interview (1-5 vs. 6 or more)
2. Whether respondent or household member refused to participate during call attempts (refused once or more vs. never refused)
3. Respondent recruited after protocol change (by internet or refusal conversion vs. by standard telephone)
• Actual response propensity• Differences in sample composition, responsiveness,
key survey estimates, data quality
Data quality• Indicators of survey satisficing (Krosnick, 1991)
• Item non-response (wave 1 only)• Non-differentiation of items with same response scale• Preference for midpoints in branched questions
• Item sets repeated across several ANES waves:• Condition of the country (5 pts); candidate liking,
attitudes to groups, policy attitudes, candidate policy positions (branched 7-pt scales)
• Validity checks • Consistency and accuracy of reports – e.g. voting (but see
Berent et al. 2010)
Results
• Few significant differences in refusal and protocol change comparisons
• But respondents recruited after 6+ calls are:– Younger and more likely to be Black, non-Hispanic– less likely to have Internet access– Less likely to be Republican and conservative– Slightly more likely to satisfice
• And reluctance at recruitment leads to lower cooperation in panel
Responsiveness
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
No. of waves started (mean)
6.34(.04)
5.85 ***(.07)
6.25 (.03)
5.95*** (.08)
6.26 (.03)
5.11 ***(.14)
No. of complete waves (mean)
6.14(.04)
5.61***(.07)
6.05(.04)
5.73**(.09)
6.05(.04)
4.82***(.15)
Completed all waves %
62.5 (1.38)
44.2*** (2.19)
59.5 (1.31)
47.2*** (2.72)
60.3 (1.20)
3.1***(1.77)
Missed 1 or more waves %
23.4 (1.21)
39.5*** (2.15)
26.1 (1.17)
36.8*** (2.63)
25.2 (1.07)
79.4*** (4.13)
Dropped out % 14.1(1.00)
16.3 (1.63)
14.4 (0.94)
16.0 (2.00)
14.6 (0.87)
17.5 (3.88)
SociodemographicsLess than 6
surveys completed
4-5 surveysCompleted
6-7 surveys completed
Male % 46.6 * 45.9 41.7
18-39 % 33.8*** 34.6*** 21.2
40-59 % 39.9** 39.7* 47.4
60+ % 26.4* 25.7* 31.4
White, nH % 76.5*** 77.4*** 87.6
Black, nH % 13.5*** 13.4** 7.0
Hispanic % 7.4** 7.2* 4.1
Other, nH % 6.5** 5.1 3.6
SociodemographicsLess than 6
surveys completed
4-5 surveysCompleted
6-7 surveys completed
< High Sch % 7.4** 6.5* 3.8
High Sch % 19.2** 19.5* 13.8
College % 35.9 38.0 38.5
Bachelor % 20.3 23.3 23.2
Graduate % 14.4** 12.3** 20.4
Northeast 17.0 15.4 15.6
Midwest 23.7* 28.4 28.4
South 32.5 33.2 31.7
West 19.0* 17.8* 24.1
Recruitment Variables
Less than 6 surveys
completed
4-5 surveysCompleted
6-7 surveys completed
Internet access % 83.2** 83.9* 88.0
Interest in politics %★ 37.9† 38.7 42.2
Interest in computers★ 41.0 39.0 41.4
Number of opinions★★ 31.4 31.8 29.0
Important to obey authority★
70.6 69.9 70.4
Wave 1 variablesLess than 6
surveys completed
4-5 surveysCompleted
6-7 surveys completed
Democrat 28.3* 29.8 32.7
Independent 22.9*** 24.0** 31.9
Republican 17.4*** 19.9*** 33.6
Liberal 22.7*** 24.0** 32.7
Moderate 18.5 20.2 20.5
Conservative 27.2*** 29.8*** 45.0
Interest in pol 33.6*** 38.4*** 55.7
Voted in 2008 75.1*** 77.0*** 88.4
Voted Obama 28.1*** 34.9** 43.5
W1 attitudes
Data qualityLess than 6
surveys completed
4-5 surveysCompleted
6-7 surveys completed
W1 MP (mean) .47* .48* .45
W1 ND (mean) .54* .54* .53
W2 MP (mean) .60* .61* .57
W2 ND (mean) .64* .64* .61
W6 MP (mean) .34** .33* .30
W6 ND (mean) .46* .46† .44
W9 MP (mean) .32* .32** .28
W9 ND (mean) .47* .47 .45
W10 MP (mean) .23 .23 .22
W10 ND (mean) .35 .36 .35
W11 MP (mean) .26† .27 .23
W11 ND (mean) .49* .49 .47
Summary• Level of effort analysis:
– Small differences between respondents as a function of ‘effort’ required to recruit them
– Significant differences in their cooperativeness at later panel waves
– Significant differences in demographics, on key survey estimates, and on satisficing between more and less cooperative panel recruits
– A few differences on substantive items used in satisficing indicators, but not many
Common causes• Ability:
– Education– Computer/Internet literacy
• R required MSN-TV device• Motivation:
– Recruitment difficulty– Interest in computers– Interest in politics
• Demographic characteristics – sex, age, race & ethnicity
Satisficing
Recruitment difficulty
Ability
Motivation
Demographics
• N of refusals • N of calls to complete
• Education
• Interest in politics• Interest in computers
• Sex• Age• Race• Web access
Response
• N of panels started
• Non-differentiation• Use of midpoints
Common cause?
Correlated residual
SEM EstimatesNo covariates +Recruitment +Ability +Motivation +Demographics
Resp Satis Resp Satis Resp Satis Resp Satis Resp Satis
Refusals -.11* .03 -.10 .02 -.10 .05 -.11* .17
Calls -.04*** .12 -.04** .03 -.04** .03 -.03** .023
Postgrad Degree .44** -3.0** .37** -1.90** .26* -1.66
College Degree .42** -2.4** .35** -1.60** .22* -.47
Some college .34** -1.8** .30** -.57 .27** -1.40**
Interest politics .37** -7.78** .24 -7.10**
Interest computers -.07 1.38** .04 1.03*
Female -.13* -1.05**
Age .01** -.01
White .40** -1.82**
Web TV -.29** .50
Residual correlation -.09** -.08** -.05 -.01 .02
RMSEA<.05; CFI>.95
Summary 2• SEM:
– Very weak correlation between satisficing and propensity to respond – a ‘reassuring’ result?
– Recruitment difficulty predicts response propensity but not satisficing
– Motivation variables better predict satisficing; ability better predicts response. Both together can jointly account for the weak correlation between propensity to respond and satisficing.
Discussion points
• Limitations: – absence of external records– advance release data– specification of SEM
• Can we improve measures of responsiveness and satisficing (including choice of item sets)?
• How can we best utilize the strengths and compensate for the limitations of the panel design
Thank you
Sample compositionCharacteristic Percent in sample
analyzed (n = 1738)Percent in population
(CPS, March 2008)
Male 43.2 47.8
Northeast 16.0 18.5
Midwest 27.2 23.1
South 31.9 36.7
West 22.7 21.8
18-29 9.5 21.3
30-39 15.0 16.5
40-49 21.1 19.5
50-59 24.3 18.3
60-69 18.5 12.1
70 or older 11.6 12.2
Sample compositionCharacteristic Percent in sample
analyzed (n = 1738)Percent in population
(CPS, March 2008)
Race/ ethnicity
White, non-Hispanic 81.6 73.9
Black, non-Hispanic 8.8 11.7
Hispanic 5.6 9.2
Other, non-Hispanic 4.0 5.2
Educational attainment
< High school diploma 4.8 11.9
High school diploma 15.4 31.4
Some college 38.2 29.2
Bachelor’s degree 22.7 18.4
Graduate degree 19.0 9.2
Sociodemographics
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
Male % 43.3 43.0 42.3 47.2 43.1 45.4
18-39 % 21.4 32.0*** 24.1 26.4 23.6 39.2**
40-59 % 45.8 44.4 45.6 44.5 45.7 40.2
60+ % 32.8 23.6*** 30.3 29.1 30.7 20.6*
White, nH % 83.2 77.9** 81.2 83.4 81.4 85.6
Black, nH % 7.4 12.0** 9.1 7.4 9.0 5.2
Hispanic % 5.2 6.4 5.5 5.9 5.5 6.2
Other, nH % 4.1 3.7 4.1 3.3 4.0 3.1
Sociodemographics
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
< High Sch % 4.3 5.6 4.4 6.2 4.6 6.2
High Sch % 15.8 14.0 14.7 17.5 15.2 15.5
College % 37.6 38.2 37.4 39.5 37.5 42.3
Bachelor % 22.8 21.5 22.8 20.8 22.8 16.5
Graduate % 18.6 19.4 19.5 16.0 18.8 19.6
Northeast 15.7 16.7 15.5 18.1 15.9 17.5
Midwest 24.6 28.2 27.5 25.8 27.7 17.5*
South 32.6 31.7 31.9 32.0 31.2 44.3**
West 24.2 22.1 23.0 21.7 22.9 19.6
Recruitment variables
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
Internet access %
88.2 83.3** 86.7 86.9 86.8 85.6
Interest in politics %★
41.7 39.7 41.3 40.4 40.7 47.4
Interest in computers★
40.9 42.1 42.1 37.7 41.0 46.4
Number of opinions★★
30.5 27.5 29.8 28.8 29.7 27.8
Important to obey authority★
70.9 70.3 70.7 69.7 41.0 46.4
★ Very or extremely ★★About many things or just about everything
Wave 1 variablesWave 1 variables (n=1577)
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
Democrat 31.7 31.2 31.8 30.3 31.9 25.8
Independent 29.8 28.9 29.4 30.0 29.3 34.0
Republican 30.5 26.6 29.6 28.2 29.7 22.7
Liberal 30.0 30.2 30.2 29.4 30.1 28.9
Moderate 20.1 19.6 19.6 21.7 20.2 15.5
Conservative 41.9 36.6* 41.1 37.1 40.5 38.1
Interest in pol 51.1 46.9 50.2 48.4 50.0 47.4
Planned to vote 93.6 90.5* 93.1 91.1 92.8 91.9
Voted in 2008 85.4 87.0 85.7 86.6 85.9 85.1
Voted Obama 37.6 43.8* 39.3 40.4 39.5 39.2
Data quality
1-5 calls(n=1222)
6+ calls(n=516)
No refusal
(n=1401)Refusal (n=337)
Standard recruit
(n=1641)
Protocol change (n=97)
W1 INR (mean) .01 .01 .01 .02 .01 .02
W1 MP (mean) .44 .47* .45 .46 .45 .45
W1 ND (mean) .53 .54 .53 .54 .53 .53
W2 MP (mean) .57 .59 .57 .59 .58 .59
W2 ND (mean) .61 .62 .61 .62 .62 .62
W6 MP (mean) .30 .31 .30 .32 .30 .31
W6 ND (mean) .44 .47** .44 .45 .44 .46
W9 MP (mean) .28 .29 .28 .31 .29 .30
W9 ND (mean) .46 .46 .46 .47 .46 .45
W10 MP (mean) .22 .22 .22 .23 .22 .22
W10 ND (mean) .35 .35 .35 .36 .35 .35