Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,

Responsive Design for Household Surveys:

Illustration of Management Interventions Based on Survey Paradata

Robert M. Groves, Emilia Peytcheva, Nicole Kirgis, James Wagner, William Axinn, University of Michigan,

USAWilliam Mosher, US National Center for Health

Statistics

Research partially supported by contract with the US National Center for Health Statistics, Contract No. 200-2000-07001

Definition: Responsive Design

1. Preidentify a set of alternative features potentially affecting costs and errors of statistics

2. Identify a set of indicators of the cost and error properties of those feature

3. Monitor indicators in initial stages of data collection

4. Alter the active features of the survey based on cost/error tradeoff decision rules

5. Combine data from separate phases into a single estimator

Survey designs that:

The Why’s of Responsive Designs

• One-off surveys are mounted with large uncertainties (e.g., eligibility of frame elements, effort required to contact, cooperation rate, length of interview)

• Most survey budgets are relatively fixed at start of project

• Some survey errors are functions of effort during production

Hence, quality is out of control of researcher unless designs are permitted to change based on production experience

Effort

Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey

0

5

10

15

20

25

30

35

1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI

Phase 1 Phase 2

The NSFG Dashboard

hours

% production

Active Sample

nonworked

interviews

Productivity

calls/day

% peak callsscrn’r/main calls

mean calls

8+ calls

locked bldgs

resistant

hard appt.

occupied

eligible

propensity

calls/interview

hours/interview

% with kids

response rate


0

5

10

15

20

25

30

35


Phase 1 Phase 2

I’rs working


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

calls/hourDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and

Screener HPI by Day of Survey

0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

noncontacts


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

cum. interviews


0

5

10

15

20

25

30

35


Phase 1 Phase 2

Data Set BalanceDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

% sexually activeDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2


0

5

10

15

20

25

30

35


Phase 1 Phase 2

group ratesDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and


0

5

10

15

20

25

30

35


Phase 1 Phase 2

CV group rates

X

Evaluation of Two Interventions Based on Paradata

• Survey setting: face-to-face survey screening (3-5 min.) to locate 60% of households with 15-44 year old; one eligible sampled (60-80 min.)

• Interventions– Increasing relative effort on screening interviews

versus main interviews with selected respondent– Increasing relative effort on a small subset of cases

with high selection weights and high propensities to respond

Ratio of Screener Calls to Main Interview Calls by Day by Quarter

0

1

2

3

4

5

6

1 8 15 22 29 36 43 50 57 64

Q2 7-day Average Scrn/Main calls Q1 7 day Scrn/Main calls Q3 7-day Scrn/Main Calls

Q4 7-day Scrn/Main Calls Year 1 Summary

Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Calls

(p-values for coefficients)

Q1 Q2 Q3 Q4

Intercept 279.9(<.0001)

385.0(<.0001)

345.3(<.0001)

294.8(<.0001)

Day -0.73(.48)

-3.2(0.28)

-2.1(.21)

-2.0(.17)

Screener Week

28.3(.29)

97.5(.009)

24.6(.56)

39.0(.29)

Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Interviews

(p-values for coefficients)

Q1 Q2 Q3 Q4

Intercept 63.6(<.0001)

94.5(<.0001)

64.6(<.0001)

63.4(<.0001)

Day -0.51(.032)

-1.23(0.001)

-0.54(.16)

-0.69(.09)

Screener Week

6.1(.30)

18.5(.01)

12.3(.20)

6.1(.54)

Second Intervention: Increased Emphasis on Subset of Active Cases

• In the last weeks of Phase 1, a subsample of cases with high propensities and high selection weights are identified

• These cases are chosen to improve balance of respondent pool

• Interviewers are asked to give greater emphasis to these cases

Mean Expected Probability to be Interviewed on Next Call, Screeners (Red) and Main (Green)

by Day of Data Collection by Quarter

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

1 8 15 22 29 36 43 50 57 64

Q1 Screener Mean Propensity Q1 Main Mean Propensity Q2 Screener Mean Propensity

Q2 Main Mean Propensity Q3 Screener Mean Propensity Q3 Main Mean Propensity

Q4 Screener Mean Propensity Q4 Main Mean Propensity Q2-Q4 Average Screener

Q2-Q4 Average Main

Analytic Approach

• Not all interviewers active workloads contain both “intervention” cases and “non-intervention” cases

• We limit the analysis to those who have both types of cases

• We examine two indicators of success:– Mean number of calls (imperfect)– Response rate in intervention period

Comparison of Mean Screener Calls During Intervention Period for Intervention and

Nonintervention Cases

Little evidence of increased calling on intervention cases

Comparison of Screener Response Rate During Intervention Period for Intervention

and Nonintervention Cases

Little evidence of increased response rate on intervention cases

Comparison of Mean Main Calls During Intervention Period for Intervention and


Mixed evidence on higher calls on intervention cases

Comparison of Main Response Rate During Intervention Period for Intervention and


General tendency to higher response rates for interventioncases

Logistic Regression For Likelihood of Main Interview

Quarter 2 Quarter 3 Quarter 4 Quarter 5Intercept -2.13*** -1.97*** -0.81*** -0.84**Intervention -0.07 0.72* 0.13 0.50*Weight 0.00 0.00 0.00 0.00Propensity 5.46*** 2.98** 2.02*** 0.65

Conclusions

• Intervention 1: Management direction to focus on screeners vs. main increases calls, sometimes dramatically; screener interviews follow

• Intervention 2: Effectiveness at focusing on individual cases greater for main interviews than screener interviews

Next Steps on Responsive Design with Paradata

• Responsive design requires effective central management direction of interviewer behavior

• We’re still learning how to communicate these directives consistently well

Documents

Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,