Adaptive Design for the NewNational Teacher andPrincipal Survey

Minsun Riddles, David Marker, Lou Rizzo, Erin Wiley, WestatAndrew Zukerberg, National Center for Education Statistics

• This presentation is intended to promote ideas. The viewsexpressed are part of ongoing research and do notnecessarily reflect the position of Westat or the U.S.Department of Education.

2

Disclaimer

• Starts in school year 2015-16

• Redesign of Schools and Staffing Survey (SASS)

3

National Teacher and Principal Survey (NTPS)

SASS NTPSCycle Every four years Every two yearsEstimating National and state NationalSamples School districts

Public/private schoolsPublic/private school principalsPublic/private school teachersPublic school libraries

Public schoolsPublic school principalsPublic school teachers

SASS Response Rates by Date

59% 62% 66% 70% 73%

2/7 3/6 4/3 5/1 7/10

Principals

58%67% 72%

77% 78%

4/3 5/1 5/15 6/5 7/10

Teachers

• Data collection every two years.

• Faster release of data.

• Integration with other ED collections.

• Reduce response burden.

5

Goals for NTPS

• Given a short field period, responsive design approach maynot be realistic.

• Conflict between “Go the Hard Road” vs. “Pick the LowHanging Fruit”

6

NTPS Adaptive Design Approach

“Go the Hard Road” “Pick the Low Hanging Fruit”Groves (2006)Groves & Peytcheva (2008)Schouten et al (2009)Schouten et al (2011)Särndal and Lundström (2008)

Groves and Heeringa (2006)

Low-propensity cases High-propensity casesReduce differences in responsepropensity

Maximize response rate

• Can we publish in a more timely fashion by cutting off datacollection early?

• What are the implications on estimates?

• Can we use paradata to identify methods that can reducepotential nonresponse bias?

7

NTPS Adaptive Design Research Questions

8

SASS Principal Response Rates by Cutoff Dates

Student enrollmentFree, reduced

price lunchCommunity typeSchool levelSchool type

0

20

40

60

80

100

2/7 3/6 4/3 5/1 Final (7/10)

9

SASS Teacher Response Rates by Cutoff Dates

Student enrollmentFree, reduced

price lunchCommunity typeSchool levelSchool type

0

20

40

60

80

100

4/3 5/1 5/15 6/5 Final (7/10)

10

SASS Teacher Response Ratesby Teacher Wave Group

Significantly different (p<0.005), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

All Early Middle Late

11

SASS Follow-up Effort for Principal ResponsePovertyCommunity typeSchool type

Significantly different (p<0.05), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

All Charter Traditional City Suburban Town Rural Highpoverty

Low/medpoverty

No follow-up One follow-up Multiple follow-ups Field Follow-up

12

SASS Follow-up Effort for Teacher ResponsePovertyCommunity typeSchool type School level

Significantly different (p<0.05), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

No follow-up One follow-up Multiple follow-ups Field follow-up

13

SASS Follow-up Effort for Teacher Responseby Teacher Wave Group

Significantly different (p<0.05), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

All Early Middle Late

No follow-up One follow-up Multiple follow-ups Field follow-up

• Create 8 data sets, 4 for principals, 4 for teachers, usingdifferent cutoff dates for the field period.

• Reproduce initial public release tables from each data set.

• Compare estimates to the final estimates

– Do some of the interim data sets produce similar estimates to thefinal data set?

– Ideally identify a response level at which similar results areachieved, but earlier time frame

14

SASS Counterfactual Scenarios with Early Cutoff

15

SASS Counterfactual Scenarios with Early Cutoff

*Significantly different (p<0.05), t-test

0%

2%

4%

6%

8%

10%

2/7* 3/6* 4/3* 5/1* Final (7/10)

Principal Estimates

Hispanic Non-Hispanic African American

16

SASS Counterfactual Scenarios with Early Cutoff

*Significantly different (p<0.05), t-test

0%

2%

4%

6%

8%

4/3* 5/1* 5/15* 6/5 Final (7/10)

Teacher Estimates

Non-Hispanic African American

17

SASS Survey Coordinator

• In sampled public schools, survey coordinators wererecruited.

Survey coordinator Principal response% Teacher response%No survey coordinator 26.4% 58.6%With survey coordinator 81.1% 76.4%

18

SASS Follow-up by Survey Coordinator Status

Significantly different (p<0.05), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

All principals No surveycoordinator

With surveycoordinator

All teachers No surveycoordinator

With surveycoordinator

No follow-up One follow-up

Multiple follow-ups Field Follow-up

19

SASS Experimentation with Survey Coordinator

• Among sampled teachers in public schools with a surveycoordinator, an experiment was conducted.

– Teacher letter through coordinator (treatment group)

– Teacher letter not through coordinator (control group)

Teacher experimental group Teacher response%Teacher letter through coordinator 78.1%Teacher letter not through coordinator 74.7%

Significantly different (p<0.05), Rao-Scott chi-square test

20

SASS Follow-up by Teacher Experimental Group

Significantly different (p<0.05), Rao-Scott chi-square test

0%

20%

40%

60%

80%

100%

With survey coordinator Letter through surveycoordinator

Letter not through surveycoordinator

No follow-up One follow-up Multiple follow-ups Field Follow-up

• Field period length can be shortened with small loss inresponse rate and minimal change in estimates.

• Need to prepare for short field period for teacherquestionnaires with teacher listing form received late.

• Recruiting and making contact through a survey coordinatormay help to increase response rates and to reduce follow-up efforts.

21

Conclusions from SASS Results

• Paradata to be collected and analyzed.

– Modes of contact, dates for each contact, mail out dates, …

• Variable field efforts from the beginning for classes ofschools, principals, and teachers with possibly lowerresponse propensities (city schools, high poverty schools,minority or less experienced principals and teachers).

• Reduce response rate differences (“Go the hard road”).

• Ongoing use of experimentation and simulation.

22

NTPS Planned Initiatives for 2015-16 Cycle

23

Thank youMinsunRiddles@westat.com