The NC Race to the Top Evaluation: An Update CCRESA Board of Directors May 17, 2013

Consortium for Educational Research and Evaluation–North Carolina

The NC Race to the Top Evaluation:An Update

CCRESA Board of Directors

May 17, 2013

Jessica Anderson, SERVE Center, UNCGJulie Marks, CIPP, UNC-CH

Trip Stallings, Friday Institute, NCSU


NC Race to the Top Evaluation: 2013 Forward

NC RttT is designed to be a “game changer”A coordinated set of innovative activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools

Evaluation contributes to NC RttT “changing the game” in two ways:1. Program Evaluation: Provide formative information on the

implementation of NC RttT initiatives Inform decisions to improve implementation

2. Policy Evaluation: Assess – from the perspective of students, teachers, leaders, and schools – the improvements that have occurred as a result of NC RttT initiatives collectively and individually: Includes LEA-level outcomes for some initiatives Inform decisions about sustainability and impacts

2

Ending in 2013


Focus of the Evaluation

• External evaluation provides objective analysis of the activities described in NC’s RttT grant proposal: Implementation fidelity Short-term outcomes Collective/overall impact

• Even with the shift from formative to summative, however, this is still not an evaluation of specific teachers, leaders, or schools

3


NC RttT Initiatives: Evaluation Organization1. Teacher and leader effectiveness

Integration of value-added student achievement measures into educator evaluation system

2. Equitable supply and distribution of teachers and leadersTeach for America, NC Teacher Corps, Regional Leadership Academies, Teacher Induction Program, Virtual Public School, Incentives

3. Professional developmentAll professional development activities in support of RttT initiatives, including: PD for standards and assessment, IIS, and data use; and PD delivery capacity-building efforts

4. Turnaround of LEAs and schoolsLow-achieving LEAs and schools; STEM schools

5. Local-level implementation and spending on RttTCloud computing, allocation of RttT funds, cost savings

6. Overall impact of RttT on students, teachers, and school leadersCross-Initiative: Omnibus survey of teachers and principals

4


Key Findings, 2012-13 (http://cerenc.org)

Value-Added Models – Evaluation of nine commonly-used teacher value-added models suggests that four models (a three-level hierarchical linear model with one year of pretest scores, a three-level hierarchical linear model with two years of pretest scores, EVAAS, and a student fixed effects model) are sufficient for lower-stakes purposes.

However, these models may not perform sufficiently well for high-stakes purposes. They typically do not fully account for the effects of differences in classroom composition (effects on outcomes for individual students that are related to whole-class composition, and potential differences in teacher effectiveness on the same student in different classroom configurations).

5

http://cerenc.org/



Regional Leadership Academies – RLAs currently serve high-need schools in 31 LEAs. Each RLA has a program-specific philosophy, curriculum focused on instruction and school improvement, and well-designed and integrated coursework and field work. Relative to traditional North Carolina MSA Programs, short-term outcomes suggest that RLAs appear to be much more deliberate, effective, and successful in developing and incorporating critical, research-based features into their programs. Most Cohort 1 participants interned in and were later hired by targeted (high-need) schools and LEAs.

Local Strategic Staffing – 18 LEAs developed full strategic staffing plans, and 55 others developed partial plans. Highlights: emergence of 2nd-generation strategic staffing plans; potential for sustainability; diversity in strategic staffing approaches; and opportunities for idea-sharing across LEAs.

6

http://cerenc.org/

Consortium for Educational Research and Evaluation–North Carolina 7


http://cerenc.org/


New Teacher Support Program – Most NC NTSP teachers in the first cohort reported feeling confident in their teaching ability after the Summer Institute, and early evidence from classroom observations suggests that they may be slightly more proficient as instructors, but that they also may have slightly less self-efficacy than comparison teachers.

NCVPS First-Year Report, Pt. I – Courses are reaching students typically under-represented in STEM (females, minorities, etc.), and the Forensics course in particular exhibits multiple strengths. Technology tools, online resources, and project-based learning components are being integrated into courses with increasing frequency. Face-to-face and online teacher relationships are strong and constructive.

8


http://cerenc.org/


Statewide Professional Development – State-provided professional development in 2011-12 focused on four RttT priorities: (1) transition to new state standards; (2) implementation of formative and summative assessments; (3) use of data to support instruction; and (4) effective use of the North Carolina Educator Evaluation System. Data indicate that the overall quality of 2011-12 events was high. The State is clarifying state and local professional development needs, though providing adequate time and resources to meet these needs remains a challenge. Participants generally believed that these events helped them develop knowledge and skills related to these priorities.

LEA-Level Professional Development – Overall, schools and districts appear to have taken initial steps toward crafting processes for providing professional development that supports the four RttT priorities listed above.

9


http://cerenc.org/


Distinguished Leadership in Practice – In Y1, DLP sessions were conducted for 194 principals (157 completed the program). Principal feedback and observation data indicate that face-to-face and online sessions were of high quality. Principal feedback and focus group data suggest that partici-pants developed specific leadership knowledge and skills.

Online Professional Development – Approximately half of the state’s educators completed at least one online module by the end of the 2011-12 school year. Most participants agreed that webinars and modules were relevant to their profess-ional development needs, though some content was redundant and not always content- or grade-specific, and some modules did not meet NSDC standards. Most modules were completed independently and not in PLCs.

10


http://cerenc.org/


NC Teacher Corps & TFA – NCTC recruitment yielded 441 candidates, of whom 34 were selected (29 remain the in program; 22 have teaching jobs). TFA placed 157 corps members in 2011-12, and 219 in 2012-13—many in teaching “pods” of 3 or more. Ca. 90% of TFA corps members complete two-year commitments.

LEA Expenditures – Total RttT funds allocated to LEAs averages to $36 per pupil per year. Based on analyses of DSWs, over the full four years of the grant, LEAs plan to use the largest proportion of their RttT funds (49%) for technology, followed by professional development (21%) and strategic staffing (15%). In terms of planned activities, LEAs plan to prioritize RttT funds for professional development (43%) followed by technology (24%) and strategic staffing (20%).

11


http://cerenc.org/


DST – In low-achieving districts, connections are weak or missing between and within central offices, schools, and classrooms. DST’s strategy at the district level is essentially the process of changing a disconnected district into a productively connected district. Early qualitative and quantitative evidence suggests that DST is making a measurable contribution to the improvement of student performance rates and student graduation rates in the schools it serves.

DST PD – Data suggest that, overall, DST successfully constructed and delivered a high-quality professional development program to participants through its School Leaders Professional Development Series.

12


http://cerenc.org/


STEM Affinity Network: Second-Year Report – Anchor schools have improved instruction and implemented STEM features (such as project-based learning and partnering for improvement of student learning) internally. Structures for networking, PD, curriculum development, and partnerships are in place to support affinity schools, though some of these activities have been delayed. There is not yet universal buy-in among anchor school staff, but many report improved instruction and implementation of STEM strategies. Students in anchor schools enjoy personalized attention and exhibit high motivation, engagement, and a passion for learning.

13


http://cerenc.org/


Omnibus Teacher and Leader Survey

• Administered to probability sample of 365 schools across the state – a selection of schools that is reflective of the state as a whole

• Assesses “Instructional Climate” 23 dimensions on leadership and organizational

conditions affecting instruction

• Baseline and Second Round Surveys completed – Fall 2011, Spring 2012

• Next administration in progress now (launched February 2013) and again in 2014 (last administration)

14


School-Level Activities Reports:https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html

15

https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html

https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html



CERE-NC Reports:http:cerenc.org

17

http://cerenc.org/



NC RttT Evaluation: TeamConsortium for Educational Research and Evaluation-NC (CERE-NC):

SERVE Center, Carolina Institute for Public Policy, and Friday Institute

Steering Committee: Gary Henry, Terri Shelton, & Glenn Kleiman

Principal Investigator: Gary Henry

Management Committee: Julie Marks, Jessica Anderson, and Trip Stallings

Team Leaders: Teacher and Leader Effectiveness – Heather HigginsEquitable Supply and Distribution of Teachers and Leaders – Trip StallingsProfessional Development – Jeni CornTurnaround of LEAs and Schools – Charles Thompson Local-Level Implementation and Spending – Nate BarrettOverall Impact – Gary Henry and Julie Marks

Other Leadership Roles:LEA Coordinator – Lynn Amwake ([email protected]; 336 574-8739);State Liaison - Trip Stallings ([email protected]; 919 513-8576)

19

mailto:[email protected]



Jess Anderson. SERVE Center, [email protected]; (336) 543-6185

Julie Marks, CIPP, [email protected]; (919) 962-7422

Trip Stallings, Friday Institute, [email protected]; (919) 513-8576

20




Documents

The NC Race to the Top Evaluation: An Update CCRESA Board of Directors May 17, 2013