Progress Towards Reading Success: The Reading First Evaluation Prepared by: Amy Kemp, Ph.D. Research Associate and Patricia A. Muller, Ph.D. Associate

Progress Towards Reading Success: The Reading First Evaluation

Prepared by:

Amy Kemp, Ph.D.

Research Associate

and

Patricia A. Muller, Ph.D.

Associate Director

Today’s Presentation


(1) Provide overview of the three year Reading First (RF) Evaluation in Indiana

(2) Present key findings from Year 1 evaluation

(3) Provide details of Year 2 evaluation

http://www.indiana.edu/

Overview of Three Year Evaluation


Purpose: To provide information to policy makers at IDOE and other key stakeholders to improve the implementation and impact of RF

• Formative: Implementation

• Summative: Impact


Formative: Implementation


Six primary questions• To what extent do RF schools implement the five

essential reading components?

• To what extent do RF classrooms implement specific instructional strategies such as the 90 minute block, progress monitoring, etc?

• What types of reading instruction strategies are consistently applied in all K-3 classrooms?



• What types of PD activities do the district and the school support?

• How well does the district and school supported PD provide for consistent, high-quality classroom instruction?

• What types of technical assistance are provided to the schools by the district? How effective is it?



Also…To what extent are the five goals of RF are being effectively and efficiently met?

• Ensuring K-3 teachers learn SBRR instruction and skills

• Assisting districts and schools in identifying SBRR materials, strategies, etc.

• Ensuring all programs and activities in K-3 classrooms meet criteria for SBRR

• Assisting districts and schools in selection and administration of screening, diagnostic and assessment tools

• Integrating initiatives and leveraging resources to avoid duplication of programs and services


Summative: Impact


Primary Question/Focus:

What is the impact of RF on

student academic outcomes?


Summative: Impact


Impact of RF on student academic outcomes

• Measures of student achievement (e.g. DIBELS)• State standards in Reading• Retention rates, special education referrals, etc.


Summative: Impact


Impact of RF on student academic outcomes

• All K-3 students• Students disaggregated based on ethnicity, ELL

and special education


Shifting Foci Across Years


Year 1: Assessing the level and success of implementation in Reading First schools

Year 2: Assessing the degree to which RF is effectively implemented; and beginning to examine preliminary indicators of impact

Year 3: Assessing the impact of RF on student outcomes


Year 1


Purpose was to assess the level and success of implementation in Reading First schools

Data Collection Methods• Surveys• Site visits of 13 RF schools• Interviews• Review of extant data


Year 1 Findings


At the end of the first year, the majority of schools hadsuccessfully begun to implement their RF plans

• Schools were using a 90-minute uninterrupted block • Staff were increasingly administering assessments• Teachers were using the five components of reading in

their instruction• Reading instruction was increasingly individualized to

meet students’ needs


Year 1 Findings


Factors critical to success included:

• Strong coach with content knowledge (SBRR)• Strong coach with interpersonal skills (reduce resistance)• Collaboration and communication among staff• Climate conducive to change (support, trust, etc.)• High-quality professional development (inc. confidence)


Year 1 Findings


Challenges to implementation• Complexity and time-intensiveness of grant requirements

(trouble attending to all details)

• Although the majority of teachers were receptive to RF, coaches spent large amounts of time with some teachers who were having difficulty accepting the changes


Year 2 Evaluation


Purpose: Assessing the degree to which RF is

effectively implemented; and beginning to examine

preliminary indicators of impact


Year 2 Evaluation


Some Key Questions

• Coaches: To what extent are coaches being effectively used in schools? How are coaches spending their time? What is the role of the coach in the classroom?

• Core teams: To what extent are core teams being effectively used in schools? To what extent can the successful implementation of RF be attributed to core teams?


Year 2 Evaluation


• Changes in the classroom: To what extent are classroom instructional strategies guided by assessment results? To what extent are SBRR strategies being used in the classroom? How are the 90 minutes being used (e.g. small groups)? Flexible grouping?

• Interventions: To what extent are diagnostics and

appropriate interventions being used in the classroom? What types of interventions are being used?

Year 2 Evaluation


• Professional Development: Are professional development choices being guided by classroom practice needs?

• Non-RF schools: To what extent has RF had an impact on non-participating schools in RF districts? To what extent are teachers at non-RF schools participating in PD, changing practices, etc.?

Year 2 Evaluation


Impact• Teacher level (e.g. changes in teacher knowledge/skills

and classroom practices)

• Student Level (e.g. Are all K-3 students meeting or progressing towards their Aimlines?)

What factors are associated with the effective implementationand impact of RF programs?

Year 2 Data Collection


• RF Consultant Interviews• District Representative Interviews (All)• Site visits for 13 RF programs• Surveys of key stakeholders (all RF teachers,

coaches, principals)• Extant data (assessment data)




District Representative Interviews • All district representatives• Semi-structured phone interview• Conducted late January/early February• Focus on questions of program implementation

and district-wide impact of the RF grant



Site visits for 13 RF programs• Sites selected and notified by early February• Two day site visits in late March/early April• Observations of classrooms and relevant

meetings; interviews with principals, coaches, teachers and other school staff; extant data and documents



Surveys of key stakeholders• Includes all RF schools• Teachers, coaches and principals• Late April/Early May• Primarily on-line surveys (with option for pencil

and paper version)



Extant Data

DIBELS and Assessment data (including progress

monitoring, and diagnostic) from Wireless

Generation



Prepared by:

Amy Kemp, Ph.D.

Research Associate

and

Patricia A. Muller, Ph.D.

Associate Director

Documents

Progress Towards Reading Success: The Reading First Evaluation Prepared by: Amy Kemp, Ph.D. Research Associate and Patricia A. Muller, Ph.D. Associate