Three ‘R’s for Evaluating the Memphis Striving Readers Project:

Relationships, Real-World Challenges, and RCT Design

Jill Feldman, RBS Director of EvaluationKelly Feighan, RBS Senior Researcher

Memphis Striving Readers

• Five-year US Department of Education grant

• Whole-school and targeted READ 180™ interventions aimed at improving middle school students’ achievement in reading and core content areas

• Research for Better Schools (RBS), evaluator

Targeted Intervention

• Eight urban, high-poverty middle schools (6-8th grades)

• Nineteen READ 180™ teachers, 48 sections

• Eligibility: no prior participation, two or more grade levels behind in reading, bottom quartile on Tennessee Comprehensive Assessment Program (TCAP)

Evaluation Design

• Random assignment of eligible students:– Treatment group participates in READ 180™

– Control group in “traditional” ELA class

• Baseline and follow-up assessments: Iowa Test of Basic Skills (ITBS), Scholastic Reading Inventory (SRI), and TCAP

Relationships with the Research Team

• School District: Office of Research Evaluation and Assessment, principals, teachers, coaches, and parents

Issues: Buy-in, compliance with randomization, and READ 180™ implementation and staff training

• READ 180™ publisher

Real-World Challenges

• High student mobility (attrition)

• Parental consent (IRB)

• Exclusion of ELL and special education students (sample bias)

Real-World Challenges

• Balancing local knowledge of individual students’ needs with identification of “eligible pool”

• Understanding specific school contexts

• Inconsistent availability of supplemental reading instruction

• Delayed startup due to slow delivery of equipment

Maintaining RCT Design

• Two waves of random assignment within first 20 days of school

• Ongoing negotiations with school personnel

• Construction of school profiles

• Monthly data “chats” between district and external evaluators

Preliminary Findings • Students eligible for READ180 ™: N = 2,277

• No differences in race, gender, ethnicity, or poverty level between conditions

• Higher percentage of ELLs in the control group (87 of 1,337 students, or 6.5%) than in READ 180™ group (35 of 940 students, or 3.7%)

• Higher percentage of special education eighth graders in READ 180™ group (28.2%) than in control group (20.9%)

Baseline Findings

• No differences in TCAP scores (N=2,277):– Control (N=1,337): M = 471.9; SD = 29.0– READ 180™ (N=940): M = 473.5; SD = 26.3

• No differences in ITBS scores (N=2,063):– Control (N=1,199): M = 187.9; SD = 16.2– READ 180™ (N=864): M = 189.3; SD = 16.3

Attrition

Combined Treatment and Control Groups Year 1 Year 2 Year 3 Year 4

RTMT (READ 180 and MCLA) 480 496 507 515

RTMC (READ 180 Only) 480 496 507 515

RCMT (MCLA Only) 720 744 761 773

RCMC (Control Only) 720 744 761 773

Estimated Number of Students

Planned Analyses

Analyses: Impact of READ 180™ participation on student reading scale ITBS score

Control Variables: school average TCAP and ITBS performance, enrollment, students’ previous ITBS score, gender, disability status, and poverty status

Treatment effects: difference between average student scale ITBS score for those in READ 180™ and those in control group

Next Steps…

• ITBS assessment: May 2007

• TCAP assessment: April 2007

• Analysis of SRI scores for READ 180™ students: Summer 2007