Upload
bruce-carter
View
221
Download
0
Embed Size (px)
DESCRIPTION
Pressures 181 First Days of School Growing Poverty Rate Manage Expectations in City – Article for upcoming ‘Chamber Views’ – Ongoing speaking and publications on challenges Focusing work of 1000 people Rising AYP Standards and City confidence in schools: can we ‘get over the bar’?
Citation preview
Fast Track Program Evaluation Using Assessments Diagnostically
Philip A. Streifer, Ph.D.Bristol Superintendent of Schools; UCONN Executive Leadership
ProgramYvel Crevecoeur
Doctoral Student, Department of Educational Psychology, program in Special Education
Primary Issues
• Bristol an urban leader in reform• Next phase of development: program
evaluation and focus on instructional excellence• Primary reading intervention program:
Fundamentals of Literacy Development (FOLD)• Rising CT AYP Standard for 2008: Need to track
student gains over time vs. NCLB/AYP year by year analysis
Pressures
• 181 First Days of School• Growing Poverty Rate• Manage Expectations in City
– Article for upcoming ‘Chamber Views’– Ongoing speaking and publications on challenges
• Focusing work of 1000 people• Rising AYP Standards and City confidence in
schools: can we ‘get over the bar’?
Challenges
• Migration• Economically Disadvantaged• Rising State NCLB Standards for 2008
Reading – 68 to 79% proficiency Math – 74 to 82% proficiency
• Special Education Placements and Costs
181 First Days of School One School’s Experience
Transfers In• July-Aug- Sep 53• October 16• November 11• December 4• January 24• February 12• March 11• April 1• May 0• June 0• TOTAL 132
Transfers Out• July-Aug- Sep 61• October 8• November 14• December 3• January 8• February 12• March 10• April 2• May 3• June 1• TOTAL 122
GRAND TOTAL for ONE YEAR: 254 Children in and out/School Population +/-350
Data Driven Reform and Improvement
Past, Present and Future
8
POWER STANDARDS
UNWRAPPED
STANDARDS
BIG IDEASESSENTIAL QUESTIONS
CURRICULUM &
PACING GUIDESEFFECTIVE
TEACHING STRATEGIE
S
GRADE/DEPT DATA
TEAM
ASSESSMENT OF
STUDENT LEARNING
SCHOOL DATA TEAM
DISTRICT DATA TEAM
IMPROVEDSTUDENT
ACHIEVEMENT
Data-Driven
Decision
Making
MAKING STANDARDS WORK
The Bristol Accountability Initiative
CT Mastery Test and CAPT
• CMT – new version for 2005-06• CAPT – new version for 2006-07• Both are tougher tests• Each year NCLB/AYP standard rises• NCLB/AYP standard for 2007-08 (spring 2008)
is rising dramatically
Matched Cohort Comparison on the Connecticut Mastery Test
Percent of Students Scoring At or Above ProficiencyGrade/Cohort Year Math Reading Writing
3 2006 82.7 68.5 85.3
4 2007 85.6 74.5 87.9
Change + 2.9 + 6.0 + 2.6
4 2006 86.0 71.1 83.25 2007 83.8 74.8 89.2 Change - 2.2 + 3.7 + 6.05 2006 80.8 74.2 89.9
6 2007 84.0 75.5 80.9
Change + 3.2 + 1.3 - 9.0
6 2006 81.7 74.5 79.37 2007 83.5 80.2 80.3 Change + 1.8 + 5.7 + 1.07 2006 79.6 80.0 78.0
8 2007 82.1 79.7 82.0
Change + 2.5 - 0.3 + 4.0
Percent Increase
Matched Cohort Data (2006 to 2007) Writing*
Total MatchedStudents Tested
Grade /Cohort
Percent Moved FromBELOW Proficient toProficient and Above
Number ofStudentsMoved toProficient
and Above
590 Gr 3 - 4 48% 41 out of 85
634 Gr 4 - 5 57% 61 out of 107
619 Gr 6 -7 34% 43 out of 128
679 Gr 7-8 38% 56 out of 146
*Grade 5 to 6 matched cohort did NOT make gains in Writing.
Matched Cohort Comparison on the Connecticut Mastery TestPercent of Students Scoring At or Above Goal
Grade/Cohort Year Math Reading Writing3 2006 62.7 52.6 62.8
4 2007 66.4 58.4 68.9
Change + 3.7 + 5.8 + 6.1
4 2006 68.6 58.6 61.75 2007 68.7 61.6 69.7 Change +0.1 + 3.0 + 8.05 2006 58.0 58.9 68.1
6 2007 63.7 63.1 59.3
Change + 5.7 + 4.2 - 8.8
6 2006 60.2 60.5 55.37 2007 60.9 69.7 56.3 Change + 0.7 + 9.2 + 1.07 2006 56.3 67.3 51.7
8 2007 59.9 71.3 62.3
Change + 3.6 + 4.0 + 10.6
Percent Increase
District CAPT 2007 ResultsDistrict CAPT 2007 Results
Mathematics ScienceReading Across The
DisciplinesWriting Across the
Disciplines
% At/Above
Goal% At/Above
Proficient% At/Above
Goal% At/Above
Proficient% At/Above
Goal% At/Above
Proficient% At/Above
Goal% At/Above
Proficient
Bristol 51.6 88.1 40.8 88.1 50.2 87.2 50.2 86.8
Male 54.3 87.6 44.9 86.8 45.6 81.3 40.1 81.7
Female 48.5 88.6 36.2 89.6 55.5 94.0 61.6 92.6
Black 28.3 71.7 20.8 77.4 46.2 71.2 36.5 71.2
Hispanic 25.4 69.8 10.8 70.8 26.6 79.7 27.0 73.0
White 57.1 91.7 47.1 91.3 53.4 89.6 54.0 89.7
F/R Meals 27.2 67.6 19.0 73.7 26.3 74.5 23.9 69.4
Full Price 58.2 93.6 46.8 92.0 56.7 90.7 57.2 91.4
Special Ed. 14.3 50.8 11.1 49.2 11.3 38.7 11.5 44.3
Not Special Ed. 55.7 92.2 44.1 92.4 54.4 92.4 54.3 91.3
Not ELL 51.9 88.7 41.2 88.4 50.7 87.5 50.5 87.3
*ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup.
Areas highlighted in yellow show where Bristol exceeded State averages.
Performance Level Graph – District vs. State AveragesPerformance Level Graph – District vs. State Averages
Gains Analysis: Matched Cohorts
• Grade 3 to 4 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Expected Gain
• Grade 4 to 5 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Loss Below Expected– Writing: Expected Gain
Gains Analysis: Matched Cohorts
• Grade 5 to 6 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Expected Gain
• Grade 6 to 7 (2006 to 2007)– Math: Expected Gain– Reading: Expected Gain– Writing: Expected Gain
Gains Analysis: Matched Cohorts
• Grade 7 to 8 (2006 to 2007)– Math: Expected Gain– Reading: Moderate Gain Beyond Expected– Writing: Moderate Bain Beyond Expected
• Grade 8 to 10 (2004 to 2007)– Math: Moderate Gain Beyond Expected– Reading: Expected Gain– Writing: Expected Gain
Role of Program Evaluation
• FOLD: – working; needs to be expanded
• Read-180 vs. ReadAbout:– Gains Analysis
• ReadAbout Deployed in More Classes 2007-08 with Grant Funds
Return on Investment
• Bristol Per Pupil Expenditure is 125th out of 169 (upper end of lower third)
• CMT 2007: at or just above state averages• CAPT 2007: above state average• City of Bristol getting a good return on its
educational investment
New Pressing Issue• CT AYP standard rises dramatically in 2008 in reading and math• Projection of 2008 AYP indicates many schools will be cited under NCLB• Current district performance at state average on CMT; above state
average on CAPT– Poor AYP performance in 2008 will likely cause erosion of public
confidence and City support• Deeper analysis ongoing to determine:
– Areas of focus for each school– Safe Harbor status and why it is important– Gains analysis to show progress of students over time who remain in
the district; to shift focus and opinion and show staff that they are doing a good job
Key Strategies Going Forward
• AYP and Safe Harbor – achieving improvement for all children
• Program Evaluation – FOLD (first pilot)• Gains Analysis – Cohort Improvement Over
Time and Across Disparate Tests • Instructional Excellence
Preliminary 2008 AYP Analysis
23
FOLD Evaluation• Why were Hubbell Elementary School’s reading
scores so high?• Development of the Foundations of Literacy
Development (FOLD) evaluation process • Identifying Growth Model of Improvement for each
elementary school• “SWAT Team” and focused plans on other
intervnetions• Long term objective: Maximizing what really works
for students participating in FOLD.
24
FOLD Evaluation Process
• Semi-structured interview of principal who originally implemented FOLD:– Key design features– Developed pilot survey questions
• Reviewed and refined survey questions• Administered survey to literacy teachers and
principals
25
FOLD Evaluation Process• Preliminary method of analysis:
– Identified each school’s performance on indicators of DRA2, CMT, and history of achievement
• Ranked each school’s performance to identify:– Differences across high and low performing schools– Differences between respondents (i.e., principals and
literacy teachers) to identify features of implementation that need revision
Gains Analysis Logic
• AERA paper presentation 2007• Data must be matched pairs and equal interval
scales• Rescale data if using different tests• Must know the absolute possible range of each
dataset• Apply Modified Effect Size analysis• Interpret with reconceptualization of Cohen’s d
Gains Analysis – Different Scales• Rescale
– Reset to zero scale
– Ratio of one scale to the other
• Analysis– Cohen’s
denominator for pooled SD for populations (for unequal variances
range
range
rescalefinal2
122 T
TT T
2T T X -X
21
T1 T2sg
ES
Interpretation:
Less Overlap = More Difference in Scores +/-
We would expect little overlap and a positive Cohen’s d
Transformation Logic – Step 1
• Two Scales:– 0 to 10: range = 10– 20 to 40: range = 20
• Step 1: Put scales on same starting point:– 0 to 10: range = 10– 0 to 20: range = 20
• [Scores are reset to start at a value of zero. That is, tests with possible scoring ranges of A→B are rescaled such that scores range from 0→(B-A).]
| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
R1
R2
0 10 20 30 40 50
| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
R1
R2
0 10 20 30 40 50
Ratio of R1 to R2 = .5
| | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | 10| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
R1
R2
0 10 20 30 40 50
Step 2: Multiply R2 scores by ratio of the scales (.5) to reset to R1 scale
| | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | 5 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
R1
R2
0 10 20 30 40 50
Rescale
• Range1/Range2 = Resets one score to the other• Example:
– R1/R2 = 5/10 = 0.5 (or…R2/R1 = 10/5 = 2.0)
Reconceptualizing Cohen’s d in this Context
Gains Analysis Findings…So Far
Read Mean SD Read Mean SD Math Mean SD Math Mean SD
3 to 4 3 4 3 4 4 to 5 4 5 4 5 3 to 4 3 4 3 4 4 to 5 4 5 4 5
Hi Poverty, Low Performing 0.68 215 237 30 34 -0.24 227 219 40 34 0.26 235 246 42 44 -0.11 238 232 54 55
Hi Poverty, High Performing 0.61 244 263 32 32 -0.21 261 252 43 43 0.42 259 276 43 39 -0.13 284 277 46 47
Low Poverty 0.43 238 258 43 50 -0.21 258 248 52 48 0.13 260 267 52 49 -0.15 277 268 55 63
Low Poverty 0.43 229 245 37 41 -0.30 256 243 43 42 -0.10 261 256 53 39 0.20 265 274 45 47
All Schools Improving the Same Over Time: Poorer Performing Schools Not Catching Up – Need Faster Growth – May be Unreasonable to Achieve
Using Data Mining to go Deeper - Variables
• School• Grade 3 Teacher• Grade 4 Teacher• Grade 5 Teacher• Gender• Ethnicity• SPED status• F-R Eligible• ELL status
• GR5 CMT Math Scale• GR5 CMT Read Scale• GR5 CMT Writing Scale• GR4 CMT Math Scale• GR4 CMT Read Scale• GR4 CMT Write Scale• GR3 DRA Reading Level
Results So Far:• Poverty not a significant factor in achievement; all
schools performing: one year gain (+/-) for one year instruction
• Teacher is associated with achievement – but not clear as to how– Strong teachers are assigned in many cases to lower
performing students which probably accounts for gains analysis
• Need to focus interventions on specific groups of students; identified through data mining and ‘teacher/administrator knowledge’
Summary• Past work by district on DDDM and Cultural
acceptance critical• Next Steps:
– Program evaluation and advanced data analysis– Focus on instructional excellence
• Marzano: Effective Teaching Strategies• Saphier & Gower: The Skillful Teacher
– Maintain City confidence – get to ‘safe harbor’ for 2008
• Change the Conversation from NCLB/AYP to Growth Over Time