8/7/2019 Parthenon rpt March 6 BTO Update (1)
1/20
New York CityDepartment of Education
Beat-the-Odds HS Update
March 6, 2008
DRAFT WORK PRODUCTFor Discussion Only
8/7/2019 Parthenon rpt March 6 BTO Update (1)
2/20
2
Beat-the-Odds HS Analysis Update
Objectives for Today
Review structure and initial findings from updated regression model
- What does this model allow us to do, and what questions do we want to ask?
Discuss data on specific follow-up questions from last data review
Gather group input on key areas of potential implications: which findings shouldbe highlighted, and which applications of the data are most powerful?
Discuss key areas of analytical next steps:
- Variables to test which might be common across Beat-the-Odds HSs
- Secondary portfolio strategies to model and assess projected impact
8/7/2019 Parthenon rpt March 6 BTO Update (1)
3/20
3
How Does the New Regression Model Work?
Student-Level Factors School-Level FactorsTrue Over- and Under-
Performance by Schools
For each individual student,what are the key demographicand performance variables
that shape the odds ofgraduation?
Given each studentsparticular profile, what is eachindividuals expectedgraduation rate?
What are the aspects ofschool structure that have anadditional peer effect on the
chances of graduation?
By how much is eachindividual students odds ofgraduation changed byattending schools withdifferent structures andpopulations?
Once we have leveled theplaying field in terms of bothstudent-level and school-level
factors, how much variationremains in the graduation rateacross schools?
Which schools are thegreatest over- and under-performers, and are therefactors that appear to enablesuch performance?
8/7/2019 Parthenon rpt March 6 BTO Update (1)
4/20
4
What Are the Variables that Drive the New Model?
Student-Level Factors School-Level FactorsTrue Over- and Under-
Performance by Schools
Gender
Ethnicity (White/Asian vs.African-American/Hispanic)
Average 8th ELA + Math
Proficiency Level
Age at Entry to High School
8th grade attendance rate
ELL Status (Y/N)
SPED Status (LRE vs. Self-Contained vs. Non-SPED)
Lack of 8th grade attendanceor 8th grade test scores
School enrollment
A series of concentrationvariables:
- Percent of students with a
Low-Level 2 or below oneither ELA or Math
- Percent of studentsentering 9th grade overage
- Percent of students with 8thgrade attendance below90%
- Tested but not significant:
Percent ELL, Percent SPED(total and Self-Containedonly), Title I eligibility, Avg.teacher tenure, Principaltenure, General Ed class size,Special Ed class size, facilitiescapacity utilization
The model explains 71% ofthe variation in high schoolgraduation rates
However, 22% of schools
differ from their predictedgraduation rate by +/- 10 ormore percentage points
8/7/2019 Parthenon rpt March 6 BTO Update (1)
5/20
5
Individual Student Odds of Graduation Are Predicted Based on a
Combination of Factors
88%84%
76%
37%
27%
14% 14% 12%
0%
20%
40%
60%
80%
100%
PredictedGraduationRate
Predicted Graduation Rate byStudent-Level Factors, 2007 Cohort
Note: Controlling for all other factors, students who are ELL during high school are 9% more likely to graduate than non-ELL studentsSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
SPED Status:
8th Gr. Att.:
Age at Entry:
Test Level:
Gender:
Ethnicity:
GE
95%
On
3.0
Female
Wh./As.
GE
95%
On
3.0
Female
Af-Am/His.
GE
95%
On
3.0
Male
Af-Am/His.
GE
95%
On
2.0
Male
Af-Am/His.
GE
95%
Over
2.0
Male
Af-Am/His.
GE
85%
Over
2.0
Male
Af-Am/His.
LRE
85%
Over
2.0
Male
Af-Am/His.
SC
85%
Over
2.0
Male
Af-Am/His.
Number of Factors Related to High Graduation RateMany Few
Note: Predicted graduation rates below are calculated independent of the schools that students attend
8/7/2019 Parthenon rpt March 6 BTO Update (1)
6/20
6
The Model Can Isolate the Effect of Demographic Variables on
Student Performance
1 2 3 4
0%
20%
40%
60%
80%
100%
Pre
dictedGraduationRate
Predicted Graduation Rate by Ethnicity and Gender, 2007 Cohort
Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
White/Asian Females
White/Asian Males
African-American/Hispanic Females
African-American/Hispanic Males
Average of 8th Grade Math & ELA Proficiency Rating
At the widest point (low-mid Level 2), white and Asian females outperform African-American and Latino males by morethan 20% points controlling for all other factors
Note: Predicted graduation rates below are calculated independent of the schools that students attend
8/7/2019 Parthenon rpt March 6 BTO Update (1)
7/207
SC
LR
E
C
TT
L1 LL2 HL2 L3
29%27%
37%
31%36%
41%
29%
41%
48%
32%
53%49%
0%
20%
40%
60%
80%
100%
Pe
rcentEarning10orMore
Credits
CTT
LRE
SC
0.9K
1.1K
2.5K
0.6K
1.7K
0.7K
0.2K
0.9K
0.2K
0.1K
0.3K
0.0K
SPED Students in CTT Environments Appear to Outperform
Peers in Both LRE and Self-Contained
Credit Performance of Special Education Populations byIncoming 8th Grade ELA Proficiency Level, First-Time Freshmen, 2006-07
Note: Districts 75 and 79 excluded; Excludes students with missing 8th Grade ELA proficiency dataSource: DOE Internal Data
Number of Students
31%
Non-Special Ed.
However, at higher levels of incoming proficiency, CTT students still under-perform general ed students
44%
57%
76%
8/7/2019 Parthenon rpt March 6 BTO Update (1)
8/208
School Structure Can Also Drive Wide Changes in Expected
Performance
Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
The Median Student
Female
African-American/Hispanic
Median 8th Grade Test Scores (2.7)
Med. 8th Grade Attendance (92%)
On-Age at Entry to HS (14 yo)
Neither ELL nor SPED
70% Predicted Graduation Rate
School 4(e.g. John F Kennedy HS)
3,000 students
Concentrations ofchallenged students15% points abovesystem average
55% Graduation Rate
School 3(e.g. William Grady CTE HS)
1,500 students
Concentrations ofchallenged students5% points abovesystem average
70% Graduation Rate
School 2
(e.g. HS for Leadership andPublic Service)
800 students
Concentrations ofchallenged students5% points belowsystem average
78% Graduation Rate
School 1
(e.g. Science, Tech andResearch Early College HS)
400 students
Concentrations ofchallenged students15% points belowsystem average
83% Graduation Rate
(Effect of 400-StudentEnrollment Alone 78%)
8/7/2019 Parthenon rpt March 6 BTO Update (1)
9/209
The Model Allows Us to Compare the Impact of Different
Strategies on Expected Student Performance
Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
8th Grade Test Score Remains at 2.0
Attend a 400 Student School With AverageConcentration of Low Achievement, Overage &Low-Attending Students
42% Graduation Rate
Improve 8th Grade Test Score to 2.5
Attend a 3,000 Student School with LowAchievement, Overage & Low-Attending StudentConcentrations 10 p.p. Above System Average
40% Graduation Rate
African-American/Hispanic Male
8th Grade Test Score of 2.0
31% Predicted Graduation Rate(independent of school attended)
Improve Preparation in MS HS Portfolio Development
8/7/2019 Parthenon rpt March 6 BTO Update (1)
10/2010
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
ActualGradRate
Predicted Grad Rate
Traditional High Schools& Career/Technical
New Small High Schools
R = 0.71
# of Obs. = 230
The New Model Highlights a Greater Number of Schools with
Low Predicted Grad Rates
Relationship Between Predicted GraduationRate and Actual Graduation Rate, 2007 Cohort
Source: ATS Data; Progress Reports Data; NYC DOE Website
Schools with theToughest Odds
Less than 40% predictedgrad rate: 5 schools (vs.0 in original model)
40-45% predicted gradrate: 12 schools (vs. 3 inoriginal model)
45-50% predicted gradrate: 24 schools (vs. 14 inoriginal model)
8/7/2019 Parthenon rpt March 6 BTO Update (1)
11/2011
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
ActualGradRate
Predicted Grad Rate
Traditional High Schools& Career/Technical
New Small High Schools
Over-Performance(15% of schools)
Underperformance(7% of schools)
R = 0.71
# of Obs. = 230
22% of Schools Diverge from Their Predicted Graduation
Rate by +/- 10 or More Percentage Points 78% of New Small High Schools have an actual graduation rate that exceeds their predicted graduation rate; 37% of
New Small High Schools exceed this prediction by 10 percentage points or greater
Relationship Between Predicted GraduationRate and Actual Graduation Rate, 2007 Cohort
Note: 10% points represents approximately one standard error in the final regression modelSource: ATS Data; Progress Reports Data; NYC DOE Website
8/7/2019 Parthenon rpt March 6 BTO Update (1)
12/2012
2
4%
23
%
22%
20%18%
16%16%
16%
14%
14%
14%
8%8%7%6%6%6%5%5%4%4%
1%1%1%0%
-0%
-2%-2%-2%-2%-3%-3%-3%-3%
-4%-4%-4%-5%-5%-6%-6%
-7%
-8%-8%-8%-8%-8%-8%-9%-9%-9%-9%
-10%
-10%
-10%
-10%
-12%
-12%
-15%
-19
%
-19
%
-23%
-20%
-10%
0%
10%
20%
30%
40%
31%
29%
24%
24%
23%
20%
19%
18%
18%
16%
16%
15%
14%
12%
12%
12%
12%
10%
10%
9%9%
8%8%
8%
8%6%
6%6%
4%
4%3%
3%2%
1%
0%0%
-0%
-
1%
-1%
-2
%
-2
%
-2
%
-5%
-6%
-15%
-15%
-20%
-10%
0%
10%
20%
30%
40%
Average = -0.3%
Average = 8.1%
Source: DOE Internal Data; Parthenon Regression Model
Residual
Residual
Distribution of 2007 Over/Under-performance for New Small Schools (n=46)
Performance of New Small Schools Differs from Small Schools
Opened During the 1990s
Distribution of 2007 Over/Under-performance for Old Small Schools (n=62)
8/7/2019 Parthenon rpt March 6 BTO Update (1)
13/2013
New Schools Outperform Most with Challenged Students, but
Much of the Gap Is in Local Diplomas
LocalDiploma
Regents
or Above
NewSmallHSs
OtherSchools
NewSmallHSs
OtherSchools
NewSmallHSs
OtherSchools
NewSmallHSs
OtherSchools
NewSmallHSs
OtherSchools
37%
19%
66%
40%
77%
60%
89%
80%
86%*
91%
0%
20%
40%
60%
80%
100%
4-YearGraduationRate
Overall Gap 18% 26% 17% 9% N/A
Regents+ Gap 4% 7% 6% 3% N/A
Class of 2007 Four Year Graduation Rate by 8th Grade ELA Level:New Small Schools vs. All Other Schools, Local vs. Regents Diplomas
* Limited sample (n=49) for Level 4 students in New Small SchoolsNote: Only includes schools that meet requirements for regression analysis (i.e., 20-student minimum cohort size, progress report grade). ExcludesOther diploma types from non-New Small Schools, including Proof Of Receipt Of HS Diploma, Already Had Diploma, Tried For Regents But Left,Career Ed Regents Diploma, Confirmed Completion Of HS Requirements, Advanced Diploma CTE, and Local Diploma CTESource: DOE Internal Data
Level 1 Low Level 2 High Level 2 Level 3 Level 4
8/7/2019 Parthenon rpt March 6 BTO Update (1)
14/2014
Over-Performance
Under-Performance
Under-Performance
Over-Performance
Underperformed Then, Over-
Performs Now
(8% of schools)
Over-Performed Then, Over-
Performs Now
(35% of schools)
Underperformed Then,
Underperforms Now
(36% of schools)
Over-Performed Then,
Underperforms Now
(21% of schools)
2005 Cohort Regression
20
07CohortRegressio
n
Regression Is Run Over Time to Assess Performance Trajectory
Note: Percentages are taken of all schools that met requirements for regression analysis for both2005 and 2007 cohorts; percentages may not total to 100% due to roundingSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
Of the schools who wereunderperforming in the Classof 2005, 82% were stillunderperforming in the Class
of 2007
Of the schools who wereover-performing in the Classof 2005, 63% were still over-performing in the Class of2007
Of schools with graduatingclasses in both 2005 and2007, 57% are under-
performing in 2007 drivenby the disproportionate rateof over-performance amongnew small schools
School Over- vs. Underperformance in New Regression Model, Class of 2005 vs. 2007
8/7/2019 Parthenon rpt March 6 BTO Update (1)
15/2015
New Small Schools Appear Mixed in Sustaining Their Performance
Going Forward
Increasedby 3+
PercentagePoints
LittleChange
Decreasedby 3+
PercentagePoints
Schools Opening in 2001
n=17
0%
20%
40%
60%
80%
100%
Increasedby 3+
Percentage
Points
Little
Change
Decreasedby 3+
Percentage
Points
Schools Opening in 2003
n=27
0%
20%
40%
60%
80%
100%
Biggest Improvement
1. Belmont Preparatory HS (+13%)
2. Fordham HS For The Arts(+11%)
3. Jonathan Levin HS For MediaAnd Communications (+10%)
4. Fordham Leadership Academy
For Business And Technology(+4%)
5. Bronx HS For Medical Science(+3%)
Biggest Decline
1. School For Excellence (-26%)
2. Frederick Douglass Academy IISecondary School (-22%)
3. Community School For SocialJustice (-19%)
4. Bronx Leadership Academy II HS(-16%)
Biggest Improvement
1. HS For Violin And Dance (+20%)
2. Brooklyn Academy Of ScienceAnd The Environment (+12%)
3. Manhattan Bridges HS (+11%)
4. Brooklyn HS For Music AndTheater (+8%)
5. New York Harbor School (+5%)
Biggest Decline
1. Pelham Preparatory Academy(-28%)
2. International Arts Business HS(-23%)
3. Discovery HS (-17%)4. Bushwick HS For Social Justice
(-15%)
5. HS For Contemporary Arts(-12%)
PercentofNewSmallSch
ools
PercentofNewSmallSch
ools
Change in Percent of Students Earning 20+ Credits After Two Years, 2007 vs. 2009 Cohort
Source: DOE Internal Data
Overall, the percent of students earning 20+ credits by the end of the 2 nd year went from 64% to 62% in the new small schools (vs.49% to 54% in the rest of the portfolio)
More analysis would be required to adjust these figuresfor changes in school population
8/7/2019 Parthenon rpt March 6 BTO Update (1)
16/2016
Three High-Performing Categories of Schools Present KM
Opportunities
Proposed Key Categories of High Schools
Turnaround SchoolsEmerging Beat-the-Odds
Schools withPockets of Success
Turnaround schools shoulddemonstrate dramaticimprovement over the period
from 2005-2007:
- Underperforming in 2005and over-performing in2007
- Improved by 10%+ pointsin both relative andabsolute grad rates
- Now have a graduationrate at or above the
citywide average
Proven Beat the OddsSchools demonstrate asustained ability to
outperform their predictedgraduation rate:
- Exceed predictedgraduation rate by atleast 5 percentage pointsin both Class of 2005 and2007
- Also exceed citywideaverage graduation rate
Schools that do not meet thecriteria of Beat the Oddsschools overall, but which
perform as well as the bestProven Beat the Oddsschools with key sub-groups
Groups tested thus farinclude:
- Low achieving 8th graders
- 9th graders who areoverage at entry to HS
- ELL students- Self-Contained SPED
students
- African-American andLatino males
Proven Beat-the-Odds
Beat-the-Odds Schools
Emerging Beat the OddsSchools outperform in themost recent cohort (2007) by
the same margin as ProvenBeat the Odds schools (atleast 5% points)
- Also exceed citywideaverage graduation rate
These schools also show signsof sustaining theirperformance going forward:
- Percent of 2nd year
students with 20+ creditshas improved or remainedsteady within 5percentage points
25 schools 21 schools 6 schools
Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model
17 schools(across 5 categories)
8/7/2019 Parthenon rpt March 6 BTO Update (1)
17/2017
Matching Schools with a Counterpoint Provides a Unique
Opportunity for Improvement
Proposed Key Categories of High Schools
Turnaround SchoolsEmerging Beat-the-Odds
Schools withPockets of SuccessProven Beat-the-Odds
Beat-the-Odds Schools
Struggling New SchoolsSchools with ChronicUnderperformance
Schools in DeclineSchools with SpecificAreas of Weakness
Schools in declinedemonstrate a dramaticdeterioration in performance:
- Went from over- to
underperforming theirpredicted rate from 2005to 2007
- Decline in performancerelative to expectations of10%+ percentage points
Schools that underperformtheir predicted graduationrate by 5 or more percentagepoints in both the Class of
2007 and the Class of 2005
Schools that are over-performing their predictedgraduation rate by 5 or morepercentage points, but lag
the citywide average in a keystudent sub-group
New small schools that:
- Underperform theirpredicted graduation rateby 5 or more percentage
points or
- Show substantial declinesin credit accumulationbetween the Class of 2007and Class of 2009
12 schools(excluding Phaseouts) 15 Schools 12 schoolsTBD (In Process)
Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model
8/7/2019 Parthenon rpt March 6 BTO Update (1)
18/2018
Beat-the-Odds High Schools Mostly Score Highly on Progress
Reports and Quality Reviews
School Name PR QR
Over-Performed by 10% or Greater On Both
Proven Beat-the-Odds Schools
Note: In order to be considered for beating the odds, a school must exceed the citywide average graduation rateSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website
Grad Rate
Over-Performed by 10% for Class of 2007
Emerging Beat-the-Odds Schools
School Name PR QR Grad Rate
School Name PR QR Grad Rate
School Name PR QR Grad Rate
Over-Performed by 5% or Greater On Both
Over-Performed by 5% for Class of 2007
19 out of 25 schools with either an A or W 12 out of 21 schools with either an A or W
HS For Enterprise, Business, And Tech 83% A W
HS For Media And Communications 65% A W
Wadleigh SS For The Performing & Vis 70% A P
Harry S Truman HS 66% B P
HS For Law And Public Service 74% A W
East Side Community HS 67% A W
Fannie Lou Hamer Freedom HS 59% B P
Park East HS 75% B P
Riverdale/Kingsbridge Academy 84% A PBrooklyn International HS 79% A W
University Heights SS At Bronx CC 82% A W
Manhattan Village Academy 83% A W
Bronx International HS 59% A P
Repertory Company HS For Theatre Arts 85% B P
Susan E. Wagner HS 79% B W
El Puente Academy For Peace And Justice 71% B W
School Of The Future HS 96% B W
Automotive HS 57% F P
Forest Hills HS 76% B P
Curtis HS 70% B W
HS For Health Careers And Sciences 58% A W
Brooklyn College Academy 93% B W
Michael J. Petrides HS 96% A W
Landmark HS 61% A P
The HS Of Fashion Industries 78% A W
Millennium Art Academy 86% A P
Marble Hill HS For International Studies 96% A W
Morris Academy For Collaborative Studies 79% B P
Bronx Aerospace HS 88% A P
Manhattan Bridges HS 68% A W
HS For Violin And Dance 85% A P
Bronx Academy Of Letters 89% A W
Bronx HS For Medical Science 80% B W
Bronx Theatre HS 83% B PNew Explorers HS 64% C P
Mott Haven Village Preparatory HS 74% A P
Bedford Academy HS 94% A W
Academy Of Urban Planning 56% C U
HS For Public Service: Heroes Of Tomorrow 98% B P
New Design HS 76% B P
Brooklyn School For Music & Theatre 80% B PHS For Dual Language & Asian Studies 78% A W
Renaissance HS Of Musical Theater & Tech 71% B P
All City Leadership SS 91% B W
Sci, Tech And Research Early College HS 97% B P
Bronx School Of Law And Finance 74% A P
8/7/2019 Parthenon rpt March 6 BTO Update (1)
19/2019
0%
20%
40%
60%
80%
100%
0 1,000 2,000 3,000 4,000 5,000
StudentsOverageatEntryto
High
Sc
hoolasPercentageofTo
tal
Total High School Enrollment
In Addition to Predicting AveragePerformance, Size and
Concentration Appear to Enable Beat-the-Odds Performance
Relationship Between Total Enrollment andProportion of Overage Students, 2007 Cohort
Note: Only includes schools that meet requirements for regression analysis (e.g., 20-student minimum cohort size, progress report grade)Source: DOE Internal Data
Harry S Truman HS
Other Schools
Proven BTO
Legend:
Emerging BTO
1 2
3 4
Percentage of SchoolsIn Each Quadrant That
Beat-the-Odds
1 2
3 4
3%34%
14%18%
The chart below looks at school size in relation to percent of students who are overage at entry to HS
The same finding applies for large schools and above-average concentration of low-proficiency students (only Harry
Truman HS is the only such school beating the odds)
8/7/2019 Parthenon rpt March 6 BTO Update (1)
20/20
Potential Areas of Implication
PortfolioDevelopment
HS AdmissionsKnowledge
Management
What targets should be set forthe overall evolution of theportfolio with respect to keystructural variables?
Can we isolate the factors thathave made the new schoolcreation process more effectivethan the 1990s schoolopenings?
Should we consider constraintson the HS admissions processthat take into consideration thepredicted graduation rate ofthe school? (e.g. dont allow
any school to have a predictedrate less than 45%)
What processes and structuresshould be created that:
- Allows on-site investigationto determine what drivesbeat-the-oddsperformance?
- Enables beat-the-oddsschools to codify their ownpractices?
- Encourages collaborationamong like schools withdifferential performance?
What is the nature of thequestions we should be askingto define those inquiries?
What other implications emerge from todays findings?
What additional questions should we be pursuing to informthe broader Secondary (or Middle School) strategy?