Parthenon rpt March 6 BTO Update (1)

8/7/2019 Parthenon rpt March 6 BTO Update (1)

1/20

New York CityDepartment of Education

Beat-the-Odds HS Update

March 6, 2008

DRAFT WORK PRODUCTFor Discussion Only


2/20

2

Beat-the-Odds HS Analysis Update

Objectives for Today

Review structure and initial findings from updated regression model

- What does this model allow us to do, and what questions do we want to ask?

Discuss data on specific follow-up questions from last data review

Gather group input on key areas of potential implications: which findings shouldbe highlighted, and which applications of the data are most powerful?

Discuss key areas of analytical next steps:

- Variables to test which might be common across Beat-the-Odds HSs

- Secondary portfolio strategies to model and assess projected impact


3/20

3

How Does the New Regression Model Work?

Student-Level Factors School-Level FactorsTrue Over- and Under-

Performance by Schools

For each individual student,what are the key demographicand performance variables

that shape the odds ofgraduation?

Given each studentsparticular profile, what is eachindividuals expectedgraduation rate?

What are the aspects ofschool structure that have anadditional peer effect on the

chances of graduation?

By how much is eachindividual students odds ofgraduation changed byattending schools withdifferent structures andpopulations?

Once we have leveled theplaying field in terms of bothstudent-level and school-level

factors, how much variationremains in the graduation rateacross schools?

Which schools are thegreatest over- and under-performers, and are therefactors that appear to enablesuch performance?


4/20

4

What Are the Variables that Drive the New Model?

Student-Level Factors School-Level FactorsTrue Over- and Under-

Performance by Schools

Gender

Ethnicity (White/Asian vs.African-American/Hispanic)

Average 8th ELA + Math

Proficiency Level

Age at Entry to High School

8th grade attendance rate

ELL Status (Y/N)

SPED Status (LRE vs. Self-Contained vs. Non-SPED)

Lack of 8th grade attendanceor 8th grade test scores

School enrollment

A series of concentrationvariables:

- Percent of students with a

Low-Level 2 or below oneither ELA or Math

- Percent of studentsentering 9th grade overage

- Percent of students with 8thgrade attendance below90%

- Tested but not significant:

Percent ELL, Percent SPED(total and Self-Containedonly), Title I eligibility, Avg.teacher tenure, Principaltenure, General Ed class size,Special Ed class size, facilitiescapacity utilization

The model explains 71% ofthe variation in high schoolgraduation rates

However, 22% of schools

differ from their predictedgraduation rate by +/- 10 ormore percentage points


5/20

5

Individual Student Odds of Graduation Are Predicted Based on a

Combination of Factors

88%84%

76%

37%

27%

14% 14% 12%

0%

20%

40%

60%

80%

100%

PredictedGraduationRate

Predicted Graduation Rate byStudent-Level Factors, 2007 Cohort

Note: Controlling for all other factors, students who are ELL during high school are 9% more likely to graduate than non-ELL studentsSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

SPED Status:

8th Gr. Att.:

Age at Entry:

Test Level:

Gender:

Ethnicity:

GE

95%

On

3.0

Female

Wh./As.

GE

95%

On

3.0

Female

Af-Am/His.

GE

95%

On

3.0

Male

Af-Am/His.

GE

95%

On

2.0

Male

Af-Am/His.

GE

95%

Over

2.0

Male

Af-Am/His.

GE

85%

Over

2.0

Male

Af-Am/His.

LRE

85%

Over

2.0

Male

Af-Am/His.

SC

85%

Over

2.0

Male

Af-Am/His.

Number of Factors Related to High Graduation RateMany Few

Note: Predicted graduation rates below are calculated independent of the schools that students attend


6/20

6

The Model Can Isolate the Effect of Demographic Variables on

Student Performance

1 2 3 4

0%

20%

40%

60%

80%

100%

Pre

dictedGraduationRate

Predicted Graduation Rate by Ethnicity and Gender, 2007 Cohort

Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

White/Asian Females

White/Asian Males

African-American/Hispanic Females

African-American/Hispanic Males

Average of 8th Grade Math & ELA Proficiency Rating

At the widest point (low-mid Level 2), white and Asian females outperform African-American and Latino males by morethan 20% points controlling for all other factors

Note: Predicted graduation rates below are calculated independent of the schools that students attend


7/207

SC

LR

E

C

TT

L1 LL2 HL2 L3

29%27%

37%

31%36%

41%

29%

41%

48%

32%

53%49%

0%

20%

40%

60%

80%

100%

Pe

rcentEarning10orMore

Credits

CTT

LRE

SC

0.9K

1.1K

2.5K

0.6K

1.7K

0.7K

0.2K

0.9K

0.2K

0.1K

0.3K

0.0K

SPED Students in CTT Environments Appear to Outperform

Peers in Both LRE and Self-Contained

Credit Performance of Special Education Populations byIncoming 8th Grade ELA Proficiency Level, First-Time Freshmen, 2006-07

Note: Districts 75 and 79 excluded; Excludes students with missing 8th Grade ELA proficiency dataSource: DOE Internal Data

Number of Students

31%

Non-Special Ed.

However, at higher levels of incoming proficiency, CTT students still under-perform general ed students

44%

57%

76%


8/208

School Structure Can Also Drive Wide Changes in Expected

Performance


The Median Student

Female

African-American/Hispanic

Median 8th Grade Test Scores (2.7)

Med. 8th Grade Attendance (92%)

On-Age at Entry to HS (14 yo)

Neither ELL nor SPED

70% Predicted Graduation Rate

School 4(e.g. John F Kennedy HS)

3,000 students

Concentrations ofchallenged students15% points abovesystem average

55% Graduation Rate

School 3(e.g. William Grady CTE HS)

1,500 students

Concentrations ofchallenged students5% points abovesystem average

70% Graduation Rate

School 2

(e.g. HS for Leadership andPublic Service)

800 students

Concentrations ofchallenged students5% points belowsystem average

78% Graduation Rate

School 1

(e.g. Science, Tech andResearch Early College HS)

400 students

Concentrations ofchallenged students15% points belowsystem average

83% Graduation Rate

(Effect of 400-StudentEnrollment Alone 78%)


9/209

The Model Allows Us to Compare the Impact of Different

Strategies on Expected Student Performance


8th Grade Test Score Remains at 2.0

Attend a 400 Student School With AverageConcentration of Low Achievement, Overage &Low-Attending Students

42% Graduation Rate

Improve 8th Grade Test Score to 2.5

Attend a 3,000 Student School with LowAchievement, Overage & Low-Attending StudentConcentrations 10 p.p. Above System Average

40% Graduation Rate

African-American/Hispanic Male

8th Grade Test Score of 2.0

31% Predicted Graduation Rate(independent of school attended)

Improve Preparation in MS HS Portfolio Development


10/2010

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

ActualGradRate

Predicted Grad Rate

Traditional High Schools& Career/Technical

New Small High Schools

R = 0.71

# of Obs. = 230

The New Model Highlights a Greater Number of Schools with

Low Predicted Grad Rates

Relationship Between Predicted GraduationRate and Actual Graduation Rate, 2007 Cohort

Source: ATS Data; Progress Reports Data; NYC DOE Website

Schools with theToughest Odds

Less than 40% predictedgrad rate: 5 schools (vs.0 in original model)

40-45% predicted gradrate: 12 schools (vs. 3 inoriginal model)

45-50% predicted gradrate: 24 schools (vs. 14 inoriginal model)


11/2011

0%

20%

40%

60%

80%

100%

0% 20% 40% 60% 80% 100%

ActualGradRate

Predicted Grad Rate

Traditional High Schools& Career/Technical

New Small High Schools

Over-Performance(15% of schools)

Underperformance(7% of schools)

R = 0.71

# of Obs. = 230

22% of Schools Diverge from Their Predicted Graduation

Rate by +/- 10 or More Percentage Points 78% of New Small High Schools have an actual graduation rate that exceeds their predicted graduation rate; 37% of

New Small High Schools exceed this prediction by 10 percentage points or greater

Relationship Between Predicted GraduationRate and Actual Graduation Rate, 2007 Cohort

Note: 10% points represents approximately one standard error in the final regression modelSource: ATS Data; Progress Reports Data; NYC DOE Website


12/2012

2

4%

23

%

22%

20%18%

16%16%

16%

14%

14%

14%

8%8%7%6%6%6%5%5%4%4%

1%1%1%0%

-0%

-2%-2%-2%-2%-3%-3%-3%-3%

-4%-4%-4%-5%-5%-6%-6%

-7%

-8%-8%-8%-8%-8%-8%-9%-9%-9%-9%

-10%

-10%

-10%

-10%

-12%

-12%

-15%

-19

%

-19

%

-23%

-20%

-10%

0%

10%

20%

30%

40%

31%

29%

24%

24%

23%

20%

19%

18%

18%

16%

16%

15%

14%

12%

12%

12%

12%

10%

10%

9%9%

8%8%

8%

8%6%

6%6%

4%

4%3%

3%2%

1%

0%0%

-0%

-

1%

-1%

-2

%

-2

%

-2

%

-5%

-6%

-15%

-15%

-20%

-10%

0%

10%

20%

30%

40%

Average = -0.3%

Average = 8.1%

Source: DOE Internal Data; Parthenon Regression Model

Residual

Residual

Distribution of 2007 Over/Under-performance for New Small Schools (n=46)

Performance of New Small Schools Differs from Small Schools

Opened During the 1990s

Distribution of 2007 Over/Under-performance for Old Small Schools (n=62)


13/2013

New Schools Outperform Most with Challenged Students, but

Much of the Gap Is in Local Diplomas

LocalDiploma

Regents

or Above

NewSmallHSs

OtherSchools

NewSmallHSs

OtherSchools

NewSmallHSs

OtherSchools

NewSmallHSs

OtherSchools

NewSmallHSs

OtherSchools

37%

19%

66%

40%

77%

60%

89%

80%

86%*

91%

0%

20%

40%

60%

80%

100%

4-YearGraduationRate

Overall Gap 18% 26% 17% 9% N/A

Regents+ Gap 4% 7% 6% 3% N/A

Class of 2007 Four Year Graduation Rate by 8th Grade ELA Level:New Small Schools vs. All Other Schools, Local vs. Regents Diplomas

* Limited sample (n=49) for Level 4 students in New Small SchoolsNote: Only includes schools that meet requirements for regression analysis (i.e., 20-student minimum cohort size, progress report grade). ExcludesOther diploma types from non-New Small Schools, including Proof Of Receipt Of HS Diploma, Already Had Diploma, Tried For Regents But Left,Career Ed Regents Diploma, Confirmed Completion Of HS Requirements, Advanced Diploma CTE, and Local Diploma CTESource: DOE Internal Data

Level 1 Low Level 2 High Level 2 Level 3 Level 4


14/2014

Over-Performance

Under-Performance

Under-Performance

Over-Performance

Underperformed Then, Over-

Performs Now

(8% of schools)

Over-Performed Then, Over-

Performs Now

(35% of schools)

Underperformed Then,

Underperforms Now

(36% of schools)

Over-Performed Then,

Underperforms Now

(21% of schools)

2005 Cohort Regression

20

07CohortRegressio

n

Regression Is Run Over Time to Assess Performance Trajectory

Note: Percentages are taken of all schools that met requirements for regression analysis for both2005 and 2007 cohorts; percentages may not total to 100% due to roundingSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

Of the schools who wereunderperforming in the Classof 2005, 82% were stillunderperforming in the Class

of 2007

Of the schools who wereover-performing in the Classof 2005, 63% were still over-performing in the Class of2007

Of schools with graduatingclasses in both 2005 and2007, 57% are under-

performing in 2007 drivenby the disproportionate rateof over-performance amongnew small schools

School Over- vs. Underperformance in New Regression Model, Class of 2005 vs. 2007


15/2015

New Small Schools Appear Mixed in Sustaining Their Performance

Going Forward

Increasedby 3+

PercentagePoints

LittleChange

Decreasedby 3+

PercentagePoints

Schools Opening in 2001

n=17

0%

20%

40%

60%

80%

100%

Increasedby 3+

Percentage

Points

Little

Change

Decreasedby 3+

Percentage

Points

Schools Opening in 2003

n=27

0%

20%

40%

60%

80%

100%

Biggest Improvement

1. Belmont Preparatory HS (+13%)

2. Fordham HS For The Arts(+11%)

3. Jonathan Levin HS For MediaAnd Communications (+10%)

4. Fordham Leadership Academy

For Business And Technology(+4%)

5. Bronx HS For Medical Science(+3%)

Biggest Decline

1. School For Excellence (-26%)

2. Frederick Douglass Academy IISecondary School (-22%)

3. Community School For SocialJustice (-19%)

4. Bronx Leadership Academy II HS(-16%)

Biggest Improvement

1. HS For Violin And Dance (+20%)

2. Brooklyn Academy Of ScienceAnd The Environment (+12%)

3. Manhattan Bridges HS (+11%)

4. Brooklyn HS For Music AndTheater (+8%)

5. New York Harbor School (+5%)

Biggest Decline

1. Pelham Preparatory Academy(-28%)

2. International Arts Business HS(-23%)

3. Discovery HS (-17%)4. Bushwick HS For Social Justice

(-15%)

5. HS For Contemporary Arts(-12%)

PercentofNewSmallSch

ools

PercentofNewSmallSch

ools

Change in Percent of Students Earning 20+ Credits After Two Years, 2007 vs. 2009 Cohort

Source: DOE Internal Data

Overall, the percent of students earning 20+ credits by the end of the 2 nd year went from 64% to 62% in the new small schools (vs.49% to 54% in the rest of the portfolio)

More analysis would be required to adjust these figuresfor changes in school population


16/2016

Three High-Performing Categories of Schools Present KM

Opportunities

Proposed Key Categories of High Schools

Turnaround SchoolsEmerging Beat-the-Odds

Schools withPockets of Success

Turnaround schools shoulddemonstrate dramaticimprovement over the period

from 2005-2007:

- Underperforming in 2005and over-performing in2007

- Improved by 10%+ pointsin both relative andabsolute grad rates

- Now have a graduationrate at or above the

citywide average

Proven Beat the OddsSchools demonstrate asustained ability to

outperform their predictedgraduation rate:

- Exceed predictedgraduation rate by atleast 5 percentage pointsin both Class of 2005 and2007

- Also exceed citywideaverage graduation rate

Schools that do not meet thecriteria of Beat the Oddsschools overall, but which

perform as well as the bestProven Beat the Oddsschools with key sub-groups

Groups tested thus farinclude:

- Low achieving 8th graders

- 9th graders who areoverage at entry to HS

- ELL students- Self-Contained SPED

students

- African-American andLatino males

Proven Beat-the-Odds

Beat-the-Odds Schools

Emerging Beat the OddsSchools outperform in themost recent cohort (2007) by

the same margin as ProvenBeat the Odds schools (atleast 5% points)

- Also exceed citywideaverage graduation rate

These schools also show signsof sustaining theirperformance going forward:

- Percent of 2nd year

students with 20+ creditshas improved or remainedsteady within 5percentage points

25 schools 21 schools 6 schools

Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model

17 schools(across 5 categories)


17/2017

Matching Schools with a Counterpoint Provides a Unique

Opportunity for Improvement

Proposed Key Categories of High Schools

Turnaround SchoolsEmerging Beat-the-Odds

Schools withPockets of SuccessProven Beat-the-Odds

Beat-the-Odds Schools

Struggling New SchoolsSchools with ChronicUnderperformance

Schools in DeclineSchools with SpecificAreas of Weakness

Schools in declinedemonstrate a dramaticdeterioration in performance:

- Went from over- to

underperforming theirpredicted rate from 2005to 2007

- Decline in performancerelative to expectations of10%+ percentage points

Schools that underperformtheir predicted graduationrate by 5 or more percentagepoints in both the Class of

2007 and the Class of 2005

Schools that are over-performing their predictedgraduation rate by 5 or morepercentage points, but lag

the citywide average in a keystudent sub-group

New small schools that:

- Underperform theirpredicted graduation rateby 5 or more percentage

points or

- Show substantial declinesin credit accumulationbetween the Class of 2007and Class of 2009

12 schools(excluding Phaseouts) 15 Schools 12 schoolsTBD (In Process)

Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model


18/2018

Beat-the-Odds High Schools Mostly Score Highly on Progress

Reports and Quality Reviews

School Name PR QR

Over-Performed by 10% or Greater On Both

Proven Beat-the-Odds Schools

Note: In order to be considered for beating the odds, a school must exceed the citywide average graduation rateSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

Grad Rate

Over-Performed by 10% for Class of 2007

Emerging Beat-the-Odds Schools

School Name PR QR Grad Rate



Over-Performed by 5% or Greater On Both

Over-Performed by 5% for Class of 2007

19 out of 25 schools with either an A or W 12 out of 21 schools with either an A or W

HS For Enterprise, Business, And Tech 83% A W

HS For Media And Communications 65% A W

Wadleigh SS For The Performing & Vis 70% A P

Harry S Truman HS 66% B P

HS For Law And Public Service 74% A W

East Side Community HS 67% A W

Fannie Lou Hamer Freedom HS 59% B P

Park East HS 75% B P

Riverdale/Kingsbridge Academy 84% A PBrooklyn International HS 79% A W

University Heights SS At Bronx CC 82% A W

Manhattan Village Academy 83% A W

Bronx International HS 59% A P

Repertory Company HS For Theatre Arts 85% B P

Susan E. Wagner HS 79% B W

El Puente Academy For Peace And Justice 71% B W

School Of The Future HS 96% B W

Automotive HS 57% F P

Forest Hills HS 76% B P

Curtis HS 70% B W

HS For Health Careers And Sciences 58% A W

Brooklyn College Academy 93% B W

Michael J. Petrides HS 96% A W

Landmark HS 61% A P

The HS Of Fashion Industries 78% A W

Millennium Art Academy 86% A P

Marble Hill HS For International Studies 96% A W

Morris Academy For Collaborative Studies 79% B P

Bronx Aerospace HS 88% A P

Manhattan Bridges HS 68% A W

HS For Violin And Dance 85% A P

Bronx Academy Of Letters 89% A W

Bronx HS For Medical Science 80% B W

Bronx Theatre HS 83% B PNew Explorers HS 64% C P

Mott Haven Village Preparatory HS 74% A P

Bedford Academy HS 94% A W

Academy Of Urban Planning 56% C U

HS For Public Service: Heroes Of Tomorrow 98% B P

New Design HS 76% B P

Brooklyn School For Music & Theatre 80% B PHS For Dual Language & Asian Studies 78% A W

Renaissance HS Of Musical Theater & Tech 71% B P

All City Leadership SS 91% B W

Sci, Tech And Research Early College HS 97% B P

Bronx School Of Law And Finance 74% A P


19/2019

0%

20%

40%

60%

80%

100%

0 1,000 2,000 3,000 4,000 5,000

StudentsOverageatEntryto

High

Sc

hoolasPercentageofTo

tal

Total High School Enrollment

In Addition to Predicting AveragePerformance, Size and

Concentration Appear to Enable Beat-the-Odds Performance

Relationship Between Total Enrollment andProportion of Overage Students, 2007 Cohort

Note: Only includes schools that meet requirements for regression analysis (e.g., 20-student minimum cohort size, progress report grade)Source: DOE Internal Data

Harry S Truman HS

Other Schools

Proven BTO

Legend:

Emerging BTO

1 2

3 4

Percentage of SchoolsIn Each Quadrant That

Beat-the-Odds

1 2

3 4

3%34%

14%18%

The chart below looks at school size in relation to percent of students who are overage at entry to HS

The same finding applies for large schools and above-average concentration of low-proficiency students (only Harry

Truman HS is the only such school beating the odds)


20/20

Potential Areas of Implication

PortfolioDevelopment

HS AdmissionsKnowledge

Management

What targets should be set forthe overall evolution of theportfolio with respect to keystructural variables?

Can we isolate the factors thathave made the new schoolcreation process more effectivethan the 1990s schoolopenings?

Should we consider constraintson the HS admissions processthat take into consideration thepredicted graduation rate ofthe school? (e.g. dont allow

any school to have a predictedrate less than 45%)

What processes and structuresshould be created that:

- Allows on-site investigationto determine what drivesbeat-the-oddsperformance?

- Enables beat-the-oddsschools to codify their ownpractices?

- Encourages collaborationamong like schools withdifferential performance?

What is the nature of thequestions we should be askingto define those inquiries?

What other implications emerge from todays findings?

What additional questions should we be pursuing to informthe broader Secondary (or Middle School) strategy?

Documents

Parthenon rpt March 6 BTO Update (1)