A-F School Grades

July 25, 2012

Better Accountability for Schools

0

100

200

300

400

500

600

700

800

900

AYP Schools Failing in 2012 (Approx.)AYP Schools Failing in 2011Schools Graded F in2012

69 (8.3%)

715 (87%)

811 (97.6%)

What’s New?

• Key Changes in School Grading– Calculation changes mandated by the federal

government in New Mexico’s NCLB waiver.

– Bonus points.

– Inaugural Opportunity to Learn (OTL) survey completed by over 194,000 New Mexico students.

Validity of School Grade Results

• Do school grades meaningfully distinguish among schools? Yes.

• Grades made up of a composite of several components – the following charts demonstrate that grades do, in fact, differentiate schools well.

• Schools consistently perform well or poorly across all the components of the grade.

F D C B A0

10

20

30

40

50

60

70

80

Percent Proficient and Above by School Grade:Elementary and Middle Schools

Percent Proficient and Above, mathPercent Proficient and Above, reading

Perc

ent

F D C B A0

10

20

30

40

50

60

70

Percent Proficient and Above by School Grade:High School

Percent Proficient and Above, mathPercent Proficient and Above, reading

Perc

ent

F D C B A-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

Relationship between Student Growth and School Grades

Top 3 quarters of students MathTop 3 quarters of students Read-ingBottom Quarter of Students MathBottom Quarter of Students Reading

Overall Final 2012 School Grade

Grow

th p

er Y

ear (

scal

e sc

ore

poin

ts)

F D C B A0

10

20

30

40

50

60

70

80

90

Graduation Rates by School Grades

Graduation Rate

Perc

ent

F D C B A0

10

20

30

40

50

60

70

80

90

College and Career Readiness by School Grade

Particpation RateSuccess Rate

Perc

ent

Student Performance and School Grades

• AYP forced focus on percent proficient and above.

• School grades includes a component of percent proficient and above, but school grades also include growth.

• Initial reaction to a grade is to look at percent proficient and above, but this is not the sole basis for a grade – percent proficient and above does not tell the whole story.

Decomposing Performance

• Several components of school grades depend upon student SBA results.

• It is important to consider underlying student performance for each year used in grades.– Correlation between Performance level and Scale Score

= 0.91. – Correlation between Scale Score from year to year =

0.77.– How much does prior performance "explain" current

year? 59.3% .

Decomposing Performance

• Previous slide indicated that:– Performance levels and scale scores are not

perfectly related; that is, they always don’t tell the exact same story.

– Student performance changes from year to year. In fact, a student’s prior score explains about 59% of the variability in current scores.

– This means student performance changes – and school grades are designed to pick up those changes.

Decomposing PerformanceChange in Performance (math)

Change PL SS

Decrease 16.8% 46.6%

No Change 66.0% 7.6%

Increase 17.3% 45.8%

The table on the right indicates that students change performance levels (PL) from year to year.

In fact of those that change, half move up and half move down.

Scale scores exhibit the same pattern, except because scale scores provide a more nuanced picture of performance there is more change.

Decomposing PerformanceChange in Performance (math)

Change PL SS

Decrease 16.8 46.6

No Change 66.0 7.6

Increase 17.3 45.8

Comparison of PL and SS changes (one year)        

Change in SSDecrease No Change Increase Total No/Yes

Change

in PL

Decrease 19,271 182 60 19,513

No Change 34,572 8,309 33,907 76,788 89%Increase 443 300 19,354 20,097

Total  54,286 8,791 53,321 116,398 

The table below indicates that even among the students that stay at the same performance level, 89% them change scale scores – half up and half down.

About 86% of schools stayed within one grade.

School Grade Examples

• The following few slides provide to examples of how school grades work.

School Grades: Example

Elementary2012

Baseline Grade DPreliminary GradeGrade points 43.0Current Standing 19.5

Proficient & Above, math 44

Proficient & Above, reading 48

Average scale score, math 39.5

Average scale score, reading 39.6School Growth 3.7Individual Student Growth Q3 1.1Individual Student Growth Q1 8.0

Opportunity to Learn (OTL) 9.4Bonus Point 1.3

This school received a D and did fairly poorly in school growth and Q3 growth

School Grades: Example

Elementary2012 2011

Baseline Grade DPreliminary Grade BGrade points 43.0 64.3Current Standing 19.5 26.7

Proficient & Above, math 44 52

Proficient & Above, reading 48 59

Average scale score, math 39.5 40.1

Average scale score, reading 39.6 41.0School Growth 3.7 9.1Individual Student Growth Q3 1.1 5.5Individual Student Growth Q1 8.0 12.9

Opportunity to Learn (OTL) 9.4 10.1Bonus Point 1.3

In fact, even though percent proficient does not tell the whole story, for this school experienced a big drop in percent proficient and above

2008-2009 2009-2010 2010-2011 2011-20120.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Elementary Performance: 2008-2009 to 2011-2012

Math SSMath Pct Prof & abv.Reading SSReading Pct Prof & abv.

Preliminary Gardes (2011)

Final 2012 Grades

Estimating Student Growth

• SS_Mtij = γ000 + γ010*Q1_MIN10ij + γ100*FAYtij + γ200*YEARtij + γ210*YEARtij*Q1_MIN10ij

+ r0ij + r2ij *YEARtij+ u00j + u20j *YEARtij + u21j *YEARtij*Q1_MIN10ij + etij

Student Growth

Student Math SS Year FAY Q1

A 41 0 1 0

A 39 1 1 0

A 39 2 0 0

B 45 1 1 0

B 48 2 1 0

C 36 1 0 0

D 22 0 1 1

D 21 1 1 1

D 20 2 1 1

For each student his or her score for each year (FAY) and Q1 status are included.

Scale scores are a function of Year and so the fact that a student is missing a year is not important for estimating growth for a school.

Student Growth

• The following slide provides the overall student growth results for math.

Fixed Effect Coefficient Standard

errort-

ratioApprox.

d.f.p-

valueFor INTRCPT1, π0

For INTRCPT2, β00

INTRCPT3, γ000 42.27 0.14573 290 859 <0.001For Q1_MIN10, β01

INTRCPT3, γ010 -16.87 0.08464 -199 210197 <0.001For FAY slope, π1

For INTRCPT2, β10

INTRCPT3, γ100 0.562 0.03262 17.2 18323 <0.001For YEAR slope, π2

For INTRCPT2, β20

INTRCPT3, γ200 -0.787 0.04612 -17 859 <0.001For Q1_MIN10, β21

INTRCPT3, γ210 1.73 0.04326 39.8 859 <0.001

Note: next slide contains some explanation

Student Growth

• The results indicate the mean math scale score for Q3 students is about 42

• The average Q3-Q1 gap is about 16.87.• The mean growth for Q3 non-FAY students is -0.787 scale score points per year• FAY students grow about 0.56 scale score points

per year faster.• Q1 students grow about 1.73 scale score points per

year more quickly than Q3 students.

Example 2: Changes in Percent Proficient don’t tell the whole story

School Grades: Example 2 Valley High School

2011 2012Baseline Grade CPreliminary Grade BGrade points 69.2 61.5Current Standing 13.7 15.4

Proficient & Above, math 26 39Proficient & Above, reading 49 52Average scale score, math 33.8 36.9Average scale score, reading 38.3 39.9

School Growth Q3 13.2 7.7School Growth Q1 14.7 10.7Opportunity to Learn (OTL) 10.2 6.1Graduation 11.4 11.5College and Career Readiness 6.1 8.6Bonus Point 1.5

Q3 Q1

Growth estimate -1.7 1.3

Percetnile 3.7 29

Points 0.376 2.9

The table on the right demonstrates the example school’s growth in math.

The expectation in Q3 is for students to have a growth estimate of 0.

The students in this school grew at -1.7, which placed them in earned them 4% of the available point for math Q3 growth – or 0.376 points.

The school did slightly better in Q1 math growth and so earned 29% of school growth, or about 2.9 points for math Q1 growth.

SY 2009-2010 SY 2010-2011 SY 2011-201230.00

32.00

34.00

36.00

38.00

40.00

42.00

44.00

46.00

48.00

50.00

High School : Math SBA Performance Q3

Current Math

Scal

e Sc

ore

School Growth applies a value added model (VAM) to a school’s growth over the past three years to generate points towards a school grade. High School demonstrates some improvement over the past three years.

SY 2009-2010 SY 2010-2011 SY 2011-201230.00

32.00

34.00

36.00

38.00

40.00

42.00

44.00

46.00

48.00

50.00

High School : Math SBA Performance Q3

Prior Math Current Math

Scal

e Sc

ore

Most of the school growth demonstrated by High School is accounted for by the growth of High School’s students while they were in eighth grade. The Value Added Model appropriately levels the playing field by taking prior student performance into account for which High School is not responsible.