Longitudinal Analysis of MAP Achievement Growth: Preliminary Estimates of School Effects

1

Longitudinal Analysis of MAP Achievement Growth:Preliminary Estimates of School Effects

Sept. 15-16, 2010

Mark EhlertCory KoedelMichael PodgurskyDepartment of Economics, MU

CALDER, NCPIKansas City Area Education Research Consortium

Prepared for Missouri Technical Advisory Committee meeting. Kansas City, MO.September 15-16, 2010

2

Overview

• Examination of emerging MOSIS data system

• Patterns of Scale Score growth in MAP• A Simple VAM for School Effects

– Model and results– Covariates or not?

• Estimation of Teacher Program Effectiveness

• Future directions

3

• Missouri is developing a sophisticated P-20 data system

• IES State Longitudinal Data Grant• Ranks very favorably compared to other

states• Data quality is high

4

Data

• Matched Spring 2006-2009 student MAP scores using MOSIS ID

• Exclusions– Bad/duplicate values of MOSIS– Students retained in grade– Special districts

• Match rate for 4 years roughly 85% - 87%; match rate for 1 year regularly at 95%

5

Grade 2006 2007 2008 2009

3 a b c d

4 a a b c

5 a a a b

6 a a a a

7 a a a a

8 a a a a

10      a

EoC a       

Math MAP Testing Regime

6

620

640

660

680

700

720

3 4 5 6 7 8

Aver

age

Scal

e Sc

ore

MAP Math: Average Performance Growthby Cohort

Grade06 = 3 Grade06 = 4 Grade06 = 5

MAP Math: Average Performance

By Cohort

7

620

640

660

680

700

720

3 4 5 6 7 8

Ave

rage

Sca

le S

core

MAP Com Arts: Average Performance Growthby Cohort

Grade06 = 3 Grade06 = 4 Grade06 = 5

MAP Com. Arts : Average Performance

By Cohort

8

MAP Math: 2008-2009 Average Gain Score

By Grade and Decile of 2008 Performance

9

MAP Com Arts: 2008-2009 Average Gain Score

By Grade and Decile of 2008 Performance

10

Value-added models

• Why “value-added?”• Traditional economic definition

– Business or firm value-added• Value of output – value of inputs• VAT

• Education analogy– Control for initial (pre-treatment) performance– Estimate the effect of contemporaneous inputs on education

outcomes

11

• We want to identify causal effect of inputs– “what works”

• Treatment and control / comparison groups– Example: teacher training programs and

teacher effectiveness– Class size– Teacher credentials

12

A i j t = f (A i t – k , S i , SCH j ) + ε i j t

Educational outcome (e.g., test score, graduation, college attendance)

Lagged or baseline performance

Student Characteristics

School / classroominputs or treatment

random error

A Simple VAM

i – th student

j –th school or classroom

t – th year or grade

13

A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)

+ school effects + ε i t

Model estimated over all Missouri students, grades 3-8

Schools included if n > 20 student gain scores

3 gain scores x multiple grades per school

Gain Score Lagged Test Scores in both subjects

Average Effect by school (state mean = 0)

14

Appendix B – Regression Coefficients and Related Statistical Estimates

Dependent Variable

Math Com Arts

Coefficient T-Statistic Coefficient T-Statistic Past Scores:

First-Year Scale Score in Math - 0.34194** - 30.81 0.19145** 18.91 Squared First-Year Scale Score in Math

- 0.00003** - 3.37 0.00001 0.98

First-Year Scale Score in Com Arts

- 0.48245** - 38.19 - 0.54505** - 47.30

Squared First-Year Scale Score in Com Arts

0.00054** 55.42 0.00007** 7.84

Indicators for Student Characteristics:

American-Indian - 0.92858** - 2.72 0.08011 0.26 Asian/Pacific Islander 4.64526** 26.53 1.49822** 9.38 Black - 4.08531** - 47.43 - 1.64919** - 20.99 Hispanic - 0.89970** - 6.46 - 0.09035 - 0.71 Female - 2.16402** - 48.82 3.60541** 89.18 Special Education - 6.16846** - 86.27 - 8.07978** -123.90 Limited English Proficiency 0.24188 1.38 - 2.12545** - 13.32 Free/Reduced Price Lunch Eligibility

- 2.37397** - 46.78 - 2.38827** - 51.61

In the School Less Than a Full School Year

- 3.33210** - 29.97 - 2.63134** - 25.95

Indicators for Grade and Year: Terminal-year Grade 4 -19.46713** -157.16 - 8.14487** - 72.10 Terminal-year Grade 5 -18.22465** -161.81 - 6.14417** - 59.81 Terminal-year Grade 6 -13.95812** -170.22 -16.47484** -220.29 Terminal-year Grade 7 -17.73426** -254.94 -12.02678** -189.57 Dummy for 2006-2007 - 0.45133** - 8.40 - 1.45330** - 29.67 Dummy for 2007-2008 - 0.60767** - 11.47 - 0.24555** - 5.08

R2 = 0.251 R2 = 0.308 Number of School Effects = 1,773 (except the reference school) Sample Size(Number of Gainscores) = 926,358 ** denotes that coefficient is significant at 1% level.

15

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free-Reduced Priced Lunch

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

16

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities

Standardized School Effects on MAP Math Performancevs. Percent Minorities

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

17

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percentile Rank

Standardized School Effects on MAP Math PerformanceIn Rank Order

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

18

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Com Arts Performancevs. Percent Eligible for Free-Reduced Priced Lunch

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

19

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities

Standardized School Effects on MAP Com Arts Performancevs. Percent Minorities

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

20

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percentile Rank

Standardized School Effects on MAP Com Arts PerformanceIn Rank Order

Others District X: Significant District X: Insignificant

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Significant: Significantly different from the statewide average of school effects.

21

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

Stan

dard

ized

Sch

ool E

ffec

t for

Com

Art

s*

Standardized School Effect for Math*

Standardized School Effects on MAP PerformanceCom Arts vs. Math

Others District X

Coefficients of Correlation(ComArts, Math):

Pearson = 0.75271Spearman = 0.70142

*Standardized School Effecti = (School Effecti - Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

22

Effect of Covariates(math results)

Model 1 = student covariates

Model 2 = no student covariates

23

-1

-0.5

0

0.5

1

-1 -0.5 0 0.5 1

Stan

dard

ized

Scho

ol Eff

ect*

from

Mod

el 2

Standardized School Effect* from Model 1

Standardized School Effects on MAP Math PerformanceModel 1 vs. Model 2

Model 1. WITH student covariatesModel 2. WITHOUT student covariates

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Coefficients of Correlation (Model 1, Model 2):

Pearson = 0.97989Spearman = 0.97581

24

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year, Student level covariates)

25

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year)

26

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities in School

Standardized School Effects on MAP Math Performancevs. Percent Minorities

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year, Student level covariates)

27

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities in School

Standardized School Effects on MAP Math Performancevs. Percent Minorities

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)Note: Number of schools = 1,762 (with more than 20 valid test scores in both subjects)

Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year)

28

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch

District X

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year, Student level covariates)

29

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Eligible for Free/Reduced-Priced Lunch in School

Standardized School Effects on MAP Math Performancevs. Percent Eligible for Free/Reduced-Priced Lunch

District X

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year)

30

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities in School

Standardized School Effects on MAP Math Performancevs. Percent Minorities

District X

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Model 1. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year, Student level covariates)

31

-1

-0.5

0

0.5

1

0 20 40 60 80 100

Stan

dard

ized

Sch

ool E

ffec

t*

Percent Minorities in School

Standardized School Effects on MAP Math Performancevs. Percent Minorities

District X

*Standardized School Effecti = (School Effecti - Weighted Average of School Effects) /StDev of Level Scores(StDev of Level Scores = Weighted Average of by-grade Stdev of Level Scores, for grades 3-8)

Model 2. Gain in Math = f(Matht-1, Matht-12, ComArtst-1, ComArtst-1

2, School Dummies, Grade, Year)

32

Work Under Way

• Teacher training program effects– New teachers

• Retirement system effects– Effectiveness of teachers x retirement

behavior

33

A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)

+ school effects + teacher effects + ε i t

Model estimated over all Missouri students, grades 3-8

Within school

34

A i g - A i g-1 = f (Ai g-1 (m, ca), student char, grade, year)

+ school effects + teacher char + ε i t

Model estimated over all Missouri students, grades 3-8

Within school

35

Comparative Effectiveness of Teacher Preparation Programs

36

37 Teacher Training Programs

37

Schools with at least one newteacher graduate:Fall, 2005 – Fall, 2009

38

39

40

41

42

43

44

37 x 37 cross placement of program grads x school, gr. 4-8

PSI1 591 156 122 45 91 96 90 30 51 192 46 139 40 33 28 34 10 13 11 23 143 82 102 19 18 10 12 10 6 11 4 10 13 9 4 11 0

PSI2 156 501 193 59 41 53 38 61 28 23 201 43 15 68 160 32 68 17 11 30 37 7 7 8 19 12 40 11 4 65 2 62 0 6 6 0 2

PSI3 122 193 352 53 139 130 60 64 34 12 80 14 50 29 37 77 15 20 26 57 14 2 3 49 14 22 21 24 16 17 6 19 12 3 21 0 0

PSI4 45 59 53 157 47 32 102 13 70 20 18 11 12 10 3 20 4 9 14 12 11 2 4 6 4 0 4 2 14 5 4 3 4 12 2 0 0

PSI5 91 41 139 47 365 222 124 61 109 4 12 2 107 9 4 10 3 2 75 2 2 0 8 70 0 4 12 22 4 2 4 2 44 5 24 0 0

PSI6 96 53 130 32 222 241 88 48 77 12 11 11 62 0 13 7 6 12 52 8 2 0 6 39 2 6 0 21 20 0 3 2 22 2 2 0 0

PSI7 90 38 60 102 124 88 338 15 86 10 4 14 40 0 10 5 6 2 26 6 0 6 7 24 6 0 2 2 37 2 7 0 23 2 2 0 0

PSI8 30 61 64 13 61 48 15 97 18 4 21 4 26 10 10 8 6 10 21 6 0 4 0 18 6 2 5 12 6 6 2 0 2 0 10 0 0

PSI9 51 28 34 70 109 77 86 18 124 4 0 5 25 5 0 2 4 6 30 4 7 2 3 20 2 6 0 8 12 2 4 0 11 4 0 0 0

PSI10 192 23 12 20 4 12 10 4 4 165 4 46 5 4 8 15 2 2 2 15 20 34 30 0 6 4 2 0 0 0 0 4 0 15 0 4 0

PSI11 46 201 80 18 12 11 4 21 0 4 273 17 5 113 76 9 29 18 2 6 21 4 5 0 10 9 33 0 11 34 2 26 0 0 0 0 0

PSI12 139 43 14 11 2 11 14 4 5 46 17 115 4 4 20 6 4 2 0 6 23 11 6 0 2 5 4 0 0 2 0 5 0 3 0 0 0

PSI13 40 15 50 12 107 62 40 26 25 5 5 4 79 9 4 6 0 4 30 2 0 5 2 18 0 4 0 10 6 0 0 0 10 2 7 0 0

PSI14 33 68 29 10 9 0 0 10 5 4 113 4 9 127 20 8 6 4 2 0 8 7 3 2 4 0 18 0 10 6 0 15 0 2 0 0 0

PSI15 28 160 37 3 4 13 10 10 0 8 76 20 4 20 113 7 29 0 2 2 9 4 2 0 2 3 15 0 0 26 0 22 0 0 0 0 2

PSI16 34 32 77 20 10 7 5 8 2 15 9 6 6 8 7 65 0 0 2 26 6 0 0 3 10 13 0 0 6 0 4 0 0 2 0 0 0

PSI17 10 68 15 4 3 6 6 6 4 2 29 4 0 6 29 0 40 0 2 4 2 0 0 2 2 0 2 0 0 12 0 9 0 0 0 0 0

PSI18 13 17 20 9 2 12 2 10 6 2 18 2 4 4 0 0 0 41 0 6 0 5 0 2 2 2 0 2 24 0 2 0 0 0 0 0 0

PSI19 11 11 26 14 75 52 26 21 30 2 2 0 30 2 2 2 2 0 63 0 2 0 0 10 0 0 0 5 0 0 2 0 8 0 5 0 0

PSI20 23 30 57 12 2 8 6 6 4 15 6 6 2 0 2 26 4 6 0 65 0 0 0 0 0 2 0 2 0 2 4 0 0 0 2 0 0

PSI21 143 37 14 11 2 2 0 0 7 20 21 23 0 8 9 6 2 0 2 0 161 11 10 2 6 2 7 0 0 0 0 0 0 0 0 0 0

PSI22 82 7 2 2 0 0 6 4 2 34 4 11 5 7 4 0 0 5 0 0 11 63 10 0 2 0 0 0 10 0 0 0 0 5 0 2 0

PSI23 102 7 3 4 8 6 7 0 3 30 5 6 2 3 2 0 0 0 0 0 10 10 48 2 3 0 4 2 0 0 0 0 0 0 0 0 0

PSI24 19 8 49 6 70 39 24 18 20 0 0 0 18 2 0 3 2 2 10 0 2 0 2 54 0 0 0 13 0 0 0 0 6 0 2 0 0

PSI25 18 19 14 4 0 2 6 6 2 6 10 2 0 4 2 10 2 2 0 0 6 2 3 0 39 0 2 0 0 0 0 0 0 0 0 0 0

PSI26 10 12 22 0 4 6 0 2 6 4 9 5 4 0 3 13 0 2 0 2 2 0 0 0 0 17 0 2 0 0 2 2 0 0 0 0 0

PSI27 12 40 21 4 12 0 2 5 0 2 33 4 0 18 15 0 2 0 0 0 7 0 4 0 2 0 43 0 0 4 0 16 0 0 0 0 0

PSI28 10 11 24 2 22 21 2 12 8 0 0 0 10 0 0 0 0 2 5 2 0 0 2 13 0 2 0 20 2 0 0 0 2 0 0 0 0

PSI29 6 4 16 14 4 20 37 6 12 0 11 0 6 10 0 6 0 24 0 0 0 10 0 0 0 0 0 2 49 0 0 0 2 0 0 0 0

PSI30 11 65 17 5 2 0 2 6 2 0 34 2 0 6 26 0 12 0 0 2 0 0 0 0 0 0 4 0 0 28 0 5 0 0 0 0 0

PSI31 4 2 6 4 4 3 7 2 4 0 2 0 0 0 0 4 0 2 2 4 0 0 0 0 0 2 0 0 0 0 10 0 2 0 0 0 0

PSI32 10 62 19 3 2 2 0 0 0 4 26 5 0 15 22 0 9 0 0 0 0 0 0 0 0 2 16 0 0 5 0 37 0 0 0 0 0

PSI33 13 0 12 4 44 22 23 2 11 0 0 0 10 0 0 0 0 0 8 0 0 0 0 6 0 0 0 2 2 0 2 0 32 0 0 0 0

PSI34 9 6 3 12 5 2 2 0 4 15 0 3 2 2 0 2 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0

PSI35 4 6 21 2 24 2 2 10 0 0 0 0 7 0 0 0 0 0 5 2 0 0 0 2 0 0 0 0 0 0 0 0 0 0 19 0 0

PSI36 11 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0

PSI37 0 2 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4

45

Other research

Teacher Pension Effects

How do pension rules affect workforce quality?