-- Preliminary, Do Not Quote Without Permission -- VALUE-ADDED MODELS AND THE MEASUREMENT OF TEACHER QUALITY Douglas HarrisTim R. Sass Dept. of Ed. LeadershipDept

-- Preliminary, Do Not Quote Without Permission --

VALUE-ADDED MODELSVALUE-ADDED MODELSAND THE AND THE

MEASUREMENT OF MEASUREMENT OF TEACHER QUALITYTEACHER QUALITY

Douglas HarrisDouglas Harris Tim R. SassTim R. SassDept. of Ed. LeadershipDept. of Ed. LeadershipDept. of EconomicsDept. of Economicsand Policy Studiesand Policy Studies Florida State Florida State UniversityUniversityFlorida State UniversityFlorida State University ([email protected])([email protected]) ([email protected])([email protected])

IES Research Conference – June 2006IES Research Conference – June 2006


Evaluating Value-Added Evaluating Value-Added MethodologyMethodology

The recent availability of panel data The recent availability of panel data has produced a flood of research has produced a flood of research studies using various “value-added” studies using various “value-added” approachesapproaches

Research QuestionsResearch Questions Are assumptions underlying the value-Are assumptions underlying the value-

added approach valid? added approach valid? Are some methods more likely to Are some methods more likely to

produce reliable estimates than others?produce reliable estimates than others? What data are most important to What data are most important to

obtaining consistent estimates?obtaining consistent estimates?


Evaluating Value-Added Evaluating Value-Added MethodologyMethodology

Basic Model TypesBasic Model Types Cumulative ModelCumulative Model Unrestricted Value-Added ModelUnrestricted Value-Added Model Value-Added Models with Persistence RestrictionsValue-Added Models with Persistence Restrictions

Restricted Value-Added or “Gain-Score” ModelRestricted Value-Added or “Gain-Score” Model Contemporaneous ModelContemporaneous Model

Specification Issues for Value-Added Specification Issues for Value-Added ModelsModels Treatment of teacher heterogeneityTreatment of teacher heterogeneity Measures of classroom/school inputsMeasures of classroom/school inputs Treatment of student heterogeneityTreatment of student heterogeneity AggregationAggregation


General Cumulative Model General Cumulative Model of Student Achievementof Student Achievement

it0iiiitit ,),t(),t(),t(AA EFX

where Ait = achievement level for individual i at the end of their tth year of life Xi(t) = entire history of individual inputs Fi(t) = entire history of family inputs Ei(t) = entire history of school-based educational inputs i0 = a composite variable representing time-invariant characteristics an individual is endowed with at birth (such as innate ability) it = a normally distributed, mean-zero error.


Basic AssumptionsBasic Assumptionsof Value-Added Modelsof Value-Added Models

Cumulative achievement function does not Cumulative achievement function does not vary with age and is additively separable.vary with age and is additively separable.

Family inputs are time invariant.Family inputs are time invariant. Parents do not compensate for poor school inputs Parents do not compensate for poor school inputs

or poor outcomesor poor outcomes Todd and Wolpin (2005) reject exogeneity of parental Todd and Wolpin (2005) reject exogeneity of parental

inputs at 90 percent, but not at 95 percent confidence inputs at 90 percent, but not at 95 percent confidence levellevel

The marginal inputs of all school-based The marginal inputs of all school-based inputs, parental inputs, and the initial student inputs, parental inputs, and the initial student endowment each decline geometrically (at endowment each decline geometrically (at potentially different rates) over time.potentially different rates) over time. Lagged achievement serves as a sufficient statistic Lagged achievement serves as a sufficient statistic

for prior inputsfor prior inputs We find twice-lagged inputs do not provide additional We find twice-lagged inputs do not provide additional

informationinformation


UnrestrictedUnrestrictedValue-Added ModelValue-Added Model

iti1itit1it1it AA EβXα

where i = an individual student effect representing time-invariant student/ family characteristics

it = it - it-1 is a random error. Given it is a function of the lagged error in achievement, it is correlated with Yt-1 and ordinary least squares is inconsistent. Rarely consistently estimated with large-scale data sets. -- Ding and Lehrer (2005), Sass (2006).


Persistence RestrictionsPersistence Restrictions

itiit1it11itit AAA EβXα

Restricted Value-Added or “Gain-Score” ModelRestricted Value-Added or “Gain-Score” Model

is assumed to equal 1 (no decay in effect of past inputs)is assumed to equal 1 (no decay in effect of past inputs) Alternatively, can interpret as an achievement Alternatively, can interpret as an achievement growthgrowth model model

where growth is independent of past school inputswhere growth is independent of past school inputs..

Contemporaneous ModelContemporaneous Model

itiit1it1itA EβXα

is assumed to equal 0 (complete decay in is assumed to equal 0 (complete decay in effect of past inputs)effect of past inputs)..


Decomposition of School-Decomposition of School-based Inputs in Value-based Inputs in Value-

Added ModelAdded Modelitmki1itkt3ijmt2it1it AA TβPβXβ

Eit is decomposed into four components: classroom peer inputs, P-ijmt

race, gender, mobility and age of peers; class size

time-varying classroom teacher inputs, Tkt experience, professional development, advanced degrees

time-invariant classroom teacher inputs, k race/ethnicity, gender, pre-service education, pre-college ability

school-wide time-invariant inputs, m Assume non-teaching inputs are constant across classroom within a school and school-wide inputs do not vary over sample period.


Modeling Teacher Modeling Teacher HeterogeneityHeterogeneity

Substituting teacher time-invariant measured characteristics for teacher fixed effects

itmki1itkt3ijmt2it1it AA ρYTβPβXβ

where it = (k-Yk) + it.

This approach will produce biased estimates if unmeasured time-invariant teacher characteristics, (k-Yk), which are now part of the error term, are correlated with observed time-varying student, peer or teacher variables in the model (ie. Xit, P-ijmt, Tkt).


Classroom and School Classroom and School InputsInputs

itmki1itkt3ijmt2it1it AA TβPβXβ

Exclusion of peer variables (P-ijmt) Number of peers (class size) and peer characteristics

(gender, race, mobility, age) If peer variables are correlated with student and

teacher characteristics (Xit and Tkt), omission will produce inconsistent estimates

Exclusion of school fixed effects (m) Given that teachers do not frequently change

schools, omission of school effects will mean that teacher fixed effects will capture both teacher effects and some of the school effect, leading to inconsistent estimates


Modeling Student Modeling Student HeterogeneityHeterogeneity


Substituting measured time-invariant student characteristics for student fixed effects Race/ethnicity, foreign/native born, language parent

speak at home, free-lunch status As with teachers, if unmeasured time-invariant

student characteristics are correlated with independent variables, will get inconsistent estimates


Modeling Student Modeling Student HeterogeneityHeterogeneity


Fixed vs. random student effects Fixed effects allow for a separate intercept parameter for

each student (equal to the mean error for that student) whereas random effects assume that the student-specific intercepts are drawn from a known distribution (typically normal)

Since random effects are part of the error structure, they must be orthogonal to the model variables (Xit, P-ijmt, Tkt) in order to yield consistent estimates

Given that fixed effects estimates are always consistent (whether or not unobserved student heterogeneity is correlated with other variables in the model), can test orthogonality assumption by applying a Hausman test

Multilevel fixed effects models have been computationally burdensome


AggregationAggregation


Measuring characteristics of specific teachers vs. grade-level-within-school averages Since Texas data does not identify specific teacher,

work by Rivkin, Hanushek and Kain (2005) relies on average characteristics of teachers within a grade

Advantages/Disadvantages of aggregation Eliminates problems associated with non-random

assignment of students to teachers within a school May reduce problem of measurement error since

individual errors may cancel out at grade level May upwardly bias estimated impacts of school

resources in the presence of omitted variables Tends to reduce precision of estimates


DataData

Florida’s K-20 Education Data Florida’s K-20 Education Data WarehouseWarehouse Census of all children attending public schools in Census of all children attending public schools in

FloridaFlorida Student records linked over timeStudent records linked over time Covers 1995/1996 – 2003/2004 school yearsCovers 1995/1996 – 2003/2004 school years Includes student test scores and student Includes student test scores and student

demographic data, plus enrollment, attendance, demographic data, plus enrollment, attendance, disciplinary actions and participation in special disciplinary actions and participation in special education and limited English proficiency education and limited English proficiency programsprograms

Includes all employee records including individual Includes all employee records including individual teacher characteristics and means of linking teacher characteristics and means of linking students and teachers to specific classroomsstudents and teachers to specific classrooms


Sample for AnalysisSample for Analysis

Middle school students (grades 6-8) who took Middle school students (grades 6-8) who took SSS-NRT (Stanford-9) math test in three SSS-NRT (Stanford-9) math test in three consecutive years during 1999/2000 – consecutive years during 1999/2000 – 2003/20042003/2004 Enrolled in a single math course in the FallEnrolled in a single math course in the Fall Up to 4 years of achievement gainsUp to 4 years of achievement gains 4 cohorts of students4 cohorts of students Includes a variety of math courses, from remedial to Includes a variety of math courses, from remedial to

advanced and gifted classesadvanced and gifted classes Use random sample of 100 middle schools Use random sample of 100 middle schools

Reduces computational burden of estimating fixed Reduces computational burden of estimating fixed effectseffects

Represents about 12% of middle schools in stateRepresents about 12% of middle schools in state


Value-Added Model EstimatesValue-Added Model EstimatesWith Varying Degrees of Persistence With Varying Degrees of Persistence

Restricted Value-Added (Partial Decay) Contem- Value Added poraneous Explanatory Variable =1 =.8 =.6 =.4 =.2 =0 0 Years of Experience -2.7495 -2.2506 -1.7517 -1.2529 -0.7540 -0.2551 (0.91) (0.83) (0.72) (0.57) (0.38) (0.14) 1 Year of Experience 1.7714 1.8542 1.9369 2.0197 2.1025 2.1852* (0.82) (0.96) (1.11) (1.29) (1.48) (1.66) 2-4 Years of Experience -0.4575 -0.1030 0.2515 0.6061 0.9606 1.3152 (0.29) (0.07) (0.20) (0.52) (0.91) (1.36) Content In-service Hourst -0.0177 -0.0187 -0.1097 -0.0207 -0.0217* -0.0227** (0.92) (1.08) (1.28) (1.50) (1.74) (1.99) Content In-service Hourst-1 0.0216 0.0183 0.0150 0.0117 0.0084 0.0051 (1.07) (1.01) (0.92) (0.80) (0.63) (0.41) Content In-service Hourst-2 0.0517** 0.0434** 0.0352** 0.0269* 0.0186 0.0104 (2.37) (2.21) (2.00) (1.70) (1.30) (0.78) Content In-service Hourst-3 0.0100 0.0084 0.0068 0.0052 0.0035 0.0019 (0.42) (0.39) (0.35) (0.30) (0.23) (0.13) Advanced Degree 0.4133 0.5187 0.6241 0.7295 0.8350 0.9404 (0.24) (0.34) (0.45) (0.59) (0.75) (0.93) ______________________________________________________________________________ Student Fixed Effects Yes Yes Yes Yes Yes Yes Teacher Fixed Effects Yes Yes Yes Yes Yes Yes School Fixed Effects Yes Yes Yes Yes Yes Yes ______________________________________________________________________________ Number of Students 47,442 47,442 47,442 47,442 47,442 47,442 Number of Observations 74,196 74,196 74,196 74,196 74,196 74,196 ______________________________________________________________________________


Correlation of Estimated Teacher Correlation of Estimated Teacher Effects From Models with Varying Effects From Models with Varying

Degrees of Persistence Degrees of Persistence

Restricted Value-Added (Partial Decay) Contem- Value Added poraneous =1 =.8 =.6 =.4 =.2 =0 =1.0 1.0000 =0.8 0.9967 1.0000 =0.6 0.9828 0.9946 1.0000 =0.4 0.9492 0.9717 0.9910 1.0000 =0.2 0.8813 0.9169 0.9534 0.9852 1.0000 =0.0 0.7612 0.8115 0.8679 0.9266 0.9773 1.0000 ______________________________________________________________________________


Restricted Value-Added Model EstimatesRestricted Value-Added Model EstimatesWith Differing Controls for Teacher With Differing Controls for Teacher

Heterogeneity Heterogeneity Time-Invariant Teacher Teacher Fixed Explanatory Variable Characteristics Effects 0 Years of Experience -3.8087*** -1.5874 (3.62) (0.49) 1 Year of Experience -0.4017 2.4501 (0.58) (1.07) 2-4 Years of Experience -2.3244*** 0.1362 (3.56) (0.08) Content In-service Hourst -0.0104 -0.0153 (0.91) (0.76) Content In-service Hourst-1 0.0151 0.0210 (1.32) (1.00) Content In-service Hourst-2 0.0116 0.0511** (0.93) (2.16) Content In-service Hourst-3 0.0018 0.0127 (0.11) (0.47) Advanced Degree 1.2414** 0.0812 (2.45) (0.04) Student Fixed Effects Yes Yes Teacher Fixed Effects No Yes School Fixed Effects Yes Yes Number of Students 45,914 45,914 Number of Observations 70,437 70,437


Restricted Value-Added Model Restricted Value-Added Model EstimatesEstimates

With Differing Classroom/School With Differing Classroom/School Controls Controls

Peers,Class Size, Peers, Class Size, School F.E. No. No No School F.E. Explanatory Variable Included Peer Var. Class Size School F.E. Excluded 0 Years of Experience -2.7495 -2.6024 -2.7354 -2.2279 -1.9576 (0.91) (0.86) (0.90) (0.75) (0.66) 1 Year of Experience 1.7714 1.8920 1.7763 1.8644 2.1128 (0.82) (0.88) (0.83) (0.88) (1.01) 2-4 Years of Experience -0.4575 -0.4340 -0.4518 -0.4871 -0.3328 (0.29) (0.27) (0.28) (0.31) (0.22) Content In-service Hourst -0.0177 -0.0171 -0.0176 -0.0185 -0.0173 (0.92) (0.89) (0.92) (0.98) (0.92) Content In-service Hourst-1 0.0216 0.0190 0.0218 0.0209 0.0182 (1.07) (0.94) (1.09) (1.05) (0.92) Content In-service Hourst-2 0.0517** 0.0501** 0.0521** 0.0529** 0.0516** (2.37) (2.30) (2.40) (2.46) (2.41) Content In-service Hourst-3 0.0100 0.0127 0.0101 0.0097 0.0122 (0.42) (0.53) (0.42) (0.41) (0.52) Advanced Degree 0.4133 0.3020 0.3965 0.4384 0.3039 (0.24) (0.18) (0.23) (0.26) (0.18) ______________________________________________________________________________ F-Test on Constraints 2.53** 0.90 0.79 0.83 ______________________________________________________________________________ Student Fixed Effects Yes Yes Yes Yes Yes Teacher Fixed Effects Yes Yes Yes Yes Yes School Fixed Effects Yes Yes Yes No No ______________________________________________________________________________ Number of Students 47,442 47,442 47,442 47,442 47,442 Number of Observations 74,196 74,196 74,196 74,196 74,196 ______________________________________________________________________________


Correlation of Estimated Teacher Correlation of Estimated Teacher Effects From Models with Differing Effects From Models with Differing

Classroom/School Controls Classroom/School Controls

Peer Var., Peer Var., Class Size, Class Size, School F.E. No. No No School F.E. Included Peer Var. Class Size School F.E. Excluded Peer Var., Class Size, School F.E. 1.0000 No Peer Variables 0.9540 1.0000 No Class Size Variable 0.9134 0.9355 1.0000 No School Fixed Effects 0.5143 0.5365 0.5050 1.0000 No Peer, No Class Size, No School F.E. 0.5126 0.5318 0.5016 0.9880 1.0000 ______________________________________________________________________________


Restricted Value-Added Model Restricted Value-Added Model EstimatesEstimates

with Differing Controls for Student with Differing Controls for Student Heterogeneity Heterogeneity

Time-Invariant Student Student Diff. Between Student Fixed Random Fixed and Explanatory Variable Characteristics Effects Effects Random Effects 0 Years of Experience -2.8612*** -2.7346 -2.8788*** 0.1442 (2.70) (0.90) (2.71) (0.07) 1 Year of Experience -1.9354** 1.7988 -1.9571** 3.7560** (2.54) (0.84) (2.56) (2.48) 2-4 Years of Experience -1.8139*** -0.4381 -1.8316*** 1.3936 (3.12) (0.28) (3.15) (1.28) Content In-service Hourst -0.0057 -0.0181 -0.0064 -0.0117 (0.77) (0.95) (0.87) (0.88) Content In-service Hourst-1 -0.0044 0.0215 -0.0048 0.0264* (0.58) (1.07) (0.64) (1.85) Content In-service Hourst-2 0.0139* 0.0515** 0.0131* 0.0384** (1.77) (2.36) (1.68) (2.43) Content In-service Hourst-3 -0.0050 0.0102 -0.0060 0.0161 (0.58) (0.42) (0.69) (0.96) Advanced Degree 0.1584 0.4168 0.0575 0.3592 (0.24) (0.24) (0.09) (0.30) ______________________________________________________________________________ Student Fixed Effects No No Yes Teacher Fixed Effects Yes Yes Yes School Fixed Effects Yes Yes Yes ______________________________________________________________________________ Number of Students 47,435 47,435 47,435 Number of Observations 74,187 74,187 74,187 ______________________________________________________________________________


Correlation of Estimated Teacher Effects Correlation of Estimated Teacher Effects From Models With Differing Controls for From Models With Differing Controls for

Student Heterogeneity Student Heterogeneity

Student Student Time-Invariant Fixed Random Student Effects Effects Characteristics Student Fixed Effects 1.0000 Student Random Effects 0.4514 1.0000 Time-Invariant Student Characteristics 0.3926 0.9498 1.0000 ______________________________________________________________________________


Restricted Value-Added Model Estimates --Restricted Value-Added Model Estimates --Teacher-Specific vs. Within-School Grade-Teacher-Specific vs. Within-School Grade-

Level Averages Level Averages Within-School Grade- Teacher-Specific Level Average Teacher Explanatory Variable Characteristics Characteristics 0 Years of Experience -2.7304 -0.0499 (0.90) (0.01) 1 Year of Experience 1.7819 4.3829 (0.83) (1.23) 2-4 Years of Experience -0.5060 -2.3871 (0.32) (0.71) Content In-service Hourst -0.0185 -0.0454 (0.96) (0.75) Content In-service Hourst-1 0.0214 -0.0761 (1.06) (1.37) Content In-service Hourst-2 0.0534** 0.0251 (2.44) (0.38) Content In-service Hourst-3 0.0112 0.0679 (0.47) (0.87) Advanced Degree 0.4881 1.0405 (0.28) (0.36) Student Fixed Effects Yes Yes Teacher Fixed Effects Yes No School Fixed Effects Yes No School-by-Year Fixed Effects No Yes Grade-by-School Fixed Effects No Yes Number of Students 47,404 47,404 Number of Observations 74,013 74,013


Summary of FindingsSummary of Findings

Model SelectionModel Selection Restricted value-added model seems to be a good Restricted value-added model seems to be a good

approximation of the full cumulative modelapproximation of the full cumulative model SpecificationSpecification

Use of student and teacher fixed effects (rather Use of student and teacher fixed effects (rather than covariates) importantthan covariates) important

Random effects may yield inconsistent estimatesRandom effects may yield inconsistent estimates Important to include school fixed effects, but Important to include school fixed effects, but

classroom peer variables relatively unimportantclassroom peer variables relatively unimportant Aggregation to the grade level has some effect, Aggregation to the grade level has some effect,

though estimates not radically different from though estimates not radically different from estimates with teacher-level data estimates with teacher-level data

Documents

-- Preliminary, Do Not Quote Without Permission -- VALUE-ADDED MODELS AND THE MEASUREMENT OF TEACHER QUALITY Douglas HarrisTim R. Sass Dept. of Ed. LeadershipDept