Attracting, Preparing and Retaining Effective Teachers
Hamp LankfordNCSL Seminar on Linking Research and
Policy to Support Effective Teaching
Attracting, Preparing and Retaining Effective Teachers
Research Partners:Don Boyd and Jim Wyckoff, University at AlbanyPam Grossman and Susanna Loeb, Stanford UniversityJonah Rockoff, Columbia University
www.teacherpolicyresearch.org
Overview of Presentation
How can we identify effective teachers?
What is the role of licensure?
How can we prepare teachers to be more effective?
Examining pathways into teachingTeacher preparation and student achievement
Teacher Sorting in NYS Elementary Schools (Math 2002)
Percent of Students at Level 1 Percent of teachers Highest Quartile Lowest Quartile
0-1 years prior 21.3 12.3teaching experience
Failed general knowledge 31.0 10.1or LAST exam
BA from a least 24.0 11.4competitive college
Improving Teaching, Improving Student Outcomes
Student Outcomes
School
teacher quality
leaders
student ability &motivation
facilities
class size
District policies
Teacher preparationpathway
Student and Environment
family
Staterequirements
learningenvironment
neighborhood
teacher certification
peers
Prospectiveteachers
other teachingpolicies
teacher education
Teacher Workforce
salaryhiring
program structure
subject specificfield experiences
preparation fordiverse learners
professionaldevelopment
academic abilityprior experience
teaching
mentoringinduction
retention
Policy Challenge
How do public schools:
Attract potentially excellent teachers to teaching, especially in traditionally difficult-to-staff schools?
Provide the skills and experiences that help potentially excellent teachers develop into excellent teachers in those schools?
Retain strong teachers, especially in traditionally difficult-to-staff schools?
Teacher Qualifications and Student OutcomesIn general, only limited evidence that specific
teacher qualifications make a difference
For example, advanced degrees do not predict greater student achievement,
Qualifications such as certification exam scores have modest effects on student achievement and
Teachers from alternative certification routes perform about as well as those from traditional preparation programs.
However, some qualifications are important:Teacher experience over the first 3-5 years
0
0.04
0.08
0.12
0.16
1 2 3 4 5 6-10 11-15 16-20 > 20Years of NYC Experience
Pro
port
ion
of S
tand
ard
Devi
atio
n
Grade 4-5
Grade 6-8
In addition …
Teachers being certified appears to improve achievement (difficult to measure its effect as we don’t observe a system that does not require certification).
Taken as a group, observable teacher qualifications do appear relevant in distinguishing among more and less effective teachers
Teacher Qualifications and Student Achievement
Research question: How has the gap in teacher qualifications between poor and non-poor schools in NYC changed between 2000 and 2005 and has that affected the gap in student achievement?
Teacher qualifications: certification exam scores, SAT scores, undergraduate college ranking, experience, certification
LAST Exam Failure Rate of Elementary Teachers by Poverty Quartile of School’s Students,
2000-2005
10%
15%
20%
25%
30%
35%
40%
2000 2001 2002 2003 2004 2005
Perc
ent o
f tea
cher
s
Low est quartile 2nd quartile 3rd quartile Highest quartile
LAST Exam Failure Rate of New Teachers by Poverty Quartile of School’s Students, 2000-2005
10%
15%
20%
25%
30%
35%
2000 2001 2002 2003 2004 2005
% o
f new
teac
hers
Lowest quartile 2nd quartile 3rd quartile Highest quartile
Number of New Teachers by Pathway, 2000-2005
0
1,000
2,000
3,000
4,000
5,000
6,000
2000 2001 2002 2003 2004 2005
Num
ber o
f new
teac
hers
College recommeded and other Teaching Fellows and TFA Temporary license
Average Certification Exam Scores on First Taking (2004; Passing= 220,SD=~30)
246
247
267
276
242
200 210 220 230 240 250 260 270 280 290 300
CollegeRecommended
IndividualEvaluation
Teaching Fellow
Teach For America
Temp license
Policies Contributing to Change
In 2000 the NYS Regents created alternative certification routes
In 2000 the NYC Department of Education created its first cohort of Teaching Fellows
Effective September 2003, NYS Regents eliminated temporary licenses for uncertified teachers with only very limited exceptions
Between 2000 and 2003 starting salaries in NYC increased from $33,186 to $39,000
Using Student Achievement Tests to Examine Policies
Individual Student gainsAit - Ai(t-1)= student, class, teacher, school
Typically include about .5 to 1.5 million student tests linked to their teachers Statistical controls to make sure we isolate effects of teacherExamine lots of different estimates to make sure conclusions are consistent
05
1015
Pro
porti
on o
f Tea
cher
s
-.15 -.1 -.05 0 .05 .1 .15 .2Average Impact on Students in Standard Deviations
Rich 2001 Poor 2001Rich 2005 Poor 2005
Changes in Grades 4 & 5 Math Attributable to Teacher Qualifications, Rich and Poor Deciles 2001 & 2005
Effective and Ineffective Teachers in High Poverty Schools
Student VA AchievementLow High
Characteristics of Effective and Ineffective Teachers in High Poverty Schools (bottom quartile, math in grades 4 & 5)
VA Quintile Mean VA
Years Experience
Not Certified
Cert Exam Score
Math SAT
Verbal SAT
1 -0.11 1.78 0.66 236 421 4732 -0.04 4.67 0.33 241 434 4673 -0.01 6.85 0.06 243 434 4684 0.02 6.65 0.02 247 439 4615 0.06 5.87 0.01 252 488 462
Range 0.17 4.09 -0.66 16 67 -11
The Role of Teacher Preparation and Licensure
Preparation and licensure can improve teaching effectiveness if:They distinguish among more and less able teachersThey improve the knowledge and skills of potential teachersThey do not adversely affect the supply of potential teachers, and ifLocal hiring authorities are motivated to hire the most effective teachers
The Role of Teacher Licensure
Several points:Licensure vs. credentialsNo certification vs. out-of-field teachingDifficult to evaluate how licensure affects student outcomes as we do not observe a system without licensure.
The Role of Teacher Preparation
Emerging evidence on whether teacher preparation improves student outcomes
Substantial literature on lack of effect of graduate educationAlternative vs. traditional preparationPreparation coursework linked to student value addedParticular aspects of preparation linked to student value added
Examining Pathways Into Teaching
DEFINING PATHWAY - The route teachers follow into their first teaching job.
College RecommendedIndividual Evaluation (transcript review)NYC Teaching FellowsTeach for America Temporary License (uncertified)Other (i.e., reciprocity, other Transitional B programs)
Effects on Student Achievement by Pathway by Experience, Math, Grades 4-5 (Coll. Rec. 1 yr =0;)
-0.08
-0.04
0
0.04
0.08
0.12
1 2 3
Teacher Experience
Achi
evem
ent s
td. d
ev.
Coll. Rec.FellowsTFATemp. Lic.
*Caution – small sample size for third-year TFA makes interpretation of results tricky
Effects on Student Achievement by Pathway by Experience, Math, Grades 6-8 (Coll. Rec. 1 yr =0;)
*Caution – small sample size for third-year TFA makes interpretation of results tricky
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
1 2 3
T eacher Experience
Ach
ieve
men
t Std
Dev
.
Coll RecFellowsTFATem p Lic
Grades 4-8 Teacher Attrition Following the 1st
Four Years of Teaching, 1999-2003 Cohorts
0%10%20%30%40%50%60%70%80%90%
CollegeRec.
Indiv.Eval.
Fellows TFA Temp
1st yr2nd yr3rd yr4th yr
Adjusted for grade School and Year
Teacher Preparation and Student Achievement – Background
Large literature on Teacher Education
Very little empirical research exploring the link between teacher preparation and student learning
The following is a step in that directionFocusing on childhood educationIn one large urban district, New York City
Questions – For Childhood Education…
What is the distribution of the average value-added of teachers from different preparation programs?
How are features of preparation programs associated with the value teachers add to student achievement gains in math and ELA?
How are teachers’ reported experiences in their preparation associated their value added?
Methods – 3 analysesPrograms /InstitutionsAijst = β0+β1Aijs(t-1)+Xitβ2+Cijstβ3+Tjstβ4+πj+ωs+εijst
Achievement as a function of program/institution (π), plus : prior achievement, student characteristicsclassroom characteristics teacher characteristics (in some specifications)school fixed-effects (in most specifications)
Program Featuressubstitute Pjstβ5 for πj cluster at program level
Reports of Experiencessubstitute Rjstβ6 for πj cluster at teacher level
Data – AdministrativeStudent data
demographic fileexam file: grades 3-8, math and ELAlinked to teachers through homeroom (elementary) or section (middle)
Teacher dataTeacher experience: transaction-level data from the NYCDOE Division of Human ResourcesTeacher demographics: range of data filesTest performance: NYS Teacher Certification Exam History File (EHF). Pathway: teacher certification applications plus separate data files for individuals who participated in Teach For America or the Teaching Fellows ProgramProgram for College-Recommended: NYSED’sprogram completers data files
Data – Programs, spring & summer of 200418 institutions: 26 traditional programs + 4 TF + 1 TFADocument collection, interviews with director and director of field experiences, surveys of math and ELA methods facultyMany measures but here focus mainly on link to practice. Program data are not ideal for this.
whether or not the program required a capstone project(50%)a composite measure of the extent that the program maintains oversight over student teaching experiences
requires a minimum number of years of teaching experience for its cooperating teachers (32%)program picks the cooperating teacher (42%) program supervisor observes their participants a minimum of five times during student teaching (27%)
for comparison math and English content courserequirements (# courses) and the proportion tenure-line faculty (.45)
Data – Survey Data, spring of 2005
All first year teachers in NYC, response rate 70%Survey available at www.teacherpolicyresearch.orgVariables
extent to which preparation included links to practice;opportunities to listen to an individual child read aloud for the purpose of assessing his/her reading achievement; …Plan a guided reading lesson, and …Study or analyze student math work (each 5-point scale).
opportunities to study the New York City curriculum;whether or not the teacher had student teaching experiencesunder supervision of the teacher-of-record in the classroom; the congruence between their student-teaching placement and their current job assignment in terms of subject matter or grade level;
Data – Survey Data, spring of 2005
Additional measures of preparation as controls and comparisons
opportunities to learn about teaching math; …to learn about teaching ELA; …to learn about handling student misbehavior; and …to learn about teaching English language learners.
Programs: Institution Effects, Math (x-axis) and ELA (y-axis), First-Year Teachers 2001-2006
(40 or more teachers with value-added estimates)
-0.08
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10
-0.06 -0.04 -0.02 0.00 0.02 0.04 0.06 0.08 0.10
Program Features – Practice Consistently Positive for First Year Teachers (2005-2006)
Capstone projectMath: 0.12*ELA: 0.10*
Oversight of student teachingMath: 0.12**ELA: 0.10**
Neither significant in year 2* Significant at 0.05; ** at 0.01
Program Features – Other Measures Less Consistent, Effect of Content Requirements small
Math Content CoursesMath: 0.02**(01-06), 0.01(05&06), 0.02*(2nd year)ELA: 0.00 for all samples
ELA Content Courses Math: 0.00 (01-06), -0.03**(05&06), 0.01(2nd year) ELA: -0.01*(01-06), 0.00(05&06), 0.01*(2nd year)
Proportion Tenure-lineMath: 0.12*(01-06), .06 (05&06), .09 (2nd year)
Survey Results – Practice for MathMath:
Opportunities in Practice Variable: (positive for all samples)
0.07** - 0.11** (std dev. 0.51)
NY Curriculum:0.05** - 0.07** (std dev 0.85)
Congruence of Field Experience and JobFirst year: .02** - .04** (std dev 0.80)Nothing second year
Survey Results –Practice not as strong for ELA
ELA:Opportunities in Practice Variable: nothing
NY Curriculum: only significant in college recommended sample
0.04 - 0.07 (std dev 0.85)
Congruence of Student Teaching & Job–not significant except in college recommended
-0.07 to - 0.12 in full and elementary
Survey Results – other variables
Very little elseOpportunities to learn teaching methods in Math or ELA not significantly related and very small point estimatesOpportunities to learn about handling student misbehavior is positive and significant (0.03) in the full model for ELA but no other modelsOpportunities to learn about teaching English learners never significantEmphasis on basic skills negative, sometimes significant for ELA
ConclusionsVariation across programs
Opportunities linked to practice appear to help teachers in their first year
Caveats – only a first stepBig district but small number of programs and teachers within schoolsMay be missing important features, or not measuring them well; developing instrumentsValue-added analysis may eliminate the most important variation (e.g. between schools or very weak preparation)Tests themselves may not proxy well for full range of learning that we care about
Summary: Improving TeachingRecruiting effective teachers
Increasing the supply of potentially effective teachersProviding evidenced-based preparation Improved screening by hiring authorities
Improving the skills of in-service teachersRetain effective teachersRemove ineffective teachers
Portfolio of evaluation methods
Papers, policy briefs, surveys, and other documents available at:
www.teacherpolicyresearch.org
Teacher Labor MarketsDemand for effective teachers
student demographicspolicy initiatives costs of hiring teachersteacher retention
Supply of effective teachersrelative working conditionsrelative compensationcosts of entering teachingpreparation and licensure requirements
Research, Evidence and PolicyEvidence of teacher effectiveness can come in a variety of forms, but ultimately should be linked to outcomes for studentsResearch that links characteristics of teachers to outcomes of their students is in the very early stages.Employing student achievement test results tied to teachers for teacher, administrator or school accountability places enormous pressure on assessments, data systems and statistical methods.States should approach the use of research or accountability systems built on value-added results cautiously as doing so could well have unintended consequences.
Entering NYC Teachers by Pathway: Teaching Fellows Replace Temp License
01,000
2,0003,0004,0005,000
6,0007,0008,000
9,00010,000
1997 1998 1999 2000 2001 2002 2003 2004 2005
Num
ber o
f Tea
cher
s
College Recommended Individual Evaluation Teaching FellowTeach for America Temporary License Other
Average Certification Exam Scores First Taking: (2004; Passing= 220,SD=~30)
246
247
267
276
242
200 210 220 230 240 250 260 270 280 290 300
CollegeRecommended
Individual Evaluation
Teaching Fellow
Teach For America
Temp license
Programs: Effects in Math with no controls (x-axis) and controls (y-axis) for teacher characteristics (y-axis)
-0.10
-0.05
0.00
0.05
0.10
0.15
-0.10 -0.05 0.00 0.05 0.10 0.15
Surveys and related papers:www.teacherpolicyresearch.org
Program Analysis Implications
Variation across institutions.08 > 1st year vs. 2nd year teacher
Variation approximately the same in math and ELA
Positive correlation between math and ELA average value-added (0.65)
Not much difference in effects with and without basic teacher controls
age, gender, race, certification exam scores
Additional Slides
Proportion of Teachers with Fewer than Three Years Experience, NYC Elementary Schools, by
Poverty Quartile of School’s Students, 2000-2005
15%
20%
25%
2000 2001 2002 2003 2004 2005
Perc
ent o
f tea
cher
s
Lowest quartile 2nd quartile 3rd quartile Highest quartile
Research Questions & Samples
Institution and program value-added2004-05 & 2005-06 1st year teachers2000-01 through 2005-06 1st year teachers2000-01 through 2005-06 2nd year teachers
Program features: same as aboveSelf-reported preparation experiences
2004-05 cohort in 1st & 2nd yearSome runs with college recommended, TF and TFA
and some only college recommended; Include random effects rather than school fixed
effects
Program Features: Math, First-Year 2001-06
0.54630.54780.54810.54450.5448R-squared2523224486289803159431594Observations
(0.0075)0.0324***Oversight
(0.0503)0.1184**Percent Tenure
(0.0159)0.0410**Capstone project
(0.0050)-0.0026ELA courses
(0.0062)0.0239***Math courses
Program Features: Math, First-Year 2005&06
0.51860.53410.52550.52610.5258R-squared75816794833589688968Observations
(0.0345)0.1240***Oversight
(0.1242)0.0614Percent Tenure
(0.0545)0.1216**Capstone project
(0.0085)-0.0272***ELA courses
(0.0174)0.0098Math courses
Program Features: Math Second-Year 2001-06
0.54690.55150.55430.54820.5483R-squared2239722697265392855128551Observations
(0.0125)-0.0145Oversight
(0.0805)0.0857Percent Tenure
(0.0221)-0.0077Capstone project
(0.0056)0.0087ELA courses
(0.0091)0.0225**Math courses
Program Features: ELA for First-Year 2001-2006
0.46610.47210.46840.46660.4665R-squared2218221334254712775727757Observations
(0.0073)0.0122Oversight
(0.0338)0.0184Percent Tenure
(0.0112)0.0496***Capstone project
(0.0039)-0.0091**ELA courses
(0.0084)-0.0034Math courses
Program Features: ELA for First-Year 2005&2006
0.46450.46980.46510.45980.4597R-squared67225997739080998099Observations
(0.0387)0.1038**Oversight
(0.0874)-0.0478Percent Tenure
(0.0501)0.1019*Capstone project
(0.0096)-0.0060ELA courses
(0.0200)0.0014Math courses
Program Features: ELA for Second-Year 2001-2006
0.48290.48180.48300.48250.4823R-squared2068120377244612620926209Observations
(0.0138)0.0022Oversight
(0.0548)0.0077Percent Tenure
(0.0178)-0.0271Capstone project
(0.0051)0.0113**ELA courses
(0.0088)0.0011Math courses
Survey - Math
12446Observations0.5442R-squared-0.044No Student Teach x 20060.008misbehavior x 2006-0.02No Student Teaching0.011Exp misbehavior-0.051*FExp: Coherence x 20060.002Basic Skills x 20060.037FExp: Coherence with job-0.032Basic Skills-0.034Math x 2006-0.03NY Curriculum x 2006-0.002Math0.053***NY Curriculum-0.042Exp to teach ELs x 2006-0.058Practice x 20060.011Exp to teach ELs0.106***Practice
Survey - ELA
12612 Observations0.4923 R-squared0.062 No Student Teach x 2006-0.025 misbehavior x 2006
-0.106 **No Student Teaching0.034*Exp misbehavior0.033 FExp: Coherence x 2006-0.046 Basic Skills x 2006-0.012 FExp: Coherence with job-0.047 Basic Skills-0.003 ELA x 2006-0.008 NY Curriculum x 2006-0.004 ELA0.005 NY Curriculum-0.016 Exp to teach ELs x 2006-0.042 Practice x 2006-0.010Exp to teach ELs0.033 Practice
Opportunities to Learn Math Methods
alpha = 0.97In your teacher preparation program, prior to September 2004, how much opportunity did you have to do the following (5-point scale)?
learn typical difficulties students have with place value; learn typical difficulties students have with fractions; use representations (e.g., geometric representation, graphs, number lines) to show explicitly why a procedure works; prove that a solution is valid or that a method works for all similar cases; study, critique, or adapt math curriculum materials; study or analyze student math work; design math lessons; learn how to facilitate math learning for students in small groups; adapt math lessons for students with diverse needs and learning styles; And practice what you learned about teaching math in your teacher preparation program in your field experience.
Opportunities to Learn ELA Methods
alpha = 0.96In your teacher preparation program, prior to September 2004, how much opportunity did you have to do the following (5-point scale)?
learn about characteristics of emergent readers; learn ways to teach students meta-cognitive strategies for monitoring comprehension; learn ways to teach decoding skills; learn ways to encourage phonemic awareness; learn ways to build student interest and motivation to read; learn how to help students make predictions to improve comprehension; learn how to support older students who are learning to read; learn ways to organize classrooms for students of different reading ability; study, critique, or adapt student curriculum materials; learn how to activate students’ prior knowledge; listen to an individual child read aloud for the purpose of assessing his/her reading achievement; plan a guided reading lesson; discuss methods for using student reading assessment results to improve your teaching; and practice what you learned about teaching reading in your field experiences.