1

Teacher Performance Pay: Experimental Evidence from India

Karthik MuralidharanUniversity of California, San Diego

(with Venkatesh Sundararaman, World Bank)

Workshop on Accountability in Education

World Bank, 22 June 2009

2

Context• Very low levels of learning in India

-~60% of children aged 6-14 in India cannot read a simple paragraph, though 95% enrolled in school (PRATHAM, 2008)

• Large inefficiencies in delivery of education - In India, 25% teachers are absent, less than half are teaching -Over 90% of non-capital spending goes to teacher salaries-Teachers are very well paid

8 (~ 4 * GDP/Capita); Pay = f (rank, experience)-No performance-based component-Correlations suggest that higher ‘levels’ of pay are not associated with better teacher performance

-Strong unions, almost impossible to fire (only 1 in 3000 schools reported firing a teacher for repeated absence)

• Performance pay for teachers is a frequently suggested way for improving school quality (being tried in many countries) – but limited evidence on effectiveness- Identification of the causal impact of teacher performance pay on learning outcomes is a central limitation

3

Location of Study• Indian State of Andhra

Pradesh (AP)-5th most populous state of India

8 Population of 80 Million -23 Districts (2-4 Million each)

• Close to All-India averages on many measures of human development

India APGross Enrollment (6-11) (%) 95.9 95.3

Literacy (%) 64.8 60.5

Teacher Absence (%) 25.2 25.3

Infant Mortality (per 1000) 63 62

4

Questions/Contributions

• Does teacher performance-pay improve test scores?

• What, if any, are the negative consequences?

• Should they be at the school or teacher level?

• How does teacher behavior change?

• How cost effective is the incentive program?

• How will teachers respond to the idea?

5

Theoretical Considerations• Incentives and intrinsic motivation

-Large literature in psychology on the possibility of crowding out of intrinsic motivation due to financial incentives

-Might be especially relevant to teaching-But bigger problem here may have been the complete lack of differentiation between high and low performing teachers

-Kremer, Muralidharan et al (2006) find that teachers with higher job satisfaction were more likely to be absent

• Incentives and multi-tasking-Teaching basic as opposed to higher-order skills (Holmstrom, Milgrom 1991)

-Test preparation instead of longer-term learning (Glewwe et al 2003)

-Manipulating test-taking population (Jacob 2005)-Short-term boosting of caloric content (Figlio & Winicki, 2005)-‘Cheating to the test’ (Jacob & Levitt, 2003)

6

Addressing Concerns About Incentives on Test Scores

• Crowding out intrinsic motivation-Framing matters-“Teacher recognition” as opposed to “school accountability”

• Teaching to the test/effort diversion-Less of a concern given extremely low levels of learning (Lazear, 2006)-Test-taking is an important skill in the Indian context -Test design can get progressively more sophisticated so that you cannot do well on the test without deeper knowledge/understanding

• Threshold effects-Minimized by making bonus a function of average improvement of ALL students

• Teachers neglecting/penalizing weaker children-Mitigate/avoid this by tying incentives to “changes” from the baseline performance and assigning low scores to drop outs

• Cheating/Paper leaks etc.-Potentially a big problem – here the testing is done by an independent outside testing team with no connection to the school

7

• Teachers were given bonus payments over and above their regular salary on the basis of average improvement of test scores of all students in grade/school over base line-Subjects considered were math and language-Assessment papers were designed by an independent testing agency (EI)-All assessments were conducted by an independent NGO (APF)

• Bonus formula -Rs. 500 bonus for every 1% point improvement in average scores-Calibrated to be around 3% of annual pay (and equal to input treatments)

• Based on absolute improvement (contract/piece rate) as opposed to competing for a fixed prize (tournament)-Contracts dominate for risk-averse agents-Greater transparency of incentive system under contracts-Common shocks small relative to school/class level shocks-Same information costs-Easier to dynamically game tournaments that reward value addition

• Both group and individual level incentives were studied-Free-riding/Peer monitoring/Gains to cooperation

Incentive Design

8

Design Overview

 

INCENTIVES (Conditional on Improvement in Student Learning)

INPUTS (Unconditional)

  NONE GROUP BONUS INDIVIDUAL BONUS

NONE CONTROL (100 Schools) 100 Schools 100 Schools

EXTRA CONTRACT TEACHER

100 Schools   

EXTRA BLOCK GRANT

100 Schools   

9

Sampling

10

Summary of Experimental Design• Study conducted across a representative sample of 500 primary

schools in 5 districts of AP

• Conduct baseline tests in these schools (June/July 05)

• Stratified random allocation of 100 schools to each treatment (2 schools in each mandal to each treatment) (August 05)

• Monitor process variables over the course of the year via unannounced monthly tracking surveys (Sep 05 – Feb 06)

• Conduct 2 rounds of endline tests to assess the impact of various interventions on learning outcomes (March/April 06)

• Interview teachers after program but before outcomes are communicated to them (August 06)

• Provide bonus payments and communicate continuation of program (Sept 06)

11

Specification

)(,,, DistrictSubMandalmSchoolkClassjChildi

ijkjkkmijkmijkm ZIncentivesBLTELT )()(

12

Impact of Incentives on Test Scores

Year 1 on Year 0

Year 2 on Year 1

Year 2 on Year 0

Year 1 on Year 0

Year 2 on Year 0

Year 1 on Year 0

Year 2 on Year 0

[1] [2] [3] [4] [5] [6] [7]

Normalized Lagged Test Score 0.5 0.553 0.45 0.49 0.418 0.516 0.484(0.013)*** (0.016)*** (0.015)*** (0.017)*** (0.022)*** (0.014)*** (0.014)***

Incentive School 0.153 0.143 0.217 0.188 0.277 0.119 0.158(0.042)*** (0.035)*** (0.047)*** (0.049)*** (0.055)*** (0.038)*** (0.043)***

Observations 68702 78613 49516 34121 24592 34581 24924R-squared 0.29 0.29 0.23 0.28 0.22 0.32 0.25

Table 2: Impact of Incentives on Student Test Scores

Combined Maths TeluguDependent Variable = Normalized End of Year Test Score

13

Distribution of Gains by Treatment

0.1

.2.3

.4pd

f of t

est s

core

gai

ns d

istri

butio

n af

ter

Y2

-5 0 5Normalized Test Score Gains Y2

Incentive Schools

Control Schools

0.2

.4.6

.81

cdf o

f tes

t sco

re g

ains

dis

tribu

tion

afte

r Y2

-5 0 5Normalized Test Score Gains Y2

Incentive Schools

Control Schools

14

Impact of Incentives by Mechanical/ConceptualDependent Variable = Endline Test Score by Mechanical/Conceptual

(Normalized by Mechanical/Conceptual Distribution in Control Schools)

Year 1 on Year 0 Year 2 on Year 0Mechanical Conceptual Mechanical Conceptual

[1] [2] [3] [4]

Normalized Baseline Score 0.481 0.336 0.446 0.306(0.012)*** (0.011)*** (0.013)*** (0.013)***

Incentive School 0.135 0.135 0.167 0.178(0.038)*** (0.043)*** (0.041)*** (0.046)***

Observations 68289 68289 42884 42884R-squared 0.28 0.17 0.24 0.15Notes:* significant at 10%; ** significant at 5%; *** significant at 1%

15

Performance on Non-Incentive SubjectsNormalized Endline Score

Year 1 on Year 0 Year 2 on Year 0

Science Social Studies Science Social

Studies[1] [2] [3] [4]

Normalized Baseline Math Score 0.214 0.222 0.155 0.166(0.019)*** (0.018)*** (0.023)*** (0.023)***

Normalized Baseline Language Score 0.206 0.287 0.214 0.182(0.019)*** (0.019)*** (0.024)*** (0.024)***

Incentive School 0.107 0.135 0.112 0.177(0.052)** (0.047)*** (0.045)** (0.049)***

Observations 12011 12011 9166 9166R-squared 0.26 0.3 0.18 0.18Notes:Social Studies and Science tests were only administered to grades 3 to 5* significant at 10%; ** significant at 5%; *** significant at 1%

16

Group versus Individual IncentivesDependent Variable = Normalized Endline Test Score

Y1 on Y0 Y2 on Y1 Y2 on Y0Combined Combined Combined Maths Telugu

[1] [4] [7] [8] [9]

Normalized Lagged Score 0.5 0.554 0.451 0.418 0.485

(0.013)*** (0.016)*** (0.015)*** (0.022)*** (0.014)***

Individual Incentive School (II) 0.160 0.198 0.271 0.321 0.223

(0.049)*** (0.044)*** (0.058)*** (0.068)*** (0.053)***

Group Incentive School (GI) 0.146 0.087 0.162 0.232 0.092

(0.050)*** (0.045)* (0.058)*** (0.071)*** (0.052)*

Observations 68702 78613 49516 24592 24924F-Stat p-value (Testiing GI = II) 0.78 0.05 0.12 0.29 0.03

R-squared 0.29 0.3 0.23 0.23 0.25

Notes:* significant at 10%; ** significant at 5%; *** significant at 1%

17

How did Teacher Behavior Change?

-0.110 **

0.124 ***

0.102 ***

0.085 **

0.091 ***

0.181 ***

0.111 ***

0.017

Teacher Behavior Incentive Schools

Control Schools

p-Value of Difference

Incentive versus Control Schools (All figures in %)Table 11: Teacher Behavior (Observation and Interviews)

What kind of preparation did you do? (UNPROMPTED) (% Mentioning)

Did you do any special preparation for the end of year tests? (% Yes) 0.000***

Actively Teaching at Point of Observation (%) 0.57

0.24

0.10

Coefficient of behavior indicator on test score

Extra Homework 0.000***

Teacher Absence (%) 0.82

Extra Classes/Teaching Beyond School Hours 0.000***

0.42

0.25

0.15

0.17

0.04

Extra Classwork 0.000***

Gave Practice Tests 0.000***

0.24

0.44

0.63

0.42

0.46

0.16

0.31

Paid Special Attention to Weaker Children 0.000***0.21 0.05

18

Comparison of Inputs and Incentives

Combined Math Language Combined Math Language Combined Math Language[1] [2] [3] [4] [5] [6] [7] [8] [9]

Incentives 0.154 0.183 0.121 0.145 0.199 0.091 0.217 0.277 0.158(0.041)*** (0.049)*** (0.038)*** (0.036)*** (0.044)*** (0.033)*** (0.048)*** (0.056)*** (0.045)***

Inputs 0.096 0.11 0.082 0.047 0.047 0.047 0.084 0.092 0.076(0.037)*** (0.042)*** (0.036)** (0.03) (0.04) (0.03) (0.043)* (0.049)* (0.042)*

Difference (Incentives - Inputs)

0.06 0.07 0.04 0.10 0.15 0.04 0.13 0.19 0.08

F-Stat p-value (Inputs = Incentives)

0.09 0.08 0.23 0.01 0.00 0.17 0.00 0.00 0.04

Observations 111811 55542 56269 119836 59820 60016 82596 41053 41543R-squared 0.29 0.27 0.32 0.29 0.26 0.33 0.21 0.21 0.23

Table 10: Impact of Inputs versus Incentives on Learning OutcomesDependent Variable = Normalized Endline Test Score

Year 1 on Year 0 Year 2 on Year 1 Year 2 on Year 0

19

Teachers Liked the Program• Teachers interviewed in August each year (before they know

outcomes)

• 75% of teachers say the program increased their motivation -25% say their motivation was unchanged

• 85% of teachers had a favorable opinion about the idea of bonus payments on the basis of improvement in student performance

• 68% thought that the government should try and scale up this program in all schools

• 75% were willing to accept a performance-pay system even under neutrality of the total wage bill

• Teachers who show greater support for performance-pay (ex ante) are also likely to have performed better (ex post)-Implications for sorting into teaching profession

20

Summary of Results (Performance Pay)• Incentive schools perform significantly better (0.22 SD)

- Improvements are across the board (all grades, districts, baseline scores)- Incentive school score distribution FOSD the control school score distribution-Limited evidence of heterogeneous treatment effects

• Children in incentive schools perform better on mechanical and conceptual components of test, and also on non-incentive subjects

• No difference between group and individual incentives in the first year – but in the second year, the individual incentives start outperforming the group incentives

• Teacher absence does not change, but incentive school teachers report higher levels of teaching activity conditional on attendance-These differences in behavior are correlated with learning outcomes

• Much more cost effective than inputs of the same value

• Teachers really liked the program – ex ante preference lines up with ex post performance

21

Conclusions/Policy Implications• Performance pay for teachers is likely to be a highly cost-effective

policy for improving learning outcomes-2 years of data suggests unlikely to be a ‘novelty effect’-Continued gains on both mechanical and conceptual components as well as non-incentive subjects suggests that distortions from multi-tasking are less of a concern in a context of very low levels of learning

• Can be largely cost/budget neutral when implemented in the context of an across the board salary increase

• The broader point is that of creating a meaningful career ladder for teachers so that their professional trajectories depend on performance (can experiment with including other measures so the weight on test-score gains is not 100%)

• Implementation details are critical and any successful scale up will need to build systems and infrastructure to do this

22

Contract Teachers: Experimental Evidence from India

(Highly Abbreviated)

Karthik MuralidharanUniversity of California, San Diego

(with Venkatesh Sundararaman, World Bank)

Workshop on Accountability in Education

World Bank, 22 June 2009

23

Background• Large scale expansion of primary education in developing

countries (MDG’s, EFA, SSA, etc)•Has led to significant increases in access and enrollment•But has also led to difficulties in maintaining and improving school quality•~60% of 6-14 age cannot read at 2nd grade level though ~95% enrolled

• Hiring and deploying enough teachers has been a big challenge•Fiscal difficulties (teacher salaries are ~90% of education spending)•Logistical challenges (the most qualified teachers are less willing to be deployed to underserved areas)

• A common response has been to staff unfilled teaching positions with locally-hired contract teachers (not civil-service employees)

• Main characteristics of contract teachers include:•Fixed-term renewable contracts – no/limited job security•Typically less qualified and much less likely to be formally trained•More likely to be from the local area (and hired by school committees)•Typically paid much lower salaries•Also more likely to be female (by product - not by design)

• Different countries’ contract teacher systems have various combinations of the above, while in India all of these are typical

24

Regular vs. Contract Teachers (in sample)

Table 1: Characteristics By Teacher Type

Panel A: Regular versus Contract Teachers (Comparison Schools)

Regular Teachers Contract Teachers P-value (H0: Diff=0)

Male 63.1% 31.8% 0.004***

Age 40.35 25.81 0.000***

College Degree or Higher 84.3% 45.5% 0.000***

Formal Teacher Training Degree or Certificate 98.3% 9.1% 0.000***

Received any Training in last twelve months 93.5% 54.5% 0.000***

From the same village 7.2% 81.8% 0.000***

Distance from home to school (km) 11.9 1.1 0.000***

Teacher Salary (Rs./month) 8698 1000 (1500) 0.000***

25

Motivation/Contributions• The use of contract teachers is probably the most prominent policy

innovation in primary education in the past 20 years in India and other developing countries-Several countries use or have used some form of contract teachers (Cambodia, Indonesia, Kenya, Mali, Nicaragua, Niger, Togo, etc)

-Nearly 25% of teachers in large Indian states of UP, Bihar, MP (Popn. >300M)-This is the stock of contract teachers – the flow is even higher

• The use of contract teachers is highly controversial-Supporters: Cost-effective, superior incentives (attendance)-Opponents: Untrained teachers won’t lead to learning, de-professionalizes teaching, reduces prestige of teaching and the motivation of all teachers

• Limited evidence on effectiveness of contract teachers

• We present experimental evidence on the impact of an additional contract teacher on student learning outcomes

• First experimental study of an “as is” expansion of the current contract teacher policy in a large Indian state

• Random assignment in a representative sample of schools

26

Summary of Results (Contract Teachers)• The provision of an extra contract teacher significantly improved

average test scores in program schools in both years-After 2 years, students in program schools scored 0.16 SD higher on math and 0.10 SD higher on language across all grades

• Consistent evidence of some heterogeneous treatment effects-Students in remote schools, smaller schools, and schools with poorer facilities benefit more from the extra contract teacher (these are correlated)

-The youngest students (grade 1) benefit the most in both years of the program-But no heterogeneity based on baseline score (students at all levels benefit)

• Contract teachers seem to have better incentives than regular teachers on multiple dimensions

-Much lower rates of absence and higher rates of teaching activity-Also, contract teachers with lower rates of absence in a given year have a higher probability of having their contracts renewed the next year

• We collect data on teachers in private schools in the same districts and find that private school teacher profiles are a lot closer to contract teachers than regular teachers on several dimensions

-Suggests that expanded use of contract teachers would move education production closer to the efficient frontier

27

Salary Distribution by School and Teacher Type

00.5

11.5

22.5

33.5

44.5

55.5

66.5

77.5

88.5

99.5

10

10.5

11

11.5

12

12.5

13

13.5

14

14.5

15

15.5

16

16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

21.5

22

22.5

23

23.5

24

24.5

0

5

10

15

20

25

30Government Schools

ECTRTHT

0 0.5

11.5

2 2.5

33.5

4 4.5

5 5.5

6 6.5

77.5

8 8.5

99.5

10

10.5

11

11.5

12

12.5

13

13.5

14

14.5

15

15.5

16

16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

21.5

22

22.5

23

23.5

24

24.5

0

5

10

15

20

25

30Private Schools

RTHT

Teacher Salary in Bins of 500 Rs. Per Month

Per

cent

age

28

Policy Implications (1 of 2)• Expanding contract teacher programs (at the margin)

may be a cost effective way of improving learning outcomes in primary education in developing countries

• The relevant policy comparison is not one regular versus one contract teacher, but one regular teacher versus several contract teachers-Several developing countries have focused on policies to get highly qualified teachers to move to backward areas

-It may be more effective to hire several local contract teachers as opposed to an additional regular teacher – especially for primary schooling

• Private schools in rural India mostly follow this model-Expanded use of contract teachers could get schools closer to the productive efficiency levels of private schools while preserving the public characteristics of schools

29

Policy Implications (2 of 2)• Three main concerns about expanding use of contract teachers

-De-professionalizing education by promoting use of untrained teachers-Sustainable of such a two-tier system would be problematic-Political economy concerns – once you have enough contract teachers, they will all lobby to become civil-service teachers and the incentive and cost benefits are both lost

• One possible solution is to create a career ladder for teachers whereby all new teachers are hired locally as contract teachers, and the best ones become eligible for bonuses/promotions on the basis of performance over a period of 6 – 10 years-Modular training over a period of 3 years over the summers-Continuous measurement of performance during this period-Adjunct versus tenure-track faculty model

• Addresses most of the concerns by integrating the two types of teachers into a career track as opposed to creating a permanent cadre of non-professional teachers

• Combines our results from both projects/papers