Evaluation Evidence on English Ed i P li iEducation Policies: What Can be Learnt?What Can be Learnt?

Sandra McNallyUniversity of Surrey and 

Centre for Economic Performance, ,London School of Economics

OutlineOutline

• School Resources

• Pedagogy

• Choice and competition• Choice and competition

• School Autonomy

• Some conclusions

What works (or not) in schools: CEP research Policy Sector Project Positive Little or

no effect Additional comments

Increasing overall school

Primary Holmlund, McNally,Viarengo (2009) √

Holmlund et al. find modest effects. overall school expenditure

Gibbons, McNally,Viarengo (2011)

√ Estimated effects (for urban schools) much larger in Gibbons et al.

Excellence in Cities

Secondary Machin, McNally and Meghir (2010) √ Moderate effects but cost-effective; Highest effect for most able pupils in most disadvantaged schools

Li dLiteracy and Numeracy Strategies

Primary Machin and McNally (2008; 2010) √

Moderate effects at very low cost.

Academies

Secondary Machin and Wilson (2009) Machin and Vernoit (2011)

√

Machin and Wilson (2009) found no effect in short-term Machin and Vernoit (2011) find positive effects forMachin and Vernoit (2011)

Machin and Silva (2013)

√ Machin and Vernoit (2011) find positive effects for schools that have been in programme for some time. But Machin and Silva (2013) find no effects in the tail of the distribution

Choice and competition

Primary Gibbons, Machin, Silva (2006, 2009)

√

Suggestion that small effect might be possible in schools with greater autonomy

Special Educational

Primary Keslair, Maurin, McNally (2012) √ No effect (or even a negative effect)

Needs  

Evidence on the effects of school expenditure (Gibbons, McNally, Viarengo)

E id f l ff t f h l di hi t f• Evidence for causal effects of school spending on achievement from LA‐income differentials and Area Cost Adjustments

• £1000 per student school spending (per year) raised achievement by ~0.25 s.d. age 11

• Spending more effective in schools with more ‘disadvantaged’ students

Ch i P dChanging Pedagogy:The Literacy Houry

• National Literacy Strategy introducednationwide in September 1998(so first year of testing in Summer 1999)(so first year of testing in Summer 1999).

• Sets out a framework for teaching: termlyhi bj i fteaching objectives for 5-11 age range;

provides a practical structure of time andclass management for a daily literacy hour.

Some results

• Policy seemed to raise performance: About 0.091 of a standard deviation.

• Compare the per pupil costs of the policy i h h i b fi fl d iwith the economic benefits, as reflected in

predicted labour market earnings.

Cost-Benefit Calculations

• Translate test gains to earnings impact using h d Th di d l fanother data set. The present discounted value of

the cumulative effect of the literacy hour is b £2 103 d £5 476between £2,103 and £5,476

• The main costs were 14 local centres and literacy l Th l hconsultants. The total cost per annum was thus

£2.5 million (covering 222,261 pupils in aggregate) The cost per pupil is £25 52 peraggregate). The cost per pupil is £25.52 per annum.

Choice and Competition: Rationale (in economic terms)

hi f d d h l• Better matching of students and schools (based on the tastes and needs of the 

/student/family and what the school provides)

• Facilitates competition between schools as a mechanism to raise standards (i e parentsmechanism to raise standards (i.e. parents given freedom to choose school ‐ puts pressure on schools to attract students if theirpressure on schools to attract students if their funding is linked to student numbers).

School choice and other policiesSchool choice and other policies

• Usually school choice is linked to competition.y p• Can also be linked to school accountability (in England, the publication of ‘league tables’ of school g , p gperformance).

• Recently, in many countries, school choice has y, y ,increased as a result of the creation of new (publicly funded) schools that depart from the traditional model (characterised by their autonomy). 

• Effects of school choice may be different depending on polices relating to accountability and school autonomy.

Competition choice and attainmentCompetition, choice and attainmentGibbons, Machin and Silva (2008)

• Pupil Choice: Do children who with more choice show

faster educational progress?faster educational progress?

S h l C titi b t h l D h l th t• School Competition between schools: Do schools that

enrol children who have more choices perform better?

• They study London and surrounding area using pupil

census data for 2002 and 2003 on tests in primary school

at age 7 and age 11: ‘Value Added’

1. Geocode school and pupil home addresses

Pupil homesp

School

2. Calculate median commuting distance (straight line)

Pupil homesp

School‘Travel area’: circleat median distance

3 Count how many schools each pupil can reach if they3. Count how many schools each pupil can reach if they travel no more than the median commute to each school

23

22 1

= pupil choice index1 01

0

4. Calculate school competition index = mean of the ppupils’ choice index by school attended

23

22 1 00

1 0

0

01

0 0

Non-competitive: =0Competitive: =1.25

School Competition index

4 55-66-7

1-22-33-44-5

0-1

The strategy

• Local Authority (LA) boundaries create discontinuities in

school choice:school choice:

– Pupils living near LA boundaries have less choice than pupils

in the centrein the centre

– Schools near LA boundaries face less competition from other

schools than schools in the centre

• Use distance from LA boundary to predict choice (home)• Use distance from LA boundary to predict choice (home)

and competition (school)

(B i f I t t l V i bl St t )(Basis for Instrumental Variable Strategy)

FindingsFindings

• Pupils with more choices do no better than pupils with

restricted choicesrestricted choices

• Pupil achievement is marginally higher in schools that

i iappear more competitive

– Correlation attributable to Faith schools only

• Isolated schools close to LA boundaries face less

competition, but perform (insignificantly) better than more

central schools

Choice and Competition – Summary

• The evidence from this English study (like many others)i it i diffi lt t d t t ff t f h i dis it is difficult to detect effects of choice andcompetition on achievement, but their may be an effecton inequalityon inequality.

• See also the large literature on house prices and school• See also the large literature on house prices and schoolquality, which shows that parents are willing to paysignificant amounts more to buy houses in catchmentg yareas with better performing schools. This willingnessto pay shows a significant parental demand for placesi i l h lin particular schools.

School Types – Autonomy and Governance

• More recent reforms have increased the variety ofMore recent reforms have increased the variety ofeducation through different school types.

• Examples are:Examples are:

- Charter schools in the US;Charter schools in the US;- Free schools in Sweden;- Academies in England- Academies in England.

School Types – Autonomy and Governance

These school types have much more autonomy thanThese school types have much more autonomy thanstandard state/public schools.

Their governance structures are also more varied , withless role from the state and more from the privatepsector.

They were introduced recently, so there is not a largebody of research, but there is significant research andpolicy interest in whether different school types canenhance performance.

School Types – Some Selected StudiesStudy Question and Approach Context Findings

1). Abdulkadiroğlu et al (2011)a

Look at the impact of charter school attendance on achievement. Identification is from student

US - charter schools in Boston.

i) Large, significant gains in test scores for pupils in charter schools.ii) Larger effects where assignmentIdentification is from student

assignment lotteries.ii) Larger effects where assignment lotteries are binding.

2) Bohlmark and Did the infl of free schools S eden i) Moderate positi e impact of free2). Bohlmark and Lindahl (2007)b

Did the influx of free schools enhance academic achievement. Adopt a differences-in-differences approach before and after the reforms

Sweden -municipality panel data, 1988/9 to 2002/3.

i) Moderate positive impact of free school growth on municiplaity academic performance.ii) Bigger effects for children from

to compare areas with large growth in free schools to those with less growth.

highly educated families; no effects for low education families and immigrants.

3). Machin and Vernoit (2011)c

The introduction of Academy schools to English education. Uses future Academies as control group.

England - secondary schools in the 2000s

i) Quality of pupil intake improves relative to predecessor schools.ii) Modest positive effect on g p ) pachievement.iii) Bigger effects for schools making bigger autonomy movements.

Notes: a - Abdulkadiroglu, A., J. Angrist, S. Dynarksi, T. Kane and P. Pathak (2011) Accountability and Flexibility in Public Schools: Evidence From Boston's Chaters and Pilots, Quarterly Journal of Economics. b - Bohlmark, A. and M. Lindahl (2007) The Impact of School Choice on Pupil Achievement, Segregation and Costs: Swedish Evidence, IZA Discussion Paper. c - Machin, S. and J. Vernoit (2011) Changing School Autonomy: Academy Schools and their Introduction to England's Education, CEE Discussion Paper.

Chapter by Steve Machin and Olmo Silva in book by Paul Marshall (Google eBook): The Tail: How England's schoolsMarshall (Google eBook): The Tail: How England s schools fail one child in five ? and what can be done

• Machin and Silva (2013): School Structure, ( )School Autonomy and the Tail

• Look at the effects of ‘school academies’ acrossLook at the effects of school academies across the distribution of pupil prior attainment. Outcome is exam at age 16 (GCSE).Outcome is exam at age 16 (GCSE).

• Find no beneficial effects of school academies in the tail of the distribution (all benefits are furtherthe tail of the distribution (all benefits are further up the distribution)

The Effect of Academy Conversion on Pupils of Different Abilities(Machin and Silva, 2012)

The Effect of Academy Conversion on Pupils of Different Abilities(Machin and Silva, 2012)

School Types – Summary

• Early evidence suggests that in some circumstances,there may be beneficial effects from school types thatoffer more autonomy and more diverse governanceoffer more autonomy and more diverse governancestructures.

• But it is early days and evaluations are so far smallscale and so the generalisability of the findings isscale and so the generalisability of the findings issomewhat limited.

ConclusionConclusion

• Resources matter! There is justification for protecting• Resources matter! There is justification for protecting education budgets

• Interventions to affect pedagogy can be very costInterventions to affect pedagogy can be very cost effective (important to evaluate properly in advance)

• Choice and competition have been a blunt instrument toChoice and competition have been a blunt instrument to increase achievement – but this could change in an environment where schools have more autonomy.

• Evidence to date suggests more autonomy could have good effects – but not necessarily for ‘hard to reach’

il D ’t t thi t l ti i ltpupils. Don’t expect this to revolutionise results.