The rise of RCTs in education: statistical and practical challenges

Kevan Collins

The EEF

Summarise the existing evidence

Make grants

Evaluate projects

Share and promote

the use of evidence

Rigorous, independent evaluations

Founded with a grant from the DfE with an

aim to spend over £220 million over 15 years

Focus on projects with a good

evidence-base

115 grants made to date

EEF and Sutton Trust

Toolkit, reports, scale up campaigns

EEF’s Approach to Evaluation

Rigorous, independent evaluations

Independent evaluation

Robust counterfactual

Focus on attainment outcomes

Impact and process

evaluations

A typical EEF evaluation

Recruit schools

Pre-test / pupil data / consent

Post-test

Set-up / Planning

Intervention delivery

Random

ise Process evaluation

When RCT not possibleRDD, propensity score

matching, synthetic analysis

94/115 projects

evaluated using RCTs

Keen to include non-cognitive

outcomes

Focus on attainment (end of KS, standardised commercial tests)

115 grants made to date

All EEF evaluation data will be submitted to a national data archive

This data archive is still being established. The aim is to allow…

….the EEF to:• track the impact of projects longitudinally• look at the cumulative impact of projects• understand better its target group

….the research community to:• verify the results of EEF evaluations• conduct further analysis on subgroups and interventions• link to other datasets for research purposes

Overarching evaluators need access

Research community needs access

Some lessons from our first 4 years…

• Schools are willing to take part in RCTs. When we first set up, some people were sceptical

• But getting them to do what you need them to is difficult. Testing, passing on data, sticking to the intervention, not contaminating the control group…

• Effect sizes are often smaller than anticipated. Maybe due to the independence of the evaluation; the “real world” nature of the trials; intention to treat analysis

• This means that trials need to be very large to detect an impact. Therefore we are increasingly reliant on NPD data rather than testing 1,000s of pupils.

Next big questions: Will schools listen to the evidence? And policy makers? Will researchers use the data archive?