Building Evidence in Education:Conference for EEF Evaluators

11th July: Theory12th July: Practice

www.educationendowmentfoundation.org.uk

Panel session 3: Working with schools

Creative solutions: lessons learnt from evaluating the LIT programme

Sarah HaywoodNatCen

Richard Dorsett, NIESREEF Evaluators Conference

12 July 2013

“Mind the Gap”

What is being tested?

• a parental engagement intervention – parents work with their children to create a short

animated film– series of facilitated sessions

• whole-school intervention – Metacognition: training teachers in the principles of

“learning to learn”• Targeted at year 4 pupils in academic year 2012/13• Predicted effect size of 0.35 - 0.45

Randomisation designAll schools:

NS=50

TreatmentNS

1=25

CPD & PENC=25

CPD NC=25

ControlNS

0=25

ControlNC=25

School randomisation

Class randomisation

A B C

Treatment effect 1: CPD & PEAll schools:

NS=50

TreatmentNS

1=25

CPD & PENC=25

CPD NC=25

ControlNS

0=25

ControlNC=25

School randomisation

Class randomisation

A B C

Treatment effect 2: CPDAll schools:

NS=50

TreatmentNS

1=25

CPD & PENC=25

CPD NC=25

ControlNS

0=25

ControlNC=25

School randomisation

Class randomisation

A B C

Treatment effect 3: CPD & PE vs. CPDAll schools:

NS=50

TreatmentNS

1=25

CPD & PENC=25

CPD NC=25

ControlNS

0=25

ControlNC=25

School randomisation

Class randomisation

A B C

Recruitment and randomisation• Birmingham, Devon, Haringey, Manchester• Drop-out is a worry

– 2 controls did so before knowing treatment status– 3 controls, 1 treatment dropped out & substituted– 2 controls dropped out & were not substituted

• Substitute schools – take treatment status of dropouts they replace– excluded from the impact estimates– provide potentially useful supplementary data

• Wanted 2-form entry but not always achieved

Achieved sample

NS=43

TNS

1=24

CPD/PENC=24

CPD NC=15

CNS

0=19

CNC=19

Some lessons• RCT Design is relatively easy – practical issues are more

complicated• The process of inducting schools is important to secure full

engagement pre-randomisation• Having something to offer schools control schools in particular

may help with drop out• Minimising drop out is best. But some drop-out is inevitable –

need for a protocol?• Some implications for analysis

– Helpful to understand reasons behind dropout– Can consider nonexperimental techniques– NPD analysis may be unaffected by drop-out of controls

EEF Conference 2013Towards a Protocol for Effective Recruitment

Mary SheardJuly 12, 2013

Recruitment as a problematic and complex relationship

“Recruit schools to the evaluation not the project”

Contexts

EEF projects: Project and design, challenges and solutions

EEF Protocol and Survey Outcomes:Effective recruitment; what has worked well and what have been the challenges

Non-EEF projects: Experience across a wide range of research studies and evaluations

What do we mean by ‘effective recruitment’?

Terminology: What is meant and understood? Programme Intervention Initiative Project Evaluation …

What has worked well• Relationships: schools, LA/parent organisations, programme

developers, evaluators trainers; test providers [Ethics]• Partnerships with schools: key personnel in school; lead project

contact; teacher implementers; technical support.

• Roles: clarity, responsibilities, expectations, inclusiveness [Ethics]

What has also worked well• Information: quality, clarity, conciseness, sufficiency, inclusivity;

suitability, accessibility (audiences, ethics]

• Examples of documentation; inviting initial expression of interest; school agreement form/contract; pupil data; data protection

Challenges and resolutions

Identifying and linking with key personnel

Senior leadership involvement

Lines of communication

Information overload

Saturation of constituency/schools asparticipant partners

More challenges and resolutions

Defining/explaining and the relationship between school, programme developer (trainer) and evaluation team

Timing

The concept of random assignment

Participation as control

Testing preparation and procedures

What we have learnedNeed to systematise a comprehensive recruitmentstrategy,to establish a recruitment protocol or checklist as the prequel to a project data management plan

Need to create a recruitment database

Need to consider equity/equal opportunity and fairnessin recruitment approaches: hard to reach schools and schools that are missed out

Developing a consistent recruitment strategy

Creating a protocol/checklist for effective recruitmentin future large-scale evaluations

Contact: mary.sheard@york.ac.uk pam.hanley@york.ac.uk

bette.chambers@york.ac.uk