TRANSFORMING EDUCATION THROUGH EVIDENCE. The Centre for Effective Education SCHOOL OF Education Conducting Educational Randomised Control Trials in Disadvantaged

TRANSFORMING EDUCATION THROUGH EVIDENCE.

The Centre for

Effective Education

SCHOOL OF

Education

Conducting Educational Randomised Control

Trials in Disadvantaged Community Settings

Andy Biggart & Liam O’Hare

SCHOOL OF Medicine, Dentistry and Biomedical Sciences

Presentation Overview

1. Introduction to research context and

programmes

2. Overview of Key Results

3. Conextualising Results in Wider Literature

4. Some methodological and ethical reflections of

conducting RCT’s in community settings

The Centre for Effective Education | SCHOOL OF Education

SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education

Research Context• Educational Research & Randomised Control

Trials

• The Atlantic Philanthropies

• Some of first contemporary trials in UK and

Ireland

• The two progammes grew out of a consultation

with the local community in Tallaght (Dublin)


Two After-schools Programmes• Doodle Den

• Newly developed after-schools programme

• Aimed at 5 and 6 year olds identified as struggling readers

• Adopted a balanced literacy framework

• Delivered by teacher and youth worker/child care assistant

• Ran throughout the school year

• 3 days a week for 1.5 hours afterschool

• Highly structured programme with strong emphasis on

professional development


Two After-schools Programmes• Mate Tricks

• Very similar in delivery to Doodle Den

• Hybrid Pro-social Behaviour Programme: Coping Power and

Strengthening Families Program

• Aimed at 9 and 10 year olds

• Universally applied in schools in Tallaght

• Parental Components

• Designed to enhance children’s pro-social development,

reduce anti-social behaviour, improve parenting skills and

parent/child interactions


Common Methodology• Individually randomised trials

• Qualitative Process Evaluation

• Children recruited on basis of teacher referrals

• 3 year rolling cohort design

• Over 600 children allocated to either

intervention or to control: ‘business as usual’

• Child, parent and teacher measures

• Outcomes measured according to pre-

validated measures where possible.


Summary Outcome Results – Doodle Den

• Primary Outcomes:

– Statistically significant effects on both direct child and teacher assessed measures of literacy

(d=0.17-0.3)

• Secondary Outcomes:

– Statistically significant effect on children’s behaviour in regular school class (d=0.18)

– No impact on attendance rates at school

• Exploratory Analysis

– No significant differences in terms of literacy

– Boys behaviour improved relative to control group

– Greater attendance at the programme led to more positive outcomes


Summary Outcome Results – Mate Tricks

• Primary Outcomes:

– No statistically significant effects on pro-social behaviour

– Two significant negative effects on anti-social behaviour (d=0.18; 0.20)

• Secondary Outcomes:

– No statistically significant positive effects on wide range of secondary measures

– Some significant negative effects in relation to child reported parenting practices

• Exploratory Analysis

– Child programme attendance not related to outcomes

– Parental engagement with the programme significant predictor of a range of positive

outcomes


Conextualising Results – Doodle Den

• First randomised trial of an after-school program in UK or Ireland focused on

academic outcomes

• Mixed results of previous trials of the effectiveness on academic outcomes

– e.g. 21st Century schools U.S. national evaluation

• Reviews of evidence highlight complexity of the field

• Recent promising practice guides have concluded a need for much more

research on what works in after-school settings

• Potential features driving positive outcomes:

– Age of children; targeted at those most in need, well structured clearly defined curriculum,

qualified staff with focus on professional development and fun engaging activities


Conextualising Results – Mate Tricks

• Hybrid programme with some evidence of effectiveness behind the two

programmes

• No evidence of effectiveness of Mate Tricks in improving behavioural outcomes

and some negative effects apparent

• Some evidence of in-school programmes focusing on social and emotional

learning having promising effects e.g. Incredible years

• However, previous reviews have also highlighted null or adverse effects in after-

school contexts

• The current study adds to existing evidence that behavioural programmes have

the potential to cause adverse effects as well as benefits

• r


Some Lessons Learned: Methodological Issues

Despite the critics, it is possible to implement RCTs in real world community settings.

Making key stakeholders aware of key principle’s of RCT evaluation designs A willingness to be flexible, without undermining the core of the study and randomisation process

Developing and maintaining good relationships

Keeping attrition to a minimum without overburdening participants

Importance of measurement and agreeing analysis plan in advance

RCT’s good at assessing intended outcomes but good process data should not be neglected


Ethical Issues and RCT’s

Equipose

Randomisation perceived as a fair way of allocation

Ethical Issues may arise at any stage

Potential for burden

Dissemination and not spoiling the field, especially when faced with null or negative findings


EVIDENCE MATTERS!!!

We intervene in children’s lives on a regular basis!

We may also do damage when we intend to do good!

Interventions often driven by fashion rather than sound evidence!

Do we owe it to children to try and ensure that the policies they are subjected to are effective in improving their life chances?

Both reports available on CEE website: www.qub.ac.uk/cee

Documents

TRANSFORMING EDUCATION THROUGH EVIDENCE. The Centre for Effective Education SCHOOL OF Education Conducting Educational Randomised Control Trials in Disadvantaged