Trials and Tribulations

Peter Tymms

York 2008

Outline

• Introduction• Background issues

– Impact of educational policies– Level of intervention– Monitoring and the possibilities for RCTs– Studying cause and studying possibilities

• Three interventions:– Small scale teaching intervention– Large scale simplicity– Large scale complexity

• Conclusions

The Impact (Failure) of Educational Policies

• One example

– In England The National Literacy Strategy cost £500 million.

– Its impact on reading levels has not been detected.

Level of Interventions

• Most systematic reviews (of RCTs) deal with pupil level interventions.

• But policy level strategies deal with school level interventions

• To change a teacher's classroom approach takes 30 hours of INSET

Monitoring

Establishing cause and establishing what is possible

• Monitoring alone does not establish cause– (MLMs can delude “explain”, “effect” etc)

• Regression discontinuity & time series can.– But they may not tell us what is possible

• RCTs can, and – To know how to improve education we must try to

improve education.

First Intervention

Cross age peer tutoring in Chemistry

Design etc

• 14 2nd year students paired with 1st years

• Random assignment

• Two half hour sessions

• Post test

Results and lessons

• ES = 0.3 (n.s.)

• Lessons– Much harder than usual teaching– A learning experience– Qualitative – Mandy’s experience– Telling others and its impact

• Colleagues• Policy makers

Second Intervention

ADHD: Naming and Informing

(include in NICE)

Attention Deficit Hyperactivity Disorder

• Very common special need

• Children with ADHD fall behind academically

• Often unpopular

• Social difficulties

• Behavioural problems

Scores at age 5: DSM(IV)

Behaviour

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

Pe

rce

nt

60

50

40

30

20

10

0

SEX

boys

girls

A Dilemma

• Feeding names back to schools

• One psychologist: “You must not do this”

• Another: “You must take action”

Interventions

• Randomly assign to schools

Identification No Identification

Booklet

No Booklet

Interventions

• Randomly assigned to Districts

Nothing Information packs

Conference & packs

Numbers

• 24 Districts

• 2040 schools

• 60,000 pupils

• Over 2 years

Results

• LEA Interventions – No impact on any of the outcome measures

ResultsAll Pupils

• For booklet– Higher Y2 reading score

• Identification– No impact

ResultsPupils with ADHD characteristics

• For booklet– More positive attitudes– Better behaviour in Year 2

• For Identification– No impact

• BUT book + identification– Negative impact on maths and reading

Teachers’ Quality of Life

Your perception of work:

Relaxing Stressful

Enjoyable Unpleasant

On top of things Overwhelmed

The behaviour of the pupils in your class:

No problems Challenging

Teachers’ Quality of Life

Your perception of work:

Relaxing Stressful

Enjoyable Unpleasant

On top of things Overwhelmed

The behaviour of the pupils in your class:

No problems Challenging

Teachers more positive if given information books

Major Issues

• Implementation– The intervention was very clear– But schools using the booklet ?– Taking notice of feedback?

• Cheap with low impact

• Very cost effective

• For the future:– Implementation: Aim for wider, deeper impact

PostScript

• A follow up study by Sayal with CEM showed:– Naming appears to have long term problems

• Lesson– Once you carried out and RCT long term

follow up because a possibility.

Third Intervention

Peer Tutoring

Peer Learning

• Good research pedigree from individual RCTs– “Low hanging fruit” Boruch

• Effect Sizes of 0.4 to 0.8 in small trials

• For attainment & other outcomes

• For tutors and tutees

The Overall Plan

• Run Peer Learning in schools for two years in reading and mathematics.

• Use monitoring data to check impact

We know that ..

Peer Learning is effective

But

• Can a whole Authority change together?

And

• Which is best – Cross-age or Same-age?– Mix or separate subjects?– Intensive or light?

Allocation to forms of peer learning

• 120 schools initially agreed to be randomly assigned

• Factorial design

• Separation of allocation & analysis from intervention

Issues

• Immediately after we had sent out allocation letters one of the allocations was changes!

• Some schools and pupils changed

• No effect after one year!

• Maths tutoring did not involve maths!

Preliminary Results

MLMs

For younger children (8-9)

• Cross age (tutee only)

• Intensive

• Reading &

• Mathematics

• ES = 0.2

For older (10-11)

• Cross age (tutors only)

• Light &

• intensive

• Reading &

• mathematics

• ES = 0.2

Issues

• Practicalities and logistics– Setting the whole thing up

• Implementation– Arrangements for Peer Learning– What was being taught– Fidelity

• The effects are small but the potential is great.

Finally• Establishing cause is not sufficient;

– We need to know what we can change.

• Large scale monitoring with ready access to the data is fairly new

• We can carry out RCTs within a monitoring framework.

• Professionals can see the value in RCTs

• We need:– Qualitative data, monitoring data and RCTS

• Meaningful change to educational systems may not come from making the system more efficient but from changing its nature.

Thank you

Peter Tymms

York 2008

References• Fitz-Gibbon, C. T. (1996). The Design And Use Of Monitoring Education: Indicators,

Quality and Effectiveness. London: Cassell.• Fitz-Gibbon, C. T. (1992). Peer and Cross-Age Tutoring. In M. C. Alkin (Ed.),

Encyclopedia of Educational Research (Sixth ed., pp. 980-984). New York: Macmillan Publishing Company.

• Tymms, P. B. (1989). Peer Tutoring with 'A' level Chemistry Students. Paired Learning, 5.

• Tymms, P., & Coe, R. (2003). Celebration of the Success of Distributed Research with Schools: the CEM Centre, Durham. British Educational Research Journal, 29(5), 639-653.

• Tymms, P., & Merrell, C. (2006). The impact of screening and advice on inattentive, hyperactive and impulsive children. European Journal of Special Needs Education, 21(3), 321-337.

• Tymms, P. Merrell, C. Heron, T., Jones, P., Albone, S., and Henderson, B. (2008 in press) The Importance of Districts School Effectiveness and School Improvement

• Tymms, P., Merrell, C. and Coe, R. (2008) Educational Policies and Randomized Cotrolled Trials The Psychology of Education Review, Vol. 32, No. 2, September 2008