1

Government Social Research Unit

www.gsr.gov.uk

Randomised Controlled Trials ConferenceUniversity of York, 14-15 September 2006

Why Governments Need RCTs

Philip Davies PhDGovernment Social Research Unit

HM TreasuryLondon SW1A 2HQ

2

www.gsr.gov.uk

Answer

Because RCTs can help get policy initiatives

right more often than wrong

3

www.gsr.gov.uk

Caveats

• Governments have got to want to get things right

• ‘Getting things right’ is more than achieving an effect size

• Governments do things other than achieving impacts

• RCTs are not a panacea

• RCTs are only as good as their design and implementation

4

www.gsr.gov.uk

Governments Have Got to Want to Get Things Right

“Social science should be at the heart of policy making.

We need a revolution in the relationship between

government and the social research community - we

need social scientists to help determine what works and

why, and what type of policy initiatives are likely to be

most effective …”

(David Blunkett, 2000)

5

www.gsr.gov.uk

Governments Have Got to Want to Get Things Right

Government policy must be:

• Evidence-based

• Properly evaluated

• Based on best practice

6

Governments Have Got to Want to Get Things Right

• Policy making must be soundly based on evidence of what works

• Improve departments’ capacity to make best use of evidence

• Improve the accessibility of the evidence available to policy makers

7

Governments Have Got to Want to Get Things Right

Professional Skills for Government (2005)Analysis and Use of Evidence

• Anticipate and secure appropriate evidence• Test for deliverability of policy/practice – and evaluate• Use evidence to challenge decision making• Identify ways to improve policy/practice• Champion a variety of tools for collecting/using evidence• Ensure use of evidence is consistent with wider government

requirements• Work in partnership with a wide range of experts/analysts

8

What ‘Getting It Right’ Means

“…a society that would use social science methods and

evaluation techniques to vigorously try out possible

solutions to recurrent problems and would make hard-

headed, multidimensional evaluations of outcomes, and

when the evaluation of one reform showed it to have

been ineffective or harmful, would move on and try

other alternatives”

The ‘Experimenting Society’

(Donald T. Campbell)

9

What ‘Getting It Right’ Means

“A social experiment benefits society by providing better

information on which to base public policy. Such

information can improve policy in one of two ways: it

can lead policymakers to adopt a program or policy that

is found to have net social benefits, or it can lead to the

termination of an existing program that is found to have a

net social cost”

(Orr, L., 1999).

10

Why RCTs?(Recap)

Randomised Controlled Trials:

• Provide most precise estimates of the likely effects of policy interventions.

• Against a strong counterfactual – i.e. the net effect/impact• Reduce/account for various biases.• Establish cause and effect of policy interventions and

outcomes.• Raise the quality of evidence to support policy making.

11

So, Why Don’t Governments Use RCTS More Often?

• RCTs not on most policy makers’ radar

• RCTs not on many government researchers’ radar

• RCTS not on many academic researchers’ radar

• For some (many?) RCTs are anathema: Don’t like ‘random’ (too ‘chancey’)

Don’t like ‘control’ (no ‘wiggle room’)

Don’t like ‘trials’ (sounds inhumane / 1940s Germany)

• There are things other than ‘impact’

• There are things other than ‘evidence’

12

Things Other Than Impact Evidence

Research Evidence

Impact Evidence

Implementation Evidence

Descriptive Analytical Evidence

Economic and

Econometric Evidence

Ethical Evidence

ExperimentalQuasi-Experimental

SurveysAdmin DataComparativeQualitative

Cost-BenefitCost-Effectiveness

Cost-Utility

Experimental Qualitative

Theories of Change

Social EthicsPublic Consultation

Statistical Modelling

Multivariate Analysis

13

Things Other Than Evidence

“Research of any kind is seldom the

determining factor in shaping public

policy. Experimentation is no exception

to this rule” (Orr, L., 1999)

14

Things Other Than Evidence

Evidence

Experience & Expertise

Judgement

Resources

Values and Policy

Context

Habits & Tradition

Lobbyists & Pressure Groups

Pragmatics & Contingencies

15

Policy Makers’

Evidence

• Colloquial (Contextual)

• Anything that seems reasonable

• Policy relevant

• Timely

• Clear Message

Researchers’ Evidence

• ‘Scientific’ (Context free)

• Proven empirically

• Theoretically driven

• As long as it takes

• Caveats and qualifications

Different Notions of Evidence

16

Academics

Constituents, Consumers and Users

Media

Professional Associations

Lobbyists and Pressure Groups

Think Tanks/Opinion Formers

'Experts'

Special Advisers

Policy or Practice

Where Do Policy Makers Go For Evidence?

17

But Not All Is Lost

• Employment Demonstration Project (ERA)• Job Retraining and Rehabilitation Project (JRRP)• Restart evaluation (1990)• BA Visiting Officer Pilot• Evaluation of Employment Zones• New Deal 25+ Pilots• Evaluation of the New Deal for Young People Intensive

Gateway • Evaluation of the Restorative Justice programme

Examples of Recent UK Government RCTs

18

RCTs Are Only As Good As Their Design and Implementation

Common Design and Implementation Faults:

• Unclear policy objectives, populations, outcomes• Insufficient time (lack of strategic policy making)• Test condition often pre-determined• Sample sizes too small – no statistical power• Problems of randomisation, concealment, cross-over• High attrition (especially in control groups)• Intention to Treat analysis not used• Limited capacity (in the UK) to do high quality RCTs• Poor counterfactual too often the result

19

RCTs Are Only As Good As Their Design and Implementation

These faults can usually be overcome!

• Better strategic policy making• Better planning and protocol development• Involvement of researchers, policy makers, front-line staff• High quality piloting / pre-fieldwork testing• High quality research/project management• Careful real time data management• Use of experienced technical experts throughout• Policy Trials Service• Professional development of researchers and policy makers

20

Take Home Message

• RCTs have enormous potential for governments

• RCTs can help governments get things right more often

• RCTs are currently used considerably to UK Govnt.

• But there are still non-believers

• And those who remain to be convinced

• RCTs, like other types of research, must compete with other notions of evidence

• And with factors other than evidence

• There is a need to improve the capacity to do high quality RCTs in the UK

21

Contact

philip.davies@hm-treasury.x.gsi.gov.ukGovernment Social Research UnitHM Treasury1 Horse Guards RoadLondon SW1A HQEngland

Tel: +44 (0)20 7270 5156

www.gsr.gov.uk