Embed Size (px)
Citation preview
www.gsr.gov.uk
Government Social Research Unit
www.gsr.gov.uk
Evidenced Based Policy: The Role of Randomised Controlled Trials
Ricky TaylorGovernment Social Research Unit
HM TreasuryLondon SW1A 2HQ
www.gsr.gov.uk
“There is nothing a government hates more than to be
well-informed; it makes the process of arriving at
decisions much more complicated and difficult”
John Maynard Keynes
www.gsr.gov.uk
“Social science should be at the heart of policy
making. We need a revolution in the relationship
between government and the social research
community - we need social scientists to help
determine what works and why, and what type of
policy initiatives are likely to be most effective …”
(David Blunkett,
2000)
www.gsr.gov.uk
Government policy must be:
• Evidence-based
• Properly evaluated
• Based on best practice
The Context of More Experimental Designs in Government
www.gsr.gov.uk
• Policy making must be soundly based on evidence of what works
• Improve departments’ capacity to make best use of evidence
• Improve the accessibility of the evidence available to policy makers
The Context of More Experimental Designs in Government
www.gsr.gov.uk
Better Policy Making
November 2001
“UK policy making is now more
informed by evidence”:
The Context of More Experimental Designs in Government
www.gsr.gov.uk
“…a society that would use social science methods and
evaluation techniques to vigorously try out possible
solutions to recurrent problems and would make hard-
headed, multidimensional evaluations of outcomes, and
when the evaluation of one reform showed it to have been
ineffective or harmful, would move on and try other
alternatives”
The ‘Experimenting Society’(Donald T. Campbell)
www.gsr.gov.uk
“A social experiment benefits society by providing better
information on which to base public policy. Such information
can improve policy in one of two ways: it can lead
policymakers to adopt a program or policy that is found to
have net social benefits, or it can lead to the termination of an
existing program that is found to have a net social cost”
(Orr, L., 1999).
Why Experiment?
www.gsr.gov.uk
Why Experiment?
“…Studies using non-random assignment may
produce acceptable approximations to results from
randomised experiments under some circumstances,
but that reliance on results from randomised
experiments as the gold standard is well founded”
(Shadish and Ragsdale, 1996).
www.gsr.gov.uk
Why Experiment?
Randomised Controlled Trials:
• Provide most precise estimates of the likely effects of policy interventions.
• Against a strong counterfactual – ie, net effect/impact
• Reduces/accounts for various biases.• Establishes cause and effect of policy
interventions and outcomes.• Raises quality of evidence to support policy
making.
www.gsr.gov.uk
7 8 16 17 23 24 52 69 233 279
9255
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Number of published RCTsSource: Campbell Collaboration
www.gsr.gov.uk
• For 30 years social experiments have been the primary
method of evaluating new policies in the US.
• Lyndon Johnson’s ‘War on Poverty’ saw the evolution of
randomised trials.
• Late 1960’s the Head Start programme, and the Perry pre-
school programme were evaluated through randomised trials.
• 1968, the Mathematica Foundation conducted the New
Jersey-Pennsylvania Income Maintenance Experiment
The U.S Experience
www.gsr.gov.uk
“It is probably correct to report that - in contrast to either
Britain or the EU - RCTs are effectively the default
option in the USA ...It is mainly when such trials turn out
to be impractical for one reason or another that other
methods, such as matched area-based trials or before-
and-after studies, come into their own.” (Trying It Out: The Role of ‘Pilots’ in Policy-Making, Cabinet Office 2003)
The U.S Experience
www.gsr.gov.uk
• Too easy to blame policy makers• A longer tradition of experimentation• US academics are more interested in policy development
• Much greater capacity to do trials• Fewer social researchers in UK Government, or in academia, have the skills to conduct large scale trials.
• Capacity problems in UK social science• But things are changing…
Why Are RCTs More Common in the U.S?
www.gsr.gov.uk
• Employment Demonstration Project (ERA)• Job Retraining and Rehabilitation Project (JRRP)• Restart evaluation• BA Visiting Office Pilot• Evaluation of Employment Zones• New Deal 25+ Pilots• Evaluation of the New Deal for Young People Intensive Gateway
• Evaluation of the Restorative Justice programme
Examples of Recent Government RCTs
www.gsr.gov.uk
• “It works but how does it work?!” • ‘Black Box’ issues• Budgets (not necessarily more
expensive)• Practicalities• Ethical Issues
Some Challenges for Experimental Designs
www.gsr.gov.uk
•“Research of any kind is seldom the determining factor in shaping public policy. Experimentation is no exception to this rule” (Orr, L., 1999)
• Ideology and Beliefs• Manifesto Commitments• Special Advisers• Lobbyists and Pressure Groups• Electoral/Parliamentary timetables
• Policy cycle Timeliness
Other Factors Influencing Policy
www.gsr.gov.uk
• Recognition/acceptance of the value of
experimental designs in policy making /evaluation
• Capacity building is required in UK social science
• Need for training and professional development
• Need to promote experimentation in social
research
Next Steps
www.gsr.gov.uk
Ricky TaylorGovernment Social Research UnitHM TreasuryLondon SW1A 2HQ
Tel: +44 (0)20 7270 5462
Email: [email protected]
Contact
www.policyhub.gov.uk
