Telling policy stories:An ethnographic study of the evidence-policy link

Alex StevensUniversity of Kent

Two epigraphs on evidence-based policy

• “Nobody rational could possibly want a government based on any other type of policy-making.”

Colin Blakemore, quoted in The Observer, November 2009

• “There is no (rational or empirical) scientific procedure of any kind whatsoever which can provide us with a decision...”

Max Weber, The Methodology of the Social Sciences

Research question

• How do policy makers use evidence?• Or rather:

How do some particular policy makers use evidence in a specific policy area (illicit drugs and crime)?

• Definitions Policy making:

• The process of choosing ends and means for government action, whether rational or not.

Evidence• What counts as evidence to be treated as an empirical

question.

Method and ethics

• Method: Six months of participant observation in a team of

English policy making civil servants. During a placement funded by ESRC. Five semi-structured interviews. Analysed through ‘adaptive coding’ of fieldnotes and

transcripts (Layder 1998).

• Ethics Approved by internal ethics committee. Negotiated in the field. Informed consent from closest colleagues and

interviewees. Covert observation of others. Anonymisation of all participants.

Prevalent models and approaches to the evidence-policy link

• Models: Linear– A direct link between research evidence and government

policy. Political/tactical– Evidence is used in support of short-term political interests. Enlightenment (Weiss 1976)– Evidence informs the climate of opinion in which decisions

are taken.

– Approaches:

– Pluralist (Kingdon, Sabatier & Jenkins-Smith, Nutley).

– Post-foundationalist (Hajer, Valentine).

– Critical (Bourdieu, Habermas).

The research environment

• A small team of people, most of whom were moving quickly between jobs in search of rapid promotion.

• Mostly non-specialist, ‘fast tracked’ civil servants, with some secondees.

• Social backgrounds:– Mostly white British.– Middle class.– Male.– Young (<35).

• Advising ‘the highest levels’ of UK (English) government on crime and drug policy.

Seven themes and two cases…

• Themes: The commitment to evidence. The oversaturation of evidence. Intra-government relations. From contested norms to monetary facts. The control of uncertainty. The preference of intuition. Bureaucratic competence and the civil service career. The third face of bureaucratic reason.

• Cases:•The ‘silent silencing’ of inequality

•The call for “totemic toughness”

1. The commitment to using evidence

• The civil servants were highly committed to the use of evidence:

– “it’s the job of officials to tell truth to power”– “evidence is a prerequisite for policy”– “evidence should be the basis for options we put to

ministers… evidence-based policy is part of the way that we work.”

• All could give examples of policies that had used evidence.

• But all could also give examples of where it had not.

What counts as evidence?

• Fifteen types of evidence coded from observations:

– internally collected government data– reports by thinktanks (e.g. ippr, Demos, Policy Exchange,

Centre for Social Justice, etc.) – opinion polls– reports from management consultancies.– previous policy papers – independent inquiries– reports of the inspectorates of police and prisons– internal evaluations– external evaluations – reports from abroad– press reports– externally produced academic analysis– television programmes (e.g. The Wire)– personal or reported experience – personal or reported opinion

• Mostly found through Google searches – No on-site library or online journal access.

2. The oversaturation of (unsuitable) evidence

“a depressingly similar pattern where you look for the best - usually quantitative data - you can find, and then, as you work through the policy problem you establish that there is not the best evidence that you want and you work your way down until, at the end, you’re left with the odd case study, something which was kind of half evaluated, some anecdotal information and then what you can garner through a few field visits.”

(Policy adviser)

3. Intra-government relations (and the creation of policy stories)

• The point of policy work is to get policies accepted. Requiring iterative negotiation within government Effectiveness becomes a secondary consideration.

• This requires the creation of coherent narratives to “sell the policy” to colleagues and ministers across departments.

• Persuasive narratives: chime with existing ways of thinking. exclude uncertainty. For example…

The excision of caveats…

4. From contested norms to monetary facts

• Repeated attempts to convert contested questions of normative value into simple questions of financial value: “Fairness can’t be measured. It’s irrelevant.” (Civil

service economist) Derision of “old-style” civil servants for muddying the

water with questions on fundamental policy aims.

• Counter-example: Use of humane values to question continuing the

high rate of youth imprisonment. But did not eventually affect the policy.

5. The control of uncertainty (through the use of “killer charts”)

• “A good chart is worth ten pages in words” (Senior civil servant)

• “Killer charts”. The prevalent form of communication for

policy related, quantitative evidence. Restricted number of variables and cases. Preferably dramatic. A visual rendition of statistics which renders

invisible the process of construction of these statistics.

• For example...

On current projections, justice is going to become unaffordable…

This graph assumes that CJS demand continues to grow at the average rate of the last six years, and that MoJ total spending follows the IFS ‘equal pain’ scenario.

0

2,000

4,000

6,000

8,000

10,000

12,000

Actual IfS scenario CJS spend

* Spend on the criminal justice system includes NOMS, prisons, probation, YJB, OCJR, criminal injuries compensation, and the proportions of legal and HMCS budgets that are spent on criminal justice

1. Context for reform

Actual and projected total MoJ spend (bars) against actual and potential CJS demand* (line), £millions

Even if MoJ total spend is flat in real terms, CJS spend could swallow 90% of MoJ spending by 2019/20, compared to 76% in 2008/9.

6. The preference for intuition over evidence

• A longstanding preference: My colleagues have “an almost unanimous reliance

on intuition and a distrust of systematic argument” (civil servant quoted by Garrett, 1972)

• Exemplified by the lack of need for evidence for market solutions to public problems: E.g. Purchaser/provider split in NOMS justified by

Ministers on the basis of Carter report’s unevidenced preference.

“it just feels intuitively right that introducing competition would focus more on cost and quality” (Senior civil servant)

7. Bureaucratic competence and the civil service career.

• The three unwritten rules of civil service promotion:

• 1. Do not specialise. “Never less than two years, never more than three”

(quote on departmental intranet chat page). “Specialist experience is seen as a positive

disadvantage” (policy adviser, pub conversation)

• 2. Find superior supporters. “it’s supposed to be open and equal but it’s not.

People go for people they know. If you’ve worked with someone senior, you try and stay in touch with them; they can help you” (female policy adviser).

• 3. Be useful.

On being useful

• “I found a problem with [policy area]. My boss said ‘Well you’re young. Why don’t you suggest we look again at [policy area] and see how far that takes you in your career?’ So there are certain areas where officials will self-censor and they won’t suggest to ministers to change policy on certain areas even though the evidence suggests it.”

• “I think if you always use the evidence [when it conflicts with current policy] then you're always going to be the awkward person that’s saying, ‘the Emperor has no clothes’”. Policy adviser, grade 7.

8. The third face of bureaucratic reason

• First face: Using reason and empirical evidence to create useful

knowledge.• E.g. Discussion on imprisonment at Ministry of Justice

• Second face: Using the performance of this rational capacity to

demonstrate worthiness for status and reward.• E.g. Competitive performance of knowledge during this

discussion.

• Third face: Using only a certain selection of the range of rationally

justifiable positions in order to demonstrate usefulness, to become a protégé and to qualify oneself for acceptance and promotion.

• E.g. Omission of explicit recommendations to take direct action to reduce imprisonment from eventual policy proposals.

Case A: The (ignored) importance of inequality

The ‘silent silencing’ of inequality

• Wilkinson’s presentation described as “compelling” and “convincing”, but no action taken on inequality in my policy area.

• Actions that are taken on drugs and crime reinforce class and ethnic inequalities.

• “the Gini coefficient is not a policy lever that we can pull” (Senior civil servant, grade 5)

• “We need to keep the lid on” (same civil servant)

Case B: “totemic toughness”

• “We need to come up with totemic, tough policies”

(Special Adviser and Senior and other Civil Servants)

• Symbolic fragmentation/expurgation: “The job of policing is to protect the goodies and to

catch the baddies” (Plenary speech at a policing policy conference)

“we know who we’re talking about. It’s not the public schoolkids waiting at the bus stop, it’s those other kids.” (Civil servant during meeting on “incivility”)

Limitations

• Ethnography provides a deep but narrow picture of empirical reality. Different governments may do things differently. Other civil servants may have acted differently. Other policy areas may operate by different implicit

rules on the use of evidence. A different researcher may have created different

reactions and noticed different behaviours.

• The shaping of persuasive research narratives. E.g. the dangerous attraction of the killer quote.

So how do policy makers use evidence?

• For these policy makers:

• Evidence was used in the creation of policy stories that favoured action over contradiction, and certainty over accuracy.

• Evidence was more likely to be used if: If it did not challenge existing asymmetries of power

and wealth. If it “totemically” reinforced the ideological

fragmentation of society into deserving and undeserving citizens.

Civil Service Reform: Panglossian pluralism?

• No acknowledgement of systemic bias.

• Reiterations of need to base policy on sound evidence…

• … while continuing to roll out unevidenced policies [insert your favourite here].

• … and continuing to miss opportunities to design innovations to generate valid knowledge of effect. ‘the key test of good policy is the feasibility of implementation’

• Naïve faith in ‘open policy making’…

• … with an added dash of privatisation (‘contestability’)

Conclusions

• Policy makers are faced with an unreadable deluge of unconvincing evidence.

• They use some of this evidence to tell and sell persuasive policy stories. Which get policy accepted. Which help to advance their careers.

• Selection of evidence for these stories fits the interests of powerful groups and so represents a form of ideology

– ‘Systematically distorted communication’ (Habermas, 2002)

Acknowledgements