Evidence and the Policymaking ProcessCFHSS Congress 2006York UniversityLouise Shaxsonlouise@shaxson.com
The SeriesEffective Research for Development Policy: How researchers can maximize their influence on policy.Evidence and the Policy Making Process: How do policy makers access research, what constitutes evidence.Action Research for Maximum Impact: Some "good news" case studies, and practical research tools for practitioners.
This WorkshopExercise: Food in primary schoolsDrivers of changePolicy development trajectoryEvidence and analysisWhat policymakers wantThe role of analystsEBPM in practiceFurther information
Exercise: Food in Primary Schools
Drivers of changeIncreasing emphasis on the quality of evidence and its use (Modernising Government);To underpin & inform strategy, policy, regulatory work, foresight; and to mitigate risk;Importance of challenge to evidence (BSE inquiry, Science Advisory Committees) Depth and breadth of future evidence needs will increase given complex and overlapping strategic priorities
Policy development trajectorySmaller policy core (efficiency drive in the public sector)Getting rid of the generalist/specialist labelImproving intelligent customer capabilityFuture evidence needs, when set against strategic priorities, are complex & overlapping. How do we deal with sustainability?
Evidence for policy isany robust information that helps to turn a Departments strategic priorities into something concrete, manageable and achievable.The nature of the evidence you need is proportional to the nature of the risk associated with the decision that is being made.
Evidence is:Facts (data, known trends), judgements, opinions, analyses, syntheses, arguments, costings, reviews, qualitative & quantitative survey dataAnalysis is:Lines of argument (strategy-policy), research, interpretationWhat is evidence and analysis?
Decision makers like numbersbut the evidence base is built uponDataLines of argument (analysis)Stakeholder opinions
Evidence-based policy making is not a sacred cow:There are policies that:
Use good informationand use it welland use it poorlyUse poor information
Analysis & evidence for policyProcuring, managing and carrying out research to provide new evidenceScoping the issue, asking the question, deciding what sort of evidence is neededInterpreting & applying new or existing evidence, monitoring & evaluating the policy once implementedEvidence and analysis needed rapidly to answer pressing policy questionsLonger-term policy and strategy development
Components of robust evidence & analysis (supply side)Is the evidence credible? Can we make generalisations from it?Is it reliable enough for M&E or impact assessments?Is it objective? How do we account for bias?Is it rooted in an understanding of the framing assumptions?
Components of robust evidence & analysis (demand side)Is the evidence policy relevant? Is it timely? Has it been delivered fast enough to inform policy decisions?Is it accessible to all key stakeholders, not just researchers?Is the evidence cost-effective? Is it interdisciplinary enough to address cross-cutting issues?
The role of analysts Scoping the issue, asking the question, deciding what sort of evidence is neededProcuring, managing and carrying out new researchInterpreting & applying new or existing evidence, monitoring & evaluating the policy once implementedQuality assessment, peer review, advisory councils, G2000Negotiate how to apply the evidence; consultancy roleNegotiate the question, advise on alternative sources of evidenceis to provide appropriate evidence & analysis throughout the policy making process
A smaller policy core wont have time to do the integrationIntelligent customers need intelligent suppliersMultiple tools: social, economic, modelling, scientific, technical, institutional analysisHelp policy makers lift their eyes from their desks to see what is possiblebutHelp them maintain their focus on the deliverable.The role of analysts
Defras Evidence & Innovation Strategy:Matching supply-side and demand-side criteriaBased on the question does it make good policy? rather than is it good science?Develop a clear line of sight between policy priorities and evidence provision for efficient deliveryAnalyse in relation to the policy cycle evidence needs change from strategy through to deliveryNeed a better understanding of innovation
EBPM in practice
How can analysts balancethe need to answer policys immediate questions(with robust & cost-efficient evidence)while developing their role(as people who understand policy processes)to underpin broader & more strategic approaches to policy (involving other evidence providers, particularly other disciplines)
Selected BibliographyShaxson, L.J. (2005) Is your evidence robust enough? Questions for policy makers and practitioners. Journal of Evidence & Policy 1(1): 101-111. http://www.ingentaconnect.com/content/tpp/ep/2005/00000001/00000001/art00006Evidence-based policy making: guidance for policy makers. Available at http://www.defra.gov.uk/science/how/evidence.htm Jones, K.E. (2005) Understanding risk in everyday policy making. Defra: UK. Available at http://www.defra.gov.uk/environment/risk/policymaking0509.pdf Defra (2005) Evidence & Innovation Strategy 20052008 (Part 1: Aims of the consultation / introduction). Available at http://www.defra.gov.uk/science/how/documents/PDFs%20in%20Parts/Part%20I.pdf
RAPID StuffODI Working Papers Bridging Research and Policy BookMeeting series MonographRAPID Briefing PaperTools for Policy ImpactCommunication Tools Policymaker ToolsRAPID CDROMwww.odi.org.uk/rapid
Policy core will need to become more sophisticated in its sourcing and handling of evidence, in terms of strategy-policy linksImproving intelligent customer capability doesnt mean creating super policy wonksAbout turning the soup of information into something:Policy makers can use to develop options, focusing on the deliverable whilst being aware of the possible (future-proofing decisions)That is the basis for robust and defendable decisionsThat helps to address causality rather than simply describing trends (the why chain)That helps us examine the assumptions under which were working (technical, economic, social, international environments, for example)Dont confuse real with perceived nature of risk is perceived differently by different people. Dont operate the deficit model of understanding risk: are you sure you understand the nature of perceptions?
Example of statistics about recorded crime not reflecting statistics about the fear of crime. Note that these are actually about two completely different phenomena one is about what bad people do, the other is about how we feel about what those bad people do and the effect is has on our society. Not competing statistics one provides richness to the other. Just a framework for thinking about how to construct an evidence base, but getting away from the supremacy of metrics. If youre unconvinced of the robustness of your data (or if the data isnt there), make sure your line of argument is robust.If youre unconvinced of the robustness of your line of argument, make sure your relationships with stakeholders are robust that they can cope with challenge and dissent. Introducing a bit more rigour into the evidence base.Negotiate with policy makers and other opinion-formers to ensure that:The issue is well framed;The question is appropriate to the issue;The evidence is appropriate to the question and to any time & resource constraints that might affect how information can be collected and analysed; (alternative sources of evidence)The analytical methods are appropriate to the quality of the evidence (and thus to the question);The results of any analysis are well applied to the initial policy question;Any monitoring and evaluation is based on reliable evidence.
Who does the integration of the evidence base? A smaller policy core wont have the time to do it, so it had better come from the suppliers in response to a well-negotiated question.Far better for it to be integrated from the outset, around an agreed question and within framing assumptions that are widely understood, than for people to be left to do their own thing and to come together at the end to argue about conflicting information & results that bear no relation to each other.Policy core needs help looking at what is possible (esp. in terms of technological possibilities), but theres a clear message from policy makers that theyre not satisfied with researchers ability to answer pressing short-term questions.
Improving our understanding of what constitutes robustness will help us ensure that all decisions are supported (if not based on!) the most robust possible evidence.
E&IS is a first attempt at matching supply side and demand side criteria and working out the organisational implications of doing so.