Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Empathic evaluationCan increased understanding of policymaking
enhance the use of evaluation evidence?
Myriam Van Parijs - Evaluation Office Enabel
Enabel – Belgian Development Agency
09/04/2018 Myriam Van Parijs - WUR Conference 2018
EVAL Office strategy
Evaluation evidence
(systems -development
strategies)
Portfolio M&E+L systems
Evaluation culture
EVALUATION Office
Knowledge brokering - User oriented
Evidence informed strategies & policiesAffecting views on how change occurs
Evaluative mindset
High quality
M&E practices
09/04/2018 Myriam Van Parijs - WUR Conference 2018
INTRODUCTIONWhy have an interest in policymaking?
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence matters in making effective policies
assuming use of evidence
Evaluation phase : communicate and disseminate
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence matters in making effective policies…
• Supply - disillusionment with limited uptake of evidence
• Demand - policymakers are craving for wisdom, while drowning in information
• Evidence supporting policymaking –little proof of this link in real-world practice
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence matters in making effective policies –the GAP narrative
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence matters in making effective policies
serviceempathy?
policymaking needsUse of evidence by policymakers
EVIDENCE ISPRODUCED
EVIDENCE IS ACCESSIBLE
POLICY MAKERS USE EVIDENCE
SUPPLY SIDE DEMAND SIDE
GAP?
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Empathic evaluation - Content
• Shift attention from ‘use of evidence’ to :– the imperatives of the policymaking systems, and the use of evidence
herein– improving the contribution that evidence makes to policies
• Can evaluation processes and evaluation systems become more empathic to the needs of policymaking?– policymaking– use of evidence
09/04/2018 Myriam Van Parijs - WUR Conference 2018
ON POLICYMAKING
What do policymakers actually do when they make policies?
09/04/2018 Myriam Van Parijs - WUR Conference 2018
What is policymaking?
Lack of definitions & understanding of ‘real-world policymaking’ is part of the problem
Source : ODI
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence in policymaking : a new call?
• 1997 : a call for ‘what works’ – Labour Government UK ‘Nobody rational could possibly want a government based on any other type of policymaking.’
• 3 drivers of the modern evidence based policy movement :– enlightenment– role of evidence in the radical improvement of health outcomes– New Public Management
• What works? – challenged by wicked problems – policymaking who gets what, when and how
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Two policymaking paradigms on use of evidence
Rational - technocratic
Politicised-relational
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Two policymaking paradigms From rational and technocratic paradigm
Use of evidence inpolicymaking
Rational use of evidence - take decisions.
Policy process Several separate stages in a process.Policy issues Are seen as ‘problems’, outside the
political sphere.
No distinction in nature of issues.Management of policypractice
Rational, related to the different stages.
Relation betweenevidence and policymaker
Focuses on the policy maker as ‘individual’.
Improve use ofevidence in policypractice
Bridge the ‘gap’ between policy maker andevidence (providers).
Focus on production of evidence (supply-side).
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Two policymaking paradigms
From rational and technocratic paradigm To politicised and relational paradigmUse of evidence inpolicymaking
Rational use of evidence - take decisions.
Ratio.
Political use, negotiation over evidence.
Based on values, interests & beliefs.Policy process Several separate stages in a process. Incrementalist, messy and complex
process, policy window.Policy issues Are seen as ‘problems’, outside the
political sphere.
No distinction in nature of issues.
Are considered as ‘problematisations’,constructed within the political sphere,subjectivity. Values, interests & beliefs.
Wicked problems and distinction betweensimple – complicated – complex issues.
Management of policypractice
Rational, related to the different stages. Intractable, muddle through.
Relation betweenevidence and policymaker
Focuses on the policy maker as‘individual’.
Focuses on individual level, increasingattention for organisational andinstitutional aspects (networks).
Improve use ofevidence in policypractice
Bridge the ‘gap’ between policy makerand evidence (providers). Focus onproduction of evidence (supply-side).
Interaction between policy maker andevidence providers. Focus on demand side(understand and enhance access and useevidence).
09/04/2018 Myriam Van Parijs - WUR Conference 2018
source : Van Parijs M., 2017
Politicised-relational paradigms
Key concepts• Policy arena• Values, interests, beliefs :
framing of policy issues –policy options
• Interactions
Policymaking
Model of Kingdon
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Policymaking : evidence-informed or evidence-based?
• Use of evidence is not self-evident – results from trade-offs between competing values, beliefs, interests
• Politicised process, part of democratic debate – who gets what, when and how :
– debate needs to remain democratically representative of the public’s values - cannot remain on technical level or in a closed network,
– network of actors– policy based evidence making
• Need for trust and interactions to interpret the value of evidence – policymaking = meaning-making
process– policy-decisions : context
dependent– evaluation : from accountability
to learning– conceptual use of evidence in
evaluation– timing is seldom synchronised
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence-informed policymaking – an art?
On government based practices related to policymaking:What needs to be more wisely understood is that there is also an art to policymaking. The art consists in :• knowing when and how best to deploy evidence• understanding its limits as a means to steer
policy formation. • accommodating the political and subjective
modifications which is the inevitable consequence of creating policy in a democracy.
Frustrating though it can be, it is necessary to be both scientific and artful to make an impact in the world (Simmons, 2015). • Textbook policymaking versus ‘real-
world’ policymaking : tackle ‘machine’ approach evidence-informedpolicymaking
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Source : ODI
Bridging the gap or connecting two worlds?
EVIDENCE ISPRODUCED
EVIDENCE IS ACCESSIBLE
POLICY MAKERS USE
EVIDENCE
SUPPLY SIDE DEMAND SIDE
GAP
Evaluation
Policymaking
09/04/2018 Myriam Van Parijs - WUR Conference 2018
ON EVIDENCE – TOWARDS IMPROVED USE OF EVIDENCE
09/04/2018 Myriam Van Parijs - WUR Conference 2018
What counts as evidence?
What is or what counts as evidence?• Evidence can be independently
observed and verified, and there is a broad consensus as to its contents and even interpretation; systematic investigation towards increasing the sum of knowledge.
• ‘Multiple evidences’ on a broad continuum - policy is to be informed by a wide breadth of evidence.
• Evidence is influenced by many aspects (experience, expertise & judgment, values, resources, culture, pressure groups, …).
Challenge?• “we are still far from a consensus –
intellectually or politically – regarding what should count as evidence, how it should be produced and validated, ad how it should be used to influence policymaking” (Heinrich, cited in Head, 2015)
• Lack of clarity hampers research on use of evidence
• What counts as evidence ? Differs : type of question asked, who asks, purpose of the use of evidence, evolution in understanding evidence over time, research culture.
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Multiple evidences : ways of knowing, hierarchies, and horses for courses
• Hierarchies of evidence :
• More rigorous and systematic issues of evidence will improve efficiency & effectiveness of public policies // medicine
• Exclusive focus on methodological rigor <–> other determining factors, such as contextual factors
• Critics on hierarchies
• Focus on methods loss of other useful evidence, and thus for quality of policy decisions.
• Hierarchies ignore why & how evidence will be used, context-influence, social desirability of what is measured, and generalisability of effects – for who.
• Evidence hierarchies can’t answer what his good evidence
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Horses for courses• Different ways of knowing – different types of policy issues and policy questions• Evidence matrices – appropriate evidence according to question asked
Multiple evidences : ways of knowing, hierarchies, and horses for courses
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Understanding evidence in policymaking context
• Evidence : – no conclusive proof, objectivity, ‘truth’, wisdom– timing, context, sources, perceptions– good evidence may be ignored, bad evidence may be used misleadingly
• Quality of the evidence process – refer to context, politics, purpose and use– Beyond common ‘quality criteria’ (credibility, accuracy, relevance) – Usefulness of evidence for policymakers : persuasive and actionable evidence– Social desirability of evidence – Accept nuanced generalisability – what works here, will it work there?
Some interventions worked for some people in some situations
• Strategies for evidence use - focused on push & pull efforts to get evidence in policy arena :– Push efforts : improving dissemination of evidence (briefs, systematic review – Pull efforts : capacities of policymakers to use evidence
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Understanding evidence in policymaking context
• Knowledge management : – Focuses on transmission, mobilisation, brokering– Mostly fails to recognise that decision to promote some pieces of evidence
over others, is a political choice (Parkhurst, 2016)– Assuming that evidence use is inherently an uncontested ‘good thing’ with
‘politics seen to be a ‘negative barrier’ to research utilisation, is a barrier to overcome (Parkhurst, 2016)
• Politics of evidence :– Rational paradigm : problem is the politicisation of science– Politicised paradigm : problem is the de-politicisation of politics– Both ‘problems’ lead to biases.
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Technical bias• Pieces of evidence are ‘scientifically
robust’ or ‘flawed’• Faithful interpretation according to
methods used & results obtained : invalid conclusions from valid evidence (absence of evidence seen as evidence of absence)
• Strategic cherry picking of evidence
undermines scientific consensus, increases uncertainty, perpetuates disagreements over what the evidence actually says or means
Issue bias• Situation where choice of evidence
can bias policy decision : not the choice creates the bias, but the obfuscation or the imposition of values in evidence.
• Decision to include one or another body of evidence, or to create evidence : prioritises some concerns over others.
can lead to selection or creation of evidence according to ones goals and values, and exclusion of others.
Towards improved use of evidence : address politics of evidence (Parkhurst, 2016)
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evidence-use according to the two paradigmsFrom rational and technocratic paradigm To politicised and relational paradigm
Use of evidence inpolicymaking
Rational use of evidence - take decisions.
What works?
Political use of evidence, negotiation overevidence.
What works, for who, said by whom, onwhat basis of evidence, selected accordingto which values & interests?
Type of evidence Hierarchy of evidence on top of ‘robust’evidence.
Best evidence.
Evidence comes without hierarchy anddepends on relation to the issue.
Appropriate evidence – horses for courses.Politics of evidence ‘de-politicise’ the use of evidence –
develop evidence free from politicalinfluence – use evidence ‘properly’
Recognise the influence of politics inevidence use
Quality of evidence Relate to ‘what works’ :quality/accuracy/objectivity, credibilityand generalisability, relevance
+ processes of using information +usefulness for policymaking (persuasiveand actionable) +
09/04/2018 Myriam Van Parijs - WUR Conference 2018
source : Van Parijs M., 2017
Towards improved use of evidence : appropriate evidence (Parkhurst, 2016)
1. 2. 3.
09/04/2018 Myriam Van Parijs - WUR Conference 2018
source : Parkhurst, 2016
Good evidence ? A matter of appropriate evidence (Parkhurst, 2016)
Evidence addressing policy concerns at hand
Evidence applicable to local context
Evidence constructed in ways useful to address policy
concern
Appropriate evidence for policy making
09/04/2018 Myriam Van Parijs - WUR Conference 2018
source : Parkhurst, 2016
Bridging the gap?
• Two different worlds – two different modus operandi between evaluation and policymaking :
– Timescales : fast vs slow– Perspective : present + forward/prospective – retrospective– More incentives not to use evaluation evidence, than to use : politics - blaming
• No ‘maggic bullet’ : evidence is mostly imprecise, inconclusive, complex, whereas policymakers can only use evidence if it is precise, provides clear guidance, and is formulated in sufficiently simple terms that it can be applied (Nutley et al.)
• Top 5 Barriers & Facilitators of evidence use :– Barriers : availability & access, clarity, timing/opportunity, policymaker research skills,
costs– Facilitators : availability & access, collaboration, clarity/relevance/reliability ,
relationships with policymakers, relationships with researchers
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Providing new evidence as necessary
Framing the issuePolicy makers and evidence providers
jointly scope the questions
Re-framing the issue: policy makers
and evidenceproviders jointly
interpret the evidence
Assembling existingevidence
Multi-stakeholder
evidence-policyinterface
Policy issues
Policy optionsA longer time frame allows evidence from primary
research to complement existing knowledge
In a shorter time frame, research reviews, synthesesand evidence networks will be more cost-effective
From a bridging a gap to an interaction model
Source : by Shaxson, 2016 – adapted from Defra, 2006
WHAT CAN WE CONCLUDE?
09/04/2018 Myriam Van Parijs - WUR Conference 2018
What does this mean for bringing two realms closer?
1. The dominant model of policymaking does not fit ‘real-world’ policymaking. This fields is complex, and needs craftsmanship to interact in an agile way. Use a map instead of a GPS.
2. Public policy issues are mostly wicked problems. They are not ‘solved’, only ‘re-solved’.
3. Using evidence : from individual approach, to organisational and institutional approach
4. Evaluation evidence is valuable towards policymakers, but remain realistic on ‘uptake’ – and the potential of ‘instrumental use’ of evaluation and re-value the conceptual use/enlightenment function of evaluation.
5. Evaluation evidence is also political. Recognise politics of evidence – both technical bias and issue bias.
6. Focus on appropriate evidence and appropriate use of evidence, rather than on more evidence & increase use of evidence.
09/04/2018 Myriam Van Parijs - WUR Conference 2018
Evaluation empathic towards policymaking?1. Interaction is key! Only when the condition is met that policy makers and scientists
can negotiate the available evidence and interpret the value of this evidence for policymaking, can knowledge effectively contribute to policy processes. (Simmons, 2015).
2. Finding a ‘magic bullet’/nudge to enhance uptake of evidence might be an illusion. It may not have solved any problems, but it [may have] led to a better class of problems (Rivlin, 1971)
3. Re-recognise the value of ‘conceptual use’ of evaluation evidence – instrumental.4. Evaluation processes & principles are rooted in accountability – suited for learning?
(independence vs interaction, relativity of objectivity).5. Consider more evaluation as a ‘service’ – understand the realm of the policymaker
and accept ‘politics of evidence’.6. Improving use of evidence in policymaking is a dynamic debate in turbulent times
with both a stronger demand for evidence, and increasing contestation –‘alternative facts’ & evaluation community shouldn’t stay out.
7. Understanding use of evidence will evolve through interdisciplinary understandings (policymaking, evaluation research, cognitive sciences).
09/04/2018 Myriam Van Parijs - WUR Conference 2018