Upload
mikkel
View
47
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Reflections on Realist Evaluation: Unpacking the Mechanisms of Change. Marnie Carter Ned Hardie -Boys Ella Spittle. What we will cover. Overview of realist evaluation Principles and approach Implementation of a realist evaluation Lessons from a realist evaluation - PowerPoint PPT Presentation
Citation preview
Reflections on Realist Evaluation:
Unpacking the Mechanisms of Change
Marnie CarterNed Hardie-Boys
Ella Spittle
What we will cover
• Overview of realist evaluation– Principles and approach
• Implementation of a realist evaluation
• Lessons from a realist evaluation• Questions and discussion
What is realist evaluation?
• Realist evaluation is an approach that can be broadly applied to different types of evaluations of policies and programmes
• The term was drawn from Pawson & Tilley’s book, Realistic Evaluation (1997)
Why realist evaluation?• It addresses the limits of traditional
approaches, which– Only ask “does this programme work?”– Assume that programmes can be credited
with general cause-and-effect statements
• Realist evaluation focuses deeper on why a programme works– “What works for whom, in what contexts,
and how?”
How does realist evaluation work?
Realist evaluation focuses on
• Mechanisms: Why a programme ‘works’ (or does not work)
• Context: What conditions are required for the mechanisms to work
• Outcomes: The results of different mechanisms working in different contexts
Mechanism + Context = Outcomes
Principles of realist evaluation I1) Programmes, policies and
interventions address problems by influencing change
2) Change is instigated by allowing participants to make different choices
3) Making different choices requires changing resources and/or reasoning
4) The combination of resources and reasoning allow participants to change their behaviour (the mechanisms)
Principles of realist evaluation II5) Changes in behaviour will vary
depending on participants’ context6) Context, therefore, affects which
mechanisms are triggered and what outcomes are achieved
7) Outcomes are varied because different mechanisms are triggered in different contexts
8) Variation is why realist evaluation findings are always provisional: “What works for whom, in what contexts, and how?”
Realist evaluation in practiceEvaluation of changes to the Road User
Charges (RUC) system• Examine initial impacts – What mechanisms have been triggered?
• Evaluate outcomes– Has activation of those mechanisms resulted in
the expected outcomes?• Recommendations– How can the system better influence activation
of which mechanisms, by which stakeholders, in which contexts, to achieve desired outcomes?
RUC system – background
• RUC is a tax for road “wear and tear”• A number of changes were made, but
the main change was to reduce complexity and opportunities for evasion– The previous RUC system charged road
users based on a vehicle’s net weight– The new RUC system charges road users
based on the maximum potential net weight
Evaluation design
• Two stage– Framework– Implementation
• Why a realist approach?• Steps– Theory-building (outcomes and
mechanisms)– Questions, measures, methods and
sources
Efficiency I
Outcome
1. Reduced compliance costs for operators
2. Reduced administrative complexity for government
Mechanisms
• Behavioural response to simplify admin processes
• Organisational response by NZTA to simplify admin processes and streamline payment methods
• Organisational response by NZTA to reduce complexity of investigations to recover unpaid RUC
Efficiency II
Outcome
3. Improved efficiency in vehicle use
4. Simplified enforcement of RUC
5. Increased uptake of electronic systems
Mechanisms
• Incentive for operators to load vehicles up to their RUC weight
• Organisational response by NZ Police to reduce officer time spent on enforcement
• Simplified processes and reduced costs incentivise new providers to enter market
Integrity I
Outcome
6. Reduced RUC evasion
Mechanisms
• Deterrence: greater chance of being caught changes risk perception
• Deterrence: perception of consequence of being caught leads to change in behaviour
• Deterrence: requirement to produce EDR acts as a deterrent
Integrity II
Outcome
7. Increased understanding of the system
Mechanisms
• Clarification/ communication of changes to system
• Monitoring and evaluation of the system
Equity
Outcome
8. Revenue neutrality within groups of vehicles (operators pay fair share)
Mechanisms
• Organisational response by government to monitor impacts and ensure no disproportionate effects on operators
Cost recovery
Outcome
9. Reduced late payment
10. Improved recovery of unpaid RUC
Mechanisms
• Penalties are sufficient to incentivise people to pay on time
• Detection: NZTA receives info from WoF providers and is able to identify evasion
• Investigative: the issuing of binding assessments assists in recovery
Implementing the RUC evaluation• Evaluation questions placed emphasis on
behavioural change - What impact have the RUC changes had on the level of
evasion?- To what extent have the RUC changes contributed to
changes in operator perception and behaviour regarding evasion?
• Survey and interview tools were designed to seek evidence of behavioural change
• Analysis focused on- whether the expected mechanism(s) had been activated- for whom, how and in what context the outcome had
been achieved
Example: efficiency in vehicle use
• Previous system: transport operators purchase RUC for the exact weight they will carry
• Policy change: Fixed ‘RUC weight’ for vehicles • Mechanism: Incentive for operators
to load vehicles up to their RUC weight
• Expected outcome: Transport operators will use their vehicles more efficiently
• Required context: transport operators are able to, and do, change their business practices
Evaluation findings• Commercial transport operators were already
loading as efficiently as possible• Other considerations (safety, customer needs)
were more important than RUC in loading decisions
• Many loads ‘cubed out’ before reaching the maximum weight
• Vehicles tend to be load-specific making it difficult to ‘back load’
The mechanism was not activated
Example: Recovery of unpaid RUC
• Previous system: time-intensive manual process for NZTA to identify unpaid RUC amongst light diesel vehicles; opportunity cost
• Policy change: Odometer reading reported as part of Warrant of Fitness inspection
• Mechanism: NZTA is better able to detect evasion
• Expected outcome: Unpaid RUC will be identified and recovered
• Required context: NZTA develops a system to receive data from WoF providers, and uses the information to recover debt
Evaluation findings• NZTA developed an automatic
reconciliation and invoicing system
0
1000
2000
3000
4000
5000
6000
Jul-1
1Au
g-11
Sep
-11
Oct
-11
Nov-
11De
c-11
Jan-
12Fe
b-1
2M
ar-1
2Ap
r-12
May
-12
Jun-
12Ju
l-12
Aug-
12Se
p-1
2O
ct-1
2No
v-12
Dec-
12Ja
n-13
Feb
-13
Mar
-13
Apr-
13
Thou
sand
s ($)
• The amount of invoiced debt has increased substantially
• Only 14% of invoiced debt has been recovered
The mechanism was partially activated
Reflections
It can be easy to get bogged down labelling mechanisms
Lesson one
Focus on describing mechanisms and what is needed to activate them – not naming them
The perils of tunnel vision
Avoiding tunnel vision• Expected outcome of reduced compliance costs
for transport operators
• Transport operators were expected to modify their administrative processes in response to a simpler system
• An online survey and case studies were used to seek evidence of behavioural change
• No evidence that operators had changed their behaviour was found
Initial conclusion: outcome was not achieved
However…• Changes were made to the NZTA RUC
administration fee structure
• 30% drop in NZTA admin fees (although this was a small portion of overall RUC compliance costs)
• Transport operators had therefore achieved a small reduction in cost, despite not changing their behaviours
Revised conclusion: the outcome was partially achieved
Lesson two
Behavioural change of the intended recipients is not the only means through which outcomes can be activated
Reduced evasion of RUC• Previous system: opportunity for operators to under-
nominate weight; outdated penalties for other forms of evasion
• Policy change: Fixed RUC weight; updated offences and penalties regime
• Mechanism: The removal of weight-based evasion is built into the system (no mechanism); updated penalties act as a deterrent to other forms of evasion
• Expected outcome: Evasion will reduce
• Required context: Transport operators are aware of the new penalties, which causes their risk perception to change resulting in modified behaviour
Evaluation findings• Police Heavy Vehicle Operation data
showed that evasion had reduced from 4% in 2012 to 1.2% in 2013
• No transport operators we interviewed admitted that they had previously evaded RUC and now were not
• We were therefore unable to identify how and why this effect had occurred (other than through speculation)
Lesson three
For sensitive or morally questionable topics, a focus only on what (rather than how and why) is appropriate.
Overall reflections• The realist evaluation approach led us to lines of
enquiry we may not have otherwise considered
• Clearly defining the mechanism of change assisted us to frame what evidence we were looking for
• Provided a means to demonstrate how and why the changes to the RUC system ‘worked’
• Helped to get inside the black box
• A useful tool for the evaluation kete
[email protected]@allenandclarke.co.nz