Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Social Impact Bonds Workshop No. 2:Outcome Measures
Les HemsEY (Ernst & young)
21 September2016
Page 2
Outcomes measures: Agenda
1. Fundamental importance of outcomes and outcomes measurement
2. Identifying and defining outcomes
3. Why client cohort groups are important
4. Which measures to use from the continuum of measures
5. Key learning
6. Questions and Answers
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 3
Fundamental importance of Outcomes and Outcomes Measurement
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 4
Exam question: Potential benefits of Social Impact Bonds
“are suited best to situations where there are a number of complex, often
interrelated, factors that impact negatively on individuals and there are no clear,
effective and existing programs directly targeting the problem.”
Statement of Opportunities Page 3
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
• for clients through the measurement and collection of outcomes-driven data
Focus on agreed outcomes
• allowing providers to deliver solutions that are specifically tailored to client need
Flexibility in service delivery
• rather than providing crisis services at later stages when interventions may be more complex or costly
Evidence base for early intervention
Clients
Service providers
Government
Page 5
Identifying and defining outcomes
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 6
Tell a compelling story: Identifying and defining outcomes
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Theory of Change
• Story of how change is created in an activity/ program
• Output: Tells you an activity has taken place and is usually quantitative (e.g. number of people trained)
• Outcome: The change that occurs as a result of an activity (e.g. improved well-being of training participants)
• What is the evidence that underpins the story? Literature / research and program track record
Inputs Activities Outputs Outcomes
Page 7
Things aren’t simple: Outcomes hierarchy and different client pathways
Stable housing
Positive family, kinship & community connections
Physical & mental health (including trauma)
Life skills, parenting, financial management, tenancy management
Learning & education
Employment / productive time
Desistance Independence
Page 8
No program is an island: System & network
Referral (30%)Referral
(25%) Referral (25%)
Referral (10%)
Youth Justice (DHHS)
Child Protection (DHHS)
DOE
Home School Liaison Officer
(DOE)
TOOL
UTURN
TasTAFE
Edzone
Work Skills
Centrelink
Interagency Support Teams Clarence Plains
Red Cross Bridgewater
Youth Magistrate
Legal Aid
Police and Youth
Community Club
Tas Police Early Intervention
Unit
Royal Hobart Hospital
Sorell Youth Centre
Link Youth Health Service
Pulse Youth Centre
GlenorchyTYSS Mission Australia
Supporting Young People on Bail
Linkage into education services
Page 9
Rome was not built in a day: Short/medium /long term outcomes for young people
Physical health
Independence
Engagement with
Education and Training
Employability
Positive Social Connections
Future Aspirations
· Less smoking, alcohol and drug use· Regular gym visits· Regular attendance to medical
appointments
· Financial independence· Transport without reliance on others · Stable and secure accommodation
· Routinely attending education or training
· Progress towards completing school / obtaining work-related certification
· Sociable, attentive behaviour in class
· Interview-related skills· Feeling comfortable at work · Acceptance of authority
· Associating with new, positive circles· Less time spent with negative cycles· Ability to deal with peer pressure
· New, positive perception of life · Setting goals for the future· Less court-related stress· Less pro-criminal thoughts· Less interactions with court/police· Reduced sentencing / avoided
detention (SYPoB)
· Awareness of the benefits of regular physical activities / healthy eating habits
· Awareness of services to assist with personal needs: health / education / employment / housing)
· Receiving vital personal documentation for ID
· Support for educational / vocational activities
· Motivation to go to school
· Support to find employment · Understanding of appropriate job
opportunities / work environment· Motivation to go to work
· Support for recreational activities and equipment
· Engaging in recreational / social / community activities
· Family support· Awareness of offending behaviours
and triggers· Access to a mentor, positive adult role
model· Assistance to deal with personal issues
through “reflective communication”· Motivation to get up from bed and 'do
things'· Meeting legal obligations
Young person
· Developing a Bail Support Plan with Youth Worker
· Interacting/spending one on one time with Youth Worker
· Attending court with Youth Worker
Save the Children / Youth Worker
· Gateway to services providing support to disengaged young people
· Representing young person in court
· Providing reports to Magistrate and Youth Justice
ACTIVITIES
SHORT-TERM OUTCOMES
MEDIUM-TERM OUTCOMES
LONG-TERM OUTCOMES
· Interpersonal / communication skills· Anger management strategies· Less violent behaviour· Ability to trust
Improved Family
Relations
Page 10
Exam question +ve return for Government: Outcomes for Government
Professionalism Expertise
Commitment
SYPoB and TRaD program costs
SCA Youth Workers and
Program Manager
Specialised Youth Court (pilot)
Department of Health (grant)
Department of Education (grant)
Tasmanian Community Fund
(grant)
Law Foundation (grant)
Youth Justice (DHHS)
Specialised Youth Court
PoliceDepartment of Education
INPUTS ACTIVITIES OUTPUTSSHORT-TERM OUTCOMES
MEDIUM-TERM OUTCOMES
LONG-TERM OUTCOMES
FINAL OUTCOME
Gateway to services
Immediacy of response
Holistic problem-
solvingBuilding
rapport and trust
Consistent Court decisions
Change in Court attitude
Increased number of young people on bail
Reduced number of young people in
detention
Voluntary participation
Effective collaboration and
trust between youth justice agencies
Young people can navigate complex
youth justice system
Appropriate COMMUNITY
support
Appropriate DETENTION support
Gap in service delivery
Reduced arrests
Reduced sentencing costs
Reduced crime in Tasmania
Reduced NUMBER of
arrests
Reduced SEVERITY of
arrests
Community-based
supervision
Detention (Ashley)
Court (admin)
Police
Legal Aid
Early intervention
Customised approach
Youth justice agency
Save the Children
Save the Children
KEY
Page 11
Why client cohort groups are important
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 12
Client segmentation & outcomes matrix
Client segmentation
Identify client segments – do their referral pathways differ – do their outcomes differ?
• Many to many relationships between client segments and outcomes
How will the program/intervention respond to these different client segments and outcomes? What are the differential costs of different client segments?
• High, medium and low vulnerabilities and capabilities –complexity of service
Clients/participants are not all the same:• Demographics• Vulnerabilities• Capabilities• Expectations and aspirations
Defining client cohorts
How do clients get referred in to the program? Eligibility and profiling / assessment tools
Page 13
Understanding complexity
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Inputs Activities Outputs Outcomes
Client segment #1High
Client segment #2Medium
Client segment #3Low
How does the theory of change differ between the key client segments identified in the Stakeholder Segmentation Matrix:
Page 14
Linking client segmentation to cost savings to government
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Where do the benefits flow?
• The benefits (financial, economic and social) should outweigh the costs• There should be substantial evidence demonstrating that the program
activities have the capacity to generate the benefits claimed• The margin of benefit value over cost value needs to be sufficiently
robust to sustain the payment stream from Government
The Nature of the Benefits
The Recipient of the Benefits
The Timing of the Benefits
• Cash savings (current)• Cash savings (future)• Avoided costs• Productivity/capacity
enhancements• Other measureable
benefits
• Single Government Agency
• Multiple Government Agencies
• Commonwealth Government
• Private Individuals• Community
• Short Term• Medium Term• Long Term
Incr
ea
sin
g D
iffi
cult
y in
e
sta
blis
hin
g leve
l o
f B
en
efi
ts
Page 15
Which measures to use from the continuum of measures
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 16
Evaluation frameworks & outcomes measurement
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Key considerations
Outcomes are the long term change that occurs as a result of an activity
Inputs Activities OutputsIntermediate
OutcomesFinal
outcomes
Typically increasing in time and complexity to measure, but also in importance to policy makers and investors
Driver of benefits to individuals and governments
► Resources –labour, staff, skills, material, IT systems, facilities, funding
► Activities undertaken to deliver high quality outputs
► Short-medium term objectives that contribute to final outcomes
► End service delivered
► Intended long term impacts
Page 17
Lots of things can be measured: What is most important for young people?
Physical health
Independence
Engagement with
Education and Training
Employability
Positive Social Connections
Future Aspirations
· Less smoking, alcohol and drug use· Regular gym visits· Regular attendance to medical
appointments
· Financial independence· Transport without reliance on others · Stable and secure accommodation
· Routinely attending education or training
· Progress towards completing school / obtaining work-related certification
· Sociable, attentive behaviour in class
· Interview-related skills· Feeling comfortable at work · Acceptance of authority
· Associating with new, positive circles· Less time spent with negative cycles· Ability to deal with peer pressure
· New, positive perception of life · Setting goals for the future· Less court-related stress· Less pro-criminal thoughts· Less interactions with court/police· Reduced sentencing / avoided
detention (SYPoB)
· Awareness of the benefits of regular physical activities / healthy eating habits
· Awareness of services to assist with personal needs: health / education / employment / housing)
· Receiving vital personal documentation for ID
· Support for educational / vocational activities
· Motivation to go to school
· Support to find employment · Understanding of appropriate job
opportunities / work environment· Motivation to go to work
· Support for recreational activities and equipment
· Engaging in recreational / social / community activities
· Family support· Awareness of offending behaviours
and triggers· Access to a mentor, positive adult role
model· Assistance to deal with personal issues
through “reflective communication”· Motivation to get up from bed and 'do
things'· Meeting legal obligations
Young person
· Developing a Bail Support Plan with Youth Worker
· Interacting/spending one on one time with Youth Worker
· Attending court with Youth Worker
Save the Children / Youth Worker
· Gateway to services providing support to disengaged young people
· Representing young person in court
· Providing reports to Magistrate and Youth Justice
ACTIVITIES
SHORT-TERM OUTCOMES
MEDIUM-TERM OUTCOMES
LONG-TERM OUTCOMES
· Interpersonal / communication skills· Anger management strategies· Less violent behaviour· Ability to trust
Improved Family
Relations
Page 18
What metrics are directly linked to +vereturns for Government
Professionalism Expertise
Commitment
SYPoB and TRaD program costs
SCA Youth Workers and
Program Manager
Specialised Youth Court (pilot)
Department of Health (grant)
Department of Education (grant)
Tasmanian Community Fund
(grant)
Law Foundation (grant)
Youth Justice (DHHS)
Specialised Youth Court
PoliceDepartment of Education
INPUTS ACTIVITIES OUTPUTSSHORT-TERM OUTCOMES
MEDIUM-TERM OUTCOMES
LONG-TERM OUTCOMES
FINAL OUTCOME
Gateway to services
Immediacy of response
Holistic problem-
solvingBuilding
rapport and trust
Consistent Court decisions
Change in Court attitude
Increased number of young people on bail
Reduced number of young people in
detention
Voluntary participation
Effective collaboration and
trust between youth justice agencies
Young people can navigate complex
youth justice system
Appropriate COMMUNITY
support
Appropriate DETENTION support
Gap in service delivery
Reduced arrests
Reduced sentencing costs
Reduced crime in Tasmania
Reduced NUMBER of
arrests
Reduced SEVERITY of
arrests
Community-based
supervision
Detention (Ashley)
Court (admin)
Police
Legal Aid
Early intervention
Customised approach
Youth justice agency
Save the Children
Save the Children
KEY
Page 19
The holy grail: Robust lead indicators and metrics linked to benefits to Government
► Theory of change and existing evidence may reveal “lead indicators” –
actions that are correlated with desired outcomes e.g.
► Progress on education pathway – course attendance and completion
► Progress on employment pathways – use of jobactive metrics
► Housing stability – maintain tenancy
► Successful completion of drug / alcohol rehabilitation program
► May also reveal metrics that are directly linked to Government
benefits – cost savings / avoidance, productivity improvements
► Reduced incidence and severity of arrests
► Reduced rates of recidivism / incarceration
► Reduced use of emergency housing
► Reduced use of emergency hospital services / hospitalisation
► Reduced incidence of child protection notices
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 20
United Kingdom: Payment by Results (PbR) drug and alcohol recovery pilotsOutcome measures:
► Abstinence from all presenting substances: proportion of clients who have
stopped using all their presenting substances
► Successfully completed treatment: proportion of clients who became free
of dependency
► Resolved housing problems: proportion of clients in the pilots who started
with a housing issue (e.g. they were homeless) but can now access and
sustain suitable accommodation
► Stopped injecting
► Improved quality of life: proportion of clients who have achieved a score on
the Treatment Outcome Profile (TOP) at or above the functional range of
those clients who go on to leave treatment successfully and do not return
The TOP measures change and progress in key areas of the lives of people
being treated in your drug and alcohol services. TOP consists of 20 simple
questions focusing on the areas such as substance use, injecting risk behaviour,
crime, health and quality of life.
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 21
Building the counterfactual:
Historical baseline: - Comparison between before and after the start of the
pilot:
► Magnitude of change was assessed for each of the five outcome measures
over the pilot duration (April 2012 to February 2013). It was then compared to
the same period 12 months before the pilot.
► Service users were segmented into five complexity groups based on their
likelihood of achieving each of the outcomes. This enabled commissioners to
take into account any changes in the profile of service users.
National comparison: Comparison to average for rest of England:
► Magnitude of change was assessed for each of the five outcome measures. It
was then compared to the aggregated performance for the rest of the country
over the same period (results were also broken down by the five complexity
groups).
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 22
Activity – Develop outcomes measurement
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
How would you measure the outcomes of the GoodHope Society’s program?
EstablishDesign & measure
Analyse Value Assure Use
• What measures?• What data needed?
• What outcomes will you measure?
• How to ensure statistical significance?
• How to improve the program?
• How to foster innovation?
• How would you estimate the monetary benefits?
• Options for independent verification? Pros/cons?
Page 23
Key learning
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
Page 24
Key learning
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation
• a compelling story above how a program achieves outcomes for individuals and benefits to Government
Tell a story
• simplicity may hide costs and barriers to achieving outcomes
Embrace complexity
• referrals in, differentiated program delivery, links to support services, different outcomes mix
Recognise different client pathways
• drive different benefits for governmentClient segments
• that “predict” outcomes but can be measured in real / short time
Find lead indicators
• to validate lead indicator and critical success factors
Continue to measure long term outcomes
• cherry picking and parking difficult clientsAvoid risk of perverse
behaviours
Page 25
Question and Answers
Copyright © 2014 Ernst & Young Australia. All Rights Reserved. Liability limited by a scheme approved under Professional Standards Legislation