2012 Desiree Nangle, Evaluation Project Manager [email protected]

2012Desiree Nangle, Evaluation Project [email protected]

SIMPLE

COMMON SENSE

PLAIN LANGUAGE

MINIMUM PAPER

USEFUL

With thanks to

The Fiscal Policy Studies InstituteSanta Fe, New Mexico

What is Results Based Accountability?

• Starts with ends and works backwards to means– Ends are how customers are better off when

the program works the way it should

• RBA is meant to get you from talk to action quickly

• Used for continuous program improvement

• Inclusive process

Results Based Accountabilityis made up of two parts:

POPULATION ACCOUNTABILITYIs about the well-being ofWHOLE POPULATIONS

For Communities – Cities – Counties – States - Nations

PERFORMANCE ACCOUNTABILITYIs about the well-being of

CUSTOMER POPULATIONSFor Programs – Agencies – Services

August, 2012

7 Performance Accountability Questions

1. Who are our customers?

2. How can we measure if our customers are better off?

3. How can we measure if we are delivering services well?

4. How are we doing on the most important of these measures?

5. Who are the partners that have a role to play in boing better?

6. What works to do better, including low-cost and no-cost ideas?

7. What do we propose to do?

7 Performance Accountability Questions

1. Who are our customers?

2. How can we measure if our customers are better off?

3. How can we measure if we are delivering services well?

4. How are we doing on the most important of these measures?

5. Who are the partners that have a role to play in boing better?

6. What works to do better, including low-cost and no-cost ideas?

7. What do we propose to do?

How much service did we deliver?

Performance Measures

How welldid we

deliver it?

How much change / effect

did we produce?

What quality of change / effect

did we produce?

Quantity Quality

Effe

ct

Effo

rt

How much did we do?

How well did we do it?

Is anyone better off?

Quantity Quality

Effe

ct

E

ffort

• Skills and Knowledge• Attitude and Opinion• Behaviour• Circumstance

• Client Satisfaction• Standards • Unit Costs• Wait List• Staff Quality• Targets

• Number Clients• Number Activities

Types of Measures Found in Each Quadrant

How much did we do? How well did we do it?


Quantity Quality

Effe

ct

E

ffort

% of parents with more of a child focusoassessed using client self-reported perceptions in questionnaire

o % of parents with less parent to parent conflict

o % of parents with decreased litigationoAssessed via Family Court feedback questionnaire

o % of clients seeking other services

o % of parents with improved ability to co-parent

o % of parents with improved self-awareness

Number of groups Number of clients

No shows/Drop out ratesBookings for intakeIntake groupGroup to completion

Level of satisfaction Family Court has with MDF

Staff satisfaction Reconnects

Mums and Dads Forever2012

How much did we do?

Not All Performance Measures Are Created Equal



LeastImportant

Quantity Quality

Effe

ct

E

ffort

MostImportant

Least

Most

AlsoVery Important

How much did we do?

The Matter of Control



Quantity Quality

Effe

ct

E

ffort

LeastControl

MostControl

Program: _______________

PerformanceMeasureBaseline

Story behind the baseline

--------------------------- ---------------------------

Partners

--------------------------- ---------------------------

Three Best Ideas – What Works

1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost

ONE PAGE Turn the Curve Report: Performance

4. --------- Off the Wall

Example of a Turn the Curve Report

Results Based Accountability

Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.

Defined Service Users

All Anglicare WA staff

Data Development Agenda

1. Determine population outcomes our services contribute to

2. Determine whether or not there are questions we can ask across services that use the same wording and same scale

Headline Performance Measures

% of staff who understand what RBA is

% of programs in Phase 2 of RBA implementation

% of programs in Phase 3 of RBA implementation

% of staff who support RBA implementation within Anglicare WA

Story Behind the Baselines

While all four baseline indicators increased from April 2012 to August 2012, some had greater increases than others. While the percentage of staff understanding RBA increased significantly (from 54% to 74%) since April, the increase in staff supporting RBA use in Anglicare WA had a much less significant increase of only 2.5%. Part of the reason for the smaller increase is most likely due to the smaller response rate on this survey (~25% of staff). The percentage of programs in Phase two jumped considerably and progress was also made in moving services from Phase two to Phase three since April (an 8% increase). This increase in services moving to Phase three is most likely due to changes that were made in the process based on suggestions given by Mark Friedman while he was here in June.

Partners Who Can Help Us Do Better

All Anglicare WA Programs and Staff

The Community Sector

Other Not-for-Profit Organisations

National and Federal Government

Trained RBA Trainers

Action Plan to Continue Turning the Curves

In an effort to continue improving performance on our four headline measures, Anglicare WA will:•Revise the length of the survey and possibly how often it is being sent out to staff in an effort to increase staff response rates•Continue focusing on data that services do have to begin the process of RBA implementation and work around helping them develop tools to address each program’s data development agenda•Highlight success stories and curves turned to encourage a culture of RBA monitoring among staff

RBA HEADLINE PERFORMANCE MEASURES

RBA on RBA

(roll out with services)

2012

How much did we do? How well did we do it?


Quantity Quality

Effe

ct

E

ffort

• # of active services by phases

• # of staff involved in active services by phase

• staff satisfaction with training and support

• # RBA facilitators trained

• time to self-sustainability

• time programs spend in each stage

• validity of measures• validity of data

• % staff that understand what RBA is• % staff that support RBA within Anglicare

WA• % services implementing RBA • % services using RBA for program

improvement• % services with the capacity to use RBA

unassisted• % of curves turned

33%(48 out of 61 services) =78%

66%

81%

19 in Phase 37

75%

(1 service) = 2%

RBA on RBAUpdated: March 2013

Do staff understand what RBA is?

December 2011: 53.6% AgreeApril 2012: 68.3% Agree

August 2012: 73.5% AgreeMarch 2013: 80.9% Agree

2013

Do staff think RBA is good for Anglicare?

December 2011: 55.2% AgreeApril 2012: 60.4% Agree

August 2012: 66.3% AgreeMarch 2013: 66.1% Agree

2013

Results Based Accountability March 2013 Headline Measures Report

Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.

Defined Service Users

All Anglicare WA staff


1. Determine population outcomes our services contribute to

2. Determine whether or not there are questions we can ask across services that use the same wording and same scale

Headline Performance Measures

% of staff who understand what RBA is

% of services in Phase 3 of RBA implementation

% of staff who think RBA will help improve their service

% of staff who support RBA implementation within Anglicare WA



All Anglicare WA Programs and Staff

The Community Sector

Other Not-for-Profit Organisations

National and Federal Government

Trained RBA Trainers


In an effort to continue improving performance on our four headline measures, Anglicare WA will:•…………•…………..•…………..

RBA HEADLINE PERFORMANCE MEASURES

Open-ended response

• Challenges of RBA • Opportunities with RBA

2012

Case Study

YES! Housing Project

YES! HousingTURNING THE CURVE

How Much Did We Do? How Well Did We Do It?

# of housed clients # of referrals made by DCP # of young people on the waitlist # of allocations made by DoH # of young people who disengage after being housed # of days on waitlist # of young people who complete the series of

workshops # of clients that receive a certificate for completion

of ILS workshops

% of young people with case plans % of young people who failed tenancies/eviction rate % of qualified staff who have completed core training

including:o Attending and participating in regular supervisiono Proper ways trainingo TAS Training: Tenancy Law for Community

Workers & Advocating for Public Housing Tenantso Other relevant training as identified in supervision

% of young people satisfied with the service % of HSO from DoH who are satisfied with the program % of clients referred to other agencies % of waitlist population, housed population, and

mainstreamed population who are:o Male/femaleo Indigenouso Parents

Unit Cost (cost/#of clients) Referrals from DCP are regular and timely

Is Anyone Better Off?

% of clients who access safe, secure, and stable housing at the completion of case management % of clients that demonstrate improved knowledge of ILS skills (i.e. home, work, education, parenting, tenancy

and healthy relationships) % of clients that ustilse strategies developed through case management to address identified barriers to

maintain their housing % of clients successfully linked to appropriate specialist services % of clients regularly engaging with their support worker to develop their case plans % of clients who achieve set goals

YES! Housing July – December 2012: RBA Headline Program Measures Report

Service Description: Anglicare WA’s YES! Housing program provides permanent housing for young people who are supported to establish independent living through long term housing, including public housing properties.

RBA Headline MeasuresRBA Headline Measures


Graph 1: There continues to be an increase of young people on the waitlist trying to access safe, secure affordable accommodation. Unfortunately YES! Housing does not have the resources necessary to meet this ever-growing need.Graph 2: Data from graph 2 is based on the average difference from pre score to post score for the combined four ILS workshops (health, tenancy, parenting skills and home skills). The total average increase went down 1% in this past 6 months, mainly due to the health and relationships survey (deceased from a 31% increase Jan-June to 12% increase July-Dec). The Home Skills workshop increased along with Tenancy Support while the Parenting Skills results decreased in the past 6 months.Graph 3: 33.3% of clients with a case management plan mostly achieved or fully achieved their most important goal.Graph 4: 96.8% of clients engaged have a case management plan. Graph 5: 90% of clients self referred to the program or were referred through word of mouth. The remaining 10% were referred from DCP and Community agencies.


In an effort to continue improving performance on our headline measures, YES! Housing will:•Research other options for content, presentation methods and implementation ideas from clients participating in the Health & Relationship and Parenting workshops.•Continue to implement the case management process to assist clients in achieving their most important goals through regular outreach visits.•Continue to create case plans with housed and waitlisted clients through implementing the case management process and developing strategies to engage clients who have disengaged from the program.•Develop stronger networks with DCP, DoH, Community Organisations, other housing organisations to offer support for their clients through case management process more accommodation options for clients in other housing situations to maintain a successful tenancy.•Implement strategies developed in consultation with DCP during our recent Program Review to access alternative affordable, safe, secure housing for clients on the YES! Housing waitlist by establishing and developing relationships with private real estate.•Review our current waitlist and remove ineligible clients.


Internal partners includingY-SHAC, Foyer, YPSG, Youth Services Manager, Fundraising

External partners including Department of Housing, Department for Child Protection, Foundation Housing, Community Housing Real Estate,YACWA,, young people,


Yes! Housing will work on developing and implementing new survey tools:•to measure Independent Living Skills knowledge increase that is only post-workshop (not pre and post)•to measure the Department of Housing’s satisfaction with the work done by YES! Housing

Clients

Young people 15-17 years who are homeless or at risk of being

homeless.

Phase 1

Awareness raising

• Emails• Presentations to groups

Phase 2

Service Planning

• Scoping and readiness assessment

• Workshop • Define clients• Agree outcomes• Agree measures• Data strategy• Plot baselines

Phase 3

Service Implementation

• Data collection• Plot curves• Monitor and use

Phase 4

Normalise

• Refine• Use unassisted

Program: _______________

Performance Measure PerformanceMeasureBaseline

Story behind the baseline --------------------------- --------------------------- (List as many as needed)

Partners --------------------------- --------------------------- (List as many as needed)

Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost

ONE PAGE Turn the Curve Report: Performance

4. --------- Off the Wall

Four Phases of Implementation

What we do is an not a science. It can’t be measured

What does the data say?I know I should

be measuring but I don’t know how

What if my program doesn’t show success?

Check out these cool metrics!

The Task (1)

Quadrant exercise: Performance measures

• How much did we do?• How well did we do it?• Is anyone better off?

How much did we do?



Quantity Quality

Effe

ct

E

ffort

• Skills and Knowledge• Attitude and Opinion• Behaviour• Circumstance

• Client Satisfaction• Standards • Unit Costs• Wait List• Staff Quality• Targets

• Number Clients• Number Activities

Types of Measures Found in Each Quadrant

Documents

2012 Desiree Nangle, Evaluation Project Manager [email protected]