Helping Our Clients Identify and Measure Outcomes

Helping Our Clients Identify and Measure

Outcomes

Edward M. Haugh Jr.ESC Consultant

ESC Brown Bag # 2, April 17 2013

Introduction

By the end of this session:• Define outcomes• Differentiate outcomes from outputs• Define the logic model• Describe several uses of a logic model• Develop a logic model• Define “indicators”• Develop a logical framework

Impact

• You will love or at least tolerate outcome-based planning

• You will be able to discuss outcomes based planning, monitoring and evaluation with your ESC clients

What is an outcome

Outcomes are benefits or changes for individuals or populations during or after participating in a program:

Outcomes

• Often expressed as verbs in past tense–Improved (nutrition, morale)–Increased (in employment) –Reduced (HIV transmission)–Enhanced (knowledge)–Maintained (vaccination coverage)

Outcomes

Reduced smoking among teenagers

Changed attitudes towards smoking (by non-smoking teenagers

School readiness among children attending preschool program

Other examples

Improved board performance after an ESC engagement

Improved strategic focus after an ESC engagement

Employment after attending a dress-for- success program

Improving the Delivery of Prenatal Care and Outcomes of Pregnancy: A Randomized Trial of Nurse Home Visitation

Outcomes of Nurse Visit Program

• Became aware of more community services• Attended birth classes more frequently• Made more extensive use of the nutrition

supplementation program for women, infants and children

• Made greater dietary improvements,

Outcomes of Nurse Visit Program

• Reported that their babies’ fathers became more interested in their pregnancies

• Were accompanied to the hospital by a support person during labor more frequently

• Reported talking more frequently to family members, friends and service providers about their pregnancies and personal problems

• Had fewer kidney problems

Outcome Indicators

• Outcomes sometimes confused with outcome indicators– Outcome - girls participating in a training program

obtain formal employment– Outcome indicators – the # and % of participants

employed within 3 months of completing the program

Importance of Focusing on Outcomes

• Outcomes are what matter to stakeholders• Focusing on outcomes promotes

accountability• Government, and funders demanding real

tangible results on the ground• Measuring outcomes helps us know if we are

we really getting value for our money

Outcomes help us

• Strengthen existing services• Target effective services for expansion• Identify staff and training needs• Develop and justify budgets• Prepare long-range plans• Focus board on programmatic issues

Funder Asked ESC the Following:

• What are your measurable outcomes, i.e. number of communities impacted, number of children served?

• Mark each category that applies to your grant measures.Increased visibility/awareness of issue

Effected policy changeEngaged constituents & beneficiaries and participationIncreased resources for issueEducated constituentsImproved condition/ quality of life in communityInfluenced stakeholders

Outcomes vs. Outputs

• Outcomes are the results

• Outputs are the tangible products produced by an intervention–Usually expressed as nouns–Can be counted

Outputs

• Classes taught• Brochures distributed• Clients served• Contracts completed• Board retreats completed• Strategic plans prepared

Many Nonprofits Report Outputs Rather than Outcomes

• Main Street Homeless Shelter provided 2,903 nights of safe shelter for 173 men.

• 23,661 hot, nutritious meals were served at the Maple Avenue Kitchen.

• 100 residents received free health care and 168 residents received free mental health care.

What Wrong with Using Outputs• Mosquito bed nets might be used for

wedding dresses or fishing nets • Loans repaid might not mean income

generated• Job training might not lead to jobs

So Why Don’t We Focus on Outcomes

• Outputs are tangible• Outcomes are intangible• Its easier to count tangibles than to describe

intangibles• Outcomes take longer to produce• Measuring outcomes is challenging • Measuring outcomes can be expensive and

time consuming

But Tracking Outcomes will Not……

• Explain why an outcome is or is not being achieved– Still need to track inputs, activities, outputs

• Prove that the outcome is the result of your program

Outcome Model

Logic Model

Inputs Processes Outputs Outcomes Impact

Logic Model Definitions

• Inputs are the resources required to undertake activities

• Activities (or processes) are what the agency, program or strategy does to fulfill its mission

• Outputs – tangible products• Outcomes – changes, benefits• Impact - a longer-term outcome

Inputs ProcessesOutputs Short-term

Outcomes

Medium-term

Outcome

• Money• Staff time• Facilities• Equipment• Laws• Regulations• Funders

requirements

• Educate consumers

• Educate providers

• Distribute vaccines

• Establish data base

• Establish reminder systems

• Monitor Coverage

• Brochures• Vaccine supplies• Doses of

vaccines delivered

• Data bases established

• Providers monitoring

• Parents aware of vaccine benefits

• Provider awareness

• Changed attitudes

• Children immunized

• Reduced incidence of HPV

Reduced number of girls who are infected with HPM

Long-term

• Reduced incidence of cervical cancer

Reduced mortality for cervical cancer

A logic model for a program to distribute HPV Vaccine

Impact

Reduced premature death

Inputs Activities OutcomesOutputs Impact

It is important to note the broad environmental context of the intervention:

Political environmentMacroeconomic picture

Other “players” in field Public attitudes Environment

Policy context

Source: Imas and Rist (2009)

How can we use the logic model

• ESC Project planning• Strategic Planning• Program Monitoring• Program Evaluation

Inputs Processes OutputsShort-term Outcomes

Medium-term

Outcome

• Staff time• CRM Time• Lead time• Consultant

time• Materials• Car and

gasoline• Knowledge• Experience• Place to meet

• Contracting• Analysis• Study• Meetings w

client• Meetings with

stakeholders• Consultant

meetings

• Progress reports

• Draft report• Final report• Board

presentation

? ?

Impact

• ?

Logic model for an ESC Engagement

Deliverables

Inputs Process Outputs

Short-term Outcomes

Medium-term Outcome

Strategic Plan Approved

Impact

Example of a logic model for an ESC Strategic Planning Engagement

Strategic Plan in regular Use

Longer-term Outcome

Strategic Plan improves agency

performance

Community benefits from

agency’s improved

performance

Inputs ProgramsOutputs

Short-term Outcomes

• Money• Staff time• Facilities• Equipment• Laws• Regulations• Funders

requirements

Program A

Impact

Non-profit Logic Model

Program B

Program C

Program D

Output A

Output C

Output B

Output D

Outcome 1

Outcome 2

Outcome 3

Outcome 1:Improved Financial Position

Outcome 5:New Training

Program Developed

Outcome 2:New Facility Developed

Outcome 3:Strengthened

Volunteer force

Outcome 4:Improved

Public Recognition

Improved Ability to Fulfill Mission

Outcome 1:Improved Financial Position

Improved Ability to Fulfill Mission

Strategy 1: Capital Campaign

Completed

Strategy 2: Increased donor

base

Strategy 3: Client Fees Increased

Indicators

• Measure the quantity or quality of the outcome

Indicators

Outcome: Participants in a job training program obtain a job in a timely manner

• # (%) of participants employed within 6 weeks • # (%) of participants still employed 3 months

after completing the program

Indicators

Outcome: Homeless people obtain secure housing

• # (%) of homeless moved into temporary quarters and stayed for 6 weeks

• # (%) of homeless who obtained a job and were still employed after 3 months

Indicators Targets

• # (%) of participants employed within 6 weeks

• # (%) of participants still employed 3 months after completing the program

• 60% of participants will gain employment within 6 weeks

• 95% of participants who obtained employment will still be employed in 3 months

Logical FrameworkComponent Indicator Baseline Target

Outcome

Output

Program/Activity/Process

Input

1. Choose the Outcomes to Measure

2. Specify the Indicators for the Outcomes

3. Prepare to collect data on the Indicators

4. Try out the Measurement System

5. Analyze and Report Findings

6. Improve the System

What Can Our Clients Do to Measure Outcomes

7. Use the Findings

Two Great Resources

• Measuring Outcomes, A Practical Approach, The United Way Foundation• Logic Model Development Guide,

WK Kellogg Foundation

Thank you for participating in ESC Brownbag No. 2.Look for more Brownbags to come