Helping Our Clients Identify and Measure Their Outcomes

Edward M. Haugh Jr.ESC Consultant

By the end of the session you will be able to help your nonprofit clients: develop and begin to use a logic model,

and prepare a simple outcomes framework to

help them monitor and evaluate the results of their programs

Today’s Planned Outcomes

Review the elements of a logic model (outcomes, outputs, activities, inputs)

Prepare a logic model Define outcome indicators Develop a few sample outcome indicators Have lunch and enjoy Develop a simple outcomes framework

Outline of the session

Answers the question: What difference did the program make.

Outcome Focused Planning and Evaluation

A communication tool Ensure programs on right track Focus on programs that really make difference Helps strategic planning Puts clients in the center Produces data to show quality Useful information for decision making Communicates value to funders Promotes accountability and cost

effectiveness

Benefits of Outcome Measurement

Outcomes are what matter to the client

A tangible benefit or change for individuals or populations produced by a program, or project or other intervention

What exactly do we mean by an outcome?

A change or result produced by a project or program:

More simply:

What participants know, think, can do or behave differently after intervention

Logic Model Definitions

Outcomes

Benefits or changes for individuals or communities

What participants know, think, can do or behave differently after intervention

Improved (nutrition, morale)Increased (employment) Reduced (HIV transmission)Enhanced (knowledge)Maintained (vaccination coverage)

Outcomes – often expressed as verbs in past tense

The logic model

Logic Models are a key tool for designing, monitoring, and evaluating outcomes

Logic Models

A graphic way to organize information, and display thinking

A visual approach to the maps we carry in our minds

A description of our planned actions and anticipated results

What is a Logic Model

Logic Model Helps Define Outcomes

Inputs Processes Outputs Outcomes Impact

Click icon to add picture

Input: Equipment and Facilities

Click icon to add picture

Activity: Health Education

Click icon to add picture

Output: Fathers acquire parenting skills

Click icon to add picture

Outcome: Healthy newborns

Click icon to add picture

Impact: Healthy Community

Logic Model

Inputs Activities Outputs

Short-term

Outcome

Medium-term

Outcome

Long-term

Outcome

Imp

act

Logic Models Come In All Shapes and Sizes

What can the logic model be used for?

Design and Planning

Inputs ProcessesOutputs Outcomes Impact

Monitoring and Evaluation

Inputs ProcessesOutputs Outcomes Impact

Communications and Fundraising

Inputs ProcessesOutputs Outcomes Impact

Strategic Planning

Inputs ProgramsOutputs Outcomes Impact

Understanding a New ESC Client

Inputs ProgramsOutputs Outcomes Impact

Can be used for: Design & Planning Monitoring & Evaluation Communications Strategic Planning Understanding a new nonprofit

Logic Models

Logic Model Definitions

Inputs

The resources a program uses to achieve objectives. Staff, funds, equipment, curricula, volunteers, management, board

Logic Model Definitions

Activities or Programs

What the agency does with its inputs – programs or services

Sheltering the homeless, delivering hot meals, counselling teenagers about smoking, providing clothes for interviews

Logic Model Definitions

Outputs

The direct products of program activities

Contracts completed, classes taught, mosquito nets distributed, shelter nights provided

Outcomes are the results Outputs are the tangible products produced by an intervention◦Usually expressed as nouns◦Can be counted

Outcomes vs. Outputs

Classes taughtBrochures distributedClients servedContracts completedBoard retreats completedStrategic plans prepared

What are Outputs

Example: XYZ Job Prep Program

InputsMeeting space

Interview space

Computer stations (6)

Job counselor

Job coaches

Job prep curriculum

XYZ job assessment tool

Activities

1 on 1 intake and job skills assessment

Group training sessions

conducted daily

Link clients with select employers

Weekly 1 on 1 meetings

Recruit partner

companies

Outputs

clients enrolled

Clients attending

daily group sessions

Clients completing initial skills

training

Clients companies

signing partner

agreements

Short-term

Outcome

Clients learn:

What jobs match their skills and interests

How to dress and groom

How to prep a resume and application

Interview skills

How to complete job search

Medium-term

Outcome

Clients prepare a resume

Clients demonstrate strong interview skills during practice

Clients plan transportation and day care

Clients apply for jobs that match their skills

Clients appear well groomed and neatly dressed

Long-term

Outcome

Clients obtain and retain a full

time job

40

ESC’s Business Logic Model

Exercise 1: Develop a Simple Logic Model

(Use format)

Logic Model Definitions

Outcome Indicators

Terms that describe how success would be measured.

Logic Model Definitions

Example of Outcome Indicators

Outcome: participants lead a healthy lifestyleNo. and % of participants who maintain recommended weight, blood pressure, and cholesterol levels.

Logic Model Definitions

Example of Outcome Indicators

Outcome: reduced tobacco use by teensNo. and % of teen participants who report that they have not smoked cigarettes for six months

Outcomes sometimes confused with outcome indicators◦ Outcome - girls participating in a training program

obtain formal employment◦ Outcome indicators – the # and % of participants

employed within 3 months of completing the program

Outcome Indicators

Outcome: Participants in a job training program obtain a job in a timely manner

# (%) of participants employed within 6 weeks

# (%) of participants still employed 3 months after completing the program

Indicators

Outcome: Homeless people obtain secure housing

# (%) of homeless moved into temporary quarters and stayed for 6 weeks

# (%) of homeless who moved into permanent housing and remained there for 6 months

Indicators

◦ Outcome: A social marketing program aims to ensure that teens understand how HIV is

spread.◦ Outcomes: Young adults practice safe sex◦ Outcome: Teens in a job training program obtain

permanent employment.◦ Outcome: Adult “shut ins” have improved

nutrition◦ Outcome: Kid’s in a summer reading program are

ready for school in the fall. ◦ Outcome: Homeless people obtain secure housing

Exercise 2: Designing Outcome Indicators

Logic Model Definitions

Targets

Numerical performance objectives for achievement of outcomes80% of participants will be able to identify 6 jobs that match their interest and skill levels.

# (%) of participants employed within 6 weeks

# (%) of participants still employed 3 months after completing the program

60% of participants will gain employment within 6 weeks

95% of participants who obtained employment will still be employed in 3 months

Indicators & Targets

Indicates how the data will be gathered◦ Interviews (self reporting)◦ Reports◦ Budgets◦ Patient records◦ Government records◦ Report cards◦ Observation (walking around) ◦ Minutes of meetings◦ Scales, measurements (weights, heights)◦ Other – systems – pain scale

Data Source:

Permission will be obtained from parents/students to review report cards

Patient approval to review clinical records Review of monthly progress reports

How are you going to measure

Develop a Logical Framework for a training program that is designed to help clients learn how to get a job. It provides training, interview skills, a set of clothes for interviewing. What else might be needed?

Exercise 3:

Outcomes Indicators Performance Targets

Data Sources

Methods

Long-term OutcomeClients attain long-term financial sustainability

Medium-term OutcomeClients get a full time job

Short-term OutcomeClients learn how to conduct an interview

Outputs ???

ActivitiesSeminars on interviewing

Seminars on “dressing for success:

Provision of clothing for interviews

This is a

train

ing p

rogra

m d

esig

ned to

help

clients g

et a

perm

anent jo

b.

1. Choose the Outcomes to Measure

2. Specify the Indicators for the Outcomes

3. Prepare to collect data on the Indicators

4. Try out the Measurement System

5. Analyze and Report Findings

6. Improve the System

What Can Our Clients Do to Measure Outcomes7. Use the

Findings

A typical ESC Problem Analysis

Lack Brd/Mgt financial oversight

Low operating efficiency and high per patient costs

ED and (staff) overwhelmed and poorly focused

Insufficient numbers of volunteer dentists and poor coordination

Lack financial information for Brd/Mgt

Lack budget and operating plan no plan to expand access

No financial sustainability

Lack of Access In spite of high demand – empty dental chairs

Poor oral health for clients and potential clients

High costs of health insurers and general public due to overuse of emergency rooms

Loss of productivity and economic potential in the County due to poor health

Start with a Problem Analysis

Inputs Processes OutputsShort-term Outcomes

Medium-term

Outcome

• Staff time• CRM Time• ESC

management time

• Board time• Funds from

scholarship

• Board & Mgt meeting w ESC Consultants

• Financial review

• Develop projections for expanding access

• Develop plan for access

• Gap analysis and ED training

• Progress reports

• Meetings completed

• Projections prepared and approved

• Expansion plan approved

• Gap analysis finalized

• Final report• ED training

completed

• Improved fiduciary management

• Improved financial performance

• Improved ED performance

Improved access to

oral health

Long-term

Outcome

ImprovedHealth

Logic model for the ESC Engagement

Deliverables

ImpactImproved health,

economic productivity and lowered emergency room costs

for the county

ESC Project planning Strategic Planning Program Monitoring Program Evaluation

How ESC can use outcomes planning

Measuring Program Outcomes: A Practical Approach: http://www.nrpa.org/uploadedFiles/nrpa.org/Professional_Development/Accreditation/COAPRT/Measuring_Program_Outcomes-UW.pdf

United Way of Greater Richmond and Petersburg: Outcome Measurements: http://www.yourunitedway.org/outcome-measurements

Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources http://managementhelp.org/evaluation/outcomes-evaluation-guide.htm#anchor146822

References

Thank you!