Performance Measurement Series: Establishing the Basis for Program Performance Tracking April 17, 2012, 9am CT YWCA Great Lakes Alliance Region

Performance Measurement Series:Establishing the Basis for Program Performance Tracking

April 17, 2012, 9am CT

YWCA Great Lakes Alliance Region

Three Part Series:

1. Establishing the Basis for Program Performance Tracking and Measurement

April 17, 2012 9am CT

2. Defining Performance Indicators and Measures May 1, 2012 9am CT

3. Analyzing, Using and Reporting DataMay 15, 2012 9am CT

SESSION 1 TOPICSLaying the groundwork for

measurement….

SESSION 1 TOPICSLaying the groundwork for

measurement….

Organizational measurement systems and the role of program performance measurement

The Importance of program logic in establishing the basis for measurement

Outputs vs. outcomes Mission-driven chain of results Resources and examples

Organizational measurement systems and the role of program performance measurement

The Importance of program logic in establishing the basis for measurement

Outputs vs. outcomes Mission-driven chain of results Resources and examples

4

History of Nonprofit Performance Measurement

Financial Accountability Program products or outputs Adherence to standards of quality in

service delivery Participant related measures Client Satisfaction Program Outcomes Community Impact

5

Developments in Outcomes Measurement

Outcomes measurement processes developed

National organizations conduct research on outcomes

National organizations develop resources for local agencies

Agencies are measuring outcomes Funders are expecting outcomes

measurement New focus on community impact

Convergence of 2 Forces Driving Nonprofit Performance Measurement

1. Increased demand for accountability—boards, media, general public

2. Growing commitment of nonprofit managers to focus on results and strengthen performance

Why Measure Performance?…managing programs or agencies without performance measures has been likened to flying blind, with no instruments to indicate where the enterprise is heading.

Measures…can help managers reward success and take corrective action to avoid replicating failure.

Appropriately configured performance measures motivate managers and employees to work harder and smarter to accomplish goals and objectives.

Theodore H. Poister, 2003

Why Measure Performance?

Unless you are keeping score, it is difficult to know whether you are winning or losing.

Harry Hatry 1978

9

About Nonprofit Programs

Program – an integrated set of services conducted to meet specific, verified community needs by achieving certain specific outcomes among specific group(s) of clients in the community.Credible approachesUtilize community resourcesBased on assumptions or “theory of change”

10

Assess Mission Related Situation

Define Intended Impact and Outcomes to improve the situation

Define Programming/ Service Methods and Approaches to achieve intended impact

Determine Feasibility – Resources Needed and Available to carry out methods and approaches; Environmental Factors that could help or interfere

Establish Systems: Marketing, Performance Measurement, Improvement

Program Planning Steps

Mission

Vision

Strategic Goals & Objectives

Performance Measures



Program Management Measurement Systems

Resources consumed Activities conducted Transactions completed Clients Served Outputs produced Services Rendered Outcomes and Impact achieved

Program Theory

Reveals the critical assumptions and expectations inherent in a program’s design.

How the program services and practices are presumed to accomplish their purposes.

Provides the framework for performance measurement.

13

The “if-then” relationship

if a program provides prenatal counseling to pregnant teens, then the teens have increased knowledge of good prenatal care. If the teens have increased knowledge of good prenatal care, then this leads to changed behavior: the teens eat the proper foods, take a prenatal vitamin each day and avoid cigarettes, alcohol and other drugs. If the teens follow these practices, then the result is that the teens deliver healthy newborns.

14

Use of Logic Model for Framing Use of Logic Model for Framing Program TheoryProgram Theory

…a picture of how a program/services will work to solve identified problems

…the basis for a convincing story of a program’s/services’ expected performance

• Shows the major components of services• Establishes logical linkages among

components.• Describes the sequence of events

thought to bring about change.• Illustrates the theory and assumptions

underlying services (theory of change).

15

Logic Models help nonprofits… Construct a theory of change Intentionally address clearly defined situations, problems or

disparities Focus on achieving measurable outcomes that improve

these situations/ decrease these disparities Invest in activities that will lead to desired outcomes/

decreased disparity Assemble resources necessary to carry out activities

effectively Establish the basis for evaluation--determining, improving

and increasing success Drive program execution and development

16

Resources

Needed to effectively carry out strategies and actions

3. Strategies and Actions

that will lead to outcomes and, eventually contribute to impact

2. Intended Outcomes Impact

Reduction or elimination of problems, disparities

1. Situation/ Disparities

Mission-focused problems, disparities, circumstances

Program Theory

17

Situation: Mission/ Vision based: why we act

Problems, Disparities, Circumstances

Resources: our capacity, what we invest

StaffTimeVolunteersRevenuePartnersEquipmentFacilitiesTechnologyOther….

Activities/ Outputs:

What we doNumbers of: Workshops,

Counseling Sessions, Meals Served, Trainings, etc.

Who we reachNumbers of : Participants, Clients, Customers, Beneficiaries

Outcomes & Impact: what we hope to achieve

SHORTLearning Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation

MEDIUMActions

BehaviorPractice DecisionsPolicies Social action

LONG-TERMConditions HumanEconomicCivic Environment

Environment: factors that may help or impedePolitical, Economic, Values, Policies

Program Impact Theory

Describes the cause and effect sequence in which certain program activities, utilizing carefully chosen resources, eventually produce and contribute to social benefits.

Longer term/ultimate or “distal” outcomes are dependent on successful attainment of shorter term/intermediate or “proximal” outcomes.

Poister, 2003

Program Impact Theory

Programs rarely exercise direct control over the social conditions they are expected to improve

Programs tend to change critical but manageable aspects of a situation which is expected to lead to more far-reaching improvements.

Services affect some intermediate condition that, in turn, improves the social conditions of concern.

Poister 2003

20

Situation: Mission/ Vision based: why we act

Problems, Disparities, Circumstances

Resources: our capacity, what we invest

StaffTimeVolunteersRevenuePartnersEquipmentFacilitiesTechnologyOther….

Activities/ Outputs:

What we doNumbers of: Workshops,

Counseling Sessions, Meals Served, Trainings, etc.

Who we reachNumbers of : Participants, Clients, Customers, Beneficiaries


SHORTLearning Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation

MEDIUMActions

BehaviorPractice DecisionsPolicies Social action

LONG-TERM IMPACTConditions HumanEconomicCivic Environment

Environment: factors that may help or impedePolitical, Economic, Values, Policies

21

Inputs/ Resources

Activities/ Outputs

Outcomes Impact

Program Logic Model

Money

Staff

Expertise

Facilities

Curriculum

Equipment

Partnerships

Volunteers

Strategies

Treatments

Advocacy

Services

Customer engagement

New knowledge

Increased skills

Changed attitudes

Modified behavior

Improved conditions

Altered status

Social Change

22

Steps for Measuring Outcomes

1. Identify desired outcomes and impact. Distinguish outcomes from inputs and activities/outputs

2. Determine outcomes sequence

3. Select outcomes to measure

4. Identify indicators (evidence) and performance standards (achievement levels)

5. Determine data collection approaches

6. Collect data

7. Use and learn from data

23

Identifying Program Outcomes

24

Determine Desired OutcomesGood program development begins with end result in mind.Whose situation/ status/ needs to change in order for the problem to be reduced?

Women? Women of color?

What impact would we hope to contribute to? What would it look like if the problems/disparities driving the program were eliminated? What changes in the situation driving the program need would we like to influence? What effects do we wish to contribute?

25

What are Program Outcomes?

Direct, intended beneficial effects customers/ participants experience during or after their involvement with a program or service

Problem related

Attainable

Measurable

26

Program Outcomes

Are Not….

Efforts or Activities

Participation in services

Numbers of customers served

Customer satisfaction

27

Program Outcomes

Are Not…. Are…

Efforts or Activities Results of efforts

Participation in services Results of participation in services


Numbers of customers served who changed

Customer satisfaction Customer impact

28

What we Do The Desired ResultsEngage youth in racial

justice education programming.

Youth who complete the program understand racism and white privilege. They learn how to interrupt racist comments and behaviors.

Engage women in political leadership development.

Women learn what it takes to run for office; some decide to run.

Provide employment preparation and training.

Individuals develop resumes, engage in interviews and become employed.

Efforts vs. Outcomes

Find the Outcomes 30 older women participated in the Social Security

workshop. Provide multicultural books in the childcare locations. 50 girls learned how to manage a checkbook. 66 participants attended the racial justice conference. Program coordinator hired to manage the racial justice

project. Women are satisfied with the money management

series. Women who provide in-home day care learned cash-

flow management techniques. 20 women entered the minority apprenticeship program.

Find the Outcomes 30 older women participated in the Social Security

workshop. Provide multicultural books in the childcare locations. 50 girls learned how to manage a checkbook. 66 participants attended the racial justice conference. Program coordinator hired to manage the racial justice

project. Women are satisfied with the money management

series. Women who provide in-home day care learned cash-

flow management techniques. 20 women entered the minority apprenticeship program.

31

Managers who Measure Outcomes Report A clear definition of the program's intended outcomes, in itself,

provides focus for the program's work. Understanding their current level of outcome achievement provides

a barometer to assess progress and direct future activities. Outcome measurement provides invaluable information to improve

programs and then see if the improvements make the intended difference.

Outcome information is a powerful motivator of staff, who now can observe the progress they are making with participants in a consistent, tangible manner.

It becomes a powerful recruitment tool for volunteers who have many other choices for how they spend their time.

It helps position the agency in the community as a successful organization, which in turn leads to increased promotion and financial support

(Hatry, van Houten, Plantz, and Greenway, 1996).

32

Why Focus on Outcomes? What gets measured gets

done. If you don’t measure

results, you can’t tell success from failure.

If you can’t see success, you can’t reward it.

If you can’t reward success, you’re probably rewarding failure.

If you can’t recognize failure, you can’t correct it.

If you can’t see success, you can’t learn from it.

If you can demonstrate results, you can win public support.

Excerpt from Reinventing Government by David Osborne and Ted Gaebler

33

Clarify problem(s) as basis for outcomes

Why does this program exist? What problem(s) does the program try to reduce

or eliminate? Who is affected by the problem?

What changes do we want those affected to experience?

34

Examples of Problems

Racial Justice Women of color are more

likely to live in poverty than white women.

The median earnings for women of color is 28% less than for white women.

The mortality rate from lung cancer (48.7 per 100,000) among African American women is higher than national rates and that for white women.

Women’s Economic EmpowermentSingle-mother families have the highest poverty rates of any household type—1 of every 2 families.Women earn 69 cents for every dollar earned by white men.Low-income women are uninsured at a rate of 19.5%.

From Status of Women in Wisconsin Reports 2004, 2008

35

From Problem to Impact

Think about the problem reduced or eliminated

PROBLEM LONG TERM OUTCOME/ IMPACT

High incidence of women in poverty; low wages

Increased economic security among women.

Homelessness among single mothers.

Women and their children in safe, affordable housing.

Community geographically segregated by race.

Fully integrated community.

Outcomes or Impact?How is impact Different?

Long term effects of our outcomes on those we serve

Eventual outcomes—things that happen because of the outcomes we produce

Less predictable—often dependent upon many other variables

Impacts are what we hope for but outcomes are what we work for.

36

Outcomes to Impact

SHORT

Learning

Awareness

Knowledge

Attitudes

Skills

Aspiration

Motivation

MEDIUM

Action

Behavior

Practice

Decisions

LONG TERM IMPACT

Conditions

Human

Economic

Civic

Environment

Outcomes to Impact

SHORT

Learning

Awareness

Knowledge

Attitudes

Skills

Aspiration

Motivation

MEDIUM

Action

Behavior

Practice

Decisions

LONG TERM IMPACT

Conditions

Human

Economic

Civic

Environment

Program Outcomes

Are the links between our actions and our mission

The tangible evidence that our work is leading to mission achievement

39

Defining Outcomes for a Program What should lead to the desired long term

impact? Outcomes should represent meaningful change

Make a marked difference in the situation, create a new situation

Replace something negative with something positive Outcomes should address “root causes”--

produce lasting, rather than temporary effects—beyond the absence of the problem.

40

41

Defining Outcomes:Start with where you are… Existing program New program idea Problem we wish to address

But be sure to clarify the problem and/or program purpose

Avoid initial preoccupation with measurementUse logic model

42

Existing Programs

1. Review Program purpose and activities—why do we do this? What would success for customers look like?

2. Clarify problem/ situation and who is affected—what would it look like if the problems were resolved?

3. Clarify outcomes based on purpose and problem

1. Clarify problem/ situation and who is affected first

2. Define desired impact and outcomes

3. Determine program purpose, activities that most likely to result in outcomes

New Program Development

43

Sources of Outcomes Information

Examine outcomes of similar programs/ services.

Talk to program staff. Talk to current and former customers. Find research about the problem and proven

solutions. Use situation/problem as the basis.

Program Outcomes

The tangible evidence that our work is leading to mission achievement

Are the links between our actions and our mission

44

ACTIONS….…OUTCOMES……….IMPACT……..MISSION

Shifting from what we do to changing lives

EFFORT: Provide

financial literacy programming to 40 women

OUTCOMES 35 of the 40 women

who completed the financial literacy program developed household budgets and savings plans; of those, 30 opened savings accounts.

IMPACT: Economic

security among women

MISSION: Empowered

women

Changing lives

EFFORT Provide

100 girls with technology training.

OUTCOMES Of the 100

girls who attended technology training, 75% demonstrated software, hardware proficiency

IMPACT Girls enter

technology-focused careers

MISSION Empowering

women

Changing lives

EFFORT Racial

justice education series provided to 150 individuals.

OUTCOME 125 of those

who attended the racial justice education series increased their understanding of racism and how to work against it.

IMPACT Anti-racist

community Reduced

incidence of racism.

MISSION Eliminating

racism

Changing lives through Advocacy

EFFORT Engaged

50 individuals in campaign against hate crimes in the state.

OUTCOME 50 individuals

wrote letters to their elected officials encouraging them to introduce state legislation against hate crimes. 10 held meetings with elected officials.

IMPACT Hate crime

legislation passed.

Reduced incidence of hate crimes.

MISSION Eliminating

Racism Empowering

Women

49

Mapping the Outcomes Sequence

50

Most Programs are Designed to Achieve Multiple Outcomes

Outcomes Sequence Most programs involve a series of outcomes. Hierarchy of logically related changes or

benefits. “Chain of outcomes”.

51

Outcomes Can Be

Short term

Intermediate Term

Long Term

52

Outcomes Sequence

SHORT

What changes do we expect

to occur within the

short term?

MEDIUM

What changes do we want to see occur after that?

LONG TERM

What changes do we hope to see over time?

53

Outcomes Sequence

SHORTLearning

AwarenessKnowledgeAttitudesSkillsAspirationMotivation

MEDIUMAction

BehaviorPracticeDecisions

LONG TERM IMPACT

Conditions

HumanEconomicCivicEnvironment

Racial Justice Program Example

Short Termknowledge

Intermediatebehavior

Long Term/ Impactcondition

Increased understanding about racism, privilege and their impact

Desire to work against racism

Increased interaction among people of different racial heritage groups

Commitment to action and action on commitment

Increased relationships between whites, people of color

More integrated community

Decreased disparities based on race

55

SITUATION INPUTS ACTIVITIES and OUTPUTS

OUTCOMES

Short Term Interm.

Long Term

Women represent 15% of elected leadership positions in the community

Recent report indicates few women choose to run

Policy climate not favorable to women; back burner, erosion of choice

Women with experience in political leadership

Funds to support program coordination

Curriculum, trainers

Series of political education workshops reach 100 women

Women in leadership group mentors 25

Women increase their understanding of political leadership process, opportunities

Women believe they can run and win

More women run for office

Women elected to 50% of all positions

Policies more favorable to women

56

Staff

Money

Partners

Design parent edu curriculum

Provide 6 training sessions

Targeted

parents

attend

Parents increase knowledge of child develop

Parents learn new ways to discipline

Parents use improved parenting skills

Reduced rates of child abuse & neglect

INPUTS ACTIVITIES/OUTPUTS OUTCOMES/ IMPACT

Family Strengthening Program

Approaches to Mapping the Outcomes Sequence

Review outcomes identified for the program.

Arrange the outcomes in a logical sequence that suggest a “theory of change”

Outcomes may be arranged in more than one “chain”

Determine if outcomes are missing from a chain and make adjustments.

57

Situation: why we actSingle women in the community with families do not progress economically at the same rate as men; incomes of women of color consistently lower than white women.•Some lack higher education, training needed for higher paid positions.•Few own assets, including a home.•Many women are paid less than men for same work.

Resources: our capacity, what we investStaff with expertise in sexism, racism, economic literacy.Scholarship funds.Partnerships with community college, university.Partnerships with bank.Partnership with 9 to 5.3 year investment.

Activities: What we do, who we reach

Services for single women with families that especially target women of color:

•College assistance with scholarships;

•Financial literacy with savings match;

•Self-advocacy training.

Community education about pay equity.


Women enter higher education programs, increase financial literacy and build savings; understand and advocate for higher earnings.

Women graduate from college. Move into higher paid positions.Build savings, own homes. Community advocates for pay equity.

Decreased income disparities based on race, gender decrease.

Environment: factors that may help or impedeEconomy, state budget, federal pay equity laws, national or local incidents

Selecting Outcomes to Measure

Should we Measure all Outcomes?

Criteria for Choosing Outcomes to Measure

AchievableFits program scope – what we can reasonably

influenceData is available, can be developedThe outcome is compelling

Which Outcomes to Measure

Review the “chain of outcomes” Focus measurement on the levels where you

are accountable; where you have reasonable control.

Decide how far out the chain of outcomes you can go.

Keep an eye on the ultimate long-term outcome or impact, although it may not be easily measured.

62

Issues to Consider

The longer term the outcome, the less direct influence the program has over its achievement and the more likely other extraneous forces are to intervene.

Because other forces affect an outcome doesn't mean it shouldn't be included.

Long-term outcomes should not go beyond the program purpose or target audience.

Staff

Money

Partners



Targeted

parents

attend





INPUTS ACTIVITIES/OUTPUTS OUTCOMES


Staff

Money

Partners



Targeted

parents

attend





INPUTS OUTPUTS OUTCOMES


Which Outcomes to Measure?

Example: Racial Justice Education Program

Short Termknowledge


Long Termcondition








Example: Racial Justice Education Program

Short Termknowledge


Long Termcondition









OUTCOMES

Short Term Interm

Long Term








Women in leadership mentor those who complete the series of workshops

Women increase their understanding of political leadership process, opportunities






OUTCOMES

Short Term Interm

Long Term








Women in leadership mentor those who complete the series of workshops

Women increase their understan-ding of political leadership process, opportuni-ties





Situation: why we actSingle women in the community with families do not progress economically at the same rate as men; incomes of women of color consistently lower than white women.•Some lack higher education, training needed for higher paid positions.•Few own assets, including a home.•Many women are paid less than men for same work.

Resources: our capacity, what we investStaff with expertise in sexism, racism, economic literacy.Scholarship funds.Partnerships with community college, university.Partnerships with bank.Partnership with 9 to 5.3 year investment.

Activities: What we do, who we reach

Services for single women with families that especially target women of color:

•College assistance with scholarships;

•Financial literacy with savings match;

•Self-advocacy training.

•Community educ. and engagem. in pay equity. advocacy


Women enter higher education programs, increase financial literacy and build savings; understand and advocate for higher earnings.

Women graduate from college.

Move into higher paid positions.

Build savings, own homes.

Decreased income disparities based on race, gender decrease.

Environment: factors that may help or impedeEconomy, state budget, federal pay equity laws, national or local incidents

Preparing for Measurement Clarify the outcomes statement as a target:

Specifies the meaningful change Identifies who will experience the change Defines the period of time for achievement May reference program effort.

Single women, especially women of color, in the community with families who are low income, who participate in and complete the 6 month financial literacy program, will increase their financial literacy and develop savings by the end of the program.

The best outcomes are those that are narrowly focused and specific.

70

Preparing for Measurement Develop outcomes statements for the outcomes

you will measure: Specifies the meaningful change Identifies who will experience the

change Defines the period of time for

achievement May reference program effort.

71

Example Outcomes Statements Girls who complete the techgyrls program will

become technology savvy. Women who engage in transitional housing

will secure safe, affordable housing within 18 months.

Children who complete the pre-school program will be ready for first grade.

72

Outcomes are not….

Numbers of activities provided


Customer satisfaction

Challenges to Measuring Outcomes

Aren’t there some things that we just can’t measure?

Pre-and post tests are too cumbersome. What can we really influence? Where do we begin? How can racial justice be measured?

Why Measure Outcomes?

Determine EffectivenessAre we making progress on eliminating racism?Do our efforts help women advance

economically?

Increase EffectivenessCould our impact be greater?What should we do more of, less of?

Communicate Value What value do we add? Why is our mission

worth supporting?

It is incredibly easy to be very busy without being very effective.

Steven Covey

76

Next Session:Defining Performance Indicators and

Measures Framing Indicators Sources of Data Validity, Reliability and other Criteria Measurement Problems and Challenges Resources and Examples

Resources to Assist with Measuring Outcomes

Urban Institute www.urban.org Publishes studies on discrimination issues, including race and

gender; measuring discrimination; community indicators Proposed national report card on racial and ethnic

discrimination

Institute for Women’s Policy Research www.iwpr.org Publishes reports on status of women—US, states, specific

issues Attention to race, ethnicity

Resources for Measuring Outcomes

Poverty and Race Research Action Council www.prrac.org Links research to advocacy on issues of race and gender

The Grantsmanship Center www.tgci.com Offers guides to program evaluation

United Way www.unitedway.org/outcomes Offers outcome measurement resources, guides

Resources for Measuring Outcomes

W.K. Kellog Foundation www.wkkf.org Publishes Logic Model Development Guide and Evaluation

Handbook

Performance Management Network, inc. www.pmn.net search: logic models Instruction on logic models framework and development

Documents

Performance Measurement Series: Establishing the Basis for Program Performance Tracking April 17, 2012, 9am CT YWCA Great Lakes Alliance Region