Logic Model: A Program Performance Framework

University of Wisconsin-ExtensionCooperative ExtensionProgram Development & Evaluation

Logic Model Workshop

Logic Model: A Program Performance Framework

Madison Workshop

December 5, 2001

Ellen Taylor-Powell, Ph.D.

Evaluation Specialist

UW- Extension-Cooperative Extension



A New Era• What gets measured gets done• If you don’t measure results, you can’t

tell success from failure• If you can’t see success, you can’t

reward it• If you can’t reward success, you’re

probably rewarding failure



• If you can’t see success, you can’t learn from it

• If you can’t recognize failure, you can’t correct it

• If you can demonstrate results, you can win public support

Osborne and Gaebler, 1992 in MQ Patton, 1997:14



Logic Model is…

• Picture of a program • Graphic representation of the program

“theory” or “action” – what it invests, what it does and what results

• Logical chain of if-then relationships; if x occurs, then y will occur

• Core of program planning and evaluation



LOGIC Reasonable

To be expected

MODEL Represents reality, isn’t reality



Why Logic Models--Why the Hype?

• Shows difference between what we do and impact we are having

• Provides a common vocabulary

• Focus on quality and continuous improvement



Logic Model: Origins• Private Sector: Total quality management• Public Sector: GPRA, performance

budgeting• Non-Profit Sector: Outcomes measurement,

eg. United Way• International Arena: Results Framework of

USAID, etc. • Evaluators: Evaluability assessment,

bennett hierarchy



Logic Model

SITUATION

OUTCOMESINPUTS OUTPUTS



HEADACHE

Feel betterGet pills Take pills

Everyday Logic Model



An Extension Example: Business Counseling

Extension invests time and resources

A variety of educational activities are provided to business owners who participate

These owners gain knowledge and change practices resulting in

Improved business performance



LOGIC MODEL: Program Performance Framework

INPUTS OUTPUTSActivities Participation

What we invest

StaffVolunteersTimeMoneyMaterialsEquipmentTechnologyPartners

What we do

WorkshopsMeetingsCounselingFacilitationAssessmentsProduct dev.Media workRecruitmentTraining

Who we reach

ParticipantsCustomersCitizens

Reactions

ASSUMPTIONS1)2)3)4)

ENVIRONMENTInfluential factors

SITUATION

What the short term results are

Learning

AwarenessKnowledgeAttitudesSkillsOpinionsAspirationsMotivations

Action

BehaviorPracticeDecisionsPoliciesSocial action

What the ultimate impact(s) is

Conditions

Social Economic Civic Environmental

OUTCOMES-IMPACTShort Medium Long-term

What the medium term results are



Logical Linkages: Series of If-Then Relationships

IF THEN IF THEN IF THEN IF THEN

INPUTS OUTPUT OUTCOMES

Programinvests time

&money

Resourceinventorycan be

developed

Familieswill knowwhat is

available

Familieswill access

services

Familieswill have

needs met



Activity (p 2-12)

Completing the causal linkages



LOGIC MODEL: Program Performance

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short Medium

What we invest

What we do

Who we reach

What results

SO WHAT??

Long-term



Outcomes vs. Activities

BE OUTCOME DRIVEN,

NOT ACTIVITY DRIVEN



Activity (p 2-13)

Program Performance Levels



Situation• Situational analysis

– Need– Asset

• Problem analysis

• Priority setting

• Engaging others



INPUTS

FacultyStaff

MoneyTime

VolunteersPartners

EquipmentTechnology



OUTPUTS What we do Who we reach

ACTIVITIES Workshops Counseling On-farm research Facilitation Product development Curriculum design Trainings Field days Conferences Media

PARTICIPATION Participants Clients Customers Users



OUTCOMESWhat results for individuals, businesses, communities..…

SHORTLearning

Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation

MEDIUMAction

Behavior Practice Decisions Policies Social action

LONG-TERMConditions

Human Economic Civic Environment



Staff

Money

Partners

Design parent ed curriculum

Provide 6 training session

Targeted

parents

attend

Parents increase knowledge of child dev

Parents learn new ways to discipline

Parents use improved parenting skills

Reduced rates of child abuse & neglect




Activity (p 2-24)

Which are the Outcomes-Impacts?

Chain of OutcomesSHORT MEDIUM LONG-TERM

Seniors increaseknowledge of food contamination risks

Practice safe cooling of food; food preparation guidelines

Lowered incidence of food borne illness

Participants increaseknowledge and skills infinancial management

Establish financial goals,use spending plan

Reduced debt andincreased savings

Community increasesunderstanding ofchildcare needs

Residents and employersdiscuss options andimplement a plan

Child care needs are met

Empty inner city parkinglot converted tocommunity garden

Youth and adults learngardening skills, nutrition,food preparation and mgt.

Money saved, nutritionimproved, residents enjoygreater sense ofcommunity


Logic Model Workshop Focus of Outcomes• Individual

– Child, parent, client, resident

• Group– family, team, community

group

• Agency, organization

• System

• Community

Child is prepared to enter school; teen uses

savings/spending plan

Community group has inclusive membership policy; family increases its savings

Communications are more open; agency adopted smoke-free policy

Family serving agencies share resources

Shared community responsibility has increased; youth are valued as contributing members



How far out the outcome chain do we go?

• What is logical?

• What is realistic?

• What is meaningful?



Staff

Money

Partners

Develop parent ed curriculum

Deliver 6 interactivesessions

Targeted

parents

attend






WHICH OUTCOMES???

Research



Outcome of Interest?

• Inherently valued outcome (Mohr, 1995)

– higher level outcome is immaterial– we are willing to assume that a higher outcome

will also be attained if we achieve the outcome of interest

• Participant valued outcome: if participants experience a change or benefit that makes a a real difference to them (United Way, 1999)



Assumptions• Beliefs about the program

– the participants– the way the program will operate– how resources, staff will be engaged– the theory of action



Assumptions, cont.

• Faulty assumptions are often the reason for poor results

• Check and test assumptions– Identify potential barriers for each

‘if-then’ sequence



Environment–Influential Factors

• Extension program does not exist in a vacuum

• Context of the program – politics, family circumstances, cultural milieu,

demographics, economics, values, biophysical environment, policies, services

• What affects the program over which you have little control?



What does a logic model look like?

• Graphic display of boxes and arrows

• Any shape possible

• Circular, dynamic

• Relationships, linkages

• Level of detail

• simple

• complex

• Multiple models



Compare examples



Logic Model: Limitations• Represents reality, not reality

Programs are not linear

• Focuses on expected outcomes

• Challenge of causal attribution

Many factors influence outcomes

• Doesn’t address: Are we doing the right

thing?



Benefits• Brings detail to broad goals

• Shows the ‘chain of events that link inputs to results

• Builds understanding and consensus

• Identifies gaps in logic and uncertain assumptions

• Signals what to evaluate and when

• Summarizes complex program to communicate with externals



Building a Logic Model• New program

• Existing program

• Team; organization

• Involvement of others

• Keep it dynamic



Logic Model: WORKSHEET

ASSUMPTIONS

•

•

•

INPUTS OUTPUTS Activities Participation

OUTCOMES - IMPACT Short Medium Long-term

Program TitleSituation/Problem



Check Your Logic Model• Are the outcomes really outcomes?• Is the longest-term outcome

– meaningful?– logical?– realistic?

• Are the connections between inputs, outputs, and outcomes clear and reasonable?

• Does it represent research and best practice?• Does it represent the program’s purpose;

response to the situation?



OUTCOMESINPUTS OUTPUTS

Activities Participation Short Medium Long

term

Programmatic investments

i

EVALUATION

PLANNING

Where does evaluation fit?

Decrease in rates - agency records

Actual use - follow-up phone interview

Increase in knowledge/skill- post session survey

# parents attending/sessionwhich parents% of parents

Quality of curriculum# of sessions delivered

Staff

Money

Partners


Provide 6 training sessions

Targeted

parents

attend






EVALUATION: What do you want to know? How will you know it?

Providing Leadership forProgram Evaluation



Evaluation PlanSchedule -when/where

Method to collect info

Source of information

Indicators - how will you know it

What do you want to know?



QUESTION TYPE OF EVALUATION

Need?

Needs assessment

Process or implementation? Process evaluation

Outcomes or impact?

Outcomes evaluation Impact evaluation

Costs and efficiencies? Cost-benefit/cost effectiveness evaluation

Evaluation Questions – What do you want to know?



Indicators - How will you know it?• The evidence or measures that indicates what

you wish to know or see:

• often multiple indicators are necessary

• may be quantitative or qualitative

• culturally appropriate



Logic model with indicators for Outputs

and Outcomes

Program implemented

Targeted farmers

Farmers learnFarmers practice new techniques

Farm profitability increases

Number of workshops held

Number and percent of farmers attending

Number and percent who learned content

Number and percent who practice the recommendations

Number and percent show farm profits increase; amount of increase

OutputsOutcomes



INDICATORS: Examples

#, % with savings goal set#, % with debt reduction goal set#, % using spending plan#, % maintaining emergency fund

Improved family financial management

#, % acres managed according to BMP guidelines Quality of conservation plan implementation

Reduction in N and P application rates

#, % of Boards with youth participation

Increased youth-adult partnerships INDICATORS OUTCOME

Evaluating Your Program

Decrease in rates of abuse & neglect among these parents

#, % parents using improved skills (specify skills)

#, % parents who increased knowledge #, % parents who learned new ways

# and %parents who attended each session

Quality of curriculum

# sessions fully delivered

Staff

Money

Partners


Provide 6 training sessions

Targeted

parents

attend






Indicators



Source and Method of Data Collection

Source of information• Participants• Parents• Teachers• Key informants

Method of collecting the information

• Survey• Interview• Observation• End-of-program

questionnaire• Focus group• Records



Evaluation Plan cont. How will the data beanalyzed andinterpreted?

How will results beshared? To whom,how, when?

Who will do what, when,what resources?



Enfin…

• View evaluation as learning - integrate into the way we work

• Build evaluation in upfront• Ask ‘tough questions’• Make measurement meaningful• Be accountable for highest professional

standards

Documents

Logic Model: A Program Performance Framework