Evaluation

Assists with allocating resources what is working how things can work better

Effective Evaluations

ID strategies/initiatives working best Provide feedback for decision making Show cost-effectiveness of different

strategies Increase ability to manage

projects/programs

Why Evaluate?

To determine effectiveness understand or verify impact of service, program,

product To assess efficiency

ensuring optimal use of time & resources To be accountable

account for what has been accomplished through project funding

To identify ways to improve a program Improving usefulness of the program or service Determining what works & doesn’t (strengths &

weaknesses) Determining if program is meeting needs of

participants

Types of Evaluation

Formative Information you collect to help you plan or

implement a program/service Needs assessment Feasibility assessment Pre-testing program elements Audience analysis

Types of Evaluation

Process Assesses aspects of program activities On-going process – feedback for

decision making along the way Tracking quantity & type of service Tracking people reached (# and type) Participant satisfaction (e.g., quality of

service)

PIE focus Perfect, Improve, Eliminate

Types of Evaluation

Impact

Whether & to what extent a program has accomplished stated goals & objectives

Fiscal or Efficiency Evaluation

Compares

costs -- economic resources consumed by program

WITH consequences -- to health effects or

outcomes

Fiscal EvaluationCost-Benefit Analysis Compares

cost in $$ economic benefits/consequences in units

of currency For every $1 spent on preschool program,

there are $7 in benefits through reduced societal costs

Fiscal Evaluation – Cost-Effectiveness Analysis Compares:

costs in monetary units -- $$ health effects in units of mortality,

morbidity, health status measure Costs $x to prevent one infant death from

diarrheal disease by supporting breastfeeding

Fiscal EvaluationCost-Utility Analysis Compares

cost in $$ consequences in health terms as utilities or

the preference/desire for a specific outcome Compare costs and outcomes of

interventions Cost $x to avoid one case of diarrheal

disease through oral rehydration or clean water supply

Steps in Evaluation

DESCRIBE/CLARIFY YOUR PROGRAM Engage stakeholders Assess resources

DESIGN THE EVALUATION Determine evaluation methods Collect the data Process and analyze the data Interpret and disseminate the results

TAKE ACTION Who conducts evaluation?

Evaluation Framework

Provides a plan that links the activities (implementation objectives) and outcomes (outcome objectives) with your evaluation questions, indicators, data sources and data collection methods

Evaluation questions directly reflect program’s implementation and/or outcome objectives.

Evaluation Framework

Indicators What are the specific measures indicating

you have achieved your objectives? How will you know if you accomplished your

objectives? What would be a success?

Sources Who can provide information about this? Where can you get the information?

Tools What is/are the best way(s) to gather

information/find out from people?

Implementation or outcome objective:

Evaluation Questions

Indicators Data Sources

Data Methods

Evaluation Framework

Measurement Indicators

Outcome Indicators measures for your outcomes Short-term

Improved access Improved collaboration Policy changes Changes in awareness, knowledge or beliefs

Longer term Service utilization Morbidity/mortality Health status

Measurement Indicators

Process Indicators Measures for how program delivered

Training sessions held Type of staff Staff activities and time Resources distributed Participation Client Satisfaction

Program Logic Model

A diagrammatic representation of a program Depicts relationships between program

activities, outputs & outcomes Describes a program to stakeholders

clarifies how the program is structured Shows how different facets of a program are

linked Integrates program planning & evaluation Facilitates the assessment of the feasibility of

program activities achieving program objectives

Program Logic Model – CAT SOLO

Components – Closely related groups of activities in your program

Activities – describe what you are going to do & provide (implementation strategies)

Target – describes who you are trying to reach

Outcomes – describe impacts on the customers, clients, organizations, systems (short and long-term outcomes)

Program Logic Model – CAT SOLOComponents

Activities

Target

Short-term Outcomes

Long-term Outcomes

Communicating Evaluation Findings Communicate info to potential users Ensure report addresses issues users

perceive as important Ensure timely delivery Ensure format easily understood by users

Communicating Evaluation Findings Include copies of survey tools

Summarize computer printouts in table format

Potential Users of Evaluation Results Immediate Supervisor Senior Management/Boards of Directors Policymakers Funding agencies Peers in similar programs Client base/Community Program personnel

Communicating Evaluation Findings: Figure 4-4, p. 123