Program evaluation

Sebastian Uijtdehaagebas@mednet.ucla.edu

310.794.9009

Why do you evaluate?

Five reasons of evaluating

1. FORMATIVEinforms program development

2. SUMMATIVEdemonstrate program’s success and impact

3. UNDERSTANDINGhow and why did program work?

4. DISSEMINATIONsharing best practices

5. RECOGNITIONscholarship, academic credit, leadership

Stufflebeam 2003

What type of evaluation should I conduct?

Outcome & Impact

Did it work?

Dan Stufflebeam:CIPP Model of Program Evaluation

Needs Assessment

What should we do?

Program Planning

How shall we do it?

Implementation

Are we doing as planned?

Outcome & Impact

Does it work?

INPUT

CONTEXT

PROCESS

PRODUCT

What type of evaluation should I conduct?

Formative or Summative?

formative & summative evaluationFormative Evaluation

Context What needs to be done?

Input How should it be done?

Process Is it being done?

Product Is it working?

Stufflebeam 2003

formative & summative evaluationFormative Evaluation Summative Evaluation

Context What needs to be done?

Were important needs addressed?

Input How should it be done?

Was program plan defensible?

Process Is it being done? Was the plan well executed?

Product Is it working? Did the effort make a difference?

Stufflebeam 2003

What type of evaluation should I conduct?

Quantitative or Qualitative?

Quantitative or qualitative measures?

Quantitative measures HOW MUCH…?

• How much did the participant change?

• How much did the community change?

Qualitative measuresWHY and HOW?

• Why did the program work?

• How did participants change as result?

What makes good evaluation?

Evaluation must be planned ahead

• Evaluation should be part of program planning

• Planning and resource allocation makes evaluation more successful– Example: long-term tracking of participants

Confidence being admitted in a medical school

Category 13.4

3.6

3.8

4

4.2

4.4

4.6

4.8

Start of programEnd of program

What else makes good evaluation?

• Scientific rigor• Evaluation produces useful information that

helps make decisions– Assessment should be done with a particular

purpose in mind• Evaluation is tailored to your program

objectives• Evaluation is done with conceptual framework

in mind

LOGIC MODEL

Resources Activities Output Outcomes Impact

The Planned Work The Intended Results

CONDITIONS and DRIVERS

LOGIC MODEL

CONDITIONS and DRIVERS

Current situation that precipitates the need for the program

LOGIC MODEL

Resources Activities Output Outcomes Impact

CONDITIONS and DRIVERS

Resources and inputs needed to operate the program:personnel, facilities, materials, equipment

LOGIC MODEL

Resources Activities Output Outcomes Impact

CONDITIONS and DRIVERS

Everything the program does if it had access to the resources

LOGIC MODEL

Resources Activities Output Outcomes Impact

CONDITIONS and DRIVERS

The “deliverables”—the events, products and services

Number of participants, classes, brochures, advisors, etc

LOGIC MODEL

Resources Activities Output Outcomes Impact

CONDITIONS and DRIVERSIf the program delivers, the participants

will benefit in certain ways:

Knowledge, skills,

behavior, attitudes

Immediate, mid-term and

long-term outcomes

LOGIC MODEL

Resources Activities Output Outcomes Impact

CONDITIONS and DRIVERS

If these benefits to participants are achieved, then certain changes in organizations and communities might be expected to occur

LOGIC MODEL

Resources Activities Output Outcomes Impact

The Planned Work The Intended Results

CONDITIONS and DRIVERS

LOGIC MODEL

• Links processes and outcomes:– Funders love this!

• Great for program planning– Work backwards

• Evaluation is built in!• Ain’t perfect

Your turn…