Program Evaluation: Using evaluation data to set direction,

expand impact, and maintain accountabilityOctober 21, 2014

Presented by:

Isaac D. Castillo

Director of Data and Evaluation

DC Promise Neighborhood Initiative

Twitter: @isaac_outcomes

Why should you care about program evaluation?

What is program evaluation?

How can your organization successfully conduct program evaluation work?

Today’s Agenda

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Why Should You Care About Program Evaluation?

LAYC domestic violence story LeapOfReason.org First Do No Harm…Then Do More Good

New domestic violence program component designed to teach three things: o Partner violence is not an OK expression of loveo Partner violence is not OK in Latino cultureo There are safe ways to get out of violent

relationships

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Why Should You Care About Program Evaluation?

No human being is perfect. Staff will make mistakes Organizations will make mistakes Services will be delivered poorly

Despite the best of intentions, some people will be harmed.

How do you know you are not harming people with your services?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

What is Program Evaluation?

Process to determine if your program / intervention / approach is effective.

Need to define what ‘success’ is for your program.

Program evaluation does NOT need to be done by specialists or outsiders – but those people do add credibility and rigor (in most cases)

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

The Basics of Program Evaluation – An Example

The concept of dieting – if you understand dieting, you understand the basics of program evaluation.

What is the goal of dieting (how do you define dieting ‘success’)?

How do you know if your diet ‘works’?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Data and Dieting

• Does that data point alone tell us anything?

• Context Matters – what if person is 4 feet tall and 10 years old? • Timing Matters – is this at beginning, end, or middle of diet?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Person weighs 200 Pounds

(90 Kilograms)

Person weighs 200 Pounds

(90 Kilograms)

Could Be About More Than Weight

• Other things that could be measured:• Body Mass Index (BMI)• Physical fitness• Blood measures (cholesterol levels)• Own perceptions of health / feeling• Appearance / muscle tone

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Five Evaluation Concepts to be Covered

Types and timing of evaluation

Who or what will you evaluate, and how will they be selected?

Quantitative, Qualitative, and Mixed Methods approaches

How detailed or rigorous does it need to be?

Who does the work – internal or external?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Types of Evaluation

What do I mean by ‘type’?

Really it is about timing - When do you collect data?

What will you compare your data to?

Much of this discussion relies on:o Costs o Availability of potential comparison data o What you are trying to learn

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Traditional (Time Series)

Most common type of program evaluation.

Looking to see if things have changed over time.

What was situation before program, then what was situation after program.

Must measure same things, in same ways, at both points in time.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Before ProgramBefore Program Program DeliveredProgram Delivered After ProgramAfter Program

Comparison Group

A time series study that compares to another group (that does not receive programming).

More rigorous, but more challenging.Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Program DeliveredProgram Delivered

Before ProgramBefore ProgramNo (or minimal)

programmingNo (or minimal)

programming After ProgramAfter Program

Who / What Will You Evaluate?

Need to define the population that will be evaluated.

Need to define ‘success measures’ (outcomes) – what are you trying to achieve?

Once these questions are answered, then need to consider which participants will be part of the evaluation (and maybe who gets programming).

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

In Time Series, This is Simple

Usually just serve and evaluate those that enroll in the program:

First come, first served is what is frequently used if there are too many potential participants.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Before ProgramBefore Program Program DeliveredProgram Delivered After ProgramAfter Program

Self-selectionSelf-selection

Comparison Groups Are More Complicated

• Can select by randomizing participants into groups:

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Program DeliveredProgram Delivered

Before ProgramBefore ProgramNo (or minimal)

programmingNo (or minimal)

programming After ProgramAfter Program

Random SelectionRandom Selection

Compare across high/low dosage

• Can use self-selection:

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Program DeliveredProgram Delivered

Before ProgramBefore ProgramNo (or minimal)

programmingNo (or minimal)

programming After ProgramAfter Program

High Attendance

High Attendance

Self-selection

Self-selection

Low Attendance

Low Attendance

Quantitative, Qualitative, and Mixed Methods Evaluation

Quantitative = more numerical information.

Qualitative = less numerical information.

One not better than other, just different types of information. Both can be high quality – both can be poor.

Most modern program evaluation is Mixed Methods – both quantitative and qualitative to varying degrees.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

What does Mixed Methods Evaluation Look Like?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Quantitative•Survey responses (that can be quantified)•Numerical data (test scores, report cards, medical data, etc.) •External data (collected by others)

Qualitative•Interviews with participants•Focus groups with participants•Interviews with staff•Interviews or focus groups with key stakeholders•Process / fidelity study•Open ended survey responses

Where Does This Get Tricky?

Need to focus on outcomes – changes in knowledge, attitudes, behavior or conditions.

Satisfaction surveys do not equal evaluation.

Just because someone liked the program it does not mean the program led to successful outcomes.

Some things can be either qualitative or quantitative depending on who you ask.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

How Detailed or Rigorous Does the Evaluation Need to Be?

What do you want to do with the results? Prove to yourself the program works? Use the results to market/fundraise? Publish the results through your own materials? Publish the results in peer-reviewed journals?

How ‘certain’ do you want to be about the results? Are you fine with some doubt? Will you be comfortable answering concerns and criticisms?

Are you willing to live with negative results?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Using Evaluation Data to Inform Change

Changes in program type / delivery (this parenting program isn’t working, time for a different program)

Changes in dosage (classes are offered once a month, increase classes to once a week)

Changes in measurement tools or approaches (this survey question is flawed, let’s find a better one)

Changes in staff training (staff do not seem to know how to deliver this program – time for

training)

Changes in organizational culture (no one is taking this approach seriously – time for larger

conversation)

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Communicating Evaluation Results

Be honest about your data and the limitations of your method.

What you omit is as telling as what you communicate.

When communicating negative / ‘bad’ results, follow this formula:

Finding / Result + Theory + New Solution

Use different formats to communicate the information.

Celebrate the successes, and identify areas for improvement.

No need to share everything in detail – but have it ready if someone requests it.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Balancing Cost and Rigor

Program evaluation does have a resource cost – but so does everything else.

Simple internal evaluation / performance management can be done at low cost.

However, larger picture requires more rigorous (and more expensive evaluation)

Start small and focus on 2-3 outcomes – then expand over time.

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Internal vs. External Evaluation

Who does design of evaluation?

Who does selection / creation of data collection tools?

Who does actual data collection?

Who does the analysis of data?

Who creates the reports / charts / publications?

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Resources - Books

The Nonprofit Outcomes Toolbox By Robert M. Penna

Handbook of Practical Program Evaluation By Joseph S. Wholey, Harry P. Hatry, and Kathryn E.

Newcomer - editors

Performance Measurement: Getting Results By Harry P. Hatry

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Resources - Articles

First Do No Harm…Then Do More Good By Isaac Castillo http://tinyurl.com/isaacLOR

Good Stories Aren’t Enough By Martha A. Miles http://tinyurl.com/milesgoodstories

Yes We Can! Performance Management in Nonprofit Human Services By David E.K. Hunter http://tinyurl.com/hunteryeswecan

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Resources - Links

PerformWell www.performwell.org

Performance Management and Evaluation: Two Sides of the Same Coin By Isaac Castillo and Ann Emery https://www.youtube.com/watch?v=nC7AG8XxrI4

Leap of Reason http://www.leapofreason.org

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014

Isaac’s Contact Information

Isaac D. CastilloDirector of Data and Evaluation

DC Promise Neighborhood Initiative

On Twitter: @Isaac_outcomesEmail: Isaac.Castillo@dcpni.org

October 21, 2014

Isaac Castillo - @isaac_outcomes – DC Promise Neighborhood Initiative – October 21, 2014