Creating an Evaluation PlanFreya Bradford, Senior Consultant NorthSky Nonprofit NetworkMay 23, 2013

How is the landscape changing?

Learning has never been more important

Leadership

Adaptabilit

y

Program Capacity

Sustainabilit

y

“Anyone who keeps learning stays young.” – Henry Ford

TCC Group Sustainability Formula

Phases of Evaluation

1. Create Evaluation

Plan

2. Collect & Analyze

Data

3. Make Meaning & Adapt Program

Evaluation Plan: A written document that describes how you will assess the success of a program or project.

Evaluation Planning Process

Convene a Team

Create a Logic Model

Prioritize & Define Eval Questions

Create a Data Collection Matrix

Write & Share the Eval Plan

Convene a team• Evaluator/Facilitator (if you have one)• Project lead• Board member/leadership• Key staff person• Member of target population

Evaluation

Create a logic model

Create a logic model

Prioritize & Define Eval Questions

• “Not everything that counts can be counted, and not everything that can be counted counts.”

•Measure what matters!• Think about uses for the data - Can you

identify a decision that could be made as a result of the data? • Consider Process & Outcome

Example Questions – NS Coaching ProgramProcess Evaluation Questions1) Were project activities conducted as planned? Were the outputs achieved?2) What are the characteristics of participants that engaged in coaching?3) How satisfied are participants with the coaching that they received? Did it meet expectations?

Outcome Evaluation Questions 1) To what extent did coaching help participants meet individual and organizational goals? 2) In what ways did participants and organizations change/grow as a result of coaching?

Create a data collection matrix

•What Information will be Collected? (Indicators). From whom? (Sources)• In what way? (Methods) •By whom? (Person Responsible)•How will the information be stored and managed? (Data Management)

Create a data collection matrix

Indicators• An indicator is piece of data that you will collect to help tell

you if you are achieving what you intended• Can be Quantitative or Qualitative• You may have multiple indicators that you want to collect to

measure one part of an Activity or Outcome

Common Activity (Process) Indicators

•FTE/volunteer hours contributed•# of partners•Amount of products/services delivered •#/type of clients served•#/type of materials produced/disseminated•Timeliness of service provision•Quality of services (satisfaction data)

Common Outcome Indicators

•#/% demonstrating increased knowledge/skill•#/% demonstrating attitude or behavior change•% changes in conditions (longer-term)

IndicatorsExampleActivity of Coaching: Two (2) 45-minute introductory information sessions will be led by coach. They will be promoted through the NS weekly update.

Indicators to measure progress:- # of times promoted in weekly update- # clicked through- # of participants at intro sessions - Participant demographics – org. role, tenure in position

IndicatorsExampleOutcome of Coaching: Participants will improve in individual development goal areas (e.g. managing others, self-awareness, self-management, work-life balance).

Indicators to measure progress:- % of participants improving by goal area- % of participants meeting and exceeding desired goal- % change in goal achievement scores overall and by

goal type

Indicators

Methods• Activity tracking • Document review• Surveys• Tests• Focus groups• Interviews• Case studies

Methods – When to Use a Survey

• Need a little info about a lot of things• Have a large number in your target population• Numbers are important to your decision making• Respondents might not feel comfortable talking

about answers in a group• Need information quickly

Methods – Tips on Survey

• Have someone available who can help with survey construction• Use a variety of question types – not too

many open-ended• Consider using an online survey• Pilot test survey with group similar to your

target• Incentives!

Methods – When to Use a Focus Group

• You have deep or complex issues you want to understand more completely • You want to develop some preliminary

understanding of the issues surrounding your topics• You want to hear people’s deep feelings or insights

about your topics• Some questions need to be explained in detail or

probed to elicit good feedback• People need time to ponder questions before

responding

Methods – Tips on Focus Group

• Use a trained facilitator who is as neutral as possible• Use a recording device (with permission) and/or a

note taker in addition to facilitator• Limit groups to 6 – 12 participants• Give participants background info before the session• Carefully plan questions – 3 – 4 in-depth questions

with planned probes if needed• Incentives!• Send thank you notes

Indicators

Write & Share the Eval PlanEvaluation Plan Outline

I. IntroductionII. Evaluation Design & QuestionsIII. Evaluation MethodsMethod Description Indicators measured with method(Repeat for each method)IV. Use of Findings

Attachments:• Timeline for Evaluation Implementation• Logic Model

Resources• NorthSky Resource Center

This presentation and other free resourceshttp://www.northskynonprofitnetwork.org/resource-center

Including:University of Wisconsin-Extension, Program Development and Evaluation Unit: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

United Way Outcome Measurement Resource Network:http://www.liveunited.org/Outcomes/Library/pgmomres.cfm

W.K. Kellogg Foundation Logic Model Handbook: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf