1

Evaluation Evaluation principles:principles:

Theory-basedTheory-basedUtilization focusedUtilization focused

ParticipatoryParticipatory

Washington Mental Health Washington Mental Health TransformationTransformation

Evaluation Task GroupEvaluation Task Group

2

A call for Theory-based A call for Theory-based evaluationevaluation

“The theory-driven approach is essential to tracking the many elements of the program [or initiative], and assuring that the results identified in the evaluation are firmly connected to the program’s activities. Tracking all aspects of the system makes it more plausible that the results are due to program activities. . . .and that the results generalize to other programs of the same type”

-- Carol H. Weiss

3

Theory Driven Theory Driven EvaluationEvaluation(Weiss, 1995)(Weiss, 1995)

Concentrate attention and resources on Concentrate attention and resources on key aspects of the programkey aspects of the program

Asks the program or initiative (e.g., Asks the program or initiative (e.g., mental health transformation) to make mental health transformation) to make assumptions clearassumptions clear

Demands that consensus is reached about Demands that consensus is reached about what we are trying to do and whywhat we are trying to do and why

These types of evaluations “may have These types of evaluations “may have more influence on both policy and more influence on both policy and popular opinion”popular opinion”

4

More benefits of theory-More benefits of theory-based evaluationbased evaluation

Allows for improved program planning Allows for improved program planning and program improvementand program improvement

Knowledge generated from a theory Knowledge generated from a theory based evaluation will generalize to a based evaluation will generalize to a wide array of system change effortswide array of system change efforts

Highlights the elements of program Highlights the elements of program activity that deserve attention in the activity that deserve attention in the evaluation, thus facilitating the evaluation, thus facilitating the planning of the studyplanning of the study

5

Designing a Theory of Designing a Theory of ChangeChange

As described by the United Way of As described by the United Way of AmericaAmerica Establish the longest-term outcomeEstablish the longest-term outcome

Develop a measurement plan

Select indicators of success

Create an action plan for implementing the strategy

Identify activities necessary for achieving system outcomes

Identify system outcomes necessary for achieving the longest-term outcomeIdentify system outcomes necessary for achieving the longest-term outcome

6

INPUTS OUTPUTS

• stakeholders at the table

• grant funding

• in-kind donations

• buildings/office space for family resource center

• airtime for media campaign

Actions• coordinated

provider directories

• joint funding applications

• joint staff trainings

• bills passed

• needs/asset assessments

• changed infrastructure

• improved practices

• changed policies

• modified activities

• increased inputs

SYSTEMOUTCOMES

• new knowledge

• increased skills

• changed attitudes or values

• modified behavior

• improved condition

• altered status

Changes or improvements in systems

Benefits or changes for individuals

ACTIVITIES

Sample Logic Model

• secure funding for child care provider training

• advocate to centralize immunization records

• coordinate parent education delivery system

ProductsResources

ULTIMATEOUTCOMES

7

““Framing” a Theory of Framing” a Theory of ChangeChange

As described by Hernandez & Hodges, As described by Hernandez & Hodges, 20012001

Population Frame

Consider the issues and strengths

Strategies Frame

Consider principles and components

Desired Outcomes Frame

Consider Short- and long-term outcomes

8

A sample frameworkA sample frameworkAs described by Hernandez & As described by Hernandez &

Hodges, 2001, 2003Hodges, 2001, 2003

IssuesLack of appropriate services

Location of services

Lack of community members as service providers

Lack of knowledge of residents’ perceptions of services available

Lack of parental involvement in agency decision making

StrategiesDevelop a core group of community members

Develop and administer community needs survey

Develop volunteer parent network

Involve parents in agency decision-making

OutcomesIncreased parent representation in agency decision-making (list of meetings and results of parent participation)

Increase in appropriate services (services resulting from needs)

Increased accessibility (location, hours, transportation)

9

Increased program quality

Increased program

slots

Baltimore’s After-School Baltimore’s After-School StrategyStrategy

Theory of changeTheory of change

After- School

Programs

Adopting research-

based Standards

Leveraging public and

private funds

Capacity-building and

TA

Accountability, monitoring,

and evaluation

Increased participation

rates

Skills, competencies, and attitudes that lead to long-term

improvements in academic

and non-academic outcomes

10

Strategy: School system reform to address mental health issues of kindergartners with EBD

Children with EBD in school-based pre-kindergarten programs receive appropriate treatment.C

hil

d

ou

tco

mes

Sys

tem

ou

tco

mes

Teachers in school-based, pre-kindergarten programs are able to

recognize signs of emotional disharmony.

Develop an agreement for mental health providers to train teachers in

school-based, pre-kindergarten programs on recognizing the signs

of emotional disharmony.

Psychiatric specialists from outside mental health programs

are on staff at school-based, pre-kindergarten programs.

Convene meetings between the school system and public, private, and nonprofit providers of mental health services for young children.

Develop an agreement for mental health providers to place psychiatric staff in school-based, pre-kindergarten programs on a

part-time basis.

Init

iati

ve

Act

ivit

ies

Kindergartners with EBD participate appropriately in classroom activities.

Sample Partial Logic Model #1 for a Success By 6® Initiative

Develop an agreement for mental health providers to accept referrals and treat children with EBD from school-based, pre-kindergarten

programs.

With parental consent, school psychologists and outside psychiatric specialists assess children that show

signs of emotional dysharmony.

Services are available in quality mental health programs for pre-school children

with EBD.

Quality mental health providers make administrative arrangements so that

more slots are available for pre-school children with EBD.

With parental consent, school psychologists and outside psychiatric specialists refer Children with EBD to quality mental health providers.

Teachers in school-based, pre-kindergarten programs refer children that show signs of

emotional disharmony to staff.

Mental health providers accept referrals and treat Children with

EBD from school-based, pre-kindergarten programs.

Write grants, solicit corporate sponsors, and

advocate in the legislature for funding to

provide more slots in quality mental health

programs for pre-school children with EBD.

11

Questions to Ask in Reviewing Logic Models

Are the outcomes really outcomes?

Is the logic logical?

Is the longest-term outcome meaningful for initiative participants and the community?

Is the longest-term system outcome reasonable?

For which outcomes (system and long-term) will you create a measurement plan?

12

What to measure?What to measure?Use a Utilization-Focused Use a Utilization-Focused

ApproachApproach Wealth of potential evaluation Wealth of potential evaluation

questionsquestions

++ Presence of finite evaluation resourcesPresence of finite evaluation resources

== Goals of the evaluation should be Goals of the evaluation should be

established early … andestablished early … and Evaluation activities should be defined Evaluation activities should be defined

that will be of most usethat will be of most use

13

Utilization-Focused Utilization-Focused EvaluationEvaluation

Allows for a crystallization of the purposes of the Allows for a crystallization of the purposes of the evaluation and a focus on evaluation activities to evaluation and a focus on evaluation activities to be carried outbe carried out

Enhances likelihood that the evaluation results Enhances likelihood that the evaluation results will in fact be utilized through the course of will in fact be utilized through the course of program activities by engaging key decision-program activities by engaging key decision-makers in the evaluation and policy analysis makers in the evaluation and policy analysis processprocess

Allows for an additional opportunity for a public Allows for an additional opportunity for a public interaction between the Initiative and invested interaction between the Initiative and invested stakeholders, allowing for an exchange of ideas stakeholders, allowing for an exchange of ideas and a sharing of informationand a sharing of information

14

Questions to askQuestions to ask in in deciding deciding “What to “What to

measure?”measure?”GENERAL EVALUATION USE:GENERAL EVALUATION USE: What decisions, if any are the evaluation’s What decisions, if any are the evaluation’s

findings expected to influence?findings expected to influence? Clearly distinguish summative from Clearly distinguish summative from

formative decisionsformative decisions When will the decisions be made?When will the decisions be made? By whom?By whom? When, then, must the evaluation findings When, then, must the evaluation findings

be presented to be timely and influential?be presented to be timely and influential?

15

Questions to askQuestions to ask in in deciding deciding “What to “What to

measure?”measure?”THE CONTEXT OF THE EVALUATION:THE CONTEXT OF THE EVALUATION: What is at stake in the decisions? For What is at stake in the decisions? For

whom? What controversies surround the whom? What controversies surround the decisions?decisions?

What’s the history and context of the What’s the history and context of the decision-making process?decision-making process?

What other factors (values, personalities, What other factors (values, personalities, politics, promises already made) will affect politics, promises already made) will affect the decision-making?the decision-making?

To what extent have the outcomes of To what extent have the outcomes of decisions already been determined?decisions already been determined?

16

Questions to askQuestions to ask in in deciding deciding “What to “What to

measure?”measure?”DATA AND INFORMATION NEEDED:DATA AND INFORMATION NEEDED: What data and findings are needed to What data and findings are needed to

support decision-making?support decision-making? How available are these data?How available are these data? Are new data collection approaches Are new data collection approaches

needed, or are the data available needed, or are the data available through existing means?through existing means?

If outcomes and indicators have already If outcomes and indicators have already been defined, do they support what is been defined, do they support what is needed?needed?

17

Participatory Participatory EvaluationEvaluation

When addressing stakeholder groups to assess their evaluation When addressing stakeholder groups to assess their evaluation related questions, concerns and contributions…related questions, concerns and contributions…

Think broadly when identifying stakeholders and make an effort Think broadly when identifying stakeholders and make an effort to include non-evaluator stakeholders in the evaluation (e.g., to include non-evaluator stakeholders in the evaluation (e.g., families, youth, program directors, community groups, etc.).families, youth, program directors, community groups, etc.).

Stakeholder = Anyone who has a stake in the program that is being evaluatedStakeholder = Anyone who has a stake in the program that is being evaluated The group of stakeholders of a program and its evaluation may The group of stakeholders of a program and its evaluation may

not be static.not be static. Regularly update stakeholder groups on the data collection Regularly update stakeholder groups on the data collection

process. Learn how it can be improved, refined, enhanced.process. Learn how it can be improved, refined, enhanced. Convene stakeholder group to design data dissemination plan.Convene stakeholder group to design data dissemination plan. Disseminate results regularly to and with stakeholders.Disseminate results regularly to and with stakeholders.