[ HOW ] CAN WE JUDGE WHAT S WORKING ? T HOUGHTS ON EVALUATION For further information, please contact: Katya Fels Smyth, Principal and Founder The Full

[HOW] CAN WE JUDGE WHAT’S WORKING?THOUGHTS ON EVALUATION

For further information, please contact:

Katya Fels Smyth, Principal and Founder The Full Frame InitiativePO Box 390955, Cambridge, MA [email protected] O: (413) 625-6936 C: (617) 620-6718

December 2009Please do not use these slides without checking with us first. We will probably say

‘yes.’ If you do use these slides and/or have feedback, please let us know.

FULL FRAME INITIATIVE OVERVIEW

Nationally, human service systems are failing hundreds of thousands of people who need them the most.

This exerts an untenable financial and human cost.

It doesn’t have to be this way.

There are (a relatively small number of) organizations and interventions that work really well. We can and must learn from them.

FFI is committed to making it more likely that people who are failed by our systems

have a chance to participate in interventions that work. 2

WHAT IS OFTEN VALUED IS NOT WHAT OFTEN WORKS

Issues and problems are discrete: respond to the specific diagnosis using proven

protocols+

Friends and family are part of the problem+

Don’t allow people to become dependent on a program—graduate them fully

+People need to leave their communities to move

forward+

Transferability matters+

Consistency and fidelity +

Linear, rational models and systems+

The model matters more than who practices it+

Demonstrate efficacy+

Management matters.

Life is messy: respond to the entire reality flexibly and recognizing

that every person’s situation is different+

Friends and family matter+

Through thick and thin and differences+

Be a community within the community, not an alternative to the larger community

+Place and local specificity matter

+Some of the best work happens in the gray areas

+Change is good : adapt and evolve an approach

+It only works with the right people working

+Be accountable: focus on results that are sustained

+Leadership matters.

© 2009 The Full Frame Initiative www.fullframeinitiative.org

3

ONCE UPON A TIME, THERE WERE TWO CHOICESAnecdotesEvidence of

good intentionsIdeologyDocumentation of the

number and nature of transactions

Evidence from randomized controlled trials

vs.

“Scientific validity”

“Unaccountable” or is it enough?


4

IN FACT, “ORGANIZATIONAL EFFECTIVENESS … CAN BE VIEWED AS HIGHLY MALLEABLE.” (EBRAHIM, 2005, 64)

Concepts of scope, focus, method matrix: after Murray and Cutt, 1998Addition of purpose: Smyth, 2009

scop

e

focus

method

organization

system

Short-

term

(outputs/ acti

vities)

Mid-term

(outcomes)

Long-t

erm

(impact)

Processe

s &

implementation

(qua

si)

expe

rimen

tal

Conv

ersa

tions

, com

mon

wis

dom

, IK

mix

ed m

etho

ds

Purpose:ProvingImproving/learningLegitimizing


program

project5

Shaded: Poor fit for experimental methods

EXPERIMENTAL METHODOLOGY ALSO LIMITS THE KINDS OF INTERVENTIONS THAT WE CAN

ASSESSGood fit for experimental methods

Intervention characteristics

Require a more complete approach

Homogeneous POPULATION Heterogeneous and multiply challenged; Highly influenced by other systems

Highly transferable SETTING Community-embedded

Individual/family LEVEL OF IMPACT

Multiple, including community-change

Defined model, independent of other systems

STASIS Dynamic and evolving; may depend on other interventions

Empower individuals to help themselves

POWER Empower individuals to help themselves and change their communities

Clear pathways through model

ELEMENTS Host of elements drawn on without a recipe


6

7

relevance rigor actionable knowledgeA clear under-standing of what is known (and where the gaps are) to provide solid information for decision making

Is it the right thing to be assessing?

Are we assessing it right?

+

A WELL-STOCKED TOOL KIT


Evaluation can be a tool for learning and improving and proving and sharing

A MORE COMPLETE APPROACH TO EVIDENCE AND EVALUATION


Pays attention to systems

Attends to interactions and amplifications

Allows for complexity and

change

Uses mixed methods

Incorporates multiple wisdoms

Employs a philosophy of

“enough” Right- sized

8

Population: multiply challenged, non homogeneous, potentially involved in multiple

other interventions and systems

Setting: community embeddedness, engagement in community change; community highly

influenced by external forces

Stasis: dynamic, evolving, emergent, demand-driven; significant front-line authority and

flexibility

Levels of impact: multiple units of analysis (individual and community and system)

Power: participatory; flattened power dynamics; increasing capacity of a cohort or community,

not just an individual; increasing voice

Key elements: a host of non-additive, individual program elements are drawn on, but without

a specific recipe

Outcome indicators will be tailored to and by the individual (so may vary within an intervention).

Outcome measures will vary across interventions.

Outcome measures will specifically address structural forces.

Outcome measures and methodologies will be continually assessed for fit with intervention’s goals as they evolve; goals will be clearly set, but will be plastic enough to respond to major changes in the environment (e.g., a financial shock).

Outcome measures will attend to community or cohort-level change, as well as individual-level change.

Outcome measures will assess the optimization of the individual’s (or cohort’s) situation and trajectory, rather than simply tallying up success or results across a standard series of measures.

Characteristic of intervention Characteristic of indicator/measurement

9


THIS SOUNDS LIKE A NIGHTMARE: HOW DO WE ROLL ALL THIS UP?

Decreased social isolation and exclusion

Increased, maintained

safety

Increased , maintained

stability

Increased mastery/self-

efficacy

Meaningful access to / use of relevant

mainstream resources

10

Domains of impact

Conditions for healthy people

and healthy communities


LEVELS OF ANALYSIS:

* What is best for the individual may not be best for the family, of course.

Individual/ Family*

Community/Collective

Organization/ Agency

11

Ambitious, meaningful, predictive of long-term wellbeing, valid for comparative purposes, plastic enough to respond to major

external forces

Highly tailored to context– not entirely

predetermined

Decreased social isolation and

exclusion

Increased, maintained safety

Increased , maintained

stability

Increased mastery/self-

efficacy

Meaningful use of relevant

mainstream resources


SOME SUGGESTIONS FOR SETTING INDIVIDUAL AND FAMILY LEVEL GOALS AND INDICATORS

Allow them to be highly tailored, highly specific to the situation.

FFI suggests that every goal should address one of the five domains of impact, and that each domain should have a goal and indicator attached to it.

12

EXAMPLES OF GUIDELINES FOR SETTING COMMUNITY GOALS AND DETERMINING INDICATORSThey must be…

Ambitious, meaningful, clearly stated, measurable (for indicators), and predictive of long-term well-being.

Developed in concert with the community. Allow comparison so that changes over time (trends) or

differences between communities can be determined and are valid.

Plastic enough to change in response to major changes in the external environment.

FFI suggests that every goal should address one of the five domains of impact, and that each domain should have indicators attached to it. 13

SOME SUGGESTIONS FOR SETTING ORGANIZATIONAL/ AGENCY PERFORMANCE MEASURES AND INDICATORS

They should be in support of an explicit theory of change that includes macro and individual level change.

They should be explicit about what data are in service of performance measurement and what are indicators.

Organizational progress in helping survivors move forward on individual goals can be assessed as a performance measurement.

FFI suggests that every indicator should address one of the five domains of impact, and that each domain should have indicators attached to it.

14

FOR MORE INFORMATION, PLEASE CONTACT:

Katya Fels Smyth, PrincipalThe Full Frame InitiativePO Box 390955Cambridge, MA 02139(413) 625-6936 (office)(617) 620-6718 (cell)[email protected]

15

Documents

[ HOW ] CAN WE JUDGE WHAT S WORKING ? T HOUGHTS ON EVALUATION For further information, please contact: Katya Fels Smyth, Principal and Founder The Full