Identifying Collective Impact Outcomes

1

Identifying Collective Impact Outcomes

Isaac Castillo - @Isaac_outcomes

Isaac D. CastilloDeputy Director

DC Promise Neighborhood Initiative@Isaac_outcomes

[email protected]

December 4, 2015

mailto:[email protected]

2

Learning Objectives

• What is Collective Impact? • What type of Collective Impact are you talking

about? • What does it take to measure outcomes in a

collective impact framework?

December 2015 Isaac Castillo - @isaac_outcomes

3

What is Collective Impact?

• As defined by FSG (the nonprofit consulting firm that invented the term)…..

“Collective Impact is the commitment of a group of actors from different sectors to a common agenda for solving a complex social problem.”

December 2015 Isaac Castillo - @Isaac_outcomes

4

Five Conditions of Collective Impact• Collective impact is more rigorous and specific than collaboration among

organizations. There are five conditions that, together, lead to meaningful results from collective impact:– Common Agenda: All participants share a vision for change that includes a common

understanding of the problem and a joint approach to solving the problem through agreed-upon actions.

– Shared Measurement: All participating organizations agree on the ways success will be measured and reported, with a short list of common indicators identified and used for learning and improvement.

– Mutually Reinforcing Activities: A diverse set of stakeholders, typically across sectors, coordinate a set of differentiated activities through a mutually reinforcing plan of action.

– Continuous Communication: All players engage in frequent and structured open communication to build trust, assure mutual objectives, and create common motivation.

– Backbone Support: An independent, funded staff dedicated to the initiative provides ongoing support by guiding the initiative’s vision and strategy, supporting aligned activities, establishing shared measurement practices, building public will, advancing policy, and mobilizing resources.

December 2015

Source: FSG Consulting website (group that developed collective impact concept)

Isaac Castillo - @Isaac_outcomes

5

But is your Collective Impact the same as mine?

• Different people / groups interpret the concept of ‘collective impact’ in many different ways.

• I have found the three most common to be: – One problem / one geographic area– ‘Franchise model’ – multiple locations– Focus on core group of individuals


DCPNI – Isaac Castillo - @Isaac_outcomes 6

One Problem / One Geographic Area

• Most commonly discussed form of collective impact.

• Single large problem is identified for solution in one area.

• Multiple agencies and organizations (government, schools, community, nonprofits) ‘collectively’ try to solve problem.

• Attempt to measure ‘impact’ of their efforts in solving the problem.

3/17/2014

7

One Problem / One Geographic AreaExample


Reduce Gang Crime in City

Mayor’s Office

Government Agencies

Residents

Funding Community

Nonprofit Organizations

Law Enforcement

Schools Evaluators

8

One Problem / One Geographic AreaCharacteristics

• Commonly defined goal (with indicators) that everyone agrees with.

• Multiple agencies/groups all working towards this common goal.

• Some agency/group takes lead or serves as coordinator.

• Not every agency/group contributes the same – some do more than others.


9

Franchise Model – Multiple Locations

• Most common model (not talked about as much).

• Same program/intervention replicated by single organization in multiple locations.

• Larger organization is trying to determine the ‘impact’ across all of their sites / locations.

• The aggregated outcomes of each location is then considered ‘collective’.


10

Franchise Model – Multiple LocationsExample


Collective Impact=+++++

11

Franchise Model – Multiple LocationsCharacteristics

• Parent organization sets goal(s) and local organizations agree to goal(s).

• Same program / approach in lots of locations. • Parent organization responsible for quality

control / fidelity.


12

Core Group of Individuals

• Rare – put present in some locations.• Specified group of service recipients that are

served my multiple organizations. • Service recipients are pre-identified at the

individual level. • Attempt to measure the ‘impact’ of all service

providers on the same participants.


13

Core Group of IndividualsExample


After-school Tutoring

Pre-Identified Youth Service Recipients

RecreationExercise Programming

Health Education College ReadinessNutrition Programming

14

Core Group of IndividualsCharacteristics

• Service providers need to agree upon common set of outcomes/indicators.

• Some group / organization needs to coordinate.

• Intent is to provide extensive services to a limited set of participants.


15

Common Concerns for All Collective Impact Approaches

• What kind of data will you collect? – Demographic and/or socioeconomic– Service amounts or types– Outcomes or changes in conditions

• Who is responsible for reporting data (and what is their motivation)?

• Who collects the data and in what format? • Who trains people on proper data submissions? • Who analyzes and then shares/reports data and findings? • Are there consequences for not sharing data (or not sharing

data in correct format or on time)?


16

How the Simple Can Create a Challenge

• Define each data point along with acceptable answer categories– Gender – what will you ask for and what will you

report? – Race / ethnicity – are these two different concepts

to be measured, or will they be grouped? And what are you answer categories?

– Neighborhood, census tract, census block group, or some other geographic area?


17

Agreeing on an Outcome

• Should come to consensus on what the outcome is that you want to make progress towards.

• Three at a maximum – the fewer the better. • This will be the hardest part – but important to

gain consensus. • Shouldn’t get too detailed or worry too much

on how to measure it. First just agree on what you want to change.


18

Next Agree on Indicators

• Indicators are what you will measure to determine if you are making progress towards your outcomes.

• They are more detailed than outcomes and can exist at multiple levels and time periods.


19

Examples of Indicators and Outcomes

• Outcome: Youth are college and career ready– Indicator: Youth graduate from college– Indicator: Youth graduate from high school – Indicator: Youth are proficient in math and/or reading

on state assessments• Outcome: Reduce poverty in defined area– Indicator: % of adults at or below poverty line– Indicator: % of children at or below poverty line– Indicator: % of people living at 150% or less of poverty

level


20December 2015 Isaac Castillo - @Isaac_outcomes

21

How Will You Measure?

• Once you have your outcomes and indicators, you can think about how to measure change.

• Will you….– Use publicly available data? (And how regularly

available is it)?– Collect data from multiple sources (and how will

you aggregate it)? – Ask multiple people to submit data (and what is in

it for them)?


22

Kenilworth-Parkside Community

DCPNI’s Collective Impact Approach


Youth Parents

Residents

23

DCPNI Collective ImpactCharacteristics

• DCPNI serves as coordinating agency.• Some outcomes/indicators set by DCPNI

(Dept. of Ed.), others developed with partners. • Will determine which combinations of

partners are most effective.


24

Measuring at the Community and Individual Levels

• How do these indicators help DCPNI evaluate its work in the community? That occurs at two levels: – At the community level – we look at change that

occurs over time for community level data. – At the individual level – we look at how each child

and parent in the community improves their outcomes over time.


25

Community Level Evaluation

• Examine community level change that occurs.• Use existing data sets (Census, ACS, Vital

Statistics, Dept. of Health, Crime, etc.) • Conduct primary data collection: – Neighborhood survey– School climate survey

• Community interviews and focus groups. • Is there a clear link to changes observed and our

interventions?


26

Individual Level Evaluation• Each partner / program defines success in alignment

with one (or more) of DCPNI’s indicators. • Each partner articulates outcomes and methods to

measure outcomes at individual (child, student, parent) level.

• Each partner shares individual level demographic, attendance, and outcome data with DCPNI.

• DCPNI combines with other data sets (school data) to measure individual level change over time.


27

Individual Level Evaluation Example


Math Tutoring

Effective Education – Students Proficient in Math

Report Card Grades

Chess Instruction

Critical Thinking Skills

School

Standardized Test

28

Individual Level Evaluation Example


Tutoring

Effective Education – Students Proficient in Math

Report Card GradesChess

Critical ThinkingSchool

Standardized Test

How Does DCPNI Fit In?

Child 1

Child 3

Child 2

29

Individual Level Evaluation Insights

• Critical part is having student / resident names (or unique identifiers).

• Need to have common outcomes / indicators.• Each partner must have adequate data

collection and measurement capacity. • Can be used to identify effective combinations

of programs.


30

Isaac’s Contact Information


Isaac D. CastilloDeputy Director

DC Promise Neighborhood Initiative

On Twitter: @Isaac_outcomesEmail: [email protected]

mailto:[email protected]

Government & Nonprofit

Identifying Collective Impact Outcomes