Evaluating Collective Impact - OPEC .Evaluating Collective Impact Community Systems Group, LLC. 1

Evaluating Collective Impact

Community Systems Group, LLC. 1

Community Systems Group, LLC 2

Should I Stay?

1. What does it mean to establish shared measures?

For the our coalitions capacity and process?

For our coalitions accomplishments (the dose)?

For our coalitions outcomes?

2. How do we connect these measures to show

collective impact?

3. What software is available to support data collection

and analysis.

Goals

Community Systems Group, LLC 3

1. What does it mean to establish shared measures?

For the our coalitions capacity and process?

Goals

4

Evidence-Based Processes

Community Systems Group, LLC 4

5

Lit Review: Quality / Degree

Community Systems Group, LLC 5

6

Behavior vs.

Perception

Community Systems Group, LLC 6

7

Benchmark

Community Systems Group, LLC 7

1. What is our shared (and evidence-based) definition of capacity?

2. What are our agreed upon measures?

3. How could having common measures of capacity benefit our work?

4. Would if be beneficial if all coalitions in our community used the same measures?

Community Systems Group, LLC 8

Critical Reflection:

Community Systems Group, LLC 9

1. What does it mean to establish shared measures?

For the our coalitions capacity and process?

For our coalitions accomplishments (the dose)?

Goals

Presenting Question:

What is it that partnerships can do that their

members cannot do (or do as well) alone?

Community Systems Group, LLC 10

1. Community Change: Instances

New or improved policies, programs and practices facilitated by the initiative in the community.

Definitions

Community Systems Group, LLC 11

12

Falmouth Prevention Partnership

History of the Partnership

Relationship with CSG

Community Systems Group, LLC 12

13

Community Changes are new or modified programs, policies and practices facilitated by the partnership to reduce substance abuse in Falmouth. This includes modifications to the physical environment.

Examples: For the first time a Social Host Lesson at Falmouth High School was implemented as part

of the health curriculum for students in the 9th through the 12th grade. 375 students attended the social host laws class. (New Program: May 1, 2012)

The Falmouth Police Department dispatching added a new category to track liquor law

violations for underage persons. Prior to this there was not a specific way to collect data on underage liquor law violations. (New Practice: September 13, 2011)

The FPP, Falmouth Public Schools and the Falmouth Police Department co-sponsored and required attendance at a Social Host Pre Prom Assembly for the parents/caretakers of children attending the Junior Prom. (New Program: May 9, 2012)

13 13 Community Systems Group, LLC 13

14 14 Community Systems Group, LLC 14

The Falmouth Prevention Partnership facilitated:

64 Community changes intended to reduce underage alcohol use.

42 Community changes intended to build opportunities for pro-social involvement or that targeted all drug use.

30 Community changes intended to reduce Rx abuse.

28 Community changes to increase community capacity (such as new partnerships).

164 Documented new or modified programs, policies, and practices facilitated by the Partnership from 2009 to 2013 in order to reduce substance abuse in Falmouth, MA.

15

Dimensions of Intensity2

Dimension Scoring Rationale

Duration High Medium Low

Ongoing More than Once One Time

Emerging evidence and expert consensus suggest that community changes that are sustained are more likely to affect behavior change and population-level outcomes.

Reach High Medium Low

21% of population = 6% to 20% of population 5% of population

Emerging evidence and expert consensus suggests that exposure to community changes is important to affect outcomes.

Strategy High Medium Low

Policy, Consequences, Access Social Support Information, Skills

Emerging evidence and expert consensus suggests that community changes addressing policy and access are more likely to achieve outcomes.

2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.

15 15 Community Systems Group, LLC 15

Helping ou create a healthier communities Helping you create healthier communities 16

Dimensions of Intensity2

Community Change Duration Reach Strategy Score

Launch of the sticker shock campaign in 18 liquor retail stores with 10, 000 stickers.

Medium Medium Low 1.20

Pre Prom Assembly which is mandatory for all parents of students going to the Junior prom.

Medium Medium High 2.10

For the first time participated in "Credit for Life" Community Service Booth.

Low Low Low 0.30

2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.

Helping ou create a healthier communities 17 n = 15

i = 15.75 n = 16

i = 23.70 n = 36

i = 38.70 n = 26

i = 37.95 n = 32

i = 45.15 n = number of changes i = total intensity score

Intensity of Changes to Reduce Underage Alcohol Use facilitated by the

Falmouth Prevention Partnership

0

5

10

15

20

25

30

35

40

45

50

2009 2010 2011 2012 2013

Provide Information

Provide Support

Build Skills

Change the Physical Design

Access & Barriers

Policy Change

Rewards & Consequences

sum

an

nu

al in

ten

sity

sco

re

Year (First 3 Quarters)

1. Community Change: Instances

New or improved policies, programs and practices facilitated by the initiative in the community.

2. Services Provided: Contact Hours

Events that give information, build skills or provide social support for people in the community.

Definitions

Community Systems Group, LLC 18

Helping ou create a healthier communities Community Systems Group, LLC 19

Services Provided are events that impart information, develop skills, and give social support such as through classes, screenings, alternative activities, and workshops. Records on services provided include the number of classes conducted, participants, length, and location.

Examples: The Falmouth Prevention Partnership provided an in-service training for K-12

Adjustment and Guidance Counselors in Falmouth Public Schools on use of evidenced-based prevention strategies.

FPP participated in a "Credit for Life" Community Service Booth. The event is for high school seniors to prepare them for financial literacy and community volunteerism.

The Alcohol and Marijuana Action Team brought the "It's All About YOU" presentation by Richard Campbell to the Junior and Senior Falmouth High School students in a one-hour assembly format.

20

Helping ou create a healthier communities Helping you create healthier communities 20

0

200

400

600

800

1000

1200

1400

1600

1800

2009 2010 2011 2012 2013*

Tota

l C

on

tact

Ho

urs

FPP: Services Provided 2009-2013

Provide SocialSupport

ProvideInformation

Build Skills

3212 youth & 3120 adults received services facilitated by the Partnership.

*2013 data are for the first 9 months of the year.

Collecting Output Data

3. Media: Impressions

Coverage of the initiative, its projects or issues in any media channel such as through newspapers, newsletters, on-line, radio, and television.

Definitions

Community Systems Group, LLC 21

Helping ou create a healthier communities Community Systems Group, LLC 22

Media are instances of coverage of the initiative, its projects, or issues in the newspaper, on-line (new media), radio, television, and other channels. Records on media include the number of impressions and the media channel.

Examples:

"Program Aims To Educate Parents Who Serve Alcohol article in Falmouth Enterprise describes how parents of high school students will receive a letter in the mail warning them of the legal consequences of allowing teenagers to drink alcohol in their homes.

"Parents Who Host Lose The Most" public service announcement ran 55 times from Dec.1 -Dec.24, 2011 on local public access television, Channel 13.

Dr. Mike Bihari, FPP Chair, wrote the Friday, September 27, 2013 Risky Business column in the Falmouth Enterprise. The article contained statistics on underage drinking and the correlation between teen perception of parental disapproval and a decrease in underage drinking.

Helping ou create a healthier communities Community Systems