CHA/CHIP Demonstration Sites Strategy-Sharing

Call #2

Monitoring, Evaluating, and

Reporting on CHIP Implementation Prepared for the former CHA/CHIP

demonstration sites

May 28, 2013

Lauren Shirey, NACCHO

2

Call Topic: Your Requests

PHAB measurement and evaluation requirements

CHIP monitoring and evaluation focused on PHAB requirements

Monitoring data

Reporting system for CHIP implementation

CHIP reporting and tracking of performance measures; what to focus on to share results with

community; how to explain to community lack of change in health status indicators in short

period of time

Sustaining momentum

Success stories with lessons learned

How has project impacted other aspects of accreditation preparation

CHIP implementation structure

3

PHAB Requirements: CHIP Monitoring and Evaluation

*Be sure to review the standards listed below to identify the measures and required

documentation that PHAB seeks related to developing a CHIP.

Standard 5.2: Conduct a comprehensive planning process resulting in a

tribal/state/community health improvement plan

4

PHAB Requirements: CHIP Monitoring and Evaluation

Measure 5.2.2L: Produce a community health improvement plan as a result

of the community health improvement process.

Required documentation: CHIP dated within the last five years that includes

1a: Community health priorities, measurable objectives, improvement strategies and

performance measures with measurable and time-framed targets;

1b. Policy changes needed to accomplish health objectives;

1c. Individuals and organizations that have accepted responsibility for implementing

strategies;

1d. Measurable health outcomes or indicators to monitor progress; and

1e. Alignment between the CHIP and the state and national priorities.

5

PHAB Requirements: CHIP Monitoring and Evaluation

Measure 5.2.3 A: Implement elements and strategies of the health

improvement plan, in partnership with others.

Required Documentation:

1. Reports of actions taken related to implementing strategies to improve health

2. Examples of how the plan was implemented

[Guidance: The health department must provide reports showing implementation of the plan.

Documentation must specify the strategies being used, the partners involved, and the status or

results of the actions taken. The report could be a work plan for the community health improvement

plan showing timelines and progress. This could be in narrative or a table format.]

[Guidance: The health department must provide two examples of how the plan was implemented by the

health department and/or its partners.]

6

PHAB Requirements: CHIP Implementation

Measure 5.2.4A: Monitor progress on implementation of strategies in the

CHIP in collaboration with broad participation from stakeholders and

partners

Required documentation:

1. Evaluation reports on progress made in implementing strategies in the CHIP including:

1a. Monitoring of performance measures and

1b. Progress related to health improvement indicators

[Guidance: Description of progress made on health indicators as defined in the plan...]; and Revised

health improvement plan based on evaluation results

[Guidance: …must show that the health improvement plan has been revised based on the evaluation

listed in 1 above…]

7

Sustaining Momentum & Monitoring and Evaluation

• Data collected routinely can be utilized to mobilize community to action and for evaluation

• Celebrate small wins!

• Re-visit process and strategies

• Re-evaluate context in which you’re working; gather new information that might be important

• Have partners and community members participate in monitoring and evaluation activities (e.g. invite

them to review data, share information to the community, etc.)

• Build upon partner strengths and what is valuable to them (e.g. universities assist in complex

analyses)

8

Why CHIP evaluation?

Evaluation can measure how well you implement actions

(process evaluation) and whether your actions have an impact

(outcome evaluation).

When evaluating your CHIP, you are measuring how well you

implemented your plan (process evaluation) and whether your

actions improved health in your community (outcome

evaluation).

9

Vision

Strategic Issues/Priorities

Outcome Objectives

Activities/Tactics

Performance

Indicators/Measures

CHA Goals

Community Health Improvement Plan

CHA

10

Vision

Strategic Issues

Outcome Objectives

Activities/Tactics

Performance Measures

Goals

CHIP

What evaluation aims to answer?

Is the achievement

of our goals

addressing our

strategic issues?

Are our activities

generating the

outcomes we want

to see?

Is the achievement of

our strategic issues

leading to our vision?

Is the achievement of

our objectives leading

to our goals?

Are we doing what we

said we would do?

11

Vision

Strategic Issues

Outcome Objectives

Activities/Tactics

Performance Measures

Goals

CHIP

What evaluation aims to answer?

Are we doing what we

said we would do?

12

Vision: A community where people are mentally and physically healthy throughout the life cycle.

Strategic Issue: How can we ensure that our youth reach their full potential in life?

Goal: Reduce the use of marijuana and alcohol use by youth.

Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014.

b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014.

Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana.

b) Percentage of middle and high school students indicating that they drink alcohol.

Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol.

b) Build the skills of parents and other adults to talk with their children about the dangers and consequences

of using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community.

Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools.

b) Provide workshops for parents and create parent chat groups

c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors.

d) Set up a tip line on marijuana sales.

Performance Indicators: a) Pre- and post test results of youth participating in awareness programs.

b) Number of parents attending workshops.

c) Number of parents participating in chat groups.

d) Number of vendors who pass alcohol compliance checks.

e) Number of calls to the tip line.

13

Strategies:

a) Provide information to youth about the dangers and consequences of using marijuana and alcohol.

b) Build the skills of parents and other adults to talk with their children about the dangers and

consequences of using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community.

Tactics:

a) Provide marijuana and alcohol awareness programs to youth in middle and high schools.

b) Provide workshops for parents and create parent chat groups

c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors.

d) Set up a tip line on marijuana sales.

Performance Indicators:

a) Pre- and post test results of youth participating in awareness programs.

b) Number of parents attending workshops.

c) Number of parents participating in chat groups.

d) Number of vendors who pass alcohol compliance checks.

e) Number of calls to the tip line.

Methods for process evaluation

14

Vision: A community where people are mentally and physically healthy throughout the life cycle.

Strategic Issue: How can we ensure that our youth reach their full potential in life?

Goal: Reduce the use of marijuana and alcohol use by youth.

Objectives: a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014.

b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014.

Outcome Indicators: a) Percentage of middle and high school students indicating that they use marijuana.

b) Percentage of middle and high school students indicating that they drink alcohol.

Strategies: a) Provide information to youth about the dangers and consequences of using marijuana and alcohol.

b) Build the skills of parents and other adults to talk with their children about the dangers and consequences

of using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community.

Tactics: a) Provide marijuana and alcohol awareness programs to youth in middle and high schools.

b) Provide workshops for parents and create parent chat groups

c) Work with law enforcement to do local vendor compliance checks on alcohol sales to minors.

d) Set up a tip line on marijuana sales.

Performance Indicators: a) Pre- and post test results of youth participating in awareness programs.

b) Number of parents attending workshops.

c) Number of parents participating in chat groups.

d) Number of vendors who pass alcohol compliance checks.

e) Number of calls to the tip line.

15

Objectives:

a) Decrease the percentage of youth using marijuana from 20% to 15% by 2014.

b) Decrease the percentage of youth drinking alcohol from 50% to 30% by 2014.

Strategies:

a) Provide information to youth about the dangers and consequences of using marijuana and alcohol.

b) Build the skills of parents and other adults to talk with their children about the dangers and

consequences of using marijuana and alcohol.

c) Reduce the access of marijuana and alcohol in the community.

Outcome Indicators:

a) Percentage of middle and high school students indicating that they use marijuana.

b) Percentage of middle and high school students indicating that they drink alcohol.

Methods for outcome evaluation

16

Creating an Evaluation Plan

• Purpose and focus of evaluation (should be practical, doable, and

reflect evaluation priorities)

• Evaluation questions

• Schematic (e.g. logic model)

• Methods

• Workplan & timeline

• Participants

• Resources

• Methods for monitoring results

• How and when information will be applied

Remember:

You don’t have to evaluate

everything.

A subcommittee can take on

this task.

Enlist students to help create

a plan.

Identify a partner who can

take ownership of activities

Performance Management System

Source: Turning Point Performance Management Collaborative, 2003.

18

Considerations for Developing &

Updating a Monitoring System

• Who?

•Identify roles and responsibilities of partners in monitoring

and updating system

•Role of Health Department given PHAB requirements?

• When?

•Identify frequency of monitoring and updating activities

• How?

•Identify methods for monitoring and updating

Documentation

Spreadsheets

Dashboards

Balanced Score Cards

Assessment data collection system (e.g. Healthy

Communities Institute, Network of Care)

19

Considerations for Developing & Updating a Monitoring

System (cont’d)

What?

• Centralized vs. Decentralized

• Quantitative and qualitative data

Why?

• Specific to community health improvement process vs. embedded within or

related to a performance management system

20

Monitoring Systems Examples

*Note: Each community should consider it’s own needs and resources when considering the

different systems. This is not a comprehensive listing; it is meant to indicate a few

possibilities that are available.

Community Balanced Scorecard: “Map strategies based on cause & effect assumptions across multiple

performance perspectives or “views” that relate to strategy; strategic alignment of a community-wide

system

Web-based Dashboards (and more): Can be useful and easy to maintain; are often costly to set-up and

then potential for high maintenance costs; want to ensure that have some ability to customize

indicators. (e.g. San Francisco Health Matters http://www.healthmattersinsf.org/, Network of Care)

PARTNER Tool: Social Network Analysis Tool http://www.partnertool.net/ (e.g. New Orleans)

Databases-Microsoft Excel and Access, SQL, etc.

Software: Lyon Software, CBISA, WePlanWell (Sedgwick County Health Department), etc.

21

Considerations in Designing or Purchasing Monitoring

Systems for Health Improvement Plans (and More)

• [Agency] Vision, Goals, and Performance Measures

•e.g. How will system fit with or enhance existing systems?

• Stakeholders and Governance

•e.g. Who is charged with system accountability? Who will populate the system?

• Value and Program Benefits

•e.g. How often is data updated? From what sources? Sub-population data?

• Project Management and Risk Management

•e.g. what are the potential negative impacts? How will these be mitigated?

• Financial Considerations

•e.g. what is initial development cost, if any? What is cost to update and maintain? At what

frequency? Financial sustainability?

Source: NACCHO and Public Health Informatics Institute. http://www.naccho.org/toolbox/tool.cfm?id=790

22

Considerations in Designing or Purchasing Monitoring

Systems: An Initial Planning Checklist

Source: All -Systems-Go

Resource Sheet ,

NACCHO and Public

Health Informatics Institute

http://www.naccho.org/toolb

ox/tool.cfm?id=790

Performance Management System Resources

• Performance Management Self-Assessment Tool:

http://www.collaborativeleadership.org/pages/pdfs/CL_self-

assessments_lores.pdf

• Turning Point Resources:

http://www.turningpointprogram.org/Pages/perfmgt.html

• PHF’s Performance Management & QI Website:

http://www.phf.org/focusareas/PMQI/Pages/default.aspx

• Public Health Performance Management Centers for Excellence:

• Developing, Monitoring, and Using Performance Measures:

http://www.doh.wa.gov/PHIP/perfmgtcenters/modules/Year2/11-09-

11_PerfMeas_public_main.htm

24

Community Reporting

• Format (e.g. oral, written)

• Frequency

• Who will be the lead? Are multiple leads helpful?

• Involve community members and partner organizations

• Managing Expectations

•Consider in CHIP rollout and in implementing strategies

• What is your hypothesis for how your strategies will lead to change?

• Illustrating how short, intermediate, and long-term objectives and goals relate to each

other and how each are important

• Don’t make annual report the first time they hear about the goals and objectives and

performance measures

•Involve community members and partner organization in efforts to improve credibility

•Consider delivery: presentation methods (e.g., oral, written; context; who is the “voice” of the

information?)

25

Revising the CHIP Based on Monitoring and Evaluation

Efforts

Set criteria ahead of time for when you’ll make revisions

• Will it solely be based on numeric information or also on feedback ? (Feedback is a two-way street)

• Broad-scale changes (e.g. Healthy People 2020)

• Only annually or will you consider in shorter time frame?

• Who will need to approve? How will you announce?

What will you change?

• Objectives—actual aim, or just number you were trying to reach?

• Strategies—if what you’re doing isn’t working, will you change it?

• Indicators—will you change indicators if you realize that it is not measurable or indicator is not

appropriate?

26

Reaching Out to Other Communities

CHA/CHIP Demonstration Sites Google Group or E-mail sites directly:

For more information contact Reena at rchudgar@naccho.org

MAPP Network: http://mappnetwork.naccho.org/

NACCHO Accreditation Support Initiative (ASI) sites: Forthcoming

performance management system information and lessons

QUESTIONS AND DISCUSSION