Implementing and Evaluating Evidence-based Strategies

Assessing your

Community

Implementing & Evaluating

Establishing Goals &

Objectives;Planning for Evaluation

Finding Evidence

Selecting Best Fitting EBAs

Adapting

Where do We Stand?

Session Objectives

• Describe successful program implementation• Discuss the importance of community

engagement, partnerships, and collaborations to implement programs

• Describe key tasks in planning, implementation, evaluation and maintenance

• Develop an implementation and evaluation work plan integrating RE-AIM measures that match program objectives and activities

Intervention Planning Model

Community Assessment

DevelopGoals &

Objectives

Plan the InterventionFinding, Selecting and Adapting an Evidence-based Approach

ImplementEvaluate

Outcomes

Share your successes

***Apply lessons

learned to Maintain and Sustain the intervention

Assess Readiness

Evaluate Processes

Planning Steps• Enlist community/stakeholder input

– Use existing committees– Understand current context, policy or environment– Best communication/outreach strategies

• Understand context, fit, and your readiness– Incorporate previous needs assessment data– Know your organizational culture related to health

• Draft work plan for program, policy, strategies– Research any previous interventions

for useful content/language – Receive feedback from expert panel

Planning your implementation and evaluation: Review Materials

Policy Process Guides

http://www.prevent.org/data/files/initiatives/smokefreepolicies.pdf

Assess Readiness

• Do you have:– Supportive leaders? – Engaged partners?– Resources (money, staff, supplies)?– Staff with knowledge, skills, and enthusiasm? – Accounting and administrative structures?– System for monitoring progress/evaluation?

• Refer to implementation guides and protocols for resources needed

Assess Organizational Readiness

Readiness Checklist for Body and Soul

What leads to implementation quality?

1. Community level engagement

(funding, policy, politics)

2. Provider/implementer characteristics– Need for innovation– Benefits of innovation– Self-efficacy– Skill proficiency for implementation

Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.

What leads to implementation quality?

3. Implementers’ organizational capacity– General organizational factors

(vision, norms, climate)– Specific processes

(decision-making, partnerships, communication)– Staff considerations

(leadership, champion)

4. Prevention support– Training– Technical assistance

Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.

Create a Work Plan

Ideas from your Organizational Readiness

Assessment

Your Work Plan for Implementing and Evaluating

Body and Soul

Components of a Work Plan for Implementation

• SMART Objectives• List of tasks or activities for each Objective• Person responsible• Resources needed• Target completion dates• Indicator(s)/Measures of completion or progress

Example: Work Plan for Body and Soul

Example Work Plan

Example Work Plan

Definitions of Workplan Steps: Policy & Environmental Change

• Review evidence/data on the problem and potential solutions• Gain stakeholder agreement on priority problems and

preferred approaches, • Draft policies in the form of rules/laws

Formulation

• Identify decision makers who will sponsor or promote the policy• Formally pass the policy in the form of a law, regulation, or

budget• Put environmental change in place (ex: inaugurate walking trail)• Communicate the new policy/environmental change

Enactment

• Monitor and document the use of the policy• Continue promotion, operationalization, and use of the policy• Conduct activities that support compliance to the policy• Establish a plan for enforcement

Implementation

• Maintain policy implementation• Ensure sustained effect through monitoring and enforcement• Possibly make further policy modification

Maintenance

Longest Jr. BB. (2009) Health Policymaking in the United States, 5th Ed.

Activity: Create an Implementation Work Plan

Activity: Create an Implementation Work Plan

Activity: Create an Implementation Work Plan

1. Review the handout on the Tobacco Use Prevention Policy Strategy

2. Choose at least 1-2 of the objectives already provided or write your own (for a worksite or faith setting)

3. For the objective(s) you chose, add:– One or more activities– Who would be responsible for each activity? – What resources are needed? – What is the completion date or due date?– What are your measures/indicators of completion or

progress?

Implementation Work Plan Discussion

4. Share different examples from the groups.

a) Specify if you chose a work site or faith setting.

b) Explain which objective you chose.

c) Discuss:– activities, – responsible persons, – resources, – dates, and – measures of completion or progress

EVALUATION Work planshave many of the same elements• SMART Objectives

• Activities for each Objective

• Indicators/Measures (of completion or progress)

• Person responsible

• Resources needed

• Target completion dates

• Processes for monitoring, reporting, modifying implementation and evaluation plans

Data sources

for evaluation activities

Actual completion dates

If you are adopting an existing evidence-based program, policy or other strategy...• Conduct formative evaluation on adapted materials

– Feedback from expert panel – Focus groups/discussion with target populations– Pilot testing

• If it was evaluated before, you can use prior process and outcome indicators, methods, and tools

– What outcomes were measured?

– How rigorous was the original evaluation?

– Do you have resources to do a similar evaluation?

• Monitor fidelity of implementing core elements and key process steps (process evaluation)

Evaluation – Using RE-AIM

• REACH your intended target population• EFFECTIVENESS or efficacy• ADOPTION by target staff, settings, or institutions• IMPLEMENTATION consistency, costs and

adaptations made during delivery• MAINTENANCE of intervention effects in

individuals and settings over time

http://www.re-aim.org/

Evaluation PhasesPlanning •Consult with the program stakeholders or

resources•Develop the questions •Identify sources of data •Make a timeline

Data Collection

•Pilot test•How will the questions be asked? •Who will ask them?

Data Analysis

•Who will analyze the data and how?

Reporting •Who will report and how? •Who will receive the data and when? •How will it affect the program?

Gather Credible Evidence

(not the other way around)

IndicatorsEvaluation questions

Methods and Measures

Gather Credible Evidence

• Link each indicator to a data source

• Look for existing data – If there are no data available – collect your own

• Look for existing data collection tools– If there are no existing tools – create your own

• Data collection plan– Who will collect?

– How will it be collected?

– When and How often will it be collected?

Evaluation for Programs

• Track program promotion and/or recruitment(# flyers, frequency of announcements)

• Document implementation of program core elements or key process steps (e.g., checklists, length & # of sessions, meetings)

• Collect process measures (e.g., attendance, timeline/work plan updates, etc.)

• Monitor program activities and performance (# of new screenings, timeliness of diagnosis)

Evaluation for Policies

• Document and publicize policy

• Track implementation of key process steps of the policy– Documentation of policy– Procedures for policy– Communication– Enforcement

• Collect process measures on benchmarks – Track participants in policy planning (partners)– Revisions of policies– Advocacy efforts (e.g., media outlets, events)– Adoption and enforcement

Example Evaluation Plan: Body and Soul

Example Evaluation Plan: Body and Soul

Discussion: Develop an Evaluation Plan

Discussion: Develop an Evaluation Plan

1. Refer to the SMART objective you chose in the work plan activity.

2. Discuss evaluation question(s) that measure success of that objective.

3. What are potential data sources to find this information?

4. What are some methods you can use to measure these processes or outcomes? (e.g. logs, surveys, etc.)

Intervention Planning Model

Community Assessment

DevelopGoals &

Objectives

Plan the InterventionFinding, Selecting and Adapting an Evidence-based Approach

ImplementEvaluate

Outcomes

Share your successes

***Apply lessons

learned to Maintain and Sustain the intervention

Assess Readiness

Evaluate Processes

Share Evaluation Results

• Use your evaluation findings

– to modify your program

– describe program benefits and challenges

• Reach all stakeholders inside and outside your organization

– In language and formats that everyone understands

– In time to make decisions

Maintenance…Plan for Sustainability

Sustainability is…

… the existence of structures and processes which allow a program to leverage resources to most effectively implement evidence-based policies and activities.

Luke D, Calhoun A, Herbers S. (Feb 2012). Building & Managing Sustainability in Public Health Programs. CDC Roundtable Presentation.

Maintenance Activities

Leverage resources to sustain interventions over time:• Provide referrals for further service, when needed• Seek additional funding• Maintain or secure a program champion• Make program a part of organizational services/policies• Surveillance of the impact over time

(e.g., tracking of smoking rates, BMI of children, screening)

• Demonstrate return on investment/cost savings to justify investment of other stakeholders

• Continue partnerships or build new ones to share costs and implementation delivery

Sustainability Planning Resources

https://sustaintool.org/

Schell, S.F., Luke, D.A., Schooley, M.W., Elliott, M.B., Herbers, S.H., Mueller, N.B., & Bunger, A.C. (2013). Public health program capacity for sustainability: A new framework. Implementation Science, 8(15).

Take-home Points: Importance of Implementation and Evaluation

• Successful implementation and evaluation requires detailed, advanced planning

• Involve partners from the beginning to implement with high quality

• Assess organizational readiness and build capacity • Use and adapt pre-existing guidance and tools• RE-AIM can be helpful to measure success in

meeting objectives• Use evaluation findings and lessons learned to plan

for long-term sustainability

Questions?