Assessing Fidelity: A Necessary Component of Effective Implementation

Sandra F. Naoom, MSPHNational Implementation Research Network

Frank Porter Graham Child Development Institute

University of North Carolina- Chapel Hill

Assessing Fidelity: A Assessing Fidelity: A Necessary Component of Necessary Component of Effective Implementation Effective Implementation

Child Care Policy

Research Consortium

Annual Meeting 2009

Copyright © Dean L. Fixsen and Karen A. Blase, 2008

What is Implementation?What is Implementation?

The act of The act of accomplishing some aimaccomplishing some aim or or executing some orderexecuting some order

To To put into practical effectput into practical effect; carry out; carry out

Pursue to a conclusionPursue to a conclusion– – Dictionary.comDictionary.com


Stages of ImplementationStages of Implementation

Major Implementation Initiatives occur in stages:

Exploration

Installation

Initial Implementation

Full Implementation

Innovation

Sustainability

2 – 4 Years


INTEGRATED & COMPENSATORY

CONSULTATION & COACHING

CONSULTATION & COACHING

STAFF EVALUATION

STAFF EVALUATION

FACILITATIVE ADMINISTRATIVE

SUPPORTS

FACILITATIVE ADMINISTRATIVE

SUPPORTS

RECRUITMENT AND SELECTION

RECRUITMENT AND SELECTION

PRESERVICE TRAINING

PRESERVICE TRAINING

SYSTEMSINTERVENTIONS

SYSTEMSINTERVENTIONS

Implementation Drivers Implementation Drivers (Processes)(Processes)

DECISION SUPPORT DATA SYSTEMS

DECISION SUPPORT DATA SYSTEMS


Implementation and FidelityImplementation and Fidelity

“The bridge between a promising idea The bridge between a promising idea and the impact on students is and the impact on students is implementation, but innovations are implementation, but innovations are seldom implemented as intended” .seldom implemented as intended” .

-Berman & McLaughlin, 1976-Berman & McLaughlin, 1976


Fidelity and Fidelity MeasuresFidelity and Fidelity Measures

Fidelity is the degree to which a Fidelity is the degree to which a program as implemented corresponds program as implemented corresponds with the program as described.with the program as described.

(Fixsen, Naoom, Blase, Friedman & Wallace, 2005, (Fixsen, Naoom, Blase, Friedman & Wallace, 2005,

National Implementation Research Network)National Implementation Research Network)

Fidelity measures are tools to assess Fidelity measures are tools to assess the adequacy of the implementation of the adequacy of the implementation of the program as described. the program as described.


Fidelity in the Field: The PrevalenceFidelity in the Field: The Prevalence

Implementation efforts in the ‘real’ world, are often Implementation efforts in the ‘real’ world, are often unsuccessful and highly variable both in terms of unsuccessful and highly variable both in terms of the fidelity and sustainability of the program or the fidelity and sustainability of the program or practice. (Elliott & Mihalic, 2004). practice. (Elliott & Mihalic, 2004).

About 23%-81% of program activities may be About 23%-81% of program activities may be omitted during implementation (Durlak, 1998).omitted during implementation (Durlak, 1998).

““Surprisingly, many of the highest-quality programs Surprisingly, many of the highest-quality programs fail to take adequate steps to monitor and verify fail to take adequate steps to monitor and verify program integrity...This weakens the conclusions program integrity...This weakens the conclusions that can be drawn regarding the program outcomes that can be drawn regarding the program outcomes and reduces the likelihood that replications will and reduces the likelihood that replications will resemble the original program” (Domitrovitch and resemble the original program” (Domitrovitch and Greenberg, 2000).Greenberg, 2000).


Why is Fidelity Important?Why is Fidelity Important?

Basis for program effectivenessBasis for program effectiveness

Helps to avoid Type III ErrorHelps to avoid Type III Error

Promotes Internal and External ValidityPromotes Internal and External Validity

Research has linked fidelity with positive Research has linked fidelity with positive outcomesoutcomes

Adult Mental Health- ACT; Children’s Adult Mental Health- ACT; Children’s Services- FFT, MST, Wraparound; Medicine- Services- FFT, MST, Wraparound; Medicine- DOTS, Texas AlgorithmDOTS, Texas Algorithm


Why Measure Fidelity?Why Measure Fidelity?

Measure Fidelity AND Outcomes BECAUSEMeasure Fidelity AND Outcomes BECAUSE

you need to know:you need to know:

Are we having an implementation problem?Are we having an implementation problem?

Low fidelity + Poor outcome =Low fidelity + Poor outcome =

Implementation problemImplementation problem

Are we having an effectiveness problem?Are we having an effectiveness problem?

High fidelity + Poor outcome = High fidelity + Poor outcome =

Effectiveness problemEffectiveness problem


Common Dimensions of Intervention Common Dimensions of Intervention Fidelity Fidelity

AdherenceAdherence

Dosage/ExposureDosage/Exposure

Quality of Program DeliveryQuality of Program Delivery

Participant Responsiveness Participant Responsiveness

Program Differentiation Program Differentiation

(e.g. Dane & Schneider, 1998; Dusenbury, (e.g. Dane & Schneider, 1998; Dusenbury, 2003)2003)


NIRN’s Fidelity DimensionsNIRN’s Fidelity Dimensions

Context Context

Pre-requisites and pre-conditions for a Pre-requisites and pre-conditions for a program to operate.program to operate.

ComplianceCompliance

Extent to which practitioner uses intervention Extent to which practitioner uses intervention processes prescribed by the program processes prescribed by the program and and avoids those proscribed by the program.avoids those proscribed by the program.

CompetenceCompetence

Level of skillLevel of skill demonstrated by practitioner demonstrated by practitioner while delivering services or treatment to the while delivering services or treatment to the consumer.consumer.


Fidelity AssessmentsFidelity Assessments

Fidelity assessments should include aspects Fidelity assessments should include aspects of :of :

Structure Structure

The framework for service delivery.The framework for service delivery.

Process Process

The way in which services are delivered.The way in which services are delivered.


Methods to Develop a Fidelity Assessment

Requires the specification of a theory of change; Requires the specification of a theory of change; core components of intervention need to be core components of intervention need to be identified and measured.identified and measured.

Drawing from a specific program model with Drawing from a specific program model with proven efficacy and effectivenessproven efficacy and effectiveness

Gathering expert opinionGathering expert opinion

Survey of experts and/or literature Survey of experts and/or literature reviews to identify active ingredients or the core reviews to identify active ingredients or the core components of the program.components of the program.

Opinions of users and advocates regarding what Opinions of users and advocates regarding what works or site visits of diverse currently operating works or site visits of diverse currently operating programs. programs.


Measuring Intervention Fidelity

Most common methods:Most common methods:

Ratings by experts, based on project Ratings by experts, based on project documentation, client records, site documentation, client records, site observations, interviews and/or observations, interviews and/or videotaped sessions.videotaped sessions.

Surveys or interviews completed by the Surveys or interviews completed by the individuals delivering the services or individuals delivering the services or those who receive them.those who receive them.


For More InformationFor More InformationSandra F. Naoom, M.S.P.H.

919-636-2035

[email protected]

At the Frank Porter Graham Child Development Institute

University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/ http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/


For More Information

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:

http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

To order the monograph go to:

https://fmhi.pro-copy.com/

Documents

Assessing Fidelity: A Necessary Component of Effective Implementation