Measuring Fidelity in Early Childhood Scaling-Up Initiatives: A Framework and Examples Carl J....

Measuring Fidelity in Early ChildhoodScaling-Up Initiatives:

A Framework and Examples

Carl J. Dunst, Ph.D.

Orelena Hawks Puckett InstituteAsheville, North Carolina

Presentation made at the “Fidelity: How Much Devotion is Enough? (and Other Methodological Virtues)” session

at the Office of Special Education Program’s Project Directors Conference,Washington, DC, July 2008

Purposes of the Presentation

• Describe a framework for measuring fidelity as part of scaling-up early literacy learning practices.

• Define different types of fidelity at different levels of training and practice.

• Illustrate how variations in fidelity can be related to variations in practice adoption and outcomes.

Center for Early Literacy Learning

The main aims of CELL are to: (1) Synthesize researchevidence on effective early literacy practices, (2) developevidence-based practices from the research syntheses,(3) evaluate the validity and use of the practices, and (4)conduct both generalized and specialized technicalassistance promoting adoption and use of CELL practices.

From Evidence-Based Practices to Fidelity to Outcomes

Developing Evidence-Based Early Literacy Learning Practices

Scaling-Up the Use of the Practices

Measuring the Fidelity of Use of the Practices

Child Outcomes

Early Literacy Learning

Specialized Technical Assistance

Framework for Measuring Fidelity

Types of Fidelity

Implementation

Diffusion

Intervention

Implementation Fidelity

Implementation fidelity is defined as the degree to whichCELL training activities of known characteristics areimplemented as planned and promote participantunderstanding of the known characteristics of the evidence-based practices constituting the focus of training.

Diffusion Fidelity

Diffusion fidelity is defined as the degree to which secondand third generation technical assistance providers useCELL training methods and procedures of known characteristics to promote end-user understanding of thekey characteristics of evidence-based practices.

Intervention Fidelity

Intervention fidelity is defined as the extent to which endusers adopt and use instructional practices of knowncharacteristics for implementing evidence-based practicesof known characteristics.

Elements of Fidelity

Each type of fidelity includes two elements:

• Fidelity of the training processes (how)

• Fidelity of practice understanding and adoption (what)

Elements of Fidelity

Training processes are HOW we promote adoption and useof WHAT we want implemented by end-users. As state byFixsen et al. (2005), it is important to be aware of thedifference between the practices constituting the focus oftraining and the training methods used to promote adoptionof the practices.

Framework for Structuring Fidelity Assessments

CELL Fidelity Checklist for MeasuringImplementation and Diffusion Fidelity

Fidelity of both implementation and diffusion training isbeing measured by a CELL fidelity checklist that assessesthe degree to which CELL training processes and CELLpractices are accomplished with (1) sufficient dose (2)and with ample quality exposure to (3) key training and practice characteristics, and (4) are viewed by participantsas socially valid.

Measuring Fidelity of Intervention

Fidelity of intervention is measured in terms of practitioneror parent, or both, use of instructional practices andevidence-based practices of known characteristics as partof the provision of everyday early childhood intervention. Fidelity is measured by the frequency and number ofpractices that are used and the development-enhancingqualities (active ingredients, core features, etc.) of theintervention practices. The social validity of both thepractices and the child outcomes from using the practicesare also assessed.

0

10

20

30

40

50

60

70

80

Not At All the Case

Somewhat the Case

Mostly/Very Much the Case

DEVELOPMENT-ENHANCING CHARACTERISTICS

PE

RC

EN

T O

F P

AR

TIC

IPA

NT

S

Parents

Practitioners

Relationship Between Fidelity and Child Outcomes

Practice MeasuresLow Users High Users

Mean Mean F-Test Cohen’sa a d

Number of Weeks -.21 .16 1.26 .37

Number of Learning Games -.32 .32 5.70* .67

aMean standardized score.

*p < .05.

Game Characteristics -.61 .51 5.73* 1.32

Further Evaluation of the Effectivenessof the Practice

Additional information about the effectiveness of thepractices was determined by a post-test only quasi-experimental between group design that included twooutcomes, one of which was a “dependant variable that atreatment should affect” and a second dependantmeasure that was not expected to be related to the use ofthe practice (Shadish et al., 2002, p.184). Obtaining anexpected group by outcome measure interaction was usedto ascertain effectiveness.

Conclusions

• Fidelity is important as part of ascertaining the adoption and use of evidence-based practices.

• A fidelity framework can help structure the development and collection of fidelity measures.

• It is important to measure both implementation and intervention fidelity.

• Relating variation in fidelity to variation in outcomes can help identify what matters most in terms of targeted practices.

For More Information Visit

www.puckett.org for methods and procedures for identifying evidence-based practices

www.earlyliteracylearning.org for information on the Center for Early Literacy Learning.