Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011

Toward a Technology of Treatment Integrity

Ronnie DetrichWing Institute

APBS, Denver, Colorado2011

Goals for Today

• Review the link between data-based decision making and treatment integrity.

• Discuss the dimensions of treatment integrity.

• Review methods for assuring treatment integrity.

Data-based Decision Making and Treatment Integrity

• Data-based decision making at heart of PBS and RtI.• The impact of PBS and RtI depends on the

effectiveness of specific interventions.• The effectiveness of interventions is a function of the

integrity with which they are implemented.• The quality of decisions about effects of an

intervention is directly linked to the quality of implementation.

Data-based Decision Making and Treatment Integrity

• Student data provides feedback about progress.• If we know about adequacy of treatment integrity

then can make decisions: Adequacy of intervention Adequacy of implementation

If implementation is inadequate then focus should be on improving implementation.

If implementation is adequate then focus should be on changing intervention so student can succeed.

Grade Level Standard

Aim Line

Trend Line

Data-based Decision Making and Treatment Integrity

Positive NegativeH

igh

Low

Continue Intervention Change Intervention

Unknown reason Unknown reason• Intervention problem?

• Implementation problem?

• Other life changes?

• Unknown intervention?

• Intervention is effective?

OutcomeIn

tegr

ityPositive Negative

Hig

hLo

w

Assumptions about Treatment Integrity

• Students cannot benefit from interventions they do not experience.

• Unless educators know what they are supposed to be doing they cannot do more of it.

Dimensions of Treatment Integrity(Dane & Schneider,

1998)

• Exposure (Dosage): the extent to which participants are exposed to the intervention as prescribed. Curricula usually prescribe frequency and duration of

exposure that is necessary for benefit. Ex: 3/week for 30 minutes/session.

Failing to satisfy either can impact student benefit.Ex: 1/week for 30 minutes.

Dimensions of Treatment Integrity(Dane & Schneider,

1998)

• Adherence: the extent to which the components of an intervention are delivered as prescribed. Most commonly measured dimension. It is necessary but not sufficient to produce benefits.

Adherence with low dosage not likely to produce positive outcomes.

Dimensions of Treatment Integrity(Dane & Schneider,

1998)

• Quality of delivery: qualitative measure of how well the intervention is implemented. Importance has been acknowledged for years.

Have not developed good measures or how to influence it. Possible measures through social validity methods:

Enthusiasm Sincerity Variations in inflection and content of speech.

Variables that Influence Treatment Integrity

• Organizational Systems that are perceived by teachers to be supportive

and provide strong leadership have higher levels of integrity.

• Personal characteristics Burnout: Inverse relation between measures of teacher

burnout and treatment integrity. Personal efficacy: Positive correlation between measures

of personal efficacy and treatment integrity.

Variables that Influence Treatment Integrity

• Characteristics of Intervention Complexity

As complexity increases integrity decreases.

Relationship between Complexity, Precision, and Treatment Integrity

• Be as precise as necessary but no more.co

mpl

exity

precision

Catch’em being good

Check inCheck out

Individualized support plan

Integrity

Variables that Influence Treatment Integrity

• Characteristics of Intervention Perceived effectiveness. Acceptability. Contextual fit. Necessary resources available.

Variables that Influence Treatment Integrity

• Training Training is necessary but not sufficient. Not all training is equal.

 OUTCOMES

(% of Participants who demonstrate knowledge, demonstrate new skills in a training setting,

and use new skills in the classroom)

TRAININGCOMPONENTS

Knowledge SkillDemonstration

Use in the Classroom

Theory and Discussion

10%

5% 0%

..+Demonstration in Training 30% 20% 0%

…+ Practice & Feedback in Training

60% 60% 5%

…+ Coaching in Classroom 95% 95% 95%

Joyce and Showers, 2002

Not All Training is Equal

What We Know About Treatment Integrity

• It is estimated that drug prevention programs are implemented with integrity only 19% of the time. (Hallfors & Godette, 2002) Programs have extensive research base as being effective. This may be a generous estimate.

• No reason to assume that other programs are immune to poor implementation.

What We Know About Treatment Integrity

• Most of what we know has been developed at the level of individual student support plans.

• SET is a measure of treatment integrity at the school level. Does not address what is happening at the level of the

individual classrooms or individual students.

What Do We Know About Treatment Integrity and Student Behavior?

• Different levels of integrity result in different levels of student behavior.

(Wilder, Atwell, & Wine, 2006) • High integrity followed by declines in integrity has

limited disruptive effect on student behavior. (Northup, Fisher, Kahng, Harrel, & Kurtz, 1997)

• Low levels of integrity followed by increases in integrity do not produce the same level of student response as when integrity high from the beginning.

(Groskreutz,, Groskreutz, & Higbee, 2011)

What Do We Know About Treatment Integrity and Student Behavior?

• Implications Make sure that integrity is high at the beginning of

intervention. It is better to start with high levels of integrity and let it decline

than to start with low integrity and try to increase it. Maximizes impact of intervention.

What Do We Know About Integrity of Interventions at the Universal Level?

• Kovaleski, Gickling, Morrow, & Swank (1999)Evaluated high vs low implementation of Instructional Support Teams

(IST). School-wide organizational change. Students benefited from IST processes only when implemented with high

fidelity. Implementing with low fidelity resulted in no better outcomes for

students than control group not exposed to IST processes. Having structures in place was not sufficient to assure high fidelity. Fidelity assessed one time per year.

What Do We Know About Integrity of Interventions at the Universal Level?

• Horner (2005) Effect of high fidelity vs low fidelity on office discipline

referrals. Schools that implemented with high fidelity had 25% fewer

office referrals for major rule violations than schools that did not meet fidelity criterion.

Fidelity measures taken 2 times per year.

How Do We Assure High Levels of Integrity?

• Technology for assuring treatment integrity is emerging.

• Most of the research is at level of individual support plans.

• Feedback is most common approach.

Mortenson & Witt, 1998

Effective Performance Feedback

• More frequent the feedback better effects (Jones, Wickstrom, & Friman, 1997Mortensen & Witt, 1998).

• Daily better than weekly.• Immediate better than delayed.• Immediate more preferred than delayed.

Limitations of Performance Feedback

• Requires direct observation.• May be too resource intensive to implement at large

scale. Would require significant restructuring to implement

effectively.

Alternative Approaches: Teacher Self Report

• Teacher rate their own implementation. Teacher ratings tend to overestimate accuracy of

implementation ( Wickstrom, Jones, Lafleaur, & Witt). Teachers rated integrity at 54% accurate. Direct observation= 4%.

Integrity may be increased by rating immediately following intervention session.

Integrity may be improved by having teacher score video tape.

It may be possible to teach teachers to rate more accurately (see self-evaluation literature).

Alternative Approaches: Quizzes

• Quizzes (Detrich et al., 2001)

Staff quizzed weekly on elements of multi-component individualized behavior support plans.Given feedback on quiz but no feedback on actual implementation

of support plan.4 versions of the quiz. One question per element of the plan

(student preferences, antecedent interventions, teaching replacement behavior, responding to misbehavior).

Alternative Approaches: Quizzes

• Advantages: Easier to implement than direct observation. May be easy to implement at scale when standard

protocols are utilized.

• Limitations Does not produce maintenance effects in the absence of

quizzes.

Alternative Approaches: Video Modeling

• Has been utilized to train a standard protocol as well as individualized interventions. Problem solving curriculum. Discrete trial instruction. Functional assessment.

Alternative Approaches: Video Modeling

• Advantages: For standard protocols can be used to train many

implementers. Demonstration of implementation can be standardized. Flexible scheduling of training.

• Limitations Has not been evaluated at large scale. Coaching resources required to assure modeling was

effective.

Alternative Approaches: Checklists

Alternative Approaches: Checklists

• Advantages: Standardize implementation. Increase integrity especially adherence.

• Limitations: Upfront effort to develop. Often low acceptability.

Alternative Approaches: Intensive Professional Development

• University faculty/coach assigned to school for full year (Klingner, Vaughn, Hughes, & Arguelles, 1999). Provided in-depth training on three reading programs. Classroom based coaching. Problem solving implementation. Produced moderate levels of treatment integrity at three

years follow-up.

Alternative Approaches: Intensive Professional Development

• Advantages: Highly trained resource on site.

• Limitations Difficult to replicate at large scale. Expensive.

Unresolved Issues

• Adoption or adaptation?Must we implement exactly as prescribed or can we

adjust to fit local circumstances?

• Research suggests that programs are almost always adapted.

• Presumably adapted to improve outcomes. Some adaptation for other reasons:

Better fit teaching style. Do not like some elements of program.

Unresolved Issues

• What does this mean for treatment integrity? Are adaptations systematic?

If so, then we can assess integrity. What parts of program can we adapt without doing

harm to effectiveness? Core elements?

If adapted is program still research-based? If teachers allowed to adapt then program more

acceptable. Teacher’s made better adaptation of reading programs if they

were well grounded in principles. (Klingner, Vaughn, Hughes, & Arguellas, 1999).

Treatment Integrity and PBS

• Regardless of the level of the intervention, it is necessary to know that it was implemented with integrity. High integrity is necessary in a data based decision

making approach.

• Integrity should be assessed at the same level that the intervention is being evaluated.

Treatment Integrity and PBS

• A program or intervention is a set of protocols that guides behavior of the adult. If protocols are not followed then by definition the

program has not been implemented or sustained.

• PBS is an excellent model for making decisions about when, where, and how to intervene. Intervention without process for assuring integrity is likely

to result in wasted effort.

Taking Treatment Integrity Measurement to Scale: Data-based Decision Making

• Insufficient resources to frequently observe all educators working with students to determine adequacy of implementation.

• One solution: Measure student behavior and analyze at different units of analysis:SchoolClassroomIndividual student

• Any level there is gap between obtained and expected performance assess levels of treatment integrity.

Taking Treatment Integrity Measurement to Scale: Data-based Decision Making

• Requires that data are routinely collected across all levels. RtI:

CBM on all students 3/year. PBS

Continuous measurement of Office Discipline Referrals.

• Systems have to be in place to assure integrity data-based decision making process.

Where are We?

• Implementation is where good interventions go to die unless there is active plan to assess and influence integrity.

• Research-based approaches to influencing treatment integrity are emerging.

• We have ethical obligation to allocate resources to influencing treatment integrity.

Thank you

Copies may be downloaded atwinginstitute.org