CHRIS BORGMEIERPORTLAND STATE UNIVERSITY

Evaluating Researchon Challenging Behavior

From Spencer, Detrich & Slocum, 2012

www.pbis.org

Horner, R., & Sugai, G. (2008). Is school-wide positive behavior support an evidence-based practice? OSEP Technical Assistance Center on Positive Behavioral Interventions and Support. http://www.pbis.org/files/101007evidencebase4pbs.pdf.

Evidence Basics

From George Sugai

Why evidence-based?

Maximize outcomesMinimize harmIncreased accountabilityIncrease efficiencyImprove decision makingImprove resource use

Basic Approach

Start w/ what has greatest likelihood of addressing (evidence-based) confirmed problem/question Explained/supported

conceptually/empiricallyAdapt to local context/culture/needMonitor regularly & adjust based

on dataAdapt for efficient & durable

implementation

4 Evaluation Criteria

Effectiveness Has/will practice produced desired

outcome?Efficiency

What are costs (time, resources, $) to implement practice?

Relevance Is practice & outcomes appropriate for

situation?Conceptually soundness

Is practice based on theory?

Identify practicethat addressesneed/problem?

Is practiceresearchbased?

Specify features ofneed/problem

Is evidence ofeffectiveness

available?

Can practicebe adapted?

Implement &monitor effects

Consider anotherpractice

No No

Yes

Yes

No

Is adequateprogress

observed?

No

Yes

Improve efficiency& sustainability of

practiceimplementation

Review questions& data on regular

basis

Does problemexist?

Yes

No

Yes

Start

Basic Practices Evaluation

Identify practicethat addressesneed/problem?

Specify features ofneed/problem

Review questions& data on regular

basis

Does problemexist?

Yes

No

Start

Identify practicethat addressesneed/problem?

Is practiceresearchbased?

Is evidence ofeffectiveness

available?

Can practicebe adapted?

Consider anotherpractice

No No

Yes

Yes

No

Yes

Can practicebe adapted?

Implement &monitor effects

Yes

Yes

Is adequateprogress

observed?

No

Yes

Improve efficiency& sustainability of

practiceimplementation

Design Questions

Has functional or cause-effect relationship been demonstrated & replicated?

Have alternative explanations been accounted & controlled for?

Have threats or weaknesses of methodology been controlled for?

Was study implemented w/ fidelity/accuracy?

Research Designs

Experimental - RCT & SSREvaluation - Descriptive w/ baselineCase Study - Descriptive w/o

baselineTestimonial - No/Limited data

Results QuestionsWho were subjects?

How much like my participants?Where was study conducted?

How much like where I work?What measures were used?

Do I have similar data?What outcomes were achieved?

Are expected outcomes similar

Effectiveness Logic

Significance (“believe”) Likelihood of same effect by chance

Effect Size (“strength”) Size of effect relative to business as usual

Consequential Validity (“meaning”) Contextually meaningful

PBIS Evidence

Base

School-wide PBIS – Let’s compare!

www.pbis.org Click on ‘Resource Catalog’ Then ‘Literature List’

Evidence Base for SW-PBISRandomized Control Trials

Randomized Control Trials of SW-PBIS

Tier 1/ Universal SW-PBIS

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010).Examining the effects of school-wide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133-148.

Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N., & Leaf, P. (2008). The impact of school-wide positive behavioral interventions and supports (PBIS) on the organizational health of elementary schools.School Psychology Quarterly.

Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. (2008).Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 31, 1-26.

Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J., (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133-144.

Sprague, J., & Biglan, A., et al (in progress).A Randomized Control Trial of SWPBS with Middle Schools.

90-School RCT StudyHorner et al., in press

• Schools that receive technical assistance from typical support personnel implement SWPBS with fidelity

• Fidelity SWPBS is associated with▫ Low levels of ODR

▫ .29/100/day v. national mean .34▫ Improved perception of safety of the school

▫ reduced risk factor▫ Increased proportion of 3rd graders who meet state

reading standard.

RCT Project TargetBradshaw & Leaf, in press

PBIS (21 v. 16) schools reached & sustained high fidelity

PBIS increased all aspects of organizational health

Positive effects/trends for student outcomes Fewer ODRs (majors + minors) Fewer ODRs for truancy Fewer suspensions Increasing trend in % of students scoring in

advanced & proficient range of state achievement test

Impact Factor

Ask a Faculty member

Collaborative Problem Solving

Visit the website http://www.livesinthebalance.org/

What do I notice? A canoe? Advertising products for purchase Lots of testimonials Little bit of research (10 citations listed under

research) let’s take a closer, evaluative look at the research

Evaluating a Research Study

AbstractIntroduction & Literature ReviewResearch QuestionsMethod & Design

Subjects & Settings / Measures/ ProceduresResultsDiscussion & ConclusionsReferences

Results QuestionsWho were subjects?

How much like my participants?Where was study conducted?

How much like where I work?What measures were used?

Do I have similar data?What outcomes were achieved?

Are expected outcomes similar

Collaborative Problem Solvingdata from CPS website on 6/18/12

Johnson, M., Ostlund, S., Fransson, G., Landgren, M., Nasic, S., Kadesjo, B., Gillberg, C., and Fernell, E. (2012).  Attention-Deficit/Hyperactivity Disorder (ADHD) with Oppositional Defiant Disorder (ODD) in Swedish Children:  An Open Study of Collaborative Problem Solving.  Acta Paediactrica, in press. 

Ollendick, T. H. (2011). Invited Address: Effective Psychosocial Treatments for Emotional and Behavioral Disorders in Youth. University of Stockholm, Sweden..

Fraire, M., McWhinney, E., & Ollendick, T. (2011). The effect of comorbidity on treatment outcome in an ODD sample. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland.

Halldorsdottir, T., Austin, K. & Ollendick, T. (2011). Comorbid ADHD in children with ODD or specific phobia: Implications for evidence-based treatments. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland.

Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors: Evaluation of a Collaborative Problem Solving pilot program. Journal of Clinical Psychology Practice, 27-40.

Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and seclusion through Collaborative Problem Solving: A five-year, prospective inpatient study. Psychiatric Services, 59(12), 1406-1412.

Greene, R.W., Ablon, S.A., & Martin, A. (2006). Innovations: Child Psychiatry: Use of Collaborative Problem Solving to reduce seclusion and restraint in child and adolescent inpatient units. Psychiatric Services, 57(5), 610-616.

Greene, R.W., Ablon, J.S., Monuteaux, M., Goring, J., Henin, A., Raezer, L., Edwards, G., & Markey, J., & Rabbitt, S. (2004). Effectiveness of Collaborative Problem Solving in affectively dysregulated youth with oppositional defiant disorder: Initial findings. Journal of Consulting and Clinical Psychology, 72, 1157-1164.

Greene, R.W., Biederman, J., Zerwas, S., Monuteaux, M., Goring, J., Faraone, S.V. (2002). Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with oppositional defiant disorder. American Journal of Psychiatry, 159, 1214-1224.

Greene, R. W., Beszterczey, S. K., Katzenstein T., Park, K., & Goring, J. (2002). Are students with ADHD more stressful to teach? Patterns of teacher stress in an elementary school sample. Journal of Emotional and Behavioral Disorders, 10, 27-37.

These studies to not evaluate

effectiveness of CPS

NO studies have been conducted in school settings; all research is either

with parents or in-patient clinical settings

Evaluating the Research studies

2004; Greene et al

2006; Greene , Ablon, Martin

2008; Martin et al

2010 – Epstein & Saltzmann

SubjectsAge & #

47 kids w ODD4-12 yrs. old 3-14 yrs. old School-age

12 Kids w ODD

Under 12 yrs.

Settings Outpatient MH clinic @ hospital

Inpatient Psyc unit (13 beds)

Inpatient Psyc unit (15 beds)

Outpatient clinic

Procedure

Compare CPS w parent training (PT) group

Trained unit staff (pre/post)

Trained unit staff (pre/post)

Group CPS parent training (pre/post)

Outcome measure

ODDRS (unpublished rating scale created by Greene; Improvement ratings (maternal & therapist)

Restraints & seclusion

Restraints & Seclusions

Eyberg Child Beh. Inv. Parent Stress Index

Outcome Improved slightly more than PT

Reduced Reduced Improvement pre to post

What does the research tell us?

So what do we know?

Based on 4 evaluation studies All include children ages 12 or less (2008 study does not

specify an age range; simply ‘school age’) 2 are in inpatient psychiatric hospitals 1 is an outpatient mental health clinic 1 is a parent training program # in school settings = 0

The research tells us nothing about the efficacy of CPS in school settings

What does the research tell us?

Outcome measures ODD Rating Scale (unpublished assessment created by the

author) & improvement ratings from parent & therapist Similar scores to parent training

Reductions in restraint & seclusion (Pre/Post) Is this due to student behavior change or adult behavior

change?

Eyberg CBI & Parent Stress Index (Pre/Post)

No studies directly measure changes in student behavior

Concerns

Only 4 research studies evaluating CPS in 4 years 2 on parent training (1 individual training & 1 group training) 2 in inpatient psyc facilities

Make sure research you are looking at takes place in settings that match your application E.g. school settings v. treatment centers

2 of 4 studies have been conducted by the author of the program Concern if authors are benefiting financially from sale of the

program

Research on CPS in Schools!

Schaubman, A., Stetson, E., & Plog, A. (2011). Reducing Teacher Stress by Implementing Collaborative Problem Solving in a School Setting. School Social Work Journal, 35(2), 72-93.

Abstract Student behavior affects teacher stress levels and the student-teacher

relationship. In this pilot study, teachers were trained in Collaborative Problem Solving (CPS), a cognitive-behavioral model that explains challenging behavior as the result of underlying deficits in the areas of flexibility/adaptability, frustration tolerance, and problem solving. It was hypothesized that teacher stress would be reduced when teachers' understanding of the underlying causes of student behavior shifted to a framework of skills development, and they began using a proactive, positive approach to misbehavior (CPS), with the support of mental health consultation. Results showed a significant decrease in teacher stress, as measured by self-report. Further, discipline referrals were significantly reduced. Limitations of the study and implications for school mental health consultation are also discussed. (Contains 4 figures.)