Building a Science on Building a Science on Implementation of Evidence-based Implementation of Evidence-based Practices in Children’s Mental Practices in Children’s Mental Health Health

Kimberly Eaton Hoagwood, Ph.D.

Columbia University

December 11, 2005

Turning Points in Child & Adolescent Services Turning Points in Child & Adolescent Services Research: A Very Brief HistoryResearch: A Very Brief History

Unclaimed Children (Knitzer, 1982) documents lack of community-based care (soon to be updated)

Systems of Care Monograph published (Stroul & Friedman, 1986); CASSP established, 1986

Tripling of funding for research on children’s mental health at NIMH (1989-2001)

Meta-analyses of psychotherapies document effect sizes (Weisz et al, 1995, 1998, 2003; Kazdin et al., 1998, 2003)

System of Care study results published (Bickman, 1996) Healthcare reform in US and Britain spurs growth of evidence-based

practice movement (1996-present) Surgeon General’s Reports (1999; 2000; 2001) highlight disparities between

research and practice Identification of >550 psychosocial therapies (Kazdin, 2003); medication

trials for ADHD, OCD, aggression, depression Methods developed for assessing organizational context applied to youth

mental health services and found to predict child outcomes (Glisson, 2002, 2005; Schoenwald et al, 2003)

EBPs spread into state and federal policy planning (2004-present)

Yet…sobering facts about mental Yet…sobering facts about mental

health services for childrenhealth services for children Unmet need as high now as 20 years ago Unmet need highest among minority youth Receipt of mh services increased but only 1/5 of

children with the most severe needs receive mh services (Kessler et al 2005; Foster, Rollefson, Doksum et al., 2005)

Onset by age 14 for 90% of adults with SMI (Kessler et al., 2005)

Lack of availability and access a major barrier for most families

There is no system: use and need don’t match up

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

White African-American

Latino Other

Ringel &Sturm, 2000; NIMH, 2001

Unmet Need for Mental Health Services

Unmet Need for Mental Health Services

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

Alaba

ma

Califo

rnia

Color

ado

Florid

a

Massa

chus

etts

Michiga

n

Minnes

ota

Mississ

ippi

New Je

rsey

New Y

ork

Texa

s

Was

hingt

on

Wisc

onsin

MH care use MH service need

National Average MH Need for Children at 6-17: 7.09%

National Average MH Use for Children at 6-17: 7.45%

Data Source: NSAF wave 1 and 2, Sturm, 2001

National Averages of Use and National Averages of Use and Need Don’t MatchNeed Don’t Match

Beyond the Linear ModelBeyond the Linear Model

Basic Research

Clinical Trial

(Efficacy)

TreatmentDevelopment

EffectivenessTrial

TreatmentDeployment

Schoenwald & Hoagwood, 2001Schoenwald & Hoagwood, 2001

WHERE TRANSPORTABILITY QUESTIONS ARISE

EFFECTIVENESSTO

DISSEMINATION

EFFICACY

TO

EFFECTIVENESS

WHOCAN

DO IT?

WHATIS

IT?

WHOWILL

DO IT?

Practitioner

Organization

Referral

AgenciesPayers

Practitioner

Organization

The Rise in Popularity of the The Rise in Popularity of the term “Evidence-Based” term “Evidence-Based” (Hoagwood & Johnson, 2003)(Hoagwood & Johnson, 2003)

   

EBT EBP EBM 1900-1990 0 0 0 1990-1995 3 7 76 1995-2002 63 459 5,425

Psychotherapies provided in Psychotherapies provided in routine clinical care have little routine clinical care have little to no effect (Weisz et al., 1995)to no effect (Weisz et al., 1995)

Psychotherapies provided in Psychotherapies provided in routine clinical care have little routine clinical care have little to no effect (Weisz et al., 1995)to no effect (Weisz et al., 1995)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Smith &Glass,1977

Weisz etal., 1987

Weisz etal., 1995

Weisz etal, 1995

Mea

n E

ffec

t S

izes

Mea

n E

ffec

t S

izes

Weisz et al., 1995

Children & AdolescentsChildren & AdolescentsAdultsAdults

UniversityUniversity

Clinic settingsClinic settings

What is Evidence?What is Evidence? Lonigan, Ebert & Johnson, 1998; Lonigan, Ebert & Johnson, 1998;

Chambless et al., 1998Chambless et al., 1998

What is Evidence?What is Evidence? Lonigan, Ebert & Johnson, 1998; Lonigan, Ebert & Johnson, 1998;

Chambless et al., 1998Chambless et al., 1998 At least two controlled group design studies or a large

series of single-case design studies

Minimum of two investigators (for well-established)

Use of a treatment manual

Uniform therapist training and adherence

Tested with clinical samples of youth

Tests of clinical and functional outcomes

Long-term outcomes beyond termination of treatment

At least two controlled group design studies or a large series of single-case design studies

Minimum of two investigators (for well-established)

Use of a treatment manual

Uniform therapist training and adherence

Tested with clinical samples of youth

Tests of clinical and functional outcomes

Long-term outcomes beyond termination of treatment

....

Grading the Quality of Grading the Quality of Evidence Evidence

Biglan, Mrazek, Carnine, Flay (Am Psych 2003) Grades 1-7

1 = multiple RCTs or multiple time series experiments by 2 or more indep teams + implementation effectiveness

2 = 1 without implementation effectiveness 3 = no indep teams 4 = 1 RCT or time series 5 = comparisons w/o randomization 6 = pre-post 7 = endorsement by authorities

Kazdin (2004) criteriaKazdin (2004) criteria

Not evaluated Evaluated but unclear, no or possibly

negative effects at this time Promising (some evidence) Well-established (parallel to well-established

in conventional schemes) Better/Best Treatments (treatments shown to

be more effective than other evidence-based treatments)

14 Major Reviews of Evidence-14 Major Reviews of Evidence-based Interventions for Children based Interventions for Children (1998-2004)(1998-2004)

Chambless & Hollon (1998) Defining empirically-supported therapies. Journal of Consulting & Clinical Psychology

Surgeon General’s Mental Health Report (1999) Weisz & Jensen (1999) Mental Health Services Research Journal of the Am. Academy of Child/Adol. Psychiatry, 1999 Olds et al., (1999) Review of Preventive Interventions,

Center for Mental Health Services Burns, Hoagwood, & Mrazek (2000) Effective treatments for

mental disorders in children and adolescents, Child Clinical and Family Psych Rvw

Rones & Hoagwood (2000) School based mental health services review. Clinical Child and Family Psychology Review

Webster-Stratton & Taylor (2001) Preventing violence in adolescence with interventions for young children

Greenberg, et al., (2001) Prevention of mental disorders in school-aged children. Prevention & Treatment

Surgeon General’s Youth Violence Report (2001) Burns & Hoagwood (2002) Community treatments for youth: Oxford

University Press Kazdin & Weisz (2003) Evidence-based psychotherapy for children and

adolescents Weisz, JR. (2004) Psychotherapy for children and adolescent:

Evidence-based treatments and case examples. Cambridge University Press.

Burns, BJ & Hoagwood, K (in press) Update on evidence-based practices. Two Volumes. Psychiatric Clinics of North America

14 Reviews of EBPs14 Reviews of EBPs

More than 1500 published clinical trials on outcomes of psychotherapies for youth

More than 550 different named psychotherapies (Kazdin, 2000; Kazdin & Weisz, 2003; Weisz, 2004)

6 meta-analyses of their effects: effects as robust as for adults

More than 300 published clinical trials on safety/efficacy of psychotropic medications and growing

Approx 50 field trials of community-based services 47 effective school-based interventions cited by Rones &

Hoagwood (2000) 34 effective preventive interventions cited by Greenberg

et al, 2001

Strength of the evidenceStrength of the evidence

EVIDENCE-BASED PSYCHOSOCIAL TREATMENTSEVIDENCE-BASED PSYCHOSOCIAL TREATMENTSWell-Established Well-Established Probably Efficacious Probably EfficaciousDEPRESSION

None Self-Control (children)Coping with Depression (adolescents)IPT (adolescents)

ADHDBehavioral Parent Training Behavioral Management TrainingBehavioral Interventions in the Classroom Behavioral Modification in Classroom

ANXIETYNone Cognitive-Behavioral Therapy

PhobiaParticipant Modeling Imaginal and In Vivo DesensitizationReinforced Practice Live and Filmed Modeling

DISRUPTIVE BEHAVIORLiving with Children Delinquency Prevention ProgramVideotape Modeling Parent-Child Interaction Therapy

Parent Training ProgramTime-Out Plus Signal Seat Treatment

Anger Coping TherapyProblem Solving Skills Training

Anger Control Training w/ Stress Innoc Assertiveness TrainingMultisystemic TherapyRational-Emotive Therapy

PreschoolPreschool

AdolescentAdolescent

School AgeSchool Age

Source: Journal of Clinical Child Psychology, Volume 27,

Number 2, 1998 + additional studies for depression and PTSD

Evidence-Based Psychosocial TreatmentsEvidence-Based Psychosocial TreatmentsWell-Established Well-Established Probably Efficacious Probably Efficacious

PTSDPTSD

None Cognitive-Behavioral Therapy for sexual & physical abuse

Source: Journal of Clinical Child Psychology, Volume 27, Number 2, 1998 + additional studies for depression and PTSD

Ineffective Psychosocial Ineffective Psychosocial TreatmentsTreatments

Non-behavioral interventions for disruptive behavior disorders and/or ADHD (Weisz et al., 1995; Pelham et al., 1998)

Group, peer-based interventions for disruptive disorders

DARE (5th and 6th grade curriculum) Gun Buyback programs Boot Camps Peer counseling programs Summer job programs (at risk youth) Home detention with electronic monitoring Wilderness / challenge programs Casework / counseling

What produces negative (iatrogenic) outcomes Waivers to adult (criminal courts) Scared Straight Shock Probation / Parole

Ineffective programs to prevent Ineffective programs to prevent youth violence youth violence (Elliot, 2000)(Elliot, 2000)

Psychopharmacology Evidence for Childhood

Disorders

STRONGSTRONG

ADHDADHD StimulantsStimulantsTCAsTCAs

MODERATEMODERATE

WEAKWEAK

DEPRESSIONDEPRESSION SSRIsSSRIsAUTISM AUTISM AntipsychoticsAntipsychoticsOCDOCD SSRIs, TCAsSSRIs, TCAsODD/CDODD/CD Antipsychotics, Mood stabilizers, Antipsychotics, Mood stabilizers, StimulantsStimulantsANXIETYANXIETY SSRIsSSRIsAGGRESSIONAGGRESSION Atypical antipsychoticsAtypical antipsychotics

BIPOLAR BIPOLAR LithiumLithium

TOURETTE’S TOURETTE’S AntipsychoticsAntipsychotics

Evidence-based Home & Community-based ServicesEvidence-based Home & Community-based ServicesEvidence-based Home & Community-based ServicesEvidence-based Home & Community-based Services

Multisystemic TherapyMultisystemic Therapy(Henggeler et al., Schoenwald (Henggeler et al., Schoenwald et al)et al)

Intensive Case Management Intensive Case Management (including Wraparound)(including Wraparound)(Evans; Burchard; Burns)(Evans; Burchard; Burns)

8 RCTs and 1 quasi-exper. 8 RCTs and 1 quasi-exper. •fewer arrestsfewer arrests•fewer placementsfewer placements•decreased aggressiondecreased aggression•cost-savingscost-savings

4 RCTs and 3 quasi-exper.4 RCTs and 3 quasi-exper.

•less restrictive placementsless restrictive placements

•some increased functioningsome increased functioning

8 RCTs and 1 quasi-exper. 8 RCTs and 1 quasi-exper. •fewer arrestsfewer arrests•fewer placementsfewer placements•decreased aggressiondecreased aggression•cost-savingscost-savings

4 RCTs and 3 quasi-exper.4 RCTs and 3 quasi-exper.

•less restrictive placementsless restrictive placements

•some increased functioningsome increased functioning

Evidence-based Home & Community-based ServicesEvidence-based Home & Community-based ServicesEvidence-based Home & Community-based ServicesEvidence-based Home & Community-based Services

Treatment Foster CareTreatment Foster Care(Chamberlain)(Chamberlain)

Nurse home visitation Nurse home visitation Program (Olds et al)Program (Olds et al)

Functional family therapy Functional family therapy (Alexander & Sexton)(Alexander & Sexton)

4 RCTs4 RCTs•more rapid improvementmore rapid improvement•decreased aggressiondecreased aggression•better post-discharge better post-discharge

outcomesoutcomes

6 RCTs6 RCTs•Long term improvements Long term improvements

in reducing child abusein reducing child abuse

3 RCTs3 RCTs•Reduced recidivism 6-42 Reduced recidivism 6-42

months outmonths out

4 RCTs4 RCTs•more rapid improvementmore rapid improvement•decreased aggressiondecreased aggression•better post-discharge better post-discharge

outcomesoutcomes

6 RCTs6 RCTs•Long term improvements Long term improvements

in reducing child abusein reducing child abuse

3 RCTs3 RCTs•Reduced recidivism 6-42 Reduced recidivism 6-42

months outmonths out

Evidence-based Home & Community-based Services (cont’d)Evidence-based Home & Community-based Services (cont’d)Evidence-based Home & Community-based Services (cont’d)Evidence-based Home & Community-based Services (cont’d)

Parent Empowerment (fam ed)Parent Empowerment (fam ed)

(Heflinger & Bickman)(Heflinger & Bickman)

Mentoring (Vance)Mentoring (Vance)

Respite Services Respite Services

(Bruns & Burchard)(Bruns & Burchard)

Crisis ServicesCrisis Services

Parent Empowerment (fam ed)Parent Empowerment (fam ed)

(Heflinger & Bickman)(Heflinger & Bickman)

Mentoring (Vance)Mentoring (Vance)

Respite Services Respite Services

(Bruns & Burchard)(Bruns & Burchard)

Crisis ServicesCrisis Services

1 RCT1 RCT•increased knowledge and self-efficacyincreased knowledge and self-efficacy

1 RCT1 RCT•less substance use and aggressionless substance use and aggression•better school, peer, and family funcbetter school, peer, and family func

2 wait-list control studies2 wait-list control studies•fewer placementsfewer placements•reduced family stressreduced family stress

0 controlled, 1 pre-post0 controlled, 1 pre-post•placement prevented in 60-90% of placement prevented in 60-90% of

casescases

1 RCT1 RCT•increased knowledge and self-efficacyincreased knowledge and self-efficacy

1 RCT1 RCT•less substance use and aggressionless substance use and aggression•better school, peer, and family funcbetter school, peer, and family func

2 wait-list control studies2 wait-list control studies•fewer placementsfewer placements•reduced family stressreduced family stress

0 controlled, 1 pre-post0 controlled, 1 pre-post•placement prevented in 60-90% of placement prevented in 60-90% of

casescases

Evidence for Institutionally-Based CareEvidence for Institutionally-Based CareEvidence for Institutionally-Based CareEvidence for Institutionally-Based Care

HospitalHospital

ResidentialResidentialTreatmentTreatmentCenterCenter

Group HomeGroup Home

Partial Partial HospitalizationHospitalization

HospitalHospital

ResidentialResidentialTreatmentTreatmentCenterCenter

Group HomeGroup Home

Partial Partial HospitalizationHospitalization

3 RCTs 3 RCTs

•findings in favor of community comparison findings in favor of community comparison conditionsconditions

2 quasi-experimental 2 quasi-experimental

•Project Re-Ed: gains versus untreatedProject Re-Ed: gains versus untreated

•Gains in residential treatment center were equalGains in residential treatment center were equalto treatment foster care (TFC @ one-half cost)to treatment foster care (TFC @ one-half cost)

2 quasi-experimental 2 quasi-experimental

•mixed findings -- gains and mixed findings -- gains and deterioration (arrest rates)deterioration (arrest rates)

1 RCT1 RCT

•partial hospital versus wait-list controlspartial hospital versus wait-list controls

•benefits at 6 months for behavior symptoms,benefits at 6 months for behavior symptoms,and familyand family

3 RCTs 3 RCTs

•findings in favor of community comparison findings in favor of community comparison conditionsconditions

2 quasi-experimental 2 quasi-experimental

•Project Re-Ed: gains versus untreatedProject Re-Ed: gains versus untreated

•Gains in residential treatment center were equalGains in residential treatment center were equalto treatment foster care (TFC @ one-half cost)to treatment foster care (TFC @ one-half cost)

2 quasi-experimental 2 quasi-experimental

•mixed findings -- gains and mixed findings -- gains and deterioration (arrest rates)deterioration (arrest rates)

1 RCT1 RCT

•partial hospital versus wait-list controlspartial hospital versus wait-list controls

•benefits at 6 months for behavior symptoms,benefits at 6 months for behavior symptoms,and familyand family

Cost Benefit Analysis Washington Institute on Public Policy

CJS and Crime Costs Per Victim Benefit Per

Participant Dollar of Cost Early Childhood Programs

Perry Pre-School (P) $ 13,938.00 $ 1.50 Syracuse Family Development (P) 45,092.00 0.34 Olds Nurse Home Visitation (BP) 7,403.00 1.54

Middle Childhood Programs

Seattle Social Development Project (P) 3,017.00 1.79

Adolescent (Non-Offender) Programs Quantum Opportunities (BP) 18,292.00 0.13 Big Brothers/Big Sisters (BP) 1,009.00 2.12

Juvenile Offender Programs

Multi-Systemic Therapy (BP) 4,540.00 13.45 Functional Family Therapy (BP) 2,068.00 10.99 Aggression Replacement Training 404.00 31.40 Adolescent Diversion Project 1,509.00 13.61 Multidimensional Treatment 1,934.00 22.58

Foster Care (BP) Juvenile Intensive Supervision 1,500.00 1.49 Juvenile Boot Camp 1,964.00 0.26

Challenges to Putting EBPs into PracticeChallenges to Putting EBPs into Practice

No science to guide implementation Policies are way ahead of the knowledge

base Pressures to hold providers accountable for

outcomes Poorly trained workforce Incentives misaligned

5 Core Components for State 5 Core Components for State System ChangeSystem Change

Train practicing clinicians to deliver clinical EBPs with proven effectiveness

Engage families in services by removing barriers to access: Target clinician outreach

Empower families with tools, guidelines, and support: Target families and advocates

Support core organizational processes Create system-wide incentives to support change

New York State Implementation Model

System & Policy ContextFinancial policies, methods of reimbursement, state policies

Organizational ContextCulture

Climate

Structure

Clinical Care ImprovementTraining on EBP’s,

supervision, consultation and support

Engagement EmpowermentAttitudes, Beliefs &

Expectancies of Families & Youth

Improved Child & Family Outcomes

Attitudes, Beliefs & Expectancies of Clinicians and Supervisors

Improved Implementation Efficiency & Effectiveness

Implementation of Trauma-Implementation of Trauma-focused CBT for Children: The focused CBT for Children: The CATS StudyCATS Study

Developed in response to the World Trade Center Disaster

ID’s Generated1068

Assigned650

Declined 50

Ineligible165

CATS450

Conferenced-In51

Comparison149

Children287

Adolescents163

Children108

Adolescents41

Unassigned204

Children and Youth participating in the CATS research project

AcknowledgementsAcknowledgementsCATS Consortium

The CATS Consortium is a cooperative multi site treatment study performed by nine independent teams in collaboration with the New York State Office of Mental Health.  The New

York State collaborators are Kimberly Eaton Hoagwood, Ph.D., Chip Felton, M.S.W., Sheila Donahue, Ph.D., Anita Appel, M.S.W., James Rodriguez, Ph.D., (NYSPI), Laura Murray, Ph.D.,

(NYSPI), David Fernandez, M.A., (NYSPI), Joanna Legerski, B.S. (NYSPI), Michelle Chung, B.A., Jacob Gisis, B.S., Jennifer Sawaya, B.A., Sudha Mehta, M.P.H. (OMH), Jessica Mass

Levitt, Ph.D. (NYSPI).  The Principal investigators and co-investigators from the nine sites are Robert Abramovitz, M.D., (JBFCS),), Reese Abright, M.D., (St. Vincents Hospital), Peter

D’Amico, (North Shore/Long Island Jewish), Giussepe Constantino, Ph.D., (Lutheran Hospital), Carrie Epstein, C.S.W.-R., (Safe Horizon), Jennifer Havens, M.D., (Columbia

University), Sandra Kaplan, M.D., (North Shore/LIJ), Jeffrey Newcorn, M.D., (Mt. Sinai), Moises Perez, Ph.D., (Alianza Dominicana), Raul Silva, M.D., (NYU/Bellevue), Heike Thiel de

Bocanegra, Ph.D., (Safe Horizon),), Juliet Vogel, Ph.D. (North Shore/LIJ).The Scientific Advisors to the project are:  Leonard Bickman, Ph.D., (Vanderbilt University),

Peter S. Jensen, M.D., (Columbia University), Mary McKay, Ph.D., (Mount Sinai Medical School), Susan Essock, Ph.D., (Mount Sinai Medical School), Sue Marcus, Ph.D., (Mount Sinai Medical School), Wendy Silverman, Ph.D. (Florida International University), Robert Pynoos, M.D. (University of California, Los Angeles); Allan Steinberg, Ph.D. (University of California, Los Angeles); Lawrence Palinkas, Ph.D. (University of California at San Diego); and Joseph

Cappelleri, Ph.D., (Pfizer Corporation).  The Treatment Developers and Scientific Consultants to the project are:  Judy Cohen, M.D., (University of Pittsburgh), Anthony Mannarino, Ph.D., (University of Pittsburgh), Christopher Layne, Ph.D., (Brigham Young University), William

Saltzman, Ph.D., (UCLA). 

CATS DesignCATS Design

81 routine practice therapists working in schools and clinics trained in 2 EBP trauma models

Ongoing consultation/support provided by treatment developers

Children and families offered EBP trauma or treatment as usual

Baseline, 3, 6, 12 month follow-up Assessments of PTSD, other anxiety, depression, behavior

problems, strengths, school functioning Regression discontinuity design + propensity analyses to

assess predictors of outcome improvement

Child Trauma-focused TreatmentChild Trauma-focused Treatment Cohen, Judith; Mannarino, A, Deblinger, E. et al.

(2002) Child and Parent Trauma-Focused Cognitive Behavioral Therapy Treatment Manual. Trauma Focused Interventions:

Psychoeducation Stress inoculation/Relaxation Cognitive triangle Trauma Narrative/Gradual exposure Cognitive processing

Parallel parent treatment sessions

Adolescent TreatmentAdolescent Treatment Layne, Christopher M.; Saltzman, William R.;

Pynoos, Robert S.; (2002) Trauma/Grief-Focused Group Intervention for Adolescents.

Module I (6 sessions): Psychoeducation Coping Skills Cognitive work Communication skills

Module II (8-12 sessions): Trauma narrative/Exposure Cognitive restructuring

Module III (3 sessions): Resuming developmental progression Problem-solving

Intensive consultationIntensive consultation

81 clinicians trained on these models 3 day training + 2 booster sessions + bi-

weekly consultation calls for 1 year Bi-weekly site visits Weekly steering committee calls with PIs Weekly site coordination meetings

Engaging Families in Treatment Engaging Families in Treatment

• “Triple threat condition”: poverty, single parent status, and stress

• Rates of service use are lowest in low-income, urban communities. No show rates can be as high as 50% (Armbruster & Kazdin)

• Trained all teams on McKay’s engagement protocol (McKay & Bannon, 2005)

72%

49%

26%

9%0%

10%20%30%40%50%60%70%80%90%

100%

Number of Sessions

Pe

rce

nt

of

Yo

uth

Re

ma

inin

g in

Se

rvic

es

Treatment as Usual Show Rates McKay et al., 2005

Empirically supported Empirically supported engagement interventionsengagement interventions

• Reminders reduced missed appointments by 32% (Kourany et al., 1990; McLean et al., 1989; Shivack et al., 1989)

• Intensive family-focused telephone engagement associated with 50% decrease in initial show rates and a 24% decrease in premature terminations (Szapocznik, 1988; 1997)

• Combined telephone and first interview engagement interventions associated with attendance rates of 74%, representing a 16 to 25% increase above the clinic comparison families (McKay

et al., 1998).

First Interview ResultsFirst Interview Results

0

20

40

60

80

100

120

Accepted 1st Appt. 2nd Appt. 3rd Appt.

% for first interview(n=33)% for comparison(n=74)

Key elements of engagement Key elements of engagement trainingtraining

Help clinicians and intake staff examine their perceptions of barriers

Practice skills related to the initial face-to-face interview with a child and their family

Support clinicians and intake staff abilities to form collaborative working relationships with adult caregivers and youth

Help them identify an immediate and practical concern that can be addressed in the first interview

Learn skills related to the development of a shared commitment, language and understanding with the family

89

80 81

50

60

70

80

90

100

assessment treatment total

CATS Assessment vs. Treatment Show Rates

8993 93

100 95 95

7984

80

0

10

20

30

40

50

60

70

80

90

100

site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8 site 9

Assessment Rate by Site

83 84

76

95

68

91

67

83

76

0

10

20

30

40

50

60

70

80

90

100

site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8 site 9

Treatment Show Rate by Site

Treatment SettingsTreatment Settings

88

73

82

77

50

55

60

65

70

75

80

85

90

95

100

School Community OP Clinic Other (n=7)

1413 13 13

12

14

12

14

12

-1

1

3

5

7

9

11

13

15

site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8 site 9

Average Number of Treatment Sessions for Treatment Completers

Treatment SessionsTreatment Sessions

85

1015

25 2515

10

0

10

2030

40

50

60

70

8090

100

Any Tx 22+ 18 to 22 13 to 17 8 to 12 5 to 7 1 to 4

N=446

Implementation challenges: Implementation challenges: Matching EBPs to individual Matching EBPs to individual casescases

Strategy: Bruce Chorpita and State of Hawaii level of evidence

5 levels of evidence Detailed information about sample

demographics Practical and flexible menu of options

Autism

Conduct

Depression

Oppositional

Substance

None

None

CBT

Parent/Teacher Training

CBT

None

Multisystemic Therapy

CBT + parents; IPT; Relaxation

Anger Coping; Assertiveness;

PSST

Behavior Tx; Family Tx

ABA; FCT

None

None

None

None

Play Therapy; GIST

Juvenile Justice; Individual Tx

Family Tx; Individual Tx

Relaxation; Individual Tx

Individual Therapy

ADHDBehavior Therapy

None NoneBiofeedback;

Play Tx; GISTNone

None

Group Therapy

None

Group Therapy

Group Therapy

AnxietyCBT; Exposure;

ModelingCBT+ parents;

Ed supportNone

EMDR; Play Tx; GIST

None

ProblemLevel 1

best supportLevel 2

good supportLevel 3

some supportLevel 4

no supportLevel 5

known risks

Example: Chorpita (2002) EBT Analysis

Example: Clinical applicationExample: Clinical application

14 year old Depressed Puerto Rican Male Late in semester

Level 2

CBT + parents

Interpersonal

Relaxation

88%

85%

100%

MA; PhD

MA; PhD; MD

MA; PhD

clinic

clinic

school

CBT 94% MA; PhDClinic; school

1.74

1.40

1.51

1.48

Level 1

Intervention Finish

14 to 18

12 to 18

11 to 18

9 to 18

Age Staff Setting Effect

NS

49% PR; 41% HA;

10% C

NS

84% NS; 18%PR; 3%AA

Ethn

7 to 8 weeks

12 weeks

5 to 8 weeks

5 to 16 weeks

Length

Evidence:Interventions for Depression

Level 2

CBT + parents

Interpersonal

Relaxation

88%

85%

100%

MA; PhD

MA; PhD; MD

MA; PhD

clinic

clinic

school

CBT 94% MA; PhDClinic; school

1.74

1.40

1.51

1.48

Level 1

Intervention Finish

14 to 18

12 to 18

11 to 18

9 to 18

Age Staff Setting Effect

NS

49% PR; 41% HA;

10% C

NS

84% NS; 18%PR; 3%AA

Ethn

7 to 8 weeks

12 weeks

5 to 8 weeks

5 to 16 weeks

Length

Evidence:Interventions for Depression

Level 2

CBT + parents

Interpersonal

Relaxation

88%

85%

100%

MA; PhD

MA; PhD; MD

MA; PhD

clinic

clinic

school

CBT 94% MA; PhDClinic; school

1.74

1.40

1.51

1.48

Level 1

Intervention Finish

14 to 18

12 to 18

11 to 18

9 to 18

Age Staff Setting Effect

NS

49% PR; 41% HA;

10% C

NS

84% NS; 18%PR; 3%AA

Ethn

7 to 8 weeks

12 weeks

5 to 8 weeks

5 to 16 weeks

Length

Evidence:Interventions for Depression

ExampleExample

16 year old Female Anxiety problems Both parents available

Level 2

CBT + parents

Edu support

93%

85%

MA; PhD

N/A

clinic

clinic

CBT 95%UG; MA;

PhDClinic; school

1.05

1.78

N/A

Level 1

Intervention Finish

14 to 18

6 to 17

2 to 17

Age Staff Setting Effect

NS

92% C

54% NS; 33% C; 7% Arm;

6%AA

Ethn

12 weeks

12 weeks

3 to 16 weeks

Length

Evidence:Interventions for Anxiety

Level 2

CBT + parents

Edu support

93%

85%

MA; PhD

N/A

clinic

clinic

CBT 95%UG; MA;

PhDClinic; school

1.05

1.78

N/A

Level 1

Intervention Finish

7 to 12

6 to 17

2 to 17

Age Staff Setting Effect

NS

92% C

54% NS; 33% C; 7% Arm;

6%AA

Ethn

12 weeks

12 weeks

3 to 16 weeks

Length

Evidence:Interventions for Anxiety

Strategy: Distillation approach (Chorpita & Weisz, 2005; Dalaiden & Chorpita, 2005)

Cross tabulate studies with intervention elements

Use all studies; code each study Yields a matrix demonstrating protocol

overlaps

Implementation Challenges: Too Implementation Challenges: Too many models to choose amongmany models to choose among

0%20%40%60%80%100%

Directed PlayLimit Setting

Time OutCost Response

Educational SupportActivity Scheduling

MaintenceSkill Building

Social Skills TrainingTherapist Praise/Rew ards

Natural and Logical ConsequencesCommunication Skills

Assertiveness TrainingParent-monitoring

ModelingIgnoring or DRO

Parent PraiseProblem Solving

Parent copingPsychoed-Parents

RelaxationTangible Rew ards

Self-monitoringCognitive/CopingPsychoed-Child

Exposure

0% 20% 40% 60% 80% 100%

Internalizing Externalizing

Example (Chorpita et al., 2005)Example (Chorpita et al., 2005)

All

ExtInt

0%20%40%60%80%100%

Directed PlayLimit Setting

Time OutCost Response

Educational SupportActivity Scheduling

MaintenceSkill Building

Social Skills TrainingTherapist Praise/Rew ards

Natural and Logical ConsequencesCommunication Skills

Assertiveness TrainingParent-monitoring

ModelingIgnoring or DRO

Parent PraiseProblem Solving

Parent copingPsychoed-Parents

RelaxationTangible Rew ards

Self-monitoringCognitive/CopingPsychoed-Child

Exposure

0% 20% 40% 60% 80% 100%

Anxiety and Phobias(Chorpita et al)

Depression

All

DepA/P

ExtInt

Strategies for making EBP Strategies for making EBP implementation practical implementation practical

Integrated Psychotherapy Consortium (Center for the Advancement of Children’s Mental Health– P. Jensen, E. Goldman)

Michigan’s clinical outcome assessments (K. Hodges, J. Wotring)

Casey Blue Sky Project (P. McCarthy, S. Henggeler, S. Schoenwald, T. Sexton, P. Chamberlain)

Hawaii’s clinical decision-making system (B. Chorpita, E. Dalaiden)

New York State Implementation Model

System & Policy ContextFinancial policies, methods of reimbursement, state policies

Organizational ContextCulture

Climate

Structure

Clinical Care ImprovementTraining on EBP’s,

supervision, consultation and support

Engagement EmpowermentAttitudes, Beliefs &

Expectancies of Families & Youth

Improved Child & Family Outcomes

Attitudes, Beliefs & Expectancies of Clinicians and Supervisors

Improved Implementation Efficiency & Effectiveness

Family-based servicesFamily-based services

Hoagwood (2005) review of 4,000 articles since 1980 identified 41 rigorous studies of family-based services 3 categories:

Families as recipients of services Families as co-therapist Processes of involvement, engagement,

empowerment

Conclusions from family-based Conclusions from family-based services reviewservices review

Broader view of outcomes is needed Absence of robust literature on process

variables limits conclusions Evidence ambiguous as to whether these

services improve child outcomes Linkage of these services to EBP

implementation may be needed to amplify effects

Parent Empowerment in New YorkParent Empowerment in New York 4 year process: scientific review of the literature Identified one controlled trial of empowerment (Bickman et al,

1998) Adapted for parent advocates and for multi-ethnic families Added modules about EBPs for child mental health (ADHD,

depression, conduct, treatment efficacy) Collaborative partnership: Mental Health Assn., Columbia

University, NYS Office of Mental Health Added engagement strategies Developed 4 manuals for advocates and parents Conducting 2 NIMH-funded effectiveness trials to examine

impact of program on knowledge, skills, self-efficacy and use of services (behavior)

PEP Manual ContentPEP Manual ContentParent Advocate Manual

Introduction Getting Ready Building Engagement, Listening,

and Boundary Setting Skills Building Your Teaching and Group

Management Skills Developing Priority Setting Skills Specific Disorders and Their

Treatments The Mental Health System of Care:

What to Expect and How to Prepare

Services and Options Through the School System

Teaching Tools for Parent Advocates

Parent Handbook Introduction Knowing Yourself Knowing Your Child Treatment Management Skills:

How to be Your Child’s Case Manager

Specific Disorders and Their Treatments

The Mental Health System of Care: What to Expect and How to Prepare

Services and Options Through the School System

Helpful Tools for Parents

Parent Empowerment Parent Empowerment Research Study Basic Research Study Basic

DesignDesign40 Parent 40 Parent

Advocates/Family Advocates/Family Support Specialists Support Specialists

(PA/FSS)(PA/FSS)

40 Parent 40 Parent Advocates/Family Advocates/Family

Support Specialists Support Specialists (PA/FSS)(PA/FSS)

20 PA/FSS20 PA/FSSPEP TrainingPEP Training

20 PA/FSS20 PA/FSSPEP TrainingPEP Training

20 PA/FSS20 PA/FSSTraining as Training as

UsualUsual

20 PA/FSS20 PA/FSSTraining as Training as

UsualUsual

120 120 Parent/Caregivers Parent/Caregivers Receiving PA/FS Receiving PA/FS

Services Services 6 per PA/FSS6 per PA/FSS

120 120 Parent/Caregivers Parent/Caregivers Receiving PA/FS Receiving PA/FS

Services Services 6 per PA/FSS6 per PA/FSS

120 120 Parent/Caregivers Parent/Caregivers Receiving PA/FS Receiving PA/FS

Services Services 6 per PA/FSS6 per PA/FSS

120 120 Parent/Caregivers Parent/Caregivers Receiving PA/FS Receiving PA/FS

Services Services 6 per PA/FSS6 per PA/FSS

Post TrainingPost TrainingSelf EfficacySelf Efficacy

1

2

3

4

5

Total*

Pre

Post

N=31*=p< .05

5 = Greater Efficacy

1 = Less Efficacy

Post TrainingPost TrainingSelf-EfficacySelf-Efficacy

1

2

3

4

5

Pessimism

***

Effectiveness

Confidence*

Vicarious Lrng

Pre TrainingPost Training

N=31+ = p< .10* = p< .05*** = p<.001

5 = Greater Efficacy

1 = Less Efficacy

Embedding effective clinical Embedding effective clinical practices in settings and systemspractices in settings and systems

Family support services (e.g., engagement, empowerment) and effective clinical treatments are part of larger work environments

Studies of environmental contexts have identified characteristics that improve or interfere with service delivery

Key Constructs in Measurement of Key Constructs in Measurement of Organizational ContextsOrganizational Contexts

Organizational climate reflects perceptions of the work environment and has been linked with child outcomes in studies of child welfare agencies (Glisson & Himmelgarn, 1998)

Organizational culture refers to the ways things are done in a work environment—the norms and shared expectations

Organizational structure refers to the hierarchy of power

Organizational context affects Organizational context affects uptake of EBPs and outcomes uptake of EBPs and outcomes Organizational context affects Organizational context affects uptake of EBPs and outcomes uptake of EBPs and outcomes

Three decades of studies by Glisson and colleagues Glisson & Himmelgarn’s (1998) study of child welfare

agencies found that the strongest predictor of child improvement was organizational climate

Organizational culture, not climate, explained variations in service quality (Glisson & James, 2002)

Organizational level interventions can improve climate and reduce staff turnover (Glisson, in press)

Organizational factors affect youth outcomes (Schoenwald et al., 2003)

Glisson & Himmelgarn (1998) Parameter Estimates for Hypothesized Six-Variable Model

ServiceQuality

CountyDemographic

s

ServiceOutcomes(problem

levels)

-.13*

.12* -.05

-.24*

-.03

.02

-.36*

.01.06

-.20*

* p < .05

OrganizationalClimate

Interorganizational Services

Coordination

Interorganizational

Relationships

Organizational Context:Organizational Context:Implications for the Transport of Implications for the Transport of

Evidence-Based Treatments To Mental Evidence-Based Treatments To Mental Health Provider OrganizationsHealth Provider Organizations

Sonja K. Schoenwald, Ph.D.Sonja K. Schoenwald, Ph.D.Family Services Research CenterFamily Services Research CenterPsychiatry & Behavioral SciencesPsychiatry & Behavioral Sciences

Medical University of South CarolinaMedical University of South Carolina

Organizational structure & climate Organizational structure & climate (Schoenwald et al., 2003)(Schoenwald et al., 2003)

Multi-site study of 40+ community clinics delivering MST

Examine impact of organizational context on therapist adherence and outcomes

Organizational structure and climate factors were not associated with adherence scores

Organizational factors were associated directly with youth outcomes.

And some associations were in unexpected directions

Aims of Transportability StudyAims of Transportability Study

To examine: the association of MST adherence to outcomes in

field sites organization’s impact on adherence extra-organizational factors’ impact on

organizational factors affecting adherence the impact of clinician training & experience on

adherence a mediation model of effectiveness

Social Ecological Model of Social Ecological Model of TreatmentTransportabilityTreatmentTransportability

Extra-Organizational Context(Referral, Reimbursement, Disposition)

Organization Clinician Child

Adherence Outcomes

Clinician Variables Professional Training & Experience

Transportability of MST - Evidence of Multi-Level Transportability of MST - Evidence of Multi-Level Treatments*Treatments*

First 666 referred youth (juvenile justice, child welfare, and

mental health)

14.7 years old, 67% male, 61% Caucasian

57% one bio parent, 15% both bio parents

48% less than 20k/yr

*Schoenwald, Sheidow, Letourneau, & Liao (2003). Mental Health Services Research

Treatment OutcomesTreatment Outcomes

Significant pre-post reductions in child behavior problems and functioning

Discharge was based on achievement of treatment goals in 73% of cases

Discharge decisions were made by the therapist and family (versus external entity) in 64% of cases

Adherence-Outcomes LinkagesAdherence-Outcomes Linkages

Higher adherence predicted post-treatment decreases in child behavior problems and child functioning problems.

Higher adherence predicted positive discharge circumstances.

Therapist Effects Pre-Post Differences In CBCL Total Therapist Effects Pre-Post Differences In CBCL Total Scores by Adherence Scores by Adherence

56

58

60

62

64

66

68

Pre-treatment Post-treatment

Low Adherence High Adherence

Organizational Structure & Climate Organizational Structure & Climate FindingsFindings

Organizational structure and climate factors were not associated with adherence scores

Organizational Factors Predicted Organizational Factors Predicted Youth OutcomesYouth Outcomes

Organizational factors were associated with youth outcomes.

And, some associations were in unexpected directions.

Adherence & Organization: Adherence & Organization: Direct Effects on OutcomesDirect Effects on Outcomes

CBCL VFI DISCHARGE

EXTERNALIZING INTERNALIZING ENTITY REASON

THERAPIST ADHERENCE - - + +ORGANIZATIONAL CLIMATE

Energized & Effective -Opportunities for Advancement &

Reward+ + - -

ORGANIZATIONAL STRUCTURE

Participation in Decision Making +Hierarchy of Authority -

Moderation of Organizational Effects Moderation of Organizational Effects by Adherence Level by Adherence Level (1)(1)

Sample of adherence scores was split into upper and lower adherence quartiles

Advancement & Reward x low adherence = increased child problems

Advancement & Reward x high adherence = unrelated to child problems

Greater Procedural Specification x high adherence = increased child problems

Greater Procedural Specification x low adherence = unsuccessful discharge

Adherence Moderates Organization Adherence Moderates Organization Effects on OutcomesEffects on Outcomes

Opportunities for Advancement & Reward appear to matter little when adherence is high, but translates into poorer outcomes when adherence is low

Hierarchical Authority and Procedural Specification may interfere with positive outcomes when therapists are adhering to MST, but matters little when adherence is low.

ImplicationsImplications

Need to better understand criteria used in organizations for advancement and reward, and to consider including adherence and outcome indicators in those criteria

Need to better understand how organizational hierarchy and procedures may interfere with adherence to a specific EBP

Key Factors Associated with Adoption and Diffusion Key Factors Associated with Adoption and Diffusion (Greenhalgh et al, 2004): A Meta-Narrative Synthesis (Greenhalgh et al, 2004): A Meta-Narrative Synthesis of Evidenceof Evidence

Characteristics of the innovation Characteristics of the individual adopter Sources of communication and influence Structural and cultural characteristics of

potential organizational adopters Characteristics of the external environment Innovation uptake practices Linkage among components of the model

Organizational change is personalOrganizational change is personal Interpersonal influence through social networks is the

dominant mechanism for diffusion (Valente, 1996) Champion roles:

Organizational maverick Transformational leader Organizational buffer Network facilitator

Boundary spanners (social networkers) Organizations that promote boundary-spanning roles are

more likely to assimilate innovations (Barnsley, Lermieux-Charles & McKinnet, 1998; Ferlie et al., 2001)

Sustaining organizational change Sustaining organizational change (Gustafson et al., 2003; Rogers 1995; Plsek 2003; Champagne et (Gustafson et al., 2003; Rogers 1995; Plsek 2003; Champagne et al., 1991)al., 1991)

Tension for change: Staff want a change Innovation-system fit: Innovation fits norms and

values of organization Assessment of implications of innovation:

Implications are thought about in advance Support and advocacy—Existence of champions

and boundary spanners Dedicated time and resources Capacity to evaluate the innovation: Ability to

monitor and evaluate the impact of the innovation

Areas for further studyAreas for further study

What are the key factors that improve the uptake and sustainability of efficacious treatments?

What factors improve the fidelity of implementation efforts? What are the most effective outcome measures and suitable

methodologies for dissemination and implementation? How do different stakeholder perspectives about EBPs affect

organizational readiness to adopt new practices? What are the mediators and moderators of organizational effects? Can organizational context be changed to improve adoption of new

practices? What are effective interventions for changing organizational culture and climate?

How do family and consumer perspectives affect organizational readiness to adopt new practices?

Important Contributing FieldsImportant Contributing Fields

Social Marketing—packaging EBPs? Behavioral Change—why use an EBP? Anthropology—fit in different communities? Organizational behavior—can organizational

environments be changed? Finance/Economics—is there an economic

argument for EBPs? Technology development: EBPs are a kind of

technology—how to efficiently incorporate new technologies in new environments?

Culture

Structure

PsychologicalClimate

OrganizationalClimate

AttitudesSocial NormsSelf-Efficacy

Beliefs & Expectations

BehavioralIntention

Models of Diffusion, Organizational Implementation & Social Processes

Systems Context

OrganizationalProperties

Individual & SharedPerceptions

Behavior

Structural Determinants ofOrganizational Innovation

Social Determinants ofOrganizational Innovation

Adapted from Glisson 2002

Concluding thoughtsConcluding thoughts

“There is no practice without theory, however much that theory is suppressed, unformulated, or perceived as obvious.” Northrup Frye/Belsey

Closing thoughtClosing thought

“New technologies alter the structure of our interests: the things we think about. They alter the character of our symbols: the things we think with. And they alter the nature of community: the arena in which thoughts develop.”

Neil Postman, Technopoly

Dimensions of Organizational Dimensions of Organizational ReadinessReadiness

What factors are considered important to the uptake of evidence-based practices?

Do all stakeholders agree on the relative importance of specific factors?

Intervention Practitioner

Client Service Delivery

Organization

ServiceSystem

Nature of intervention

theory

Specialized training

Nature of referral problems

Frequency of sessions

Structure, hierarchy

Policies of referral source, pay

Focus of

intervention

Adherence monitoring

Family context

Length of sessions

Personnel policies

Financing methods

Intervention

specification

Manual?

Supervisor/

Researcher

Source of referral

Physical location of sessions

Organiza-

tional

culture

Legal mandate for

referrals

Similarity of int to std practice

Training of practitioner

Age and dev-

elopmental status

Source of payment

Organiza-tional climate

Interagency working relationship

Complexity of intervention

Endorsement of intervention

Gender Organiza-tional

mission

Clarity of intervention

Salary level/

Criteria for increases

Ethnicity/

cultural iden

Organiza-tional mandates

Schoenwald Schoenwald & & Hoagwood, Hoagwood, 20012001

Intervention Characteristics Theoretical foundation, strength of research support, clinical foundation, precision,availability of manual, specificity of manual, clarify of model Practitioner Characteristics Clinical adherence to model, frequency of clinical supervision, structure of clinical supervision, type of clinician, treatment orientation of clinician

Client CharacteristicsReferral problem(s), family context, client’s ethnicity/cultural identification

Service Delivery CharacteristicsReferral source, frequency of treatment sessions, length of treatment sessions, setting/location of treatment sessions, setting/location of the clinic or school

Service System CharacteristicsSalary incentives to adopt EBPs, policies and practices of referral sources, source of payments for the specific EBP, financing/payment mechanisms, legal mandates of referral sources, strength of interagency relationships

DOMAIN 1

DOMAIN 2

DOMAIN 3

DOMAIN 4

DOMAIN 5

DOMAIN 6

Key Readiness Factors

Service Agency CharacteristicsEndorsement by site leadership, structure of organization, size of organization, culture and climate of organization, policies and practices within the organization

Dimensions of Organizational Dimensions of Organizational Readiness (DOOR) Readiness (DOOR) Question: how important are the Question: how important are the following factors:following factors:

To Me Researchers Consumers Clinicians Clinic Directors

Support for the EBT by ….          

… clinical staff (e.g., therapists, social workers, psychologists, psychiatrists)          

… families or youth in the service setting          

… consumer advocacy          

… state mental health authorities          

… outside agencies          

Strength of the research supporting the EBT

Length of each treatment session required to deliver the EBT

Frequency of clinical supervision required to deliver the EBT (e.g., consultations between clinicians & supervisors)

Mean Ratings of Mean Ratings of Readiness FactorsReadiness Factors

22.5

33.5

44.5

55.5

66.5

7

All Respondents

Scale

Total Mean Ratings of Readiness FactorsTotal Mean Ratings of Readiness Factors

3.5

4

4.5

5

5.5

6

6.5

7

Clin

icia

ns

Con

sum

ers/

Fam

ilyA

dvoc

ates

Res

earc

her/

Tre

atm

ent

Dev

elop

ers

Sta

te/L

ocal

Pol

icym

aker

s

Clin

ic A

dmin

istr

ator

s

ALL

Intervention Practitioner Client Service Delivery Agency System

4

4.5

5

5.5

6

6.5

Int Pra Client Deliver Agency System

Clinicians

Consumers/Advocates

Researchers

State/LocalPolicymakersClinicAdministrators

Comparison of Respondent Group RatingsComparison of Respondent Group Ratings

Scale