Improving Practice through use of Improving Practice through use of Research and Evaluation: Research and Evaluation:

Sacramento County’s Experience Sacramento County’s Experience Measuring Effectiveness and Measuring Effectiveness and

Wraparound Fidelity Part II (CPS)Wraparound Fidelity Part II (CPS)Belle Darsie, Stanford Youth SolutionsBelle Darsie, Stanford Youth SolutionsGordon Richardson, EMQ Families FirstGordon Richardson, EMQ Families First

Jennifer Shebesta, Stanford Youth SolutionsJennifer Shebesta, Stanford Youth SolutionsJohn Woolcott, Sacramento Children’s HomeJohn Woolcott, Sacramento Children’s HomeKaren Vang, River Oak Center for ChildrenKaren Vang, River Oak Center for Children

Kristy Schwee, River Oak Center for ChildrenKristy Schwee, River Oak Center for ChildrenMaria Pagador, Mental HealthMaria Pagador, Mental Health

Mary Ann Wong, EMQ Families FirstMary Ann Wong, EMQ Families FirstRikke Addis, Sacramento Children’s HomeRikke Addis, Sacramento Children’s Home

Stephen Wallach, CPSStephen Wallach, CPSJune 14, 2012 at 1:30pmJune 14, 2012 at 1:30pm

OutlineOutline Part II CPSPart II CPS

– Desired OutcomesDesired Outcomes– Youth StoryYouth Story– How we measure fidelity and outcomesHow we measure fidelity and outcomes– How we achieve those outcomesHow we achieve those outcomes– Caregiver StoryCaregiver Story– Summary Summary

Ultimate Wraparound Ultimate Wraparound OutcomesOutcomes

Youth are…Youth are…At homeAt homeIn schoolIn schoolOut of troubleOut of trouble

Goals of WraparoundGoals of Wraparound 1. Community living situation

2. Improve quality of life

3. Improve functioning and resolve unmet needs

4. Maintain fidelity to Wraparound standards

5. Suspension and expulsion free or reoffend

free at discharge from the program

6. Attend school or work regularly

Specific Outcomes required by the CourtSpecific Outcomes required by the Court

What does Wraparound Success What does Wraparound Success Mean to CPSMean to CPS

SafetySafety Permanency/Lifelong Permanent Permanency/Lifelong Permanent

ConnectionConnection Placement StabilityPlacement Stability Well BeingWell Being AccountabilityAccountability

WraparoundWraparound

Spanish – Cubriendo TodoSpanish – Cubriendo Todo Italian – Andare a capoItalian – Andare a capo German – UmwickelGerman – Umwickel Filipino – NakapalibotFilipino – Nakapalibot Portuguese – WrapPortuguese – Wrap

WraparoundWraparound

A planning process that is strengths-A planning process that is strengths-based, family centered, and needs based, family centered, and needs drivendriven

Identified nationally as a promising Identified nationally as a promising practicepractice

Used in Sacramento County for Used in Sacramento County for youth in, or at imminent risk of youth in, or at imminent risk of placement in RCL 10-14 level of careplacement in RCL 10-14 level of care

Wraparound Providers in Wraparound Providers in Sacramento CountySacramento County

EMQ Families FirstEMQ Families First (since 1999) (since 1999) River Oak Center for Children (since 1999)River Oak Center for Children (since 1999) Stanford Youth SolutionsStanford Youth Solutions (since 2000) (since 2000) Sacramento Children’s Home (since 2003)Sacramento Children’s Home (since 2003)

Referral ProcessReferral Process

How CPS has been able to maintain a steady stream of How CPS has been able to maintain a steady stream of referrals to Wraparoundreferrals to Wraparound– What does not workWhat does not work– A single methodA single method– Making referrals mandatoryMaking referrals mandatory– What worksWhat works– Multiple and simultaneous strategiesMultiple and simultaneous strategies– SupportiveSupportive

– WRAP-a-ThonsWRAP-a-Thons– WRAP Walk AroundWRAP Walk Around– WRAP Resource FairsWRAP Resource Fairs– Celebrate successesCelebrate successes– Food!Food!

– DirectiveDirective– Communicate expectationsCommunicate expectations– Discuss, discuss, discussDiscuss, discuss, discuss– Use data; outcomes, Safe MeasuresUse data; outcomes, Safe Measures

– Resolve conflicts quickly and openlyResolve conflicts quickly and openly

Danny’s StoryDanny’s Story

ProfileProfile Prior to WRAP ReferralPrior to WRAP Referral Reasons for WRAP ReferralReasons for WRAP Referral WRAP ServicesWRAP Services Post WRAP OutcomePost WRAP Outcome

Demographics of Youth Referred by Sacramento County CPS (N=1053)

Median Age = 14.3 years

40% Black/African American, 38% White/Caucasian

Living Arrangement at Intake: 66% in Group Home RCL 12-14, 8% with Biological/Adoptive Parent(s)

Primary Diagnosis: 36% Attention-Deficit and Disruptive Behavior Disorders, 29% Anxiety Disorders (incl. PTSD), 24% Mood Disorders

Assessments/Measures/Assessments/Measures/Tools/InstrumentsTools/Instruments

Child and Adolescent Needs and Strengths Child and Adolescent Needs and Strengths (CANS)(CANS)

Wraparound Fidelity Index (WFI)Wraparound Fidelity Index (WFI) Team Observation Measure (TOM)Team Observation Measure (TOM) Child and Adolescent Functional Assessment Child and Adolescent Functional Assessment

Scale (CAFAS)Scale (CAFAS) Youth Outcome Questionnaire (Y-OQ) and Youth Outcome Questionnaire (Y-OQ) and

Self Report (Y-OQ-SR)Self Report (Y-OQ-SR) Post-Discharge Research InterviewPost-Discharge Research Interview

Engagement and AssessmentEngagement and Assessment

Strengths, needs, cultural discoveryStrengths, needs, cultural discovery Child & Adolescent Needs and Child & Adolescent Needs and

Strengths (CANS)Strengths (CANS) Countywide implementation as of Countywide implementation as of

January 2011January 2011

CANS DomainsCANS Domains

Life Domain FunctioningLife Domain Functioning Child StrengthsChild Strengths AcculturationAcculturation Caregiver Strengths and NeedsCaregiver Strengths and Needs Child Behavioral/Emotional NeedsChild Behavioral/Emotional Needs Child Risk BehaviorsChild Risk Behaviors Transition to AdulthoodTransition to Adulthood

The Promise of CANS: Engagement, The Promise of CANS: Engagement, Assessment and Outcome DataAssessment and Outcome Data

Comparing Mean Domain ScoresComparing Mean Domain Scores

Using the Reliable Change IndexUsing the Reliable Change Index

CANS – Comparing Mean Domain Scores

CANS – Reliable Change Index

Implementation Evaluation Implementation Evaluation

Wraparound Fidelity Index InterviewWraparound Fidelity Index Interview

Team Observation MeasureTeam Observation Measure

““Are we doing what we say we are doing?”Are we doing what we say we are doing?”Assessing Fidelity to the Wraparound ModelAssessing Fidelity to the Wraparound Model

Team Observation Team Observation Measure:Measure:

Observation Form Observation Form includes 20 items includes 20 items assessing fidelity in the assessing fidelity in the areas of: areas of:

Team BasedTeam Based CollaborativeCollaborative IndividualizedIndividualized Natural SupportsNatural Supports PersistencePersistence Cultural competenceCultural competence Outcomes-BasedOutcomes-Based Voice and ChoiceVoice and Choice Strengths-basedStrengths-based Community-basedCommunity-based

Wraparound Fidelity Index Interview:

4 Interview Forms:Facilitator Youth Caregiver Team Member

Structured, Open-ended Interview format

Average length is 20 minutes

In-Person or Phone Interview

Team Observation Measure Team Observation Measure

Examples of Assessment areas:

“Team Members demonstrate consistent willingness to compromise or explore further options when there is disagreement”

“The youth, caregiver, and family members are given time to talk about the family’s values, beliefs, and traditions”

“The youth prioritizes life domains, goals, or needs on which he or she would like the team to work”

Wraparound Fidelity Index Wraparound Fidelity Index

Facilitator Interview:

• ““Does the caregiver Does the caregiver feel comfortable feel comfortable expressing her or his expressing her or his opinions even if they opinions even if they are different from the are different from the rest of the team?”rest of the team?”

• ““Do the youth and Do the youth and family receive the family receive the supports and services supports and services stated in the plan?”stated in the plan?”

Youth Interview:

•“Do you feel comfortable expressing your opinions even if they are different from the rest of the team?”

•“Does it ever seem like someone on the team does not respect who you are and the things that you believe in?”

Caregiver Interview:

•“Did you take part in creating a written plan that identifies supports and services that meet your child’s needs at home, at school, and in the community?”

•“Does the team review your child’s progress toward specific goals at every team meeting?”

Examples of Interview Questions:

Outcome EvaluationOutcome Evaluation

PlacementPlacement CAFASCAFAS Y-OQY-OQ Y-OQ-SRY-OQ-SR

For Longitudinal DataFor Longitudinal Data Post-Discharge Research InterviewPost-Discharge Research Interview

Outcomes as Measured by CPSOutcomes as Measured by CPS

CPS Discharge Outcomes (N=929)CPS Discharge Outcomes (N=929)

Median duration of Wraparound = 14 monthsMedian duration of Wraparound = 14 months 58% Living in Community Placement58% Living in Community Placement 79% Maintained Placement Level or 79% Maintained Placement Level or

Transitioned to a Lower Level of CareTransitioned to a Lower Level of Care

Child and Adolescent Functional Child and Adolescent Functional Assessment Scale (CAFAS) SubscalesAssessment Scale (CAFAS) Subscales

School/Work Role PerformanceHome Role PerformanceCommunity Role PerformanceBehavior Toward OthersMoods/EmotionsSelf-Harmful BehaviorSubstance UseThinking

Outcomes Measured by CAFAS

Statistically significant improvement (p<.001)

Y-OQ and Y-OQ-SR SubscalesY-OQ and Y-OQ-SR Subscales

Intrapersonal Distress (ID)Somatic (S) Interpersonal Relations (IR)Social Problems (SP)Behavioral Dysfunction (BD)Critical Items (CI)Total Score

Y-OQ Data: CPS Youth

Mean Y-OQ Subscale Scores & Total Score, Matched Pairs Since FY 07/08 - Dec 2011 (N=66)

4 3

20

62

7

16

84

712

453

44

5

15

5 512

16

46

0

10

20

30

40

50

60

70

ID S IR SP BP CI Total Score

Intake Discharge Cutoff

Y-OQ-SR Data: CPS Youth

Mean Y-OQ-SR Subscale Scores & Total Score, Matched Pairs Since FY 07/08 - Dec 2011 (N=54)

20

6 6 5

138

58

16

5 4 4

115

45

17

63 3

116

47

0

10

20

30

40

50

60

70

ID S IR SP BP CI Total Score

Intake Discharge Cutoff

Post-Discharge SurveyPost-Discharge Survey

>Living situation >Living situation

>Current school placement>Current school placement

>How many times has the youth run >How many times has the youth run away in the past 6 months?away in the past 6 months?

CPS Post-Discharge Survey – Living Arrangement

28%

46%42%

46%

0%0%4%4%

13%

29%

17%21%

58%

13%

21%17%

0%4%4%4%

0%

8%

0%0% 0%0%

13%8%

0%

20%

40%

60%

80%

100%

Bio/Adopt Kincare Foster Care Group Home Juvenile Hall AWOL Other

Admit LA Discharge LA Current LA Predominant LA

CPS Post-Discharge Survey – Current School Placement

N=24 35% in Regular School22% in Alternative School26% in Special Education9% in Residential/DTP9% in Home-Based Program

100% of youth are in school

CPS Post-Discharge Survey – Run Away Information

N=2477% youth with 0 runaway9% youth with 1 runaway6% youth with 3 runaways6% youth with 5 runaways6% youth with 6 runaways

A Parent’s PerspectiveA Parent’s Perspective

◦◦

SUMMARYSUMMARY Belle: Research+program+partners=better systemsBelle: Research+program+partners=better systems Gordon: Transparent data-driven decision makingGordon: Transparent data-driven decision making Jennifer: Unwavering commitment to families and Jennifer: Unwavering commitment to families and

permanencypermanency John: Team-work is essentialJohn: Team-work is essential Karen: The proof is in the outcome dataKaren: The proof is in the outcome data Kristy: Collaboration and data collection are keys to Kristy: Collaboration and data collection are keys to

successful outcomessuccessful outcomes Maria: Promoting wellness of families by serving their Maria: Promoting wellness of families by serving their

needs through collaboration and outcome evaluationneeds through collaboration and outcome evaluation Mary Ann: Wraparound works!Mary Ann: Wraparound works! Rikke: Assessing fidelity, we do what we say we doRikke: Assessing fidelity, we do what we say we do Stephen: Wrap is a necessary service to achieve the Stephen: Wrap is a necessary service to achieve the

outcomes we needoutcomes we need Yolanda: Open arms and meeting families and permanencyYolanda: Open arms and meeting families and permanency

WraparoundWraparound

Q & AQ & A

Thank you!Thank you!

ReferencesReferences Hodges, K. (2000) Child and Adolescent Functional Assessment Scale. Hodges, K. (2000) Child and Adolescent Functional Assessment Scale.

Functional Assessment Systems, L.L.C. Ann Arbor , Michigan.Functional Assessment Systems, L.L.C. Ann Arbor , Michigan.

Wells, G.M., Burlingame, G.M. and Lambert, M.J. (2005) OQ Measures, Wells, G.M., Burlingame, G.M. and Lambert, M.J. (2005) OQ Measures, L.L.C. Salt Lake City, UT.L.L.C. Salt Lake City, UT.

Lyons, J. (1999) Child and Adolescent Needs and Strengths Tool, The Lyons, J. (1999) Child and Adolescent Needs and Strengths Tool, The Praed Foundation, Chicago, IL.Praed Foundation, Chicago, IL.

Bruns, E. J. and Sather, A. (2007) Team Observation Measure. Bruns, E. J. and Sather, A. (2007) Team Observation Measure. Wraparound Evaluation and Research Team, Seattle, W.A. Wraparound Evaluation and Research Team, Seattle, W.A.

Bruns, E. J. and Sather, A. (2007) Wraparound Fidelity Index - 4.0. Bruns, E. J. and Sather, A. (2007) Wraparound Fidelity Index - 4.0. Wraparound Evaluation and Research Team, Seattle, W.A. Wraparound Evaluation and Research Team, Seattle, W.A.