Upload
snowy
View
28
Download
0
Embed Size (px)
DESCRIPTION
Continuing Development of Wraparound as an Evidence-Based Practice (Workshop A-11) Sixth California Wraparound Institute Tran M. Ly, Ph.D. Ming H. Lee, M.P.H. Christopher J. Jarosz, Ph.D. Department of Children and Family Services County of Los Angeles June 7, 2010. Workshop Outline. - PowerPoint PPT Presentation
Citation preview
DCFSResearch and Evaluation
Continuing Development of Wraparound as an Evidence-Based Practice
(Workshop A-11)
Sixth California Wraparound Institute
Tran M. Ly, Ph.D.Ming H. Lee, M.P.H.
Christopher J. Jarosz, Ph.D.
Department of Children and Family ServicesCounty of Los Angeles
June 7, 2010
2DCFS
Research and Evaluation
Workshop Outline
• Introduction• Overview of Wraparound’s evidence base• Los Angeles County’s placement and cost outcomes studies• Program evaluations• Logic model• Methods• Implementing an evaluation• Review and wrap-up• Resources
3DCFS
Research and Evaluation
Introduction
4DCFS
Research and Evaluation
Learning Objectives
• Be cognizant of the evidence base of Wraparound, including a recent example of a Wraparound evaluation in Los Angeles County.
• Learn what is a program evaluation, and why and when to conduct one.
• Recognize how the selection of study designs, measurement tools, and sampling strategies affect the types of questions asked and conclusions drawn in an evaluation.
• Practice building a basic logic model as an initial step in devel-oping an evaluation plan.
5DCFS
Research and Evaluation
The Presenters
• Tran Ly has graduate training in psychology, education, and child development. She previously planned and conducted several studies involving families of children with various disa-bilities.
• Ming Lee has training in biology, and graduate training in public health and biostatistics. He previously planned and conducted studies in perinatal health indicators and their influences on birth outcomes.
• Christopher Jarosz has graduate training in the behavioral and biological sciences. He has planned and conducted behavioral, community health, and organizational research in the public and private sectors.
6DCFS
Research and Evaluation
DCFS Research and Evaluation
• Conduct program evaluations and applied research for DCFS.• Provide technical consulting to other DCFS programs. • Plan and conduct departmental surveys.• Serve as the department’s liaison with outside researchers.
Website:http://dcfs.co.la.ca.us/rae/index.html
7DCFS
Research and Evaluation
Overview of Wraparound’s Evidence Base
8DCFS
Research and Evaluation
Residential Care
• “Residential care” and “group home care” are terms sometimes used interchangeably within the research literature, with no consensus on a single definition.
• Residential care is a term applied to a diverse array of services that are provided to youth living in such congregate care as apartments, emergency shelters, halfway homes, secure set-tings, and group homes.
9DCFS
Research and Evaluation
Residential Care (continued)
• Within residential care, placement instability is a common occurrence associated with various negative outcomes such as child behavior problems and juvenile delinquency.
• Residential care placements in California were approximately 11% of the foster care population in 2000, but more than 50% of foster care expenditures.
• Group homes provide continuous staff supervision, which can make them an expensive placement option.
• Group home providers are categorized into Rate Classification Levels (RCLs) based on the level of care and services provided, with RCL 14 being the most restrictive.
10DCFS
Research and Evaluation
Other Research Studies
• Bruns and Suter (2010) updated their summary of experimental and quasi-experimental research studies on outcomes of the Wraparound process.
• Nine controlled studies have been presented in peer-reviewed publications encompassing child welfare, juvenile justice, and mental health.
• Bruns and Suter note that each study has methodological weak-nesses.
• The studies considered in total indicate “superior outcomes for youth who receive wraparound compared to similar youth who receive some alternative service.”
Eric J. Bruns and Jesse C. Suter. Summary of the Wraparound Evidence Base: April 2010 Update (2010). Resource Guide to
Wraparound, Theory and Research, Chapter 3.5, National Wraparound Institute, 1-9.
11DCFS
Research and Evaluation
Positive Outcomes
• Fewer placement changes, and moves to less restrictive environ-ments.
• Older youth more likely to be in a permanency plan at the time of follow-up.
• Improved ability to externalize problems, especially with boys in the study.
• Overall decline in Child and Adolescent Functional Assessment Scale (CFAS) scores.
• Increased school attendance, reduction in school disciplinary actions, and increased grade point averages.
12DCFS
Research and Evaluation
Los Angeles County’s Placement and Cost Outcomes Studies
13DCFS
Research and Evaluation
Los Angeles County Wraparound Program
• Wraparound provides Los Angeles County (LAC) with an alterna-tive to residential care for foster youth consistent with a national review panel report (Cole, 1998).
• LAC implemented Wraparound in 1998 as a 10-child pilot study as an alternative to residential care.
• The goal of the pilot was to explore the impact of Wraparound on reducing reliance on out-of-home care.
• An analysis of the outcomes suggested that Wraparound youth were more likely to return and stay at home compared to youth in residential care.
14DCFS
Research and Evaluation
LAC Wraparound Program (continued)
• With closure of the County’s children’s shelter (McLaren) in 2003, Wraparound was identified as a primary mechanism for returning youth to their communities.
• Referrals were initially slow due to the lack of understanding of Wraparound.
• Another factor was the lack of research on its potential impact on youth, which at the time consisted of only a handful of controlled studies.
15DCFS
Research and Evaluation
An Initial Study
• A LAC study was conducted in 2004 to compare the outcomes of Wraparound graduates with youth who were discharged from Rate Classification Level (RCL 12 or 14) and went into less re-strictive placements.
• RCL 12 and 14 were selected since youth must qualify at these levels for enrollment in Wraparound since they must be in or at imminent risk of placement in a residentially-based institutional setting.
• The two groups were followed for 2-1/2 years using administra-tive data.
16DCFS
Research and Evaluation
An Initial Study (continued)
• The Wraparound graduates did substantially better than the youth who were discharged from RCL 12-14.
• The Wraparound graduates had fewer subsequent out-of-home placements and therefore less financial cost to the County of Los Angeles.
• A greater percentage of Wraparound graduates also exited foster care.
17DCFS
Research and Evaluation
Wraparound Growth
• Residential care in Los Angeles County has been reduced by over 60 percent since the 2004 study.
• About 2,400 foster youth were in residential care in 2004, while the number declined to fewer than 1,000 youth in 2009.
• Wraparound enrollment was less than 200 youth in 2003, which increased to over 1,200 youth by 2009.
• The number of Wraparound providers increased from eight to 34 in 2006.
• In 2009, the Wraparound program was expanded to accommo-date up to 4,200 youth.
18DCFS
Research and Evaluation
Further Study
• In 2009, we applied greater methodological rigor in attempting to replicate and expand upon the initial study.
• The areas we focused on including using entire populations in the study, providing a more detailed set of selection criteria for the cohorts, and use of a matched-comparison design.
• We found that the placement and cost outcomes of Wraparound versus RCL 12-14 were consistent with the findings of the initial study.
Michael Rauso, Tran M. Ly, Ming H. Lee, and Christopher J. Jarosz (2009), Improving outcomes for foster care youth with
complex emotional and behavioral needs: a comparison of outcomes for wraparound vs. residential care in Los Angeles
County. Emotional & Behavioral Disorders in Youth, 9, 53-76.
19DCFS
Research and Evaluation
Further Study (continued)
• This year, we established new one-year cohorts of Wraparound graduates and youth discharged from RCL 12-14 to lower-level placements.
• Our principal goal was to determine how consistent the results were with the previous studies.
• We compared the placement outcomes and associated costs of the two cohorts as defined in the selection criteria shown on the next slide.
• We hypothesized that Wraparound would continue to result in better outcomes (less restrictive placements) and lower costs to the system.
20DCFS
Research and Evaluation
Selection Criteria
Criteria Wraparound RCL 12-14
The case record is available in CWS/CMS X X
Graduated from Wraparound between 1 July 2007 and 30 June 2008
X
Discharged from RCL 12 or 14 to a lower placement level (< RCL 10) or home between 1 July 2007 and 30 June 2008
X
Had not previously been enrolled in the Wraparound program
X
Did not receive Wraparound services in the 12 months after discharge
X
Was in a Wraparound or RCL 12-14 placement for at least six months prior to graduation or discharge
X X
Was no older than 17 years, 0 months at the time of graduation or discharge
X X
21DCFS
Research and Evaluation
Outcome Measures
• Case closures—did the case close during the 12 months after Wraparound graduation or discharge from RCL 12-14 to a lower placement level?
• Placements—how many placements, including placement types, did the youth have during the 12-month period after graduation or discharge?
• Costs—based on the placement types and time durations, what were the system costs during the 12-month period?
22DCFS
Research and Evaluation
Demographics
Demographic Characteristic
Wraparound (N = 80)
RCL 12-14 (N = 98)
Age Ranges
5-to-11 years 16% 14%
12-to-17 years 83% 86%
Gender
Female 54% 59%
Male 46% 41%
Ethnicity
African American 23% 51%
Hispanic 51% 30%
White 15% 16%
Other 11% 3%
23DCFS
Research and Evaluation
Case Closures
16.9
58.8
0.0
50.0
100.0
Wraparound RCL 12-14
Per
cent
age
of C
hild
ren
Percentages of Cases that Closedwithin 12 Months
24DCFS
Research and Evaluation
Placement Outcomes
Figure 3. Average Number of Out-of-Home Placements
0.88
2.22
0
0.5
1
1.5
2
2.5
Wraparound RCL 12-14
Ave
rage
Num
ber
of P
lace
men
ts
Average Number ofOut-of-Home Placements
Figure 4. Average Number of Days inOut-of-Home Placements
202
308
0
100
200
300
Wraparound RCL 12-14
Ave
rage
Num
ber
of D
ays
Average Number of Days inOut-of-Home Placements
25DCFS
Research and Evaluation
Placement Outcomes (continued)
41.3
51.3
7.66.1
57.1
36.7
0
20
40
60
No Placement 1-2 Placements 3 or More Placements
Pe
rce
nta
ge
of C
hild
ren
Wraparound (N=80) RCL 12-14 (N=98)
Children Who Had None Versus at Least One Placement
26DCFS
Research and Evaluation
Figure 5. Distribution of Out-of-Home Placements
47.2
14.34.3
25.7
8.617.0
2.8
25.2
42.7
12.4
01020304050
Guardian/Relative Foster Family Court-Specif ied/ Small Family
FFA-Certif ied Group Home
Perc
enta
ge o
f Pl
acem
ent T
ypes
Wraparound RCL 12-14
Distribution of Out-of HomePlacement Types
Placement Distribution
27DCFS
Research and Evaluation
Cost Outcomes
Figure 6. Average Out-of-Home Placements Costs
$9,627*
$15,872*
$0
$10,000
$20,000
Wraparound RCL 12-14
Ave
rage
Cos
t per
Chi
ld
Average Out-of-Home PlacementCosts
Statistically-significant, p < 0.01
28DCFS
Research and Evaluation
Figure 7. Distribution of Out-of-Home Placements Costs
42.5 46.3
7.51.3
19.4
2.06.12.5
62.2
10.2
0
20
40
60
No Cost $1--$20,000 $20,001-$40,000 $40,001--$60,000 $60,001--$80,000
Placement Costs by Child
Perc
enta
ge o
f C
hild
ren
Wraparound
RCL 12-14
Distribution of Out-of-Home Placement Costs
Cost Distribution
29DCFS
Research and Evaluation
Summary
• The findings support previous cost impact analyses that Wrap-around is more cost-effective and has better outcomes compared to traditional residential care.
• Relative to RCL 12-14 children, Wraparound children are more likely to have their cases closed within 12 months of graduation.
30DCFS
Research and Evaluation
Summary (continued)
• Wraparound versus RCL 12-14 children in the 12-month period after graduation had:
– No or fewer out-of-home placements.– Placements, when they did occur, were often to less restric-
tive environments and require fewer number of days.– Financial costs associated with placements were significantly
less.
• Despite recent improvements in placement and cost outcomes for the RCL children, the outcomes of Wraparound graduates remain significantly better.
31DCFS
Research and Evaluation
Summary (continued)
• By many indications, Wraparound has provided an effective mechanism for reducing the Los Angeles County’s reliance on out-of-home care.
• Wraparound has a positive impact in that youth are living in less restrictive and more stable living environments.
32DCFS
Research and Evaluation
Policy and Practice Implications
• Wraparound provides an effective service delivery option for youth with behavioral and emotional problems.
• Wraparound, endorsed as a high-priority strategy in Los Angeles County, was recently expanded to accept youth who are not currently in or at imminent risk of placement into higher levels of residential care.
• A training model for service delivery is being developed for children’s social workers that shares many of the Wraparound values and principles.
• Wraparound principles and practices will soon influence the nature of residential care in Los Angeles County from long-term placements to planned, short-term, and individualized interven-tions.
33DCFS
Research and Evaluation
Limitations
• The study design is a retrospective study, and therefore youth were not randomly assigned to groups.
• We examined placement and cost outcomes for a period of one year.
• In using administrative data, we do not have information on the behavioral characteristics or functioning of the comparison youth.
34DCFS
Research and Evaluation
Future Research Directions
• We will be conducting a third, one-year cohort study to further assess the year-to-year stability of placement and cost outcomes for Wraparound and RCL 12-14.
• We are currently analyzing two-year cohort groups to determine longer-term outcomes for Wraparound and RCL 12-14.
• We are refining the selection criteria for equivalence of emotional, behavioral, and mental health difficulties of the Wraparound and RCL 12-14 groups.
35DCFS
Research and Evaluation
Program Evaluations
36DCFS
Research and Evaluation
What is a Program Evaluation?
An evaluation involves “carefully collecting information about a program or some aspect of a program… to make necessary
decisions about the program.”
“The type of evaluation… depends on what you want to learn about the program.”
“More recently… evaluation has focused on utility, relevance and practicality at least as much as scientific validity.”
(McNamara, 2006)
37DCFS
Research and Evaluation
Possible Goals
• Measure the impact of services on clients and other stakeholders.• Improve the effectiveness and reduce the costs of service delivery.• Verify if the program is operating to plan and if the plans need to be
changed.• Facilitate meta-thinking about program goals, and if they are being
met.• Promote decision-making based on data in responding to organiza-
tional and community needs.
38DCFS
Research and Evaluation
Possible Goals (continued)
• Compare programs to determine which best meet the intended needs.
• Establish a basis for duplicating or adapting programs for other venues and uses.
Adapted from:McNamara, Carter (2006). Field Guide to Nonprofit Program Design, Marketing and Evaluation (4th edition). 252 pages.
39DCFS
Research and Evaluation
Types of Program Evaluations
• Program goals• Processes and process fidelity• Short- and long-term outcomes• Client satisfaction• Time, quality, and financial costs • Combination of two or more types
40DCFS
Research and Evaluation
Stakeholders
• Who are the formal and informal decision-makers?• Who is likely to benefit from the evaluation?• Who can facilitate or possibly hinder the evaluation?• What resources can the stakeholders offer for the effort?• Is there a shared vision and commitment to mutual goals?
41DCFS
Research and Evaluation
Shared Vision?
42DCFS
Research and Evaluation
Group Exercise #1
Let’s say you are asked to conduct an evaluation of Wraparound service delivery in your community.
Identify the primary and secondary goals that you would want to achieve. What type of evaluation would you perform? Who
would be the three most influential stakeholders in supporting or benefiting from the evaluation?
43DCFS
Research and Evaluation
Logic Model
44DCFS
Research and Evaluation
What Is a Logic Model?
• A logic model is a useful tool for program development and evaluation and is usually part of an evaluation plan.
• A logic model:
– Describes the logical links between the program’s theories or assumptions, services, and outcomes desired.
– Illustrates a sequence of cause-and-effect relationships.
• While there many versions, a logic model is often a one-page graphical or schematic representation of what you do, why you do it, what you hope to achieve, and how you will measure that achievement.
45DCFS
Research and Evaluation
Sample Template
Program vision: Population served: Population needs to be addressed by services:
Outcomes Indicators Measurement Services* Resources
* Include service assumptions
46DCFS
Research and Evaluation
Building a Logic Model
• Establish your program vision and goals or objectives.• Describe your target population that will be served, including the
needs to be addressed.• Articulate the services and resources needed, including the
assumptions you are making.• Identify your program outcomes and indicators.• Determine how you will measure these outcomes and indica-
tors.
47DCFS
Research and Evaluation
Vision Statement
• A vision statement is a brief mission statement of what you hope for and want for the families and community to achieve in the future.
• In general, vision statements are very broad and ambitious goals—the objective for the program is to be able to contribute towards that vision.
• Vision statements are not necessarily measurable—example, all children are “ready” for school by age 6.
48DCFS
Research and Evaluation
Target Population and Needs
• In building a logic model, you will need to identify your target population.
• Describe its key characteristics which may include age, gender, ethnicity, educational status, economic status, and personal risk factors.
• Be as specific as possible in identifying the people who will or have received the services to ensure that intended outcomes are meaningful and appropriate.
• It is important to describe the population needs that your program intends to address—what will your services help them to do?
49DCFS
Research and Evaluation
Service Needs and Assumptions
• In continuing the development of the model, briefly describe the services that your families will receive.
• Describe why you think the services will lead to the desired outcomes.
– What assumptions are you making?– Are the services informed by evidence-based practice?
• When possible, use theory and research to guide the develop-ment of your logic model.
50DCFS
Research and Evaluation
Outcomes versus Indicators
• An outcome is a change that is likely to take place as a result of the target population’s participation in your program.
• Three types of outcomes are: long-term, intermediate, and short-term.
• Indicators tell you whether an outcome has been achieved.• Indicators are concrete specific descriptions of what you will
measure.• Also known as performance targets, indicators can specify a
targeted number or percentage of achievement.
51DCFS
Research and Evaluation
Examples of Outcomes and Indicators
• An example of an outcome is “participants will make their home child-safe.”
• Some examples of indicators for this outcome might include:
– “Participants access and install supplies needed to have a child-safe home.”
– “85% of participants remove household hazards that jeopardize child safety.”
– “Participants have a list of emergency numbers posted where all family members can find them.”
52DCFS
Research and Evaluation
Group Exercise #2
With this brief introduction to outcomes and indicators, are you able to distinguish between the two?
On your handout, please circle if you think the example is an outcome (O) or an indicator (I).
53DCFS
Research and Evaluation
Methods
54DCFS
Research and Evaluation
• Now, you are almost ready to implement your logic model.• How do you start?• The process of implementing your logic model should include:
– Selecting a study design.– Determining your available data collection options.– Analyzing your collected data and presenting them.
Next Steps
55DCFS
Research and Evaluation
• Of the nine Wraparound research studies described by Bruns and Suter (2010), two types of research designs were used: randomized designs and quasi-experimental designs.
• In both design types, participants are grouped into treatment and comparison groups.
• In a quasi-experimental design, differences observed may be influenced by dissimilarities already present between compari-son groups.
Selecting a Study Design
56DCFS
Research and Evaluation
Some Measurement Tools
• Questionnaires, surveys, and checklists• Structured and semi-structured interviews• Rating scales (e.g., CAFAS and CANS)• Administrative data• Direct observations• Focus groups• Documentation reviews• Case studies
57DCFS
Research and Evaluation
• Will you be studying the entire population or sample from the population?
• When including the entire population is too costly or impractical, a sample from the population may be appropriate.
• When sampling, the objective is to draw a subset of the popula-tion that is representative of the characteristics of the population.
Populations vs. Samples
58DCFS
Research and Evaluation
Sampling Methods
• Simple random sampling—everyone has an equal chance of being selected.
• Stratified sampling—used to ensure that enough participants with desired characteristics are selected.
• Nonrandom sampling—includes availability, purposive, quota, and snowball techniques.
59DCFS
Research and Evaluation
Implementing an Evaluation
60DCFS
Research and Evaluation
Implementation Plan
• Construct the logic model.• Select or develop measurement and data management tools.• Administer the evaluation tools.• Analyze and report on the results of the evaluation.
61DCFS
Research and Evaluation
Group Exercise #3
Let’s say you are asked to conduct an evaluation of Wraparound service in your community. The initial step in implementing such an evaluation would be to develop a
logic model.
Please break into small groups and work on building a basic logic model using the template provided.
62DCFS
Research and Evaluation
Review and Wrap-up
63DCFS
Research and Evaluation
Learning Objectives Accomplished?
• Be cognizant of the evidence base of Wraparound, including a recent example of a Wraparound evaluation in Los Angeles County.
• Learn what is a program evaluation, and why and when to conduct one.
• Recognize how the selection of study designs, measurement tools, and sampling strategies affect the types of questions asked and conclusions drawn in an evaluation.
• Practice building a basic logic model as an initial step devel-oping an evaluation plan.
64DCFS
Research and Evaluation
Conclusions
• We have given you a general overview of these topics:
– Program evaluation– Logic model– Methods– Evaluation plan
• There is enough information to teach a semester-long course for any of these topics.
• What we want you to take away from this workshop is that evaluating and planning a program is not a one-time or a one-step process.
65DCFS
Research and Evaluation
Resources
66DCFS
Research and Evaluation
Useful Web Links
Research Methods Knowledge Base
http://www.socialresearchmethods.net/kb/evaluation.php
Basic Guide to Program Evaluation
http://www.managementhelp.org/evaluatn/fnl_eval.htm
Logic Model and Toolkit
http://www.friendsnrc.org/outcome/toolkit/index.htm