Selecting and Implementing

Evidenced Based Practices in

Child Welfare

Phoenix, AZ July 18, 2014 Dee Wilson

2

The modern child welfare system in this country is 45-50 years old, and yet

evidenced based practices continue to be widely viewed as innovations.

How were treatment programs and other interventions selected and

evaluated prior to the evidenced based practice movement?

3

There were distinct periods in the past 45-50 years when the federal government made big

investments in research on treatment outcomes for abusing and neglecting parents, for example the

mid to late 1970s and early 1980s.

Anne Cohn and Deborah Daro summarized findings from evaluations of 90 federally funded studies of treatment programs for abusing and

neglecting parents conducted by Berkeley Planning Associates. Major findings from these

studies remain relevant to this day.

4

Major Findings from Early Studies of Treatment Outcomes

• Treatment programs were effective for no more than half of maltreating parents.

• There was a high rate of recurrence (30% plus) of child maltreatment during treatment; but recurrence of maltreatment was not necessarily a good prognostic indicator.

• Treatment programs were less effective for neglecting parents than abusive parents.

5

These early federally funded studies did not use experimental / quasi-

experimental designs. Nevertheless, they tracked changes in parental behaviors, beliefs and attitudes

understood to be associated with child abuse and neglect.

6

In the 1990s, the federal government made a large investment in family preservation programs, most of which were initially

evaluated without use of randomized controlled trials (RCTs) or quasi- experimental designs.

Initially, these programs appeared to have impressive effects on placement prevention for children at imminent risk of placement. Findings from RCTs led to a reconsideration of the use

of these programs.

7

Is this a well-founded claim?

A child welfare agency makes a public commitment to reducing the number of children in foster care and

the director of the agency visits local offices to discuss the benefits of a smaller foster care system. At the same time, the agency implements statewide

a differential response (DR) system. Entries into care decline by 20% during the next year. At the end of the fiscal year, the agency sends a report to the

legislature stating that the agency’s DR initiative has resulted in a large reduction of entries-into-care.

8

Figuring out what works

As a rule, in evaluating strategies to change human behavior or improve an individual’s or group’s health or well-being, it’s often

easier to figure out what hasn’t worked than to understand the actual causes of change.

Causation is often very difficult to figure out. Why?

9

Premature conclusions

There have been some famous instances of premature conclusions

that expensive programs were achieving desired outcomes

10

Is the agency’s assertion about cost savings well-founded?

A state implements a family preservation services (FPS) program in half a dozen counties that costs more than $4,000 per family. Caseworkers are instructed to refer families for FPS only if children are at imminent risk of

placement, and supervisors must sign off on referrals. At the end of 9 months, an evaluation of the program finds

that 78% of target children remained in the home and 15% of target children were placed with relatives, most of whom were not licensed foster parents. The state agency sends a

report to the legislature asserting that the program has generated huge cost savings as each foster care

placement has been found to cost (on average) almost $30,000.

11

Looking for Evidence-based Programs

When groups viewed as authorities on EBPs select programs as evidence-based or give

programs ratings based on strength of evidence, they tend to look for randomized controlled trials (RCTs), or at least quasi-

experimental studies with control groups to designate programs as EBPs.

12

Varying Rates of Achieving Outcomes

• The description of a program as “evidence-based” usually means that the program was more effective than business as usual, or doing nothing, or a specific program with which it was compared in a study.

• EBPs can (and do) have highly varying rates of achieving outcomes for which they were designated.

• A careful review of evidence regarding an EBP should determine the rate of persons referred to the program who achieved a desired outcome.

13

EBPs and Effect Size

• Scholars tend to describe the effectiveness of an EBP in terms of its effect size, i.e., the difference between the experimental group and the control group in achieving a desired outcome.

• A number of EBPs of interest in child welfare

had large effect sizes when tested in university settings but small to moderate effect sizes when implemented in agency settings.

14

Credibility of Research

• Research on an EBP is more credible when some of the studies have been conducted by non-developers.

• The Washington State Institute of Public

Policy (WSIPP) cuts the estimated effect sizes of programs in half if the research study was conducted by a developer.

15

There are 4 main categories of EBPs important for child welfare systems

1. Programs that have been demonstrated to reduce one or more types of child maltreatment by changing parents' behaviors, beliefs and/ or attitudes

2. Programs that have been demonstrated to reduce conditions or mental health symptoms believed to have a causal effect on child abuse and/ or neglect , for example substance abuse, depression, domestic violence

3. Programs which reduce abused and neglected children's mental health problems and/ or improve their well being, for example by developing social skills, enhancing academic performance, helping traumatized children to control their emotions and calm down when under stress.

4. Programs that improve the behavioral health of other troubled but non- maltreated children, for example conduct disordered children and youth or delinquent youth

16

The list of programs that have demonstrated effectiveness in directly reducing

maltreatment rates is not lengthy; and some of the programs commonly on such lists are prevention programs which either have not been tested with child welfare populations, or which have not produced desired results

with parents who had open child welfare cases.

17

Even with programs that have demonstrated effectiveness in reducing at least one type of

child maltreatment, there is still plenty to learn about the best use of these programs. For example, PCIT is evidenced based for the most common type of physical abuse, excessive discipline. However, it is highly

questionable whether PCIT is a reasonable intervention for battered child syndrome,

torture, serial battering, etc.

18

It is an important question how evidenced based parenting practices should be

combined or not combined with substance abuse and/ or mental health treatment

programs, or how to sequence substance abuse treatment and parenting education

programs.

19

Some studies in recent years have found that better parenting skills can lead to

reduced depression, or reduced cravings to use drugs and alcohol as well as the other

way around. The child welfare field and courts continue to need research based

guidelines around sequencing of interventions for multi-problem families.

20

In reviewing, selecting and implementing evidenced based

practices, it is important to follow Eileen Gambrill's dictum:

"WE SHOULD SEEK TO BE QUESTIONERS RATHER THAN

BELIEVERS."

21

YES-BUTS re Evidenced Based Practices

22

Mental Health Programs

• Most mental health programs on EBP lists are evidence-based in reference to a relatively narrow range of symptoms.

• When used to address mental health conditions, or other conditions, for which they were not designed they are not evidence-based.

• Accurate diagnosis and targeting is critical for effective use of EBPs. Child welfare staff need to be trained in how to utilize EBP options for parents and children who do not fit neat diagnostic categories.

23

Engagement Strategies

EBPs often have high drop out rates; so that developers and implementers

have begun to experiment with engagement strategies for increasing

participation in and completion of programs.

24

Some of the most interesting - and promising -- programs in the country are substance abuse programs designed for parents with open child welfare cases, Project START in Kentucky, the PCAP program in Washington and Project On-Track in Oregon. These programs have

found ways to provide ongoing support to parents in recovery, with an emphasis on

relapse planning and child safety.

25

EBP’s and Outcomes

Some EBPs in prevention and early childhood education provide extended

contact between nurses or other professionals and parents for years; and have a wide range of positive

outcomes on child and family functioning that extend for 15-20 years

(at least).

26

Investing in EBPs…

• Because EBPs in mental health target a narrow range of symptoms, an agency must invest in several programs to be able to provide EBPs for the normal range of mental health problems.

• EBPs are often expensive and take a large amount of time and effort to implement; often, agencies can afford only one or two EBPs at a time.

27

Common Elements of EBPs

Because of the cost of EBPs and the difficulties of simultaneously

implementing several EBPs, there has been an increased interest in the

common elements of effective programs.

28

Most evidenced based programs that have demonstrated a direct effect on child maltreatment

rates or on mental health conditions associated with child maltreatment have used various forms of

cognitive behavioral treatment. These programs seek to influence parents' beliefs and attitudes about children, as well as parenting

behavior. Concretely, this means that effective programs influence parental expectations of

children, attributions of motives and understanding of children's needs, not just their disciplinary

methods or other parenting skills.

29

EBPs use a wide range of methods to achieve common goals. Effective

parenting programs teach and coach parents to greatly increase praise and

other rewards and reduce use of punishment. Changing the ratio of

praise to punishment is a common goal of these parenting programs.

30

Implementation Science Source: Lawrence Green (2009)

http://www.insa.pt/sites/INSA/Portugues/ComInf/Noticias/Documents/Semin%C3%A1rios/Lawrence_Green16012009.pdf

31

Child welfare agencies often underestimate what it takes to competently implement

EBPs and other programs. In particular, too much is expected of one time training

programs. Agencies may underestimate the amount/ extent of communication needed to ensure that eligible children and families are

referred by child welfare practitioners to available EBPs by several orders of

magnitude.

32

EBPs tend to have strict requirements described in manuals. Ensuring a high degree of adherence

to these requirements by contractors is an ongoing challenge for QA systems. At the same

time, local agencies will often have good reasons for wanting to adapt programs to local conditions and/ adapt program requirements to the different

cultural practices of various ethnic groups. Agencies need to have dedicated EBP

experts who can respond to these types of conflicting tensions.

33

• There needs to be periodic conscientious reviews of program fidelity, and ongoing discussions with contractors regarding implementation issues.

• Child welfare staff and other referral sources must be trained- and retrained- in the best use of specific EBPs.

• Agencies will often need to develop engagement strategies to increase enrollment and participation in EBPs

• Drop- out rates need to be carefully tracked

Implementing Evidenced Based Programs

34

There is no guarantee that EBPs will deliver the same results as reported in

clinical trials. There is a continuing need for evaluation of outcomes.

Evaluating EBPs and Promising Programs

35

State child welfare systems need a way to test use of EBPs and promising programs in

a few offices and or communities. A few offices can be designated as R&D sites and given the resources for program evaluation

Research and Development Sites

36

R&D can bring together researchers, educators and practitioners. R&D offices can play an important role in staff development and in the education of BSWs and MSWs,

as well as create a capacity to evaluate programs.

The Potential for R&D in Child Welfare