Evaluation of Ohio’s Title IV-E Waiver Demonstration

Project “ProtectOhio”

Summary of Findingsfrom the 5-year study

September 2003

HSRI, Westat, Chapin Hall, IHSM

Green = demonstration PCSAYellow = comparison PCSA

Ohio’s Evaluation Counties

LOGIC MODEL

WaiverSystemsReform

Interventions

ImprovedOutcomes

for children

& families•Any service

•Any clients

•$ up front

•Keep savings

•Internal organization

•Services available

•Financing patterns

•MC strategies

What’s done for what families and children

•How they go through the system

•Services received

•Use of relatives

placement days

permanency

functioning

well being

Demonstration Counties’ Response to the Waiver

Compared to their own pre-Waiver actions and comparison counties’ concurrent actions, demonstration sites made these changes:– Greater targeted focus on prevention– More attention to outcomes– Overall greater use of managed care strategies– More interagency collaboration– Slower growth in foster care spending– More growth in “other child welfare” expenditures

Demonstration sites focused more on prevention…

• Demonstrations and comparisons had similar levels of insufficiency in array of available services; moderate differences in area of prevention

• Demonstrations targeted new prevention activities to areas identified as insufficient

• Demonstration sites expressed stronger commitment to prevention

Figure 3.3: Insufficiency of Prevention Services

0%

25%

50%

75%

100%

Familypreservation

Parenting ed. Mentoring/hmmgt.

In-homecounseling

I&R, familysupport

School-based

Demonstration

Comparison

Table 3.3: Number of Service Insufficiencies in Year 3 and Creation of New Service in Year 4

Demonstration Comparison

Y3 number of counties having insufficient prevention service

12 9

Y3 count of all insufficient prevention services across those counties

37 28

Y4 number of new prevention services created in the area of insufficiency

20 10

New services created as a percent of insufficient services

20 out of 37 = 54%

10 out of 28 = 36%

Number of counties who targeted services (targeting = county created

services for at least half of their reported service insufficiencies)

9 4

Table 3.4: Prevention Groups*

New Spending onnon-foster care

Strong Theme No Strong Theme

Demos Comps Demos Comps

Above the median

Group One Group Three

6 1 3 2

Median or below

Group Two Group Four

3 2 1 8

*One demonstration and one comparison counties had incomplete fiscal data so they could not be categorized in this table.

Demonstration sites gave more attention to outcomes…

• Gathering more outcomes data:

10 demo and 6 comp reported systematically

gathering outcome information from staff (Y4)

• Using outcomes data in decision-making (Y3)

• Integrating fiscal & program outcomes information

(Y4)

Figure 2.25: Types of Outcome-Focused Activities Y3

0% 25% 50% 75% 100%

Use of outcome-based internal management reports

Planful use of outcomes data to modify practice

Systematic measurement of client outcomes

Outcome-based performance criteria in contracts

Comparison

Demonstration

Figure 2.6: Use of Fiscal and Program Data (Y4)

0

2

4

6

8

10

12

14

Program Managers Use

Fiscal Data

Fiscal Staff Use Program

Data

Demonstration

(n=14)

Comparison

(n=14)

Ohio’s Managed Care Strategies

• Service array (care criteria): new services, shifting focus

• Financing (capitation & risk): capitated & case rate contracts,

use of IV-E

• Targeting (eligibility): special initiatives, specialized units

• Case management: unit structure, transfers

• Provider competition: provider affiliations, rate changes

• Utilization review: placement reviews, service reviews

• Information management: automated MIS, mgrs use info

• Quality assurance: control & enhancement, outcome focus

Demonstration sites employ more managed care strategies

• In Year 2, significantly more use of managed care strategies by demos

• In Year 4, demos made more use of 7 of 8 managed care strategies

• Overall rankings: more demos in the “high” group, fewer in the “low” group

Table 2.6 Tukey’s Quick Test for Differences in Managed Care Index Score (Year 2)

CountyDemonstration/

Comparison Y2 Score CountyDemonstration/

Comparison Y2 Score

Lorain Demonstration 47.69 Fairfield Demonstration 25.78

Hamilton Demonstration 46.09 Stark Demonstration 25.52

Medina Demonstration 38.68 Butler Comparison 25.45

Richland Demonstration 36.91 Hancock Comparison 24.77

Portage Demonstration 35.56 Hocking Comparison 24.77

Franklin Demonstration 33.60 Belmont Demonstration 23.53

Greene Demonstration 31.66 Allen Comparison 20.64

Crawford Demonstration 29.56 Ashtabula Demonstration 20.20

Trumbull Comparison 29.48 Warren Comparison 19.75

Clark Demonstration 28.51 Mahoning Comparison 17.66

Muskingum Demonstration 28.45 Columbiana Comparison 14.59

Montgomery Comparison 28.20 Miami Comparison 13.62

Summit Comparison 28.04 Wood Comparison 9.02

Scioto Comparison 26.69 Clermont Comparison 8.95

Table 2.4: Y4 Managed Care Score as a % of Total Possible Score

Managed Care Category

Demonstration County Average

Comparison County Average

Financing 38% 18%

Utilization Review 58% 52%

Service array 45% 52%

Case Management 49% 46%

Competition 42% 29%

Targeting 40% 30%

Quality Assurance 55% 46%

Data Management 48% 37%

TOTAL 47% 39%

Counties Grouped by Level of Managed Care Activity, Year 4

Counties with High Managed Care Activity

Counties with Moderate Managed Care Activity

Counties with Low Managed Care Activity

Demo: Comp: Demo: Comp: Demo: Comp:

Franklin GreeneHamiltonLorainMedinaRichland

MontgomeryTrumbull

BelmontClark MuskingumPortageStark

Allen ButlerMiamiSciotoSummit

AshtabulaCrawfordFairfield

ClermontColumbiana HancockHockingMahoningWarrenWood

Demonstration sites have stronger collaboration…

• Overall, PCSA partners say collaboration with PCSA has become stronger

BUT• Demo and comparison PCSAs have

similarly strong relationships with juvenile court, mental health board

Table 3.19: Characterizing the Relationship between Survey Respondents and the PCSA*

Relationship Strength Demonstration Comparison

Average Strength of Relationship (scale: -2 to 2)

1.40 1.11

Rating the Strength of Relationship

Very strong/strong strong

*Difference is significant at 0.058

Table 3.20: Characterizing the Change in the Relationship between Survey Respondents and the PCSA since 1997*

Change in Relationship Strength

Demonstration Comparison

Average Change in the Strength of Relationship (scale: -1 to 1)

0.89 0.61

Rating Average Change in the Strength of Relationship

Grown stronger Stayed the same/grown

stronger

*Difference is significant at 0.000

Relationship between PCSA and Main Partners (Year 4)

PCSA relationship with Juvenile Court:PCSA relationship with Juvenile Court:

Majority of demo & comp counties have strong relationship (9D, 7C)

Inappropriate referrals continue to be issue in some demo and comp

counties

PCSA relationship with Mental Health:PCSA relationship with Mental Health:

– Relationships remain strong in most counties, no significant change over

Waiver period (9D, 7C)

– Inadequate mental health services for children and families, so PCSAs

(both demo and comp) develop and purchase own mental health services

However, demonstration sites are no different from comparisons in:

• Family involvement in case decisions – slow growth in family role and team conferencing;

• Use of managed care contracting – 3 demo;

• Increased competition in foster care – growth in foster homes and per diem rates;

• Utilization review and QA – moderate use of formal review processes.

Utilization Review (Year 4)

• Placement review processes:– Pre-placement – 9D, 8C– During placement – 8D, 5C

• Oversight of non-placement resources:– Pre-service review – 4D, 4C– Subsequent review – 2D, 3C

Changes in Patterns of Child Welfare Spending Over Time

• Both groups of counties experienced growth in paid placement days and in the average daily cost of foster care;

BUT • Demonstration sites appeared to contain growth

in foster care spending more than comparison sites;

• Demonstration sites appeared to increase other child welfare spending more than comparison sites.

Change in Placement Day Utilization

Evaluation Group County

Growth Baseline-2002

Evaluation Group County

Growth Baseline-2002

Comparison Wood (38%) Demonstration Medina 6%

Demonstration Belmont (34%) Comparison Scioto 6%

Comparison Mahoning (29%) Demonstration Greene 17%

Demonstration Lorain (26%) Comparison Clermont 17%

Comparison Butler (25%) Comparison Montgomery 17%

Comparison Trumbull (22%) Comparison Miami 22%

Demonstration Muskingum (21%) Comparison Hocking 28%

Demonstration Clark (20%) Demonstration Crawford 32%

Demonstration Ashtabula (15%) Comparison Summit 32%

Comparison Allen (9%) Comparison Columbiana 44%

Demonstration Portage (5%) Demonstration Franklin 46%

Demonstration Hamilton 2% Demonstration Fairfield 48%

Demonstration Richland 2% Comparison Warren 53%

Demonstration Stark 5% Comparison Hancock 97%

Changes in Foster Care Spending

EvaluationGroup County

Growth Baseline-2002

EvaluationGroup County

GrowthBaseline-2002

Comparison Wood (24%) Demonstration Fairfield 47%

Comparison Butler (1%) Comparison Mahoning 53%

Demonstration Portage 2% Comparison Scioto 59%

Comparison Allen 13% Demonstration Stark 59%

Comparison Trumbull 17% Comparison Summit 61%

Demonstration Richland 18% Comparison Montgomery 67%

Demonstration Clark 20% Comparison Warren 80%

Demonstration Hamilton 26% Comparison Miami 92%

Demonstration Greene 33% Demonstration Franklin 103%

Demonstration Muskingum 40% Comparison Columbiana 131%

Demonstration Belmont 43% Comparison Hocking 152%

Demonstration Ashtabula 43%

Additional Waiver Revenue

Most demonstration counties received more revenue through the Waiver than they would have received through normal Title IV-E reimbursement for foster care board and maintenance expenses.

Additional Revenue Received ($1000s)

Demonstration

Estimated Expenditures

Eligible for Title IV-E Foster Care B&M

Reimbursement 1998-2002

ProtectOhio Waiver Award

1998-2002

Total ProtectOhio

Revenue Available for Reinvestment

1998-2002

ProtectOhio Revenues as a

Percent of Total Expenditures

Ashtabula $2,504 $4,290 $1,786 7%

Belmont $2,627 $3,161 $534 3%

Clark $9,753 $12,743 $2,990 6%

Franklin $82,084 $95,740 $13,656 2%

Greene $4,900 $5,038 $138 1%

Hamilton $53,246 $76,399 $23,153 7%

Lorain $8,600 $12,061 $3,461 6%

Medina $1,726 $1,712 ($14) (0%)

Muskingum $5,594 $6,219 $625 3%

Portage $5,158 $7,883 $2,725 10%

Richland $6,345 $6,751 $406 1%

Stark $24,937 $26,212 $1,275 1%

Additional Revenue Purchased

• All but one of the demonstration counties spent its additional Waiver revenue on child welfare services other than board and maintenance payments.

• As a result, spending on all other child welfare services increased significantly more among the group of demonstration counties than among the group of comparison counties.

All Other Child Welfare Expenditures

EvaluationGroup County

All OtherCWServicesCumulativeSpending asa % of TotalExpenditures

EvaluationGroup County

All OtherCWServicesCumulativeSpending asa % of TotalExpenditures

Demonstration Portage 0% Demonstration Ashtabula 12%

Comparison Trumbull 0% Comparison Butler 12%

Comparison Miami 3% Demonstration Greene 12%

Comparison Wood 7% Comparison Allen 13%

Comparison Montgomery 8% Demonstration Stark 13%

Comparison Hocking 8% Comparison Warren 15%

Comparison Summit 9% Demonstration Richland 16%

Comparison Scioto 10% Demonstration Muskingum 19%

Demonstration Hamilton 10% Demonstration Clark 21%

Comparison Mahoning 11% Demonstration Belmont 36%

Demonstration Franklin 12%

Overall, demonstration counties have made more changes in practice than

comparison sites because the Waiver:

• allows PCSAs to spend in different ways

• makes it easier to change management practices

• may “grease the wheels” of collaboration

…But how did these changes affect child & family outcomes?

Outcomes Resulting from Organizational and Fiscal

Changes

• Changes in Caseload Trends

• Changes in Case Mix

• Waiver effects on Permanency

Abuse/Neglect Incidents Fell During Waiver Period for Both Groups

Figure 3.1: Number of Child Abuse Incidents By Federal Fiscal Year

0

5000

10000

15000

20000

25000

30000

35000

1996 1997 1998 1999 2000 2001

Fiscal Year

Nu

mb

er o

f C

hild

Ab

use

In

cid

ents

DemonstrationCounties

ComparisonCounties

During Waiver Period, Number of Children in Caseloads Increased in Demonstration

Counties

Number of Ongoing Cases at End of Federal Fiscal Year

0

5000

10000

15000

20000

25000

30000

1996 1997 1998 1999 2000 2001

Fiscal Year

Nu

mb

er

of

On

go

ing

Ca

se

s

DemonstrationCounties

ComparisonCounties

Number of Children Entering First Placements Increased in Large Demonstration Counties

Figure 3.4: First Placement by County Size and Fiscal Year, 1991-2001

0

500

1000

1500

2000

2500

3000

3500

1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001Fiscal Year

Nu

mb

er

of

Fir

st

Pla

ce

me

nts

Large DemonstrationCounties

Small DemonstrationCounties

Large ComparisonCounties

Small ComparisonCounties

Early Findings: Permanency Trends

• Both demonstration and comparison counties experienced an increase in the number of children in permanent custody over the first three years of the Waiver.

• The comparison counties experienced a much larger increase than did the demonstration counties in the number of children in PPLA.

Early Findings: Permanency Trends

• In the first 2 years of the Waiver, both demonstration and comparison county groups increased the percentage of children who experienced no moves in their first placements, to over half of all children in first placements.

• Nearly 25 percent of children in both groups experienced only one move while in placement, and this percentage increased during the first 2 years of the Waiver.

• The percentage of children who experienced five or more moves decreased in both groups

Early Findings: Permanency Trends

• During the first 2 years of the Waiver, the number of teens placed in group and residential settings in demonstration counties did not decline.

• The demonstration counties did not experience greater success than comparison counties at moving children from group and residential settings to less restrictive settings during the first 2 years of the Waiver.

Early Findings: Permanency Trends

• In the first 3 years of the Waiver, both demonstration and comparison county groups experienced an increase in the number of children placed with relatives.

• However, 9 of 14 demonstration counties experienced an increase, compared with 6 of 14 comparison counties.

Use of Kinship Care Varied by County

Table 5.4: Most Frequent Approaches to the Use of Relative/Kinship Care*

Approach Demonstration Counties Comparison Counties

Informal placement directly with relative

Muskingum Allen, Columbiana, Hancock, Hocking, Mahoning, Miami, Trumbull

Custody to relative after short time Belmont, Clark, Crawford, Greene, Medina, Richland, Stark

Butler, Clermont, Montgomery, Scioto, Wood

Placement with relative, custody to PCSA

Ashtabula, Fairfield, Franklin, Hamilton, Portage

Summit, Warren

Approach not determined Lorain None

Early Findings: Permanency Trends

• The majority of counties in both demonstration and comparison groups had an increase in the number of new children eligible for adoption subsidy.

• The overall number of adoptions increased 40% in demonstration counties and 32% in comparison counties during the first three years of the Waiver.

Early Findings: Safety Trends

• During the first 3 years of the Waiver, the recidivism rate for targeted children (those identified with moderate to high risk) declined slightly in the demonstration counties while decreasing considerably in comparison counties.

• Nine demonstration counties and seven comparison counties experienced decreases in recidivism rates for targeted children.

Early Findings: Well-being

• The balance between in-home and placement services remained stable across the baseline and the Waiver periods, and demonstration counties’ proportions were similar to comparison counties’

Changes in Placement Case MixDuring the Waiver

Table 6.2 Changes in Demographic Characteristics of Children in First Placements in Demonstration Counties, from Pre-Waiver to Waiver Period

County Sex Age Race

Male <1 1-4 5-13 14-17 White Black Other

Ashtabula

Belmont

Clark

Crawford

Fairfield

Franklin

Greene

Lorain

Medina

Muskingum

Portage

Richland

Stark

Percentage of Children Placed From Abuse/Neglect Incidents Changed for Several

Counties During Waiver Period Table 6.3 Changes in Abuse/Neglect and Disabilities of Children in First Placements

in Demonstration Counties, from Pre-Waiver to Waiver Period

County Abuse/Neglect and Disabilities

Sexually Abused Alleged Victim of Abuse or Neglect

Cognitive Disabilities Physical Disabilities

Ashtabula

Belmont

Clark

Crawford

Fairfield

Franklin

Greene

Lorain

Medina

Muskingum

Portage

Richland

Stark

Several Counties Increased First Placements with Relatives During the Waiver

Table 6.4 Changes in Settings of First Placements in Demonstration Countiesfrom Pre-Waiver to Waiver Period

County

Setting of Child’s First Placement

ResidentialTreatment Center

Group HomeFoster Home

Nonlicensed Nonrelative

Relative Independent LivingDetention Facility or

Hospital

Adoptive Home

Ashtabula

Belmont

Clark

Crawford

Fairfield

Franklin

Greene

Lorain

Medina

Muskingum

Portage

Richland

Stark

Waiver Effect on the Exit Type fromFirst Placements Overall

Table 6.5 Waiver Effects on Exit Types from First Placementsfor All Demonstration Counties

First Placements Ending With:

Percentage of Cases

Waiver Effect

Actual Under the Waiver Counterfactual Projection

Reunification 45.34 56.74 -11.40*

Custody to Relative 18.12 14.46 3.66*

Adoption 9.26 9.26 0.00

Runaway 1.04 0.54 0.50*

Othera 26.22 19.02 7.20*

Total 100.00 100.00 N/A

The Waiver Shortened the Overall Median Duration of First Placements

Table 6.6 Waiver Effects on Duration of First Placementsfor All Demonstration Counties

First Placements Ending With: Median Placement Duration in Months Waiver Effect

Actual Under the Waiver Counterfactual Projection

Reunification 3.04 2.90 0.14

Custody to Relative 6.56 6.28 0.28

Adoption 31.78 32.00 -0.22

Runaway 6.96 9.24 -2.28

Any type of exit 4.50 4.90 -0.40*

Significant County-Level Effects of Where Children Exited from Placement

• The Waiver significantly increased the percentage of first placements that ended with custody to relatives in six of the 13 demonstration counties and decreased it in one county.

• The Waiver significantly increased runaways in three counties, although the overall effect was small

• The Waiver decreased reunifications in the largest county.

• The Waiver increased adoptions in one county, with no overall effect.

• The Waiver increased “other” exit types in one county (the largest) and decreased them in two others, for an overall increase

County-Level Waiver Effects on Exit Type

Table 6.7 County-Level Waiver Effects on Exit Types from First Placements

County

Exit Type

ReunificationCustody to

RelativeAdoption Runaway Othera

Ashtabula

Belmont

Clark

Crawford

Fairfield

Franklin

Greene

Lorain

Medina

Muskingum

Portage

Richland

Stark

Overall

Some County-Level Waiver Effects on Median Duration to Each Exit Type

Table 6.8 County-Level Waiver Effects on Median Duration of First Placements

CountyExit Type

Reunification Custody to Relative Adoption Runaway Any Type of Exit

Ashtabula

Belmont

Clark

Crawford

Fairfield

Franklin

Greene

Lorain

Medina

Muskingum

Portage

Richland

Stark

Overall

No County-Level Waiver Effect on Reentry

Table 6.9 Re-entry After Reunification from First Placement a

County Actual Rate Under the Waiver (%) Median Duration Before Re-Entry (months)

Ashtabula 26.76 15.32

Belmont 36.10 11.46

Clark 27.30 9.22

Crawford 23.08 18.40

Fairfield 37.44 8.12

Franklin 41.64 9.84

Greene 30.44 14.04

Lorain 23.70 12.88

Medina 38.60 16.22

Muskingum 35.38 10.08

Portage 31.70 11.90

Richland 38.40 13.46

Stark 32.44 15.56

Overall 37.44 12.08

Putting it all together: Case Studies of 6 Demonstration sites

• Clark, Fairfield, Franklin, Lorain, Muskingum, Stark

• Major organizational changes adopted and outcomes achieved

Offers insights into dynamics of child welfare practice under the Waiver, but cannot be extrapolated to all counties

Case Study Findings (1): Lorain and Muskingum

Commitment to systemic change appears to have fostered a greater degree of success, in terms of fiscal and participant outcomes, than in other evaluation counties.– More use of managed care strategies

– More focus on prevention

– Increased spending on non-foster care

– Fewer children in PPLA

– Quicker adoption, more adoption

– More exits to relative custody

Case Study Findings (2)

Other four counties had less clear-cut results despite initiating a variety of reform efforts.

• Clark – reformed front-end of the system through expansion of home-based services and collaboration with juvenile court; result was reduced placement days.

• Franklin – focus on external reform, contracting with private providers; result was reduced LOS in foster care.

• Fairfield and Stark – less consistent reform agendas during Waiver due in part to major organizational challenges; no significant improvements in outcomes.

Overall ResultsMix of evidence of significant impact of the Waiver, areas where systematic impact is lacking, and signs of substantive change in individual counties

Confounding circumstances: – Funding context (levies, state cutbacks)– County-administered system so county-to-county variation– Lack of management tools– Low financial risk– Small sample sizes; differences among data sets

Suggests that the flexibility provided by the Waiver… facilitates reform efforts where other factors are already

conducive,

but may not itself be robust enough to generate fundamental reform of the state’s public child welfare system.

The Future

• Good likelihood of Waiver extension

• Evaluation focus:More in-depth analysis of foster care

utilizationMore examination of child safetyFurther analysis of expenditure patternsTargeted investigation of special topics –

use of relatives, mental health services, court-referred children, etc.