Linda GoodmanConnecticut Birth to Three System

Early Childhood Outcomes ConferenceJuly 31, 2010

1

Results and Performance Accountability, Decision-making and Budgeting

Mark FriedmanFiscal Policy Studies

InstituteSanta Fe, New Mexico

www.resultsaccountability.comwww.raguide.org

Use #1

2

“All Performance Measures

that have ever existed

in the history of the universe

involve answering two sets of

interlocking questions.”

3

HowMuchdid we do?

( # )

HowWelldid we do it?

( % )

Quantity Quality

4

Effort

Effect

HowMuch

HowWell

5

How much did we do?

How welldid we do it?

Is anyonebetter off?

Quantity Quality

E

ffect

E

ffort

# %

6

How much service did we deliver?

How welldid we deliver it?

How much change/effect did we produce?

What quality of change/ effect did we produce?

Quantity Quality

E

ffect

E

ffort

Ou

tpu

t

Inp

ut

7

How much did we do?

Education

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

E

ffort

Number ofstudents

Student-teacherratio

Percent of 9th graders whoenter college oremployment after graduation

Number of 9th graders whoenter college oremployment after graduation

8

How much did we do?

Part C

How well did we do it?

Is anyone better off?

Quantity Quality

E

ffect

E

ffort

Number of Eligible Children

Percentage of children under 12 months

Number of children exiting whose development was accelerated

Percent of children exiting whose development was accelerated

9

How much did we do?

Not All Performance Measures Are Created Equal

How well did we do it?

Is anyone better off?

LeastImportant

Quantity Quality

E

ffect

Eff

ort

2nd MostImportant

3rd MostImportant Most

Important

10

How much did we do?

The Matter of Control

How well did we do it?

Is anyone better off?

MoreControl

Quantity Quality

E

ffect

E

ffort Less

Control

Partnerships needed to improve performance

11

“We haven’tgot the money, so we’ve gotto think.”

Lord Rutherford1871 - 1937

“Turning the curve”

12

Contributionrelationship

Alignmentof measures

Appropriateresponsibility

THE LINKAGE Between POPULATION and PERFORMANCE

POPULATION ACCOUNTABILITY

Healthy Births Rate of low birth-weight babiesStable Families Rate of child abuse and neglectChildren Succeeding in School Percent graduating from high school on time

CUSTOMERRESULTS

# Foster ChildrenServed

% withMultiplePlacements

# RepeatAbuse/Neglect

% RepeatAbuse/Neglect

PERFORMANCE ACCOUNTABILITYChild Welfare Program

POPULATIONRESULTS

13

Quality of Life Result: All Connecticut children are healthy and ready for school success at age 5, contributing to a reduction over time in Connecticut’s achievement gap at Grade 4.

Program Contribution to Result: By providing family-centered early intervention services, the program strengthens the capacity of Connecticut’s families to meet the developmental and health-related needs of their infants and toddlers who have delays or disabilities, thereby ensuring that more of these children are ready for Kindergarten. Activities include coaching caregivers to embed intervention in the child’s daily natural routines and learning opportunities.Partners: 44 contracted providers programs; local school districts; pediatricians and family medicine practitioners, hospital NICU staff, Office of Policy and Management; Department of Public Health; Department of Insurance; State Department of Education; Department of Children and Families; Board of Education and Services for the Blind; Children’s Trust Fund, UCEDD, Early Childhood Cabinet; HMO Association Members and Anthem; United Way; State ICC; Local ICCs;

Connecticut Part C Example

14

Performance Measure 1 Percentage of infants and toddlers with disabilities or developmental delays who improve so that at exit, they function at age level in three skill areas

Skill Areas

Story Behind the baseline: The three skill areas on the graph are: (1) social/emotional; (2) acquisition and use of knowledge (including early language and literacy); and (3) use of appropriate behavior to meet their needs.

Each bar represents the percentage of children that performed at age level in each skill area at the time they exited the Birth to Three System. Data began in FY07 and some variation is due to the fact that it has taken three years for the children with the most significant disabilities, identified at birth, to exit the system.

Children that received services for at least six months prior to exit

15

Proposed actions to turn the curve: Since Birth to Three only enrolls children with significant delays or disabilities, it is not possible that 100% of children will exit the program at age-level. However, we do expect these percentages to increase from their current levels. The research shows that improvement is most highly correlated with the ability of each provider to ensure that parents and childcare givers know how to incorporate intervention techniques into daily routines so that each child gets the maximum amount of practice, all day, every day. This is a different approach than out-patient rehabilitation services, in which short sessions in therapy rooms are expected to lead to generalization and carryover into other environments. That approach does not work well for infants and toddlers. Since each local program is responsible for overseeing the delivery of early intervention services, the next step will be to rank each program on its child outcome data and then focus on improving the low-performing programs through technical assistance and additional training for their personnel.

16

Performance Measure 2: Percentage of families who report that, as a result of receiving services from the Birth to Three System, they are better able to help their children develop and learn.

Story behind the baseline: The blue bars on the graph show the percentage of families who strongly or very strongly agreed with the statement: “Birth to Three has helped me to help my child develop and learn.” The trend in this positive family response has increased from 73% to 78% in the past three years, which shows the programs’ effectiveness in achieving their mission of helping families to facilitate their children’s development. The percentage of families indicating “agree, strongly agree, or very strongly agree” for this measure was 87.5%

17

Proposed actions to turn the curve: This data, like the child outcome data, will be publicly posted by program and used to monitor any low-performing programs and drive improvement in how well staff work with families. Additional training for service coordinators will be offered both through Birth to Three and through the UCEDD.

18

Focused Monitoring

National Center for Special Education Accountability Monitoring

Alan CoulterJane Nell Luster

Use #2

19

There is something very sad in the disparity between our passion for figures and our ability to make use of them once they are in our hands.

-M.J. Moroney, 1951

20

Why?Monitoring resources are in shorter and

shorter supply yet there is a greater emphasis on general supervision

It is not possible for EI programs or districts to concentrate on the hundreds of requirements in IDEA at all times

Programs cannot work to improve hundreds of things at the same time

Focusing on important result-oriented priorities can more readily lead to improved results for young children

21

Six Components of Focused Monitoring

1. Stakeholder participation and decision-making

2. Data availability, analysis, and use3. Integration with the system of general

supervision4. Off-site and on-site monitoring processes5. Corrective actions, incentives, and sanctions6. Evaluation

22

Principles of Focused Monitoring1. A limited number of priorities are chosen by

a diverse group of stakeholders2. A limited number of indicators are identified

within each priority area3. The system is data and information-based

and is verifiable4. Data-based information is used to allocate

monitoring resources in the direction of most need

23

Principles of Focused Monitoring5. The monitoring agency provides supports

and imposes sanctions upon programs in order to achieve corrective actions

6. Standard, uniform benchmarks are used for inquiry when making monitoring decisions

7. There is a relationship between monitoring and corrective actions – solutions are linked to identified problems.

8. The system includes clear, known triggers for interventions and sanctions

24

Principles of Focused Monitoring9. Limited resources are allocated to the areas

of greatest need, which are determined by identifying what is most likely to lead to improvements in child performance

10.Available information is used to select priorities that will prove child and family outcomes

11.Monitoring strategy is systemic12.Corrective actions ensure a change in

behavior that results in improved child and family outcomes

25

Principles of Focused Monitoring13.Monitoring staff are well trained and

engage in continuous professional development

14.There is third party evaluation of the monitoring/enforcement system

26

Selecting Priorities & Key Performance IndicatorsPriority areas should align with SPPBest not to select compliance indicators

if possibleIndicators must be measureableData must be available (preferably at

least quarterly) otherwise indicator must be deferred until the required data is available

27

Many priorities...Determining priority indicators in 2004

IFSP• child and family outcomes met• resources• objective timelines met• planning

Child Find• identification by diagnosed

condition by age• correct assessment, all needs

identified• referral age• timely referral new money • families not accepting services

Transition• date of transition plans• date of referral to LEA• date of transition conference with

LEA• Inservice training on Part B

requirements• the timely preparation of families

for all transitions• training personnel and parents

about advocacy for Part B and C• adequate information shared from

Part C to Part B

Service provision• timelines of objectives met• withdrawals• competent providers• appropriate services and

support• resources/other families• measurable progress• services delivered match IFSP

Family involvement• define• direct/indirect• opportunity to be involved• knowledgeable to help their

child• information and decision-

making• advocacy

28

Little data….Determining priority indicators in 2004

IFSP• child and family outcomes met• resources• objective timelines met• planning

Child Find• identification by diagnosed

condition by age• correct assessment, all needs

identified• referral age• timely referral new money • families not accepting services

Transition• date of transition plans• date of referral to LEA• date of transition conference with

LEA• Inservice training on Part B

requirements• the timely preparation of families

for all transitions• training personnel and parents

about advocacy for Part B and C• adequate information shared from

Part C to Part B

Service provision• timelines of objectives met• withdrawals• competent providers• appropriate services and

support• resources/other families• measurable progress• services delivered match IFSP

Family involvement• define• direct/indirect• opportunity to be involved• knowledgeable to help their

child• information and decision-

making• advocacy

29

But in 2009… things changed Priority: Child & Family Outcomes Indicator: Families are more confident and gain new skills

which help their children develop and learn.

DATA FOR INDICATORresponses to the family survey measure #32 (Over the past year Birth to Three services have helped me and/or my family figure out solutions to problems as they come up)child outcome data (C3b) showing the percent of children who closed or reduced the “gap” in their acquisition and use of knowledge and skills (including early language/communication).

30

But in 2009… things changed Priority: Child & Family Outcomes Indicator: Families are more confident and gain new skills

which help their children develop and learn.

Selection criteria Programs that are more than 2 SD below the statewide mean are listed as “ very low”Programs that are 1 – 2SD below the statewide mean are listed as “low”. 0 – 1SD below the statewide mean is “mid”The first programs selected will be those that are very low in at least one measure. The next programs selected will be those that are low in at least one measure.

31

How might that look?Both High One

Mid/One High

Two Mid One or more low

One or more very low

Cheshire Abilis Rehabilitation Assoc.

Ahlbin Center Bisantz & Assoc.

Early Connections

Building Bridges

CES REM

Hill Health Children’s Therapy

Child & Family Network

Key Services

Kennedy Center CREC Easter Seals

Oak Hill Education Connections

Family Junction

LEARN SARAH

HARC

McLaughlin

Wheeler32

Program is notified that they’ve been selected for focused monitoring

At desk audit, monitoring team studies data, looks deeper into the data and related data and forms a hypothesis. (e.g. High rate of staff turnover is why children in this program are not making as much progress as children in similar programs.) Discusses with program administrator

Monitoring team goes on-site, follows established protocol that includes looking at records, talking to families, talking to staff, and sometimes talking to LEAs.

What’s the Monitoring Process?

33

Protocol excerptsWhat to look for Records Famil

yStaff Dat

aLEA

Other

Quality evaluation/assessment reports describing strengths and needs

× × ×

Families know what their service coord. can do for them

× ×

Families are connected to outside supports

× × ×

IFSP is routines-based × × ×

Families are using strategies between visits

× × ×

Where to look

34

Protocol excerptsWhat to look for Records Famil

yStaff Dat

aLEA

Other

Staff can demonstrate/describe coaching vs. consulting vs. teaching

× ×video?

Timing of visits is related to outcomes

× ×

Family outcomes are addressed

× × × ×

Updated assessments are used at transition conferences

× × × ×

Families discuss child’s progress, strengths & needs with LEA

× × ×

Where to look

35

What’s the Monitoring Process?Monitoring team members keep administrator

informedMonitoring team meets at the end of each day to

discuss. When all data is gathered, monitoring team meets

to discuss hypothesis and any other issues they have discovered

Monitoring team meets with administrator(s) for exit interview. Nothing said should be a surprise. TA offered

Final report is issued, including findings36

What’s the Process?Improvement plan (or compliance

agreement) written to correct any non-c0mpliance and improve performance. Strategies may include training and TA.

Accountability and Monitoring staff track improvement plan timelines and data until data indicates that non-compliance is corrected and performance has improved.

37