Christina KasprzakECTA/ECO/DaSy

Lauren BartonECO/DaSy

Ruth Chvojicek WI Statewide Part B Indicator 7 Child Outcomes Coordinator

September 16, 2013Improving Data, Improving Outcomes Conference - Washington, DC

Where the Rubber Hits the Road:

Tools and Strategies for Using

Child Outcomes Data for

Program Improvement

Purposes

• To describe national resources for promoting data quality and supporting program improvement

• To share Wisconsin 619 experience and strategies to promote data quality and program improvement

• To discuss potential approaches for examining data quality and using data in your state 2

Quality Assurance: Looking for Quality Data

I know it is in here somewhere

4

Do Ratings Accurately Reflect Child Status?

Pattern Checking

• We have expectations about how child outcomes data should look– Compared to what we expect– Compared to other data in the state– Compared to similar states/regions/school

districts• When the data are different than

expected ask follow up questions

Questions to Ask

• Do the data make sense?– Am I surprised? Do I believe the data?

Believe some of the data? All of the data?

• If the data are reasonable (or when they become reasonable), what might they tell us?

5

Pattern Checking for Data Quality

Strategies for using data analysis to improve the quality of state data by looking for patterns that indicate potential issues for further investigation.

http://ectacenter.org/~pdfs/eco/pattern_checking_table.pdf

7

Predicted Pattern

3b. Large changes in status relative to same age peers between entry and exit from the program are possible, but rare.

Most children served in EI and ECSE will maintain or improve their rate of growth in the three child outcomes areas over time given participation in intervention activities that promote skill development.

8

Rationale

Analysis

1. Crosstabs between entry and exit ratings for each outcome, best for COS ratings.

2. Exit minus Entry numbers.

For COS ratings we would expect most

cases to increase by no more than 3 points.

Question: Is the distribution sensible?

9

Entry

Exit 1 2 3 4 5 6 7 total

1 1 4 2         7

2 1 1 5 6 9 3 1 26

3   2 15 14 27 19 6 83

4   4 4 21 39 28 12 108

5   1 12 14 71 86 48 232

6   1   3 21 48 63 136

7       2 18 23 56 99

Review Total 2 13 38 60 185 207 186 691

Outcome 3: Crosstabs Between Entry and Exit Ratings

Outcome 1: Children that increased by 4 or more points

from entry to exit

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 State0%

5%

10%

15%

20%

25%

30%

35%

% children who increased 4 or more points from entry to exit by district with 30 or more children (N=25)

Analyzing Child Outcomes Data for Program Improvement

• Quick reference tool• Consider key issues,

questions, and approaches for analyzing and interpreting child outcomes data.

http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutcomesData-GuidanceTable.pdf

Steps in Using Data for Program Improvement

Defining Analysis Questions

Step 1. What are your crucial policy and programmatic questions?

Step 2. What is already known about the question?

Clarifying Expectations

Step 3. Describe expected relationships with child outcomes.

Step 4. What analysis will provide information about the relationships? Do you have the necessary data for that?

Step 5. Provide more detail about what you expect to see. With that analysis, how would data showing the expected relationships look?

13

Steps in Using Data for Program Improvement

Analyzing Data

Step 6. Run the analysis and format the data for review.

Testing Inferences

Step 7. Describe the results. Begin to interpret the results. Stakeholders offer inferences based on the data.

Step 8. Conduct follow-up analysis. Format the data for review.

Step 9. Describe and interpret the new results as in step 7. Repeat cycle as needed.

Data-Based Program Improvement Planning

Step 10. Discuss/plan appropriate actions based on the inference(s).

Step 11. Implement and evaluate impact of the action plan. Revisit crucial questions in Step 1.

Defining Analysis Questions

What are your crucial policy and programmatic questions?

Example:

1. Does our program serve some children more effectively than others?

a. Do children with different racial/ethnic backgrounds have similar outcomes?

15

What do you expect to see?

Do you expect children with racial/ethnic backgrounds will have similar outcomes? Why? Why not?

16

Clarifying Expectations

1. Compare outcomes for children in different subgroups:

a. Different child ethnicities/races (e.g. for each outcome examine if there are higher summary statements, progress categories, entry and/or exit ratings for children of different racial/ethnic groups).

17

Analyzing Data

Outcome 1: Summary Statements by Child’s

Race/Ethnicity

18

Summary Statement 1 Greater Than Expected

Growth

Summary Statement 2 Exit at Age Expectations

0

10

20

30

40

50

60

70

80

90

100

68

61

6764

74

69

6259

57

51

72

63

NationalStatewide (4824)Caucasian (2496)Hispanic/Latino (1018)African-American (1134)Multiple/Other (176)

Perc

enta

ge o

f Chi

ldre

n

Outcome 1: Progress Categories by Child’s Race/Ethnicity

19a - no progress b - progress

compared to selfc - narrowed the gap d - closed the gap e - maintained

0

5

10

15

20

25

30

35

40

CaucasianHispanic/LatinoAfrican AmericanMultiple/Other

• Is the evidence what you expected? • What is the inference or interpretation?• What might be the action?

20

Describing and Interpreting Results

Guidance Table

21

22

USING DATA FOR STATE & LOCAL IMPROVEMENT WISCONSIN’S PART B

Ruth Chvojicek – WI Statewide Part B Indicator 7 Child Outcomes Coordinator

KEY POINTS ABOUT WISCONSIN’S SYSTEM

Sampling strategy until July 1, 2011 Part B Child Outcomes Coordinator position funded

through preschool discretionary funds – focus on training and data

Statewide T/TA system with district support through 12 Cooperative Educational Service Agency’s – Program Support Teachers

1 2 3 4 5 6 7

Outcome 1 0.036392615221145

0.108115287554788

0.0931066542701554

0.138265373887635

0.213972639128702

0.208527028821889

0.201620401115686

Outcome 2 0.0468853765440298

0.141984327267898

0.16323548944083

0.197104529153938

0.277593305883916

0.141718687740736

0.0314782839686546

Outcome 3 0.0332049408952053

0.0753088059503256

0.0848718289281448

0.11475627573383

0.17293133218223

0.211980342674991

0.306946473635278

2.5%

7.5%

12.5%

17.5%

22.5%

27.5%

32.5%

State 11-12 Entry Rating Distribution

GERMANTOWN SCHOOL DISTRICT – LESSON’S LEARNED Jenni Last – Speech Language Pathologist

Lisa Bartolone

School Pyschologist

RESULT OF GERMANTOWN’S WORK IN JUST 2 YEARS

1 2 3 4 5 6 7

2011-2012

0.0263157894736842

0.105263157894737

0.0526315789473684

0.184210526315789

0.263157894736842

0.157894736842105

0.210526315789474

2012-2013

NaN NaN 0.0769230769230769

0.269230769230769

0.576923076923077

0.0769230769230769

NaN

5.0%

15.0%

25.0%

35.0%

45.0%

55.0%

65.0%

Germantown Outcome OneEntry Rating Comparison

GERMANTOWN – OUTCOME TWO123456

2011-2012

0.0263157894736842

0.0789473684210526

0.0526315789473684

0.394736842105263

0.394736842105263

0.0526315789473684

2012-2013

2.5%

7.5%

12.5%

Germantown Outcome TwoEntry Rating Comparison

GERMANTOWN – OUTCOME THREE

1 2 3 4 5 6 7

2011-2012 0.0263157894736842

0.0526315789473684

0.0526315789473684

0.105263157894737

0.157894736842105

0.184210526315789

0.421052631578947

2012-2013 NaN NaN 0.0384615384615385

0.269230769230769

0.346153846153846

0.269230769230769

0.0769230769230769

2.5%

7.5%

12.5%

17.5%

22.5%

27.5%

32.5%

37.5%

42.5%

Germantown Outcome ThreeEntry Rating Comparison

STATE PROGRESS IN TWO YEARS – OUTCOME ONE

1 2 3 4 5 6 7

11-12 Out-come 1

0.0400000000000001

0.11 0.0900000000000001

0.14 0.21 0.21 0.2

12-13 Out-come 1

0.0336343410548358

0.107577542206518

0.120926580290538

0.166208611438294

0.246957204554378

0.193691925140688

0.131003795314749

3%

8%

13%

18%

23%

28%

Outcome 1 Entry Rating Comparison

STATE PROGRESS – OUTCOME TWO

1 2 3 4 5 6 7

11-12 Out-come 2

0.05 0.14 0.16 0.2 0.28 0.14 0.03

12-13 Out-come 2

0.0443717277486912

0.149083769633508

0.182984293193718

0.23717277486911

0.270418848167539

0.0973821989528796

0.018586387434555

3%

8%

13%

18%

23%

28%

Outcome 2 Entry Rating Comparison

STATE PROGRESS – OUTCOME THREE

1 2 3 4 5 6 7

11-12 Out-come 3

0.03 0.08 0.08 0.11 0.17 0.21 0.310000000000001

12-13 Out-come 3

0.0261814373609112

0.0747480036654013

0.100798533839508

0.142165204869747

0.233145699698913

0.215604136667103

0.207356983898416

3%

8%

13%

18%

23%

28%

33%

Outcome 3 Entry Rating Comparison

BUT … OUTCOME ONE EXIT RATING

1 2 3 4 5 6 7

11-12 Out-come 1 Exit Rating

0.01 0.032 0.039 0.07 0.158 0.253 0.438000000000001

12-13 Out-come 1 Exit Rating

0.00668073136427568

0.019338959212377

0.0365682137834038

0.0618846694796062

0.146272855133615

0.284810126582279

0.444444444444444

2.5%

7.5%

12.5%

17.5%

22.5%

27.5%

32.5%

37.5%

42.5%

47.5%

Outcome 1 Exit Rating Comparison

OUTCOME THREE

1 2 3 4 5 6 7

11-12 Out-come 3 Exit Rating

0.013 0.025 0.03 0.039 0.09 0.238 0.564

12-13 Out-come 3 Exit Rating

0.00597959901512491

0.0140696447414703

0.0235666549419626

0.0425606753429478

0.0960253253605351

0.236721772775237

0.58107632782272

5.0%

15.0%

25.0%

35.0%

45.0%

55.0%

65.0%

Outcome 3 Exit RatingComparison

WISCONSIN PART B DATA REVIEWS

11-12 – Piloted process individually with 20 districts Discovered differences in how districts were determining

eligibility S/L and SDD Two districts who used criterion referenced tool

consistently AND provided PD on using tool showed more appropriate pattern than other 18 districts

Next steps identified by districts: Mentoring and pd for new staff More attention to formative assessment process Work on internal data tracking system

WISCONSIN PART B 12-13 DATA REVIEW

Looked at 8 data patterns including: Entry Rating Distribution Entry Rating Distribution by Disability* Comparison Entry Ratings by Outcome Exit Rating Distribution Entry / Exit Comparison* Race/Ethnicity Comparison* State Progress Categories* Summary Statements*

LOOKING AT RACE/ETHNICITY

Outcome 1 1 2 3 4 5 6 7Asian 4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%Black 6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%Hispanic 4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%American Indian Alaskan 2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%Hawaiian Other Pacific Islander 0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%Two or More Races 4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%White 2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

Outcome 1 State 11-12 Entry

State BlackState HispanicState White

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

Outcome 1 CESA 1

CESA 1 BlackCESA 1 HispanicCESA 1 White

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

Outcome 1 CESA 2

CESA 2 BlackCESA 2 HispanicCESA 2 White

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

Outcome 1 District M 11-12 Entry

District M BlackDistrict M HispanicDistrict M White

1 2 3 4 5 6 70.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

Outcome 1 District B 11-12 Entry

District B BlackDistrict B HispanicDistrict B White

TRYING OUT THE NEW TOOL - DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?

1 2 3

State Target 0.796 0.825 0.824

State n = 2830 0.782 0.798 0.791

Speech Language n = 2062 0.801 0.827 0.833

SDD n = 341 0.793 0.752 0.783

Autism n = 133 0.702 0.721 0.733

OHI n = 142 0.767 0.721 0.727

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

75.00%

85.00%

Summary Statement One 12-13 Data

DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?

1 2 3

State Target 0.701 0.703 0.806

State n = 2830 0.73 0.612 0.818

Speech Language n = 2062 0.848 0.693 0.914

SDD n = 341 0.528 0.437 0.674

Autism n = 133 0.173 0.278 0.368

OHI n = 142 0.394 0.387 0.546

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

75.00%

85.00%

95.00%

Summary Statement Two 12-13 Data

DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?

A B C D E

State n = 2830 0.007 0.114 0.15 0.282 0.448

Speech Language n = 2062 0.005 0.078 0.069 0.266 0.582

SDD n = 341 0.015 0.173 0.284 0.434 0.094

Autism n = 133 0 0.293 0.534 0.158 0.015

OHI n = 142 0.014 0.197 0.394 0.303 0.092

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

Outcome One 12-13 Progress

DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?

A B C D E

State n = 2830 0.006 0.153 0.228 0.4 0.213

Speech Language n = 2062 0.003 0.127 0.177 0.447 0.246

SDD n = 341 0.015 0.211 0.337 0.349 0.088

Autism n = 133 0 0.256 0.466 0.195 0.083

OHI n = 142 0.028 0.211 0.373 0.246 0.141

2.50%

7.50%

12.50%

17.50%

22.50%

27.50%

32.50%

37.50%

42.50%

47.50%

Outcome Two12-13 Progress

DO CHILDREN WITH SPECIFIC TYPES OF DISABILITIES SHOW DIFFERENT PATTERNS OF GROWTH?

A B C D E

State n = 2830 0.008 0.082 0.092 0.25 0.568

Speech Language n = 2062 0.005 0.042 0.038 0.199 0.715

SDD n = 341 0.018 0.155 0.152 0.472 0.202

Autism n = 133 0.015 0.218 0.398 0.241 0.128

OHI n = 142 0.035 0.199 0.22 0.404 0.142

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

75.00%

Outcome Three 12-13 Progress

WISCONSIN NEXT STEPS

Looking at the data – does the type of setting impact the progress children make? (District level analysis)

As a state T&TA system, we’re operating as a PLC to guide the work and support the District What will the Districts want to focus on? E.g. settings,

race/ethnicity, curriculum use

Local Contributing Factors Tool

Provides ideas for the types of questions a local team would consider in identifying factors impacting performance.

http://www.ectacenter.org/~meetings/outcomes2012/Uploads/ECO-C3-B7-LCFT_DRAFT-10-19-2012.docx

Relationship of Quality Practices to Child and Family Outcome

Measurement Results

Designed to assist states in identifying ways to improve results for children and families through implementation of quality practices.

51

http://ectacenter.org/~docs/eco/QualityPracticesOutcomes_4-29-11-Final.doc

Next Steps?

• Try out using these resources• Send feedback to ECO Center about the

new Analysis tool • What are your ‘take aways’ and next

steps related to analyzing your data for data quality and/or program improvement? (notes for State Team time)

52

53

Find more resources at: http://www. the-eco-center-

org