Looking for Patterns in Child Outcome Data – Examples from NYS

Looking for Patterns in Child Outcome Data – Examples from NYS

New York State Department of Health

Bureau of Early Intervention

July 30, 2010 Measuring Child & Family Outcomes 2

New York’s Child & Family Outcomes System

• New York’s Early Intervention Program (NYSEIP) is one of the largest in the nation– Provides services to nearly 75,000 children and

families each year– 58 local programs administered by 57 county

government agencies and New York City

• Given size and scope of the NYSEIP, a sampling approach has been implemented for collection of child and family outcome data



• Child outcomes data are collected using the COSF

• Each year, local programs enroll up to a designated number of children in child outcomes samples

• COSF forms are completed at entry and exit for children enrolled in child outcomes samples



• A memorandum of understanding with the University at Buffalo, School of Public Health and Health Professions, Population Health Observatory, to partner with BEI in implementing New York’s child and family outcomes system– Responsible for data entry, quality control, management,

and analyses– Maintains a suballocation with the University At

Binghamton, Institute for Child Development, for materials development, training and technical assistance, and annual family survey



• Special thanks to University at Buffalo, PHO staff who collaborated on the analyses and presentation– Li Yan, Ph.D., M.A., Data Manager– Amy Barczykowski, M.S.Ed., Project Coordinator– Randy Carter, Ph.D., Director of PHO

• Our partners at University at Binghamton staff are– Ray Romanczyk, Ph.D., Director, Institute for Child

Development– Emily Huber-Callahan– Jamie Silverio


Pattern 1a

Children will differ from one another in their entry scores in reasonable ways (few scores at the high and low ends and more in the middle)


Pattern 1aEntry Scores Across All 3 Outcome Areas

Outcome 1 Positive Social Emotion SkillsOutcome 2 Acquire and Use Knowledge and SkillsOutcome 3 Take Appropriate Action to Meet Needs


Pattern 1b

Children will differ from one another in their exit scores in reasonable ways


Pattern 1b

Exit Scores Across all 3 Outcome Areas



Pattern 1c

Children will differ from one another in their OSEP Progress Categories in reasonable ways


Pattern 1c

OSEP Categories Across all 3 Outcome Areas

a Children who did not improve functioningb Children who improved functioning but not sufficient to move nearer to functioning comparable to same age

peersc Children who improved functioning to a level nearer to same-aged peers but did not reach itd Children who improved functioning to reach a level comparable to same-aged peerse Children who maintained functioning at a level comparable to same-aged peers


Pattern 2

Functioning in one outcome area will be related to functioning in other outcome areas


Pattern 2

Comparisons Between Outcome 1 and Outcome 2


Pattern 2

Comparisons Between Outcome 1 and Outcome 2

Exit

Outcome 1 x Outcome 2

Outcome 1 Outcome 2

1 2 3 4 5 6 7 TOTAL

1 14 3 5 2 1 0 0 252 5 31 18 4 4 0 0 623 2 21 114 35 25 13 0 2104 1 5 41 77 68 17 3 2125 0 8 32 66 173 100 18 3976 1 0 13 28 122 294 89 5477 0 0 1 10 50 143 362 566

TOTAL 23 68 224 222 443 567 472 2019


Pattern 2 Comparisons Between Outcome 1

and Outcome 2 OSEP CategoriesOutcome 1 and Outcome 2 OSEP Scores

Outcome 1 x Outcome 2

Outcome 1 Outcome 2

a b c d e TOTAL

a 27 19 18 5 2 71

b 17 211 138 43 20 429

c 5 67 253 70 11 406

d 2 31 84 353 68 538

e 6 46 56 220 247 575

TOTAL 57 374 549 691 348 2019


Pattern 3

Functioning at entry within one outcome area will be related to functioning at exit (children who have higher functioning at entry in an area will be the ones who are high functioning at exit in that area)


Pattern 3

Comparisons Between Entry Outcome 2 and Exit Outcome 2

Entry and Exit

Entry Outcome 2 by Exit Outcome 2

Entry Outcome 2

Exit Outcome 2

1 2 3 4 5 6 7 TOTAL

1 6 13 25 9 15 11 3 82 2 6 22 66 41 62 39 23 259 3 5 12 80 94 135 103 59 488 4 2 6 26 38 89 109 72 342 5 2 9 19 19 87 169 103 408 6 2 4 5 16 38 106 106 277 7 0 2 3 5 17 30 106 163

TOTAL 23 68 224 222 443 567 472 2019


Pattern 4

Most children will either hold their developmental trajectories or improve their trajectories from entry to exit


Pattern 4

Outcome 2Exit Score-Entry Score

Outcome 2 Acquire and Use Knowledge and Skills


Pattern 5

Entry, exit, and OSEP progress category distributions should be similar from year to year


Pattern 5

Outcome 2 Entry Scores Across 4 Cohort Years



Pattern 5

Outcome 2 Exit Scores Across 3 Exiting Periods



Pattern 5

Outcome 2OSEP Categories Across 3 Entry Cohort Years

a Children who did not improve functioningb Children who improved functioning but not sufficient to move nearer to functioning comparable to same age

peersc Children who improved functioning to a level nearer to same-aged peers but did not reach itd Children who improved functioning to reach a level comparable to same-aged peerse Children who maintained functioning at a level comparable to same-aged peers


Pattern 7

Entry and exit scores and OSEP categories should be related to the nature of the child’s disability


Pattern 7

Children with Only a Communication Delay ENTRY Scores for 3 Outcome Areas



Pattern 7

Children with Only a Communication Delay EXIT Scores for 3 Outcome Areas



Pattern 7

OSEP Categories for Children with Only a Communication Delay

a Children who did not improve functioning

b Children who improved functioning but not sufficient to move nearer to functioning comparable to same age peers

c Children who improved functioning to a level nearer to same-aged peers but did not reach it

d Children who improved functioning to reach a level comparable to same-aged peers

e Children who maintained functioning at a level comparable to same-aged peers


Pattern 8

Scores at entry and exit should not be related to certain characteristics


Pattern 8

Outcome 2Percentage of Male and Female Entry Scores



Pattern 8

Outcome 2Medicaid Status and Entry Scores



Conclusion/Next Steps

• Pattern checking is a useful way to begin the data quality assurance/quality improvement process

• Analyzing the data in these ways has helped NY target areas for improvement

• Planning/designing a webinar to share data with local programs

• Identifying other strategies to provide training and technical assistance to programs, providers, and families

Documents

Looking for Patterns in Child Outcome Data – Examples from NYS