Analyzing COSF Data in Support of System Validity

Analyzing COSF Data in Analyzing COSF Data in Support of System ValiditySupport of System Validity

Analyzing COSF Data in Analyzing COSF Data in Support of System ValiditySupport of System Validity

Charles R. Greenwood & Dale Walker

Some of these data are published in Greenwood, C. R., Walker, D., Hornback, M., Nelson, C., Hebbeler, K., & Spiker, D. (2007). Progress developing the Kansas Early Childhood Special Education Accountability System: Initial findings using the ECO Child Outcome Summary Form (COSF). Topics Early Childhood Special Education, 27(1), 2-18.

Margy Hornback (KS) Birth-5Marybeth Wells (ID) Section 619

http://www.fpg.unc.edu/~ECO/

http://www.fpg.unc.edu/~ECO/

2

Purpose of this PresentationPurpose of this Presentation

Demonstrate analyses of COSF data

Point out how analyses inform the validity of the State’s OSEP accountability system

Help States conduct similar analyses

3

Validity of an Accountability SystemValidity of an Accountability System

An accountability system can be said to have validity when evidence is judged to be strong enough to support inferences that: The components of the system are aligned to

the purposes, and are working in harmony to help the system accomplish those purposes

The system is accomplishing what was intended (and not what was not intended) (Marion et al., 2002, pg. 105)

4

The Validity of an Accountability The Validity of an Accountability SystemSystem

Requires answers to a number of logical questions demonstrating that the parts of the system are working in harmony as planned

Validity is improved by improving the quality and integrity of the parts in the system

Validity requires continued monitoring and improvement

5

COSF Validity Questions and COSF Validity Questions and EvidenceEvidence

1. The Anchor Indicators used in the COSF Process are Mapped via a Cross-walk to the OSEP Outcomes Have the Anchor Indicators been

cross-walked to the 3 OSEP outcomes?

Do the Anchor Indicators have evidence of validity and reliability?

6


2. The COSF Process Involves Multiple Participants and Sources of Evidence Are multiple adults participating in the

process? Are parents participating in the

process? Are multiple sources of evidence being

used?

7

Team Roles Make UpTeam Roles Make UpRoles less than 3% Collapsed to OtherRoles less than 3% Collapsed to Other

8

Parents and RatingsParents and Ratings

How many children have a parent providing the rating 731 out of 2388 (31%)

Other Family Members? Foster Parent = 10 Grandparent = 12 Advocate = 5 Baby Sitter = 2

9

Evidence Sources Reported in One Evidence Sources Reported in One District (Part B)District (Part B)

10


3. COSF Ratings Should Display Differences Between Children’s Performance Is the distribution of COSF ratings

normally distributed? Are fewer children scored 1 and 7,

and more children scored 3, 4, and 5?

11

0.0

5.0

10.0

15.0

20.0

25.0

1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

COSF Rating

Pe

rce

nta

ge

Part B Social Knowledge & Skills Meets Needs

Kansas – Part BKansas – Part B

12

Idaho – Part BIdaho – Part B

13

0.0

5.0

10.0

15.0

20.0

25.0

1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

COSF Rating

Pe

rcen

tage

Part C Social Knowledge & Skills Meets Needs

Kansas – Part CKansas – Part C

14


4. OSEP Outcomes are Defined Functionally, Therefore, They Should be Highly Inter-correlated Are the three outcomes highly inter-

correlated? Does each outcome contribute unique

information?

Correlations Between Entry OutcomesCorrelations Between Entry Outcomes

State and Part

Pair ID (B)KS (B) KS (C)

Know vs Meets

.726 .732 .633

Social vs Meets

.799 .743 .620

Know vs Social

.782 .774 .758

N Children 1003 1280 1108

Entry Correlations When Controlling Entry Correlations When Controlling for the Third Outcomefor the Third Outcome

Control For Pair

ID (B)

KS (B)KS (C)

Social Know vs. Meets

.270 .371 .320

Know-ledge

Social vs. Meets

.540 .408 .276

MeetsNeeds

Know vs. Social

.488 .505 .602

N Children 1003 1280 1108

What is the commonality of shared variance in What is the commonality of shared variance in the entry Part B Social Outcome in ID?the entry Part B Social Outcome in ID?

Predictors R2

Knowledge 61%

Meets Needs 64%

Knowledge, Meets Needs 72%

Predictor/Partition Knowledge Meets Needs

Knowledge 9%

Meets Needs 11%

Knowledge, Meets Needs 53% 53%

Unique 9% 11%

Common 53% 53%

Total 61% 64%

Shared Variance in Social

Formulas for Unique and Commonality Components of Shared Variance:

U1 = R2(12) – R2(2); U2 = R2(12) – R2(1); C12 = R2(1) + R2(2) – R2(12)(Thompson, 2006 (pg 279)

18


5. COSF ratings should be at least moderately (not strongly) correlated with the anchor-primary assessment measure What is the concurrent validity

correlation with the primary assessment measure?

Is there a linear, increasing relationship between ratings and mean test scores?

19

Mean

Correlation between COSF Outcome Ratings And BDI

Domain ScoresSocial vs. PerSocial = .65Knowledge vs. Cognitive = .62 Meets Needs vs. Adaptive = .61

BDI Domain Means by COSF Rating BDI Domain Means by COSF Rating

20

By Anchor Test (ID)By Anchor Test (ID)

21

By Anchor Tests in KSBy Anchor Tests in KS

22


6. COSF Ratings Should Not Be Affected by Conditions in the State’s COSF Process Are there differences by region or

program? Are there differences due to use of

different Anchor tests? Are there differences due to quality or

intensity of training/fidelity in the COSF process?

23

District ComparisonDistrict Comparison

DistID

92 23.8 23.8 23.8

68 17.6 17.6 41.5

75 19.4 19.4 60.9

52 13.5 13.5 74.4

50 13.0 13.0 87.3

49 12.7 12.7 100.0

386 100.0 100.0

D0229

D0233

D0259

D0512

D0602

D0620

Total

ValidFrequency Percent Valid Percent

CumulativePercent

24

District ComparisonDistrict Comparison

25

Evidence Use Profiles Evidence Use Profiles

for 3 largest Part C for 3 largest Part C EntitiesEntities

C0016

C0033Z0042

26


7. Theoretically, We Might Expect COSF Ratings to be Influenced by Differences in Sociodemographics Are there differences in COSF ratings due to

type of disability? Are boys rated lower than girls on the Social

Outcome? (boys tend to have more behavior problems than girls)

Are English Language Learners rated lower on the Knowledge and Skills Outcome?

Do these variables explain significant variance in COSF Outcome at Entry and Exit?

27

By Gender (ID)By Gender (ID)

28

By Disability (ID)By Disability (ID)

29

By Race (ID)By Race (ID)

30

How much variance in entry rating do How much variance in entry rating do demographic variables explain?demographic variables explain?

Social Outcome Model Summary

.424a .180 .179 1.542 .180 219.042 1 1001 .000

.430b .185 .183 1.538 .005 6.643 1 1000 .010

Model1

2

R R SquareAdjustedR Square

Std. Error ofthe Estimate

R SquareChange F Change df1 df2 Sig. F Change

Change Statistics

Predictors: (Constant), Disability typea.

Predictors: (Constant), Disability type, Gendercodedb.

Social Outcome "Best" Model Summary

.799a .638 .638 1.024 .638 1764.829 1 1001 .000

.851b .724 .724 .894 .086 313.333 1 1000 .000

.854c .729 .728 .888 .004 15.431 1 999 .000

.854d .730 .729 .886 .001 5.177 1 998 .023

Model1

2

3

4

R R SquareAdjustedR Square

Std. Error ofthe Estimate

R SquareChange F Change df1 df2 Sig. F Change

Change Statistics

Predictors: (Constant), Meets Needs Outcomea.

Predictors: (Constant), Meets Needs Outcome, Knowledge & Skills Outcomeb.

Predictors: (Constant), Meets Needs Outcome, Knowledge & Skills Outcome, Disability typec.

Predictors: (Constant), Meets Needs Outcome, Knowledge & Skills Outcome, Disability type, Racecodedd.

31


8. Theoretically, We Expect COSF Ratings Will Be Sensitive to Growth and Early Intervention Over Time Are COSF exit rating distributions skewed

to the right, indicating children scoring higher at exit compared to entry?

Are there gains in COSF ratings when comparing entry to exit?

Are these gains statistically significant, and what are the effect sizes?

32

Sample KS Entry and Exit DataSample KS Entry and Exit Data

Part

8 7.5 7.5 7.5

98 92.5 92.5 100.0

106 100.0 100.0

Part B

Part C

Total


CumulativePercent

ProgramEntryType

113 98.3 99.1 99.1

1 .9 .9 100.0

114 99.1 100.0

1 .9

115 100.0

Part B - EC-SPED Team

Transition Team (PartsB & C)

Total

Valid

SystemMissing

Total

Frequency Percent Valid PercentCumulative

Percent

Sample ID Entry and Exit DataSample ID Entry and Exit Data

33

What growth is evident?What growth is evident?

KS IDKS ID

34

What growth is evident: KS?What growth is evident: KS?

Statistics

106 106 106 106 106 106

0 0 0 0 0 0

5.00 6.00 5.00 6.00 5.00 6.00

-.760 -1.175 -.377 -.959 -.761 -1.339

.235 .235 .235 .235 .235 .235

-.072 1.142 -.680 .398 -.244 1.707

.465 .465 .465 .465 .465 .465

1 1 1 1 1 1

7 7 7 7 7 7

Valid

Missing

N

Median

Skewness

Std. Error of Skewness

Kurtosis

Std. Error of Kurtosis

Minimum

Maximum

Entry-Social Exit-SocialEntry-

KnowledgeExit-

KnowledgeEntry-Meets

NeedsExit-Meets

Needs

Statistics

115 115 115 115 115 115

0 0 0 0 0 0

4.00 5.00 3.00 5.00 4.00 6.00

.383 -.749 .541 -.472 .053 -1.129

.226 .226 .226 .226 .226 .226

-.621 .218 -.314 -.192 -1.148 .789

.447 .447 .447 .447 .447 .447

1 1 1 1 1 1

7 7 7 7 7 7

Valid

Missing

N

Median

Skewness

Std. Error of Skewness

Kurtosis

Std. Error of Kurtosis

Minimum

Maximum

EntrySocial ExitSocialEntry

KnowlegeExit

KnowledgeEntryMeets

NeedsExitMeets

Needs

What growth is evident: ID?What growth is evident: ID?

35

What GAIN in Rating: KS?What GAIN in Rating: KS?

What GAIN in Rating: ID?What GAIN in Rating: ID?

Descriptive Statistics

115 -1.0 4.0 1.435 1.0852

115 .0 5.0 1.678 1.0966

115 -1.0 5.0 1.322 1.2179

115

gainSocial

gainKnowledge

gainMeetsNeeds

Valid N (listwise)

N Minimum Maximum Mean Std. Deviation


106 -4.00 5.00 .8208 1.34375

106 -3.00 5.00 .9528 1.48889

106 -4.00 5.00 .8302 1.51483

106

gainSocial

gainKnow

gainMeets

Valid N (listwise)

N Minimum Maximum Mean Std. Deviation

36

What growth is evident: KS?What growth is evident: KS?


106 5.01 1.636 1 7

106 4.58 1.707 1 7

106 5.08 1.616 1 7

106 5.83 1.320 1 7

106 5.53 1.422 1 7

106 5.92 1.296 1 7

Entry-Social

Entry-Knowledge

Entry-Meets Needs

Exit-Social

Exit-Knowledge

Exit-Meets Needs

N Mean Std. Deviation Minimum Maximum

Test Statisticsb

-5.752a -5.840a -5.021a

.000 .000 .000

Z

Asymp. Sig. (2-tailed)

Exit-Social -Entry-Social

Exit-Knowledge -

Entry-Knowledge

Exit-MeetsNeeds -

Entry-MeetsNeeds

Based on negative ranks.a.

Wilcoxon Signed Ranks Testb.

37

What growth is evident: ID?What growth is evident: ID?


115 3.75 1.594 1 7

115 3.33 1.485 1 7

115 4.35 1.722 1 7

115 5.18 1.374 1 7

115 5.01 1.392 1 7

115 5.67 1.394 1 7

EntrySocial

EntryKnowlege

EntryMeetsNeeds

ExitSocial

ExitKnowledge

ExitMeetsNeeds

N Mean Std. Deviation Minimum Maximum

Test Statisticsb

-8.367a -8.720a -7.855a

.000 .000 .000

Z

Asymp. Sig. (2-tailed)

ExitSocial -EntrySocial

ExitKnowledge -

EntryKnowlege

ExitMeetsNeeds -

EntryMeetsNeeds

Based on negative ranks.a.

Wilcoxon Signed Ranks Testb.

38

Change in Social Distribution: KSChange in Social Distribution: KS

39

Change in Social Distribution: IDChange in Social Distribution: ID

40

Change in Knowledge Distribution: KSChange in Knowledge Distribution: KS

41

Change in Knowledge Distribution: IDChange in Knowledge Distribution: ID

42

Change in Meets Needs Distribution: KSChange in Meets Needs Distribution: KS

43

Change in Meets Needs Distribution: IDChange in Meets Needs Distribution: ID

44

What do the OSEP outcome category What do the OSEP outcome category results look like?results look like?

OSEPSocial

12 10.4 10.4 10.4

51 44.3 44.3 54.8

35 30.4 30.4 85.2

17 14.8 14.8 100.0

115 100.0 100.0

b

c

d

e

Total


CumulativePercent

OSEPKnow

12 10.4 10.4 10.4

60 52.2 52.2 62.6

34 29.6 29.6 92.2

9 7.8 7.8 100.0

115 100.0 100.0

b

c

d

e

Total


CumulativePercent

OSEPMeets

11 9.6 9.6 9.6

29 25.2 25.2 34.8

43 37.4 37.4 72.2

32 27.8 27.8 100.0

115 100.0 100.0

b

c

d

e

Total


CumulativePercent

OSEP Category Definition Codesa: Children who did not improve functioningb: Children who improved functioning but not sufficient to move nearer to functioning comparable to same age peersc : Children who improved functioning to a level nearer to same-aged peers but did not reach it d: Children who improved functioning to reach a level comparable to same-aged peerse: Children who maintained functioning at a level comparable to same-aged peers

Note: There were no Cat a. children

45

A State’s OSEP Outcome A State’s OSEP Outcome DistributionsDistributions

46

COSF New Skills Coding Error to COSF New Skills Coding Error to CheckCheck Yes or No and the New Skills Question?

No Means no new skills acquired, no can not be

associated with ratings that go up from entry to exit (e.g., 3 to 4 always = yes)

Yes Means new skills were acquired and in COST 7 to 6 (child means child is still typical) 2 to 2, 3 to 3, etc (staying the same rating in

COSF = yes, new skills acquired) http://www.fpg.unc.edu/~ECO/pdfs/Summar

y_of_Rules_COSF_to_OSEP_8-9-07.pdf

47

COSF Validity Questions and COSF Validity Questions and Evidence (Future Inquiry)Evidence (Future Inquiry)

9 Theoretically, We Expect Gains in COSF Exit Ratings to be Explained by Early Intervention Factors Are gains in COSF ratings explained by

length of service? Are gains in COSF ratings explained by

intervention/program quality features (e.g., models, evidence-based practice, etc.)?

Are gains in COSF ratings explained by family outcomes?

48

ConclusionConclusion

9 validity questions and supporting COSF evidence were discussed

Such analyses help establish and maintain a state’s OSEP accountability system

Evidence from two states appears to support the COSF process as a valid approach

More work is needed, we need to know more from more states!

For More Information see: For More Information see: http://www.fpg.unc.edu/~ECO/http://www.fpg.unc.edu/~ECO/

For More Information see: For More Information see: http://www.fpg.unc.edu/~ECO/http://www.fpg.unc.edu/~ECO/

Documents

Analyzing COSF Data in Support of System Validity