Upload
radwan
View
48
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource Allocations Presented February 13, 2013. Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann York College. Background: Linking Data Sets. - PowerPoint PPT Presentation
Citation preview
1
Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs,
Policies, and Resource Allocations
Presented February 13, 2013
26th Annual Management Information Systems [MIS] Conference
Deborah T. Carran, Jacqueline Nunn, Sara HooksJohns Hopkins University
Stacey N. DammannYork College
2013 MIS Conference 2
Background: Linking Data Sets
Part C Early Intervention Services data (Birth – 3)Part B Special Education Services data (4, 5 – 21)General Education Services dataK Children assessed for K Readiness
WSS-K, scaled then scores at 3 levels Readiness (Developing, Approaching, Fully Ready)
Maryland State Assessment (MSA), scaled then scored at 3 levels (Advance, Proficient, Basic)– Math and Reading assessments administered annually in
grades 3 through 8
2013 MIS Conference 3
2013 MIS Conference 4
Benefits of Using the Maryland IDEA Scorecard
• Data at the State, District, and School levels that drills down to the student level
• Data that allows users to identify students in need of targeted interventions in the alert categories (attendance, academics, suspension, and mobility)
• Data to create an action plan to monitor student progress within targeted interventions
• Reporting functions to support monitoring of progress towards targets of interventions
2013 MIS Conference 5
How to use these data at the state level?
• Descriptive– Who do we serve?
• Longitudinal– What is the educational placement of children served in EIS by K
and Grade 3?– Is Fall K WSS is a successful predictor of later standardized test
performance?• Comparative– Is there a difference in standardized achievement performance
within educational services (Gen Ed and Sp Ed) for children served in EIS?
2013 MIS Conference 6
Tracking Plan
Birth Cohort09/01/2001-08/31/2002
Yes EIS
Gen Ed Grade 3
Sp Ed Grade 3
No EIS
Gen Ed Grade 3
Sp Ed Grade 3
2013 MIS Conference 7
Method
• PARTICIPANTS– 42% tracking match– Missing data
• Outcomes– WSS-K (2006-07)– RSAA and MSAA (Spring 2011)
• Procedure• Three studies presented
2013 MIS Conference 8
Missing Data
• N = 5,328 participants received EIS– 42.1% (n = 2,245) matched – 57.9% (n = 3,083) unmatched (missing data)
• Is there a significant difference between matched and unmatched participants on characteristics: gender, Part C eligibility category, Part C MA, Race, age of entry, age of exit, and/or months in EIS?
2013 MIS Conference 9
Gender
Female33%
Male67%
Unmatched
Female34%
Male66%
Matched
Results of the Chi-Square statistic assuming equal probabilities indicated no significant difference between Unmatched and Matched participants for gender [X2 (1, N = 5,328) = 0.93, p > .05]. Inspection of cell counts and percentages indicated that the distribution for gender was similar for both groups.
2013 MIS Conference 10
Part C Eligibility 25% Delay Atypical Develp/Behav High Probability
N (%) N (%) N (%)
Unmatched 1,902 (61.9%) 463 (15.1%) 707 (23.0%)
Matched 1523 (67.8%) 376 (16.7%) 346 (15.4%)
Total 3,425 (64.4%) 839 (15.8%) 1053 (19.8%)
Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Eligibility [X2 (2, N = 5,317) = 47.23, p < .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible due to a Condition with a High Probability than Matched participants. Conversely, a greater proportion of Matched participants were eligible based on 25% Delay than Unmatched participants.
2013 MIS Conference 11
Part C Eligibility cont.
62%15%
23%
Unmatched25% Developmental DelayAtypical Devel or BehaviorHigh Probability
68%
17%
15%
Matched25% Developmental DelayAtypical Devel or BehaviorHigh Probability
2013 MIS Conference 12
Part C Medical Assistance
Yes MA39%
No MA61%
Unmatched
Yes MA33%
No MA67%
Matched
Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Medical Assistance [X2 (1, N = 5,278) = 17.49, p = .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible for Medical Assistance while receiving EIS.
2013 MIS Conference 13
Race
Am Ind/Alaska Native
Asian or Pacific
Islander Black/AfAm Hispanic Multiple White
Unmatched 2 (0.1%) 121 (3.9%) 971 (31.5%) 237 (7.7%) 112 (3.6) 1640 (53.2%)
Matched 5 (.2%) 91 (4.1%) 606 (27%) 201 (9%) 75 (3.3%) 1267 (56.4%)
Total 7 (.1%) 212 (4%) 1577 (29.6%) 438 (8.2%) 187 (3.5%) 2907 (54.6%)
Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Race [X2 (5, N = 5,328) = 16.76, p < .05]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were Black or African American while a smaller proportion of participants in the Unmatched group were Hispanic than in the Matched group.
2013 MIS Conference 14
Entry/Exit Age in Months
EntryAge ExitAge Months in Program0
5
10
15
20
25
30
35
16.53
30.38
13.7412.23
28.9
16.71 UnmatchedMatched
2013 MIS Conference 15
Summary – Missing Data• Gender – the two groups are similar• Eligibility – significantly greater proportion of Unmatched
participants in High Probability Category• Medical Assistance – significantly greater proportion of
Unmatched participants received MA• Race – significantly greater proportion of Black or African
American participants in Unmatched group with significantly greater proportion of Hispanic participants in Matched group
• Age – Matched participants were significantly younger at age of entry than Unmatched participants. Matched participants received EIS longer than Unmatched participants. Age of exit was similar for both groups.
2013 MIS Conference 16
Outcome Instruments• Work Sampling System-Kindergarten (WSS-K)
Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)• Personal and Social Development• Language and Literacy• Mathematical Thinking• Scientific Thinking• Social Studies• The Arts• Physical Development
• Reading State Accountability Assessment (RSAA)– scaled then scored at 3 levels (Basic, Proficient, Advanced)
• Math State Accountability Assessment (MSAA)– scaled then scored at 3 levels (Basic, Proficient, Advanced)
2013 MIS Conference 17
Procedure• Student data tracking system links records of students
in General Ed, Special Ed (Part B) and EIS (Part C)• Identify children in 3rd Grade 2010-11 school year
with birth dates between Sept. 1, 2001 and Aug. 31, 2002
• At the state level, student identifiers entered in student data tracking system to obtain student service level (Gen Ed or Sp Ed), student outcomes, and demographic info
• Deidentified data provided to researchers
2013 MIS Conference 18
Studies and RQ for this Presentation1. Impact of Early Intervention on K Readiness– Who was served by EIS?– Does EIS impact later K Readiness Scores?
2. Tracking Children Receiving Early Intervention Services (Part C Services Birth to 3) into Elementary School to Grade 3– What is the educational placement at Grade 3 for
children who received EIS?– What is the SAA performance at Grade 3 for
children who received EIS?
2013 MIS Conference 19
Different Data Set
3. K Readiness and Grade 3 SAA Performance– Is there a relationship between K Readiness and
Grade 3 RSAA and MSAA scoring?– What subscales of the WSS predict RSAA and
MSAA scoring?
2013 MIS Conference 20
Study 1: Impact of Early Intervention Services on K Readiness
• Does level of service provided to children (Birth – 3) enrolled in early intervention services (EIS) programs enhance their later performance on the Kindergarten Work Sampling System (WSS-K)?
2013 MIS Conference 21
K Readiness: Method• 2,245 children – Who were eligible and received EIS services in MD
linked with MD MMSR scores– Born between Sept 1, 2001 and Aug 31, 2002
• EIS services• WSS-K– Summary scaled composite score
• Hierarchical Linear Regression
2013 MIS Conference 22
Participants
Birth Cohort09/01/2001-08/31/2002
Yes EISKindergarten
2006-07
2013 MIS Conference 23
EIS Services & WSS-K Average ScoresService Description Children WSS-K WSS-K
N Mean SD
Audiology 474 73.2 13.1
Family Counseling/Training 375 71.7 13.2
Occupational Therapy 467 71.0 14.4
Physical Therapy 565 73.0 13.9
Special Instruction 1145 71.8 13.5
Speech/Language Therapy 1553 74.4 12.9
Other Services Than Above 335 73.1 13.6
At least 1 Service 2245 74.5 12.8
Max WSS-K score of 90
2013 MIS Conference 24
VariablesOutcome: WSS-KPredictors:• Demographics: FaRMs, Gender, Minority
• Earliest Age of Child that Part C Services Begin(-) receive services earlier better prepares child to enter K
• Time in Program (+) indicates longer time in program better prepares child to enter K
• Total Minutes Services (+) indicates longer time in program better prepares child to enter K
2013 MIS Conference 25
Hierarchical Regression ResultsVariables Model 1 Model 2
β β
FaRMs -.170** -.171**
Gender -.048* -.044*
Minority -.120** -.103**
Age Svc Began (months) -.069*
Time in Program (days) -.066*
Total Minutes Services -.138**
R2 .061 .086
R2 Change .025**
2013 MIS Conference 26
EIS – WSS Conclusions• Demographic Controls: WSS-K was higher for students not
economically disadvantaged, higher for girls, and for White students
• Age Svc Began (-): For every month earlier a child starts receiving services, he/she is expected to score .017 SD increase on the WSS-K– Supports previous findings
• Time in Program (-): inconclusive; possible that children who are in the program for longer times have more severe disabilities
• Total Minutes Services (-): inconclusive, possible that children with more severe disabilities will have more/longer services– Correlation Time in Program & Total Min Svc r = .37, p < .001
2013 MIS Conference 27
Study 2
Tracking Children Receiving Early Intervention Services (Birth to 3) into Elementary School by Service Level:
How do they Compare with their Peers?
2013 MIS Conference 28
Background• Part C Services (IFSP)• Part b/B Services (IEP)• Little research examining outcomes of children
receiving Part C services, mostly Part B services (Cole, Dale, Mills & Jenkins, 1993; Daley & Carlson, 2009; Peterson et al., 2004; Walker et al., 1988)
– Focused on developmental progress over short term; not longitudinal or growth trajectories
– Enrollment in special education changes for children as they move through elementary school
– Limitations (disability, small samples, covariates)
2013 MIS Conference 29
First Steps: Descriptive Analyses
• Children in 3rd grade (within birth cohort)– What is their Education Service (Gen Ed, Sp Ed)?– Who were the children receiving EIS?
2013 MIS Conference 30
Participants
Birth Cohort09/01/2001-08/31/2002
Yes EIS
Gen Ed Grade 3
Sp Ed Grade 3
No EIS
Gen Ed Grade 3
Sp Ed Grade 3
2013 MIS Conference 31
Total Sample, Gen Ed and HI Sp Ed
Characteristic N %Total Sample 52,584 100.0
General Education 47,928 91.1
Special Education (Part B) 4,656 8.9
Disability Codes 4, 6, 8, and 9 3,994 86.7
04-Speech or Language Impairments 1,898 40.7
06-Emotional Disturbance 224 4.8
08-Other Health Impairments 721 15.5
09-Specific Learning Disabilities 1,151 24.7
All Other Disability Codes* 665 14.3
*Autism, Deaf, Deaf-Blindness, Developmental Delay, Hearing Impaired, Intellectual Disability, Multiple Disabilities, Orthopedic Impairment, Traumatic Brain Injury, Visual Impairment
2013 MIS Conference 32
Gender N %
Males 26,449 50.3
Females 26,135 49.7
New Race Descriptions
American Indian/Alaskan 140 0.3
Asian 3,108 5.9
Black/African American 17,044 32.4
Hispanic 6,611 12.6
Multiple Races 2,258 4.3
Native Hawaiian/Pac Islander 33 0.1
White 23,390 44.5
Eligible for Free and Reduced Meals
No 29,892 56.8
Yes 22,692 43.2
Limited English Proficiency Identified
No 47,081 89.5
Yes 4,056 7.7
Exited 1,447 2.8
2013 MIS Conference 33
Students in Grade 3 Historically Tracked to EISCharacteristic N %
Third Grade Students Received EIS Part C 2,482 100.0 General Education Grade Three 1,628 65.6
Special Education Grade Three 854 34.4
Disability Codes 4, 6, 8, and 9 (SL, ED, OHI, SLD) 617 72.2
All Other Disability Codes 237 27.8
Gender
Males 1,646 66.3
Females 836 33.7
Limited English Proficiency Identified
No 2,353 94.8
Yes 97 3.9
Exited by Grade Three 32 1.3
Free and Reduced Meals in Grade Three
No 1,627 65.6
Yes 855 34.4
Minority Status
Yes, Minority 1,086 43.8
No, Minority 1,396 56.2
34
Students in Grade 3 Tracked to EIS, EIS Characteristics
Part C Eligibility of Grade 3 Students Born Sept. 1, 2001 – Aug. 31, 2002
25% Delay Atypical Development
High Probability Condition
Student Placement Grade Three 2011 (n = 1,674) (n = 400) (n = 408)
N % N % N %
General Education Grade Three 1,093 65.3 284 71.0 251 61.5
Special Education Grade Three 581 34.7 116 29.0 157 38.5
2013 MIS Conference
65.5%, n = 1,628 enrolled in general education at grade three
2013 MIS Conference 35
Average State Assessment Scores by Educational Service at K and Grade 3
2008 WSS-K 2011 RSAA 2011 MSAA
N M SD M SD M SD
General Ed Gr 3 47928 77.9 10.9 430.8 38.2 429.9 41.1
No EIS 46300 77.9 10.9 430.9 38.2 429.9 41.1
Yes EIS 1628 77.2 11.1 427.8 39.1 428.6 41.7
Special Ed Gr 3 3994 67.8 13.5 368.0 120.6 364.6 114.4
No EIS 3377 68.0 13.4 371.5 117.3 367.1 111.2
Yes EIS 617 67.2 13.9 349.2 135.9 350.9 129.8
2013 MIS Conference 36
Hierarchical Regression Analyses
Outcome: RSAA Grade 3, MSAA Grade 3Three Models:
Model 1, 2, 3Demographics: FaRMs, Gender, MinorityModel 2, 3Part C EISModel 3EIS x FaRMsEIS x GenderEIS x Minority
Hierarchical Regression Results: WSS-K
WSS-K General Ed Special Ed
Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size
Variable B (SE) B (SE) B (SE) Partial ŋ2 B (SE) B (SE) B (SE) Partial ŋ2
Constant 81.57**(.10) 81.60**(.10) 81.60**(.10) .592 72.95** (.50) 73.20** (.52) 73.65** (.55) .248
FARMS -3.57**(.12) -3.58**(.12) -3.56**(.12) .003 -3.73** (.50) -3.83** (.50) -3.79** (.56) .002
Gender -2.46**(.11) -2.45**(.11) -2.49**(.11) .000 -1.89** (.51) -1.86** (.51) -2.54** (.55) .004
Minority -1.98**(.12) -1.99**(.12) -1.98**(.12) .001 -3.62** (.50) -3.61** (.50) -3.63** (.56) .009
EIS -.80** (.29) -1.16* (.56) .000 -1.28* (.62) -4.37** (1.37) .003
EIS * FARMS -.47 (.68) .000 -.35 (1.32) .000
EIS * Gender 1.09 (.60) .000 4.22** (1.37) .001
EIS * Minority -.40 (.66) .000 .53 (1.31) .000
R2.057 .057 .057 .056 .057 .060
R2 Change .0001** .000 .001* .003*
2013 MIS Conference 37
*p ≤ .05, **p ≤ .01
2013 MIS Conference 38
WSS-K ResultsGeneral Ed
• WSS-K higher for students not FaRMs, for girls, and for White students
• EIS students scored lower (.7 M diff)
• No interaction effects
Special Ed• WSS-K higher for students
not FaRMs, for girls, and for White students
• EIS students scored lower (.8 M diff)
• Sig interaction EIS x Gender– F > M NO EIS (2.9 M diff)– M > F EIS (1.7 M diff)
Hierarchical Regression Results: RSAA
RSAA Grade 3 General Ed Special Ed
Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size
Variable B (SE) B (SE) B (SE) Partial ŋ2 B (SE) B (SE) B (SE) Partial ŋ2
Constant 451.50**(.29) 451.70**(.30) 451.76**(.30) .592 399.95**(4.10) 404.91**(4.19) 406.34**(4.44) .248
FARMS -22.31**(.35) -22.34**(.35) -22.31**(.36) .003 -19.31**(3.99) -21.19**(3.99) -20.24**(4.36) .002
Gender -10.15**(.32) -10.04**(.32) -10.14**(.32) .000 5.16 (4.06) 5.79 (4.05) 4.08 (4.39) .004
Minority -11.26**(.35) -11.35**(.35) -11.39**(.36) .001 -46.56** (4.00) -46.75** (3.99) -48.16** (4.37) .009
EIS -5.49**(.89) -7.64**(1.71) .000 -27.86**(5.16) -39.27**(11.48) .003
EIS * FARMS -1.00 (2.08) -5.64 (10.86)
EIS * Gender 3.08 (1.84) 12.72 (11.48)
EIS * Minority 1.30 (1.98) 9.38 (10.75)
R2.153 .154
.154 .054 .061 .060
R2 Change .001** .000 .007** .000
2013 MIS Conference 39
*p ≤ .05, **p ≤ .01
2013 MIS Conference 40
RSAA ResultsGeneral Ed
• RSAA higher for students not FaRMs, for girls, and for White students
• EIS students scored lower (3.1 M diff)
• No interaction effects
Special Ed• RSAA higher for students
not FaRMs and for White students
• EIS students scored lower (22.3 M diff)
• No interaction effects
Hierarchical Regression Results: MSAA
MSAA Grade 3 General Ed Special Ed
Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size
Variable B (SE) B (SE) B (SE) Partial ŋ2 B (SE) B (SE) B (SE) Partial ŋ2
Constant 448.58**(.32) 448.77**(.32) 448.81**(.33) .550 390.64**(3.84) 394.63**(3.92) 396.59**(4.16) .248
FARMS -22.64**(.38) -22.67**(.38)-22.57**(.39) .003 -18.99**(3.74) -20.50**(3.74) -20.10**(4.08) .002
Gender -2.24**(.35) -2.13**(.35) -2.28**(.35) .000 18.68**(3.81) 19.19**(3.79) 15.52**(4.11) .004
Minority -14.36**(.38) -14.45**(.38)-14.47**(.39) .001 -53.53**(3.75) -53.68**(3.74) -53.01**(4.09) .009
EIS -5.35**(.96) -7.68**(1.85) .000 -22.53**(4.84) -37.81**(10.76) .003
EIS * FARMS -3.46 (2.24) -3.77 (10.18)
EIS * Gender 4.85* (1.99) .000
25.01* (10.76) .001
EIS * Minority 1.02 (2.15) -2.25 (10.08)
R2 .142 .142
.143 .078 .083 .084
R2 Change .001** .000* .005** .000
2013 MIS Conference 41
*p ≤ .05, **p ≤ .01
2013 MIS Conference 42
MSAA ResultsGeneral Ed
• WSS-K higher for students not FaRMs, for girls, and for White students
• EIS students scored lower (1.3 M diff)
• Sig interaction EIS x Gender– F > M No EIS (1.9 M diff)– M > F EIS (4.2 M diff)
Special Ed• WSS-K higher for students
not FaRMs, for girls, and for White students
• EIS students scored lower (16.2 M diff)
• Sig interaction EIS x Gender– M > F No EIS (10.4 M diff)– M > F EIS (42.9 M diff)
2013 MIS Conference 43
General Ed EIS SAA Conclusions• Demographic Controls: WSS-K, RSAA, & MSAA
was higher for students not economically disadvantaged, higher for girls, and for White students
• EIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Gen Ed peers.
• EIS Interactions: WSS-K & RSAA no significant interactions; MSAA EIS x Gender (F EIS)
2013 MIS Conference 44
Special Ed EIS SAA Conclusions• Demographic Controls: WSS-K, RSAA, & MSAA
was higher for students not economically disadvantaged and for White students; females scored higher on WSS-K & MSAA
• EIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Sp Ed peers
• EIS Interactions: WSS-K & MSAA significant EIS x Gender (F EIS lower in K and more at Grade 3)
2013 MIS Conference 45
Youth in EIS have lower average scores than their peers in Gen Ed and Sp Ed
WSS K RSAA G3 MSAA G3 0
5
10
15
20
25
0.67
3.1
1.30.8
22.3
16.2
Gen EdSp Ed (HI)
Aver
age
Mea
n Di
ffere
nce
Betw
een
Non
EIS
and
EIS
2013 MIS Conference 46
Female EIS in Sp Ed (HI) Services have lower average scores than their peers
WSSK RSAA G3 MSAA G30
50
100
150
200
250
300
350
400
F EISM EISFM
2013 MIS Conference 47
Study 3: K Readiness and Grade 3 MSA Performance
Is performance on the WSS-K predictive of Grade 3 high stakes testing (Reading MSA and
Math MSA)?
2013 MIS Conference 48
Background
Maryland Model for School Readiness (MMSR) Pearson Work Sampling System Assessment
Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)• Personal and Social Development• Language and Literacy• Mathematical Thinking• Scientific Thinking• Social Studies• The Arts• Physical Development
2013 MIS Conference 49
Method• Participants/Procedure– General Ed Fall K Work Sampling System student scores
(2003, 2004, 2005) N = 152,105Matched to – Grade 3 Spring MSA Math, MSA Read student scores (2006,
2007, 2008) N = 100,958Match Rate 66% for Reading & Math
• Instruments– WSS is a 30 item instrument with scaled items coded 1 - 3– MSA is a high stakes test coded 1 - 3
2013 MIS Conference 50
Kindergarten MMSR Composite Score Distribution by Grade 3 MSA Performance Level
MSA N % WSS-K M WSS-K SD Math
Basic 7,876 17.71% 63.98 5.71 Proficient 55,777 55.25% 72.21 5.52 Advanced 27,304 27.05% 78.38 5.25
Total 100,957 100.00% 72.42 3.51
Reading Basic 17,695 17.52% 64.78 5.67 Proficient 64,907 64.28% 72.63 5.62 Advanced 18,376 18.20% 79.02 4.91
Total 100,978 100.00% 72.42 3.85
2013 MIS Conference 51
Matching Students K to Grade 3Kindergarten School Year
N WSS-K Scores
Grade 3 MSAA
Grade 3 MSAA
Grade 3 RSAA
Grade 3 RSAA
N Students Match Rate (%)
N Students Match Rate (%)
2002-2003 54,452 36,032 66.17% 36,026 66.16%
2003-2004 50,024 33,986 67.94% 34,008 67.98%
2004-2005 47,629 30,940 64.96% 30,944 64.97%
Total 152,105 100,958 66.37% 100,978 66.39%
2013 MIS Conference 52
Method
• Causal Comparative Descriptive Design• CART analysis– Binary decision trees representing a series of rules
that lead to membership in a class or value– Variables in the dataset are analyzed and split on a
value to best predict outcome – By following the splits in the decision tree to the
targeted outcome, observers can discern the patterns and combinations of variables that best predict either the presence or absence of the desired outcome
MIS DC 2013 Data Conference 53
Grade 3 Math Decision Tree
All Participants
MIS DC 2013 Data Conference 54
Grade 3 Math Decision Tree
IIA3: Begin Phonemic Aware
MIS DC 2013 Data Conference 55
Grade 3 Math Decision Tree
IIA3: Begin Phonemic Aware
IIA3: Begin Phonemic Aware
MIS DC 2013 Data Conference 56
Grade 3 Math Decision Tree
IIA3: Begin Phonemic Aware
IIIB1: Number & Quantity
IIA3: Begin Phonemic Aware
MIS DC 2013 Data Conference 57
Grade 3 Math Decision Tree
IIIC2:Recog/Dup/Extend Patterns
IIA3:Begin Phonemic Aware
IIA3:Begin Phonemic Aware
IIIB1: Number & Quantity
MIS DC 2013 Data Conference 58
Grade 3 Math Decision Tree
IIA3: Begin Phonemic Aware
IIIB1: Number & Quantity
IIA3: Begin Phonemic Aware
IIIC2: Recog/Dup/Extend Pattern
IIIA1: Strategy Use for Math
MIS DC 2013 Data Conference 59
Grade 3 Math Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIIB1: Number & Quantity IIIC2: Patterns
IIIA1: Strategy Use Math IIA1: Listening
MIS DC 2013 Data Conference 60
Grade 3 Math Decision Tree
IIA3: Phonemic Aware
IIIC2: Patterns
MIS DC 2013 Data Conference 61
Grade 3 Math Decision Tree
IIIC2: Patterns
IBI: Rules & Routines
IIA3: Phonemic Aware
MIS DC 2013 Data Conference 62
Grade 3 Math Decision Tree
IIIA1: Strategy Use for Math
IBI: Rules & Routines
IIIC2: Patterns
IIA3: Phonemic Aware
MIS DC 2013 Data Conference 63
Grade 3 Math Decision Tree
IIIA1: Strategy for Math
IBI: Rules & RoutinesIIIC2: Patterns
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIA1: ListeningIIIA1: Strategy for Math
IIIC2: Patterns
2013 MIS Conference 64
Math Decision Tree Nodes DescriptionsTerminal
NodeSegment: Predicted Outcome
# Students Average Math Score
Demonstrates beginning phonemic awareness
(IIA3)
Shows understanding of number and quantity (IIIB1)
Recognizes, duplicates, and
extends patterns (IIIC2)
Begins to use and explain strategies to
solve mathematical
problems (IIIA1)
Gains meaning by listening
(IIA1)
Follows classroom rules
and routines (IB1)
1 3 2,838 1.495 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3}
2 3 4,833 1.709 1 {2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}
3 3 2,173 1.723 {2,1} {1,2,3} {3,2} 1 {1,2,3} {1,2,3}
4 2 14,620 1.927 {2,1} {1,2,3} {3,2} {2,3} {1,2,3} {1,2,3}
5 2 4,647 2.027 {2,1} {1,2,3} 1 {1,2,3} {3,2} {1,2,3}
6 2 4,857 2.188 {2,1} {1,2,3} 1 {1,2,3} 1 {1,2,3}
7 2 4,091 2.143 1 {1,2,3} {1,3} {1,2,3} {1,2,3} {1,2,3}
8 1 3,908 2.246 1 {1,2,3} 1 {1,2,3} {1,2,3} {1,2}
9 1 4,164 2.324 1 {1,2,3} 1 {1,2,3} {1,2} 1
10 1 10,019 2.453 1 {1,2,3} 1 {1,2,3} 1 1
2013 MIS Conference 65
Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Math Model
SAA MathDemonstrates beginning phonemic awareness (IIA3)Shows understanding of number and quantity (IIIB1)Recognizes, duplicates, and extends patterns (IIIC2)Begins to use and explain strategies to solve mathematical problems (IIIA1)Gains meaning by listening (IIA1)Follows classroom rules and routines (IB1)
2013 MIS Conference 66
Grade 3 Math Score Distribution: Comparing Training & In Time Validation
Predicted Outcome
Training Average
MSA Score
In Time Average
MSA Score
Actual Training Outcome: % Students
Per Segment
Actual In Time Validation Outcome:
% Students Per Segment
1 (Basic)
2 (Proficient)
3 (Advanced)
1 (Basic)
2 (Proficient)
3 (Advanced)
Advanced 435 434 6.16% 49.80% 44.04% 6.99% 48.34% 44.67%
Proficient 411 411 19.08% 59.88% 21.04% 19.14% 59.48% 21.38%
Basic 385 384 42.05% 50.82% 7.13% 43.63% 49.50% 6.88%
Total 414 414 18.95% 55.04% 26.01% 19.46% 54.17% 26.37%
2013 MIS Conference 67
Conclusion
• Fall WSS-K is a moderately successful predictor of later standardized Math test performance
MIS DC 2013 Data Conference 68
Grade 3 Reading Decision Tree
MIS DC 2013 Data Conference 69
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
MIS DC 2013 Data Conference 70
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
MIS DC 2013 Data Conference 71
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIIA1: Strategy f/Math
MIS DC 2013 Data Conference 72
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIIA1: Strategy f/Math IIA1: Listening
MIS DC 2013 Data Conference 73
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIIA1: Strategy f/Math IIA1: Listening
IIIA1: Strategy f/Math
MIS DC 2013 Data Conference 74
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IIA3: Phonemic Aware
IIIA1: Strategy f/Math IIA1: Listening
IIIA1: Strategy f/Math IIIB1: Number & Quantity
MIS DC 2013 Data Conference 75
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IB1: Rules & Routines
MIS DC 2013 Data Conference 76
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IB1: Rules & Routines
IIC2: Print Concepts
MIS DC 2013 Data Conference 77
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IB1: Rules & Routines
IIIC2: Patterns
IIB1: Effective Comm
MIS DC 2013 Data Conference 78
Grade 3 Reading Decision Tree
IIA3: Phonemic Aware
IB1: Rules & Routines
IIIC2: Patterns
IIB1: Effective Comm
IIA3: Phonemic Aware
IIIA1: Strategy f/Math IIA1: Listening
IIIA1: Strategy f/ Math IIIB1: Number & Quantity
2013 MIS Conference 79
Read Decision Tree Nodes DescriptionsTerminal
NodeSegment: Predicted Outcome
# Students Average Reading Score
Demonstrates beginning phonemic awareness
(IIA3)
Begins to use and explain strategies to
solve mathematic
problems (IIIA1)
Gains meaning by
listening (IIA1)
Shows understanding of number and quantity
(IIIB1)
Follows classroom rules and
routines (IB1)
Shows some understanding of concepts about print
(IIC2)
Speaks clearly and conveys
ideas effectively
(IIB1)
1 3 4,164 1.566 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}
2 3 3,466 1.708 3 {2,1} {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}
3 3 2,176 1.732 2 3 {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3}
4 2 15,345 1.875 2 {2,3} {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3}
5 2 4,419 1.949 2 {1,2,3} 1 {1,3} {1,2,3} {1,2,3} {1,2,3}
6 2 4,274 2.084 2 {1,2,3} 1 1 {1,2,3} {1,2,3} {1,2,3}
7 2 5,384 2.083 1 {1,2,3} {1,2,3} {1,2,3} {1,2} {1,2,3} {1,2,3}
8 2 2,609 2.123 1 {1,2,3} {1,2,3} {1,2,3} 1 {1,2} {1,2,3}
9 2 1,601 2.156 1 {1,2,3} {1,2,3} {1,2,3} 1 1 {3,2}
10 1 12,545 2.307 1 {1,2,3} {1,2,3} {1,2,3} 1 1 1
2013 MIS Conference 80
Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Reading Model
SAA ReadingDemonstrates beginning phonemic awareness (IIA3)*Gains meaning by listening (IIA1)*Begins to use and explain strategies to solve mathematical problems (IIIA1)*Shows understanding of number and quantity (IIIB1)*Follows classroom rules and routines (IB1)*Recognizes, duplicates, and extends patterns (IIIC2)*Speaks clearly and conveys ideas effectively (IIB1)
*also significant predictors in MSA math model
2013 MIS Conference 81
Grade 3 Reading Score Distribution: Comparing Training & In Time Validation
Predicted Outcome
Training Average
MSA Score
In Time Average
MSA Score
Actual Training Outcome: % Students
Per Segment
Actual In Time Validation Outcome:
% Students Per Segment
1 (Basic)
2 (Proficient)
3 (Advanced)
1 (Basic)
2 (Proficient)
3 (Advanced)
Advanced
444
444 5.35% 58.62% 36.03% 5.50% 58.44% 36.06%
Proficient
421
421 17.94% 66.41% 15.65% 17.87% 66.32% 15.81%
Basic
398
398 39.03% 56.66% 4.31% 38.27% 57.89% 3.84%
Total
422
422 18.81% 62.96% 18.23% 18.76% 63.08% 18.16%
2013 MIS Conference 82
Additional Analyses/Conclusions
• Sex– For math and reading, similar performance
• Ethnicity– For math and reading, Whites and Asian/Pac Islanders
realized predicted performance at higher levels; African-American, Hispanic, & Native American realized predicted performance at lower levels
• FaRMs– For math and reading, student receiving FaRMs realized
predicted performance at lower levels
2013 MIS Conference 83
Conclusions • Fall K WSS is a modestly successful predictor of later
standardized test performance• Math SAA performance was better predicted than
Reading SAA• Overidentification of middle group (Proficient)• Findings support the importance of early literacy
skills• Findings support the persistent ‘gap’ in student
achievement, evident from K
2013 MIS Conference 84
Summary QuestionsAre early intervention services effective in preparing
children to enter kindergarten?Value of EIS?Continue funding of Birth to five model?
Is level of K preparedness predictive of grade 3 high stakes testing performance?Value of high quality preK programs (QRIS)?Fund additional preK programs?
Are early intervention services effective in preparing children to be successful in school?Value of EIS?Further tracking studies looking at child outcomes.