Data Based Decision Making

Measuring Responsiveness to Intervention

What is a significant change in performance?

• One that results in meaningful change.• One that significantly reduces the gap.• One that meets your goal.

What is a non-responder?• Median Split of slop on standardized assessment instruments

(e.g. Vellutino, 1996; WRMT)

• Normalized Scores (e.g. Torgeson, 2001; 90 or better on WRMT)– Most Lenient

• Dual Discrepancy (Speece and Case, 2001; Slope and level = 1sd below mean)

• End of year benchmark (e.g. Good, 2001)– Most stringent

Dual Discrepancy

• Discrepant from peers at data collection point 1 (e.g. fall benchmark)

• Discrepancy continues or becomes larger at point 2 (e.g. winter benchmark)– This is referred to a student’s rate of improvement

(ROI)

2nd Grade Benchmarks

0

5

10

15

20

25

30

35

40

0 1 2 3

BENCHMARK SESSIONS

J im

District Benchmarks

State Discrepancy

• Be objective. Does it refer to an observable characteristic of behavior?

• Be clear. Can others read the discrepancy statement and observe it easily?

• Calculate the discrepancy ratio– Include statement of student’s current level of

performance.– Include statement of the expected level of

performance (e.g., peer data, teacher expectation).

Discrepancy Ratios:Elementary Example

Helps to quantify how many times the student’s current level of performance varies from that of his/her peers.

– In order to calculate a discrepancy ratio use the following formula: Peer Behavior Target Student Behavior

– Example:When given a 4th grade probe, Jessica is reading 80 correct words per minute while average 4th grade peers are reading 145 correct words per minute.

Peer Behavior = 145 = 1.81x Target Student Behavior 80

Discrepancy Ratios

Enables team to make decisions about levels of support and resource from the start.Generally speaking…

– A student who is 1.5x discrepant from his/her peers is appropriate for the problem-solving team.

– If a student is significantly discrepant from peers, additional problem-solving and intervention resources may be appropriate.

– Example: Jessica is 1.81 x discrepant from peers and MAY benefit from problem solving.

Discrepancy Ratios:Secondary Example-Negative Behavior

Helps to quantify how many times the student’s current level of performance varies from that of his/her peers.

– In order to calculate a discrepancy ratio for undesirable behavior use the following formula:

Target Student BehaviorPeer Behavior

– Example: Jessica has been disruptive 15 times per week while the average 10th grade peer is disruptive 3 times per week.

Target Student Behavior = 15 = 5x Peer Behavior 3

Provides a way to evaluate student outcomes and the effectiveness of an intervention to reduce initial performance discrepancies.

Name Grade Area Initial Performance Discrepancy

Follow Up Performance Discrepancy

Rate of Progress

Outcome Decision

Bill 3 Reading 3.5X 2.2X 2.3 WRC per week

Satisfactory Maintain Intervention

Susie 6 Math 1.2X NA NA No Severe Problem

Jess 10 Behavior 5X 4.8X .2x change

Insufficient Progress, Recycle through process

What is significant Discrepancy?

• 1.5 times discrepant• .5 SD• 25th percentile• 90% probability not passing high stakes

testing– criterion based decision making

Writing Goals and Objectives

Goals/ObjectivesWhat are Instructional Objectives and How are they Used?

A. Behavioral Objectives: Statement of skill that students are expected to know at the end of some period of instruction.

* Method of measurement should also be stated in meaningful observable methods.* Should state: Performance, condition, criteria and when possible the date.

e.g. Given a worksheet containing 20 3 digit by 2 digit multiplication problems that require carrying, students will be able to complete all of them with 90% accuracy within 20 minutes.

Goal Setting

• Based on last three data points • Based on split plot• Based on intra-individual performance

(Addition predicting subtraction)• Based on group of students (benchmarking

and rate of improvement)• Based on team decision

Determining Long range Goal

• Multiply number of weeks that you will be monitoring by the criterion (Expected ROI).

• Add this number to the median baseline point• Example:

– Median baseline point = 35– Number of weeks = 10– Expected rate of growth (based on norms or

suggestion)

0

5

10

15

20

25

30

35

40

45

50

55

60

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

WEEKS

WR

CP

MBaseline Intervention

Writing IEP Goals

• Long range GoalIn ___ (total # weeks) when presented with

math problems form (curriculum and grade level) ____ (Student’s name) will perform ____(long range goal) with _____ errors or fewer.

Writing IEP Goals

• Short term objective Each successive week, when presented with a

random selection from _____ (curriculum and grade level) ____ (Student’s name) will perform at an average increase of _____ DCM and no increase in errors.

Progress Monitoring

How long to implement intervention?

• 10-15 weeks– Perhaps a quarter system approach?

How long before tweaking?

• 3 data points: Above or below line?• 4 Data points: 3 weeks of instruction and 6 points of

data collected: Examine 4 most recent points• Trend line: 4 weeks of instruction and 8 data points

collected construct trend line• Remember:

– Stability– Trend– Level

Time to Triage

• What do you with:– Jason, 8 year old African American male who is 1.

7 x discrepant from grade level peers in math and 1.5 times discrepant from grade level peers in reading? Grades are below average in both subject areas. Jason is at the 40th percentile in reading and math within his classroom.

Decision Making Time

• What do you do with:– Jason, after 10 weeks of Standard Protocol

Intervention and his rate of improvement (ROI) is 2wcpm in reading compared to 1.5 wcpm ROI in reading but he is still only at the 40th percentile in reading compared to grade level peers?

Time to Triage

• What do you with:– Jason, 8 year old African American male who is 1.

7 x discrepant from grade level peers in math and 1.5 times discrepant from grade level peers in reading? Grades are below average in both subject areas. Jason is at the 15th percentile in reading and math within his classroom.

Decision Making Time

• What do you do with:– Jason, after 10 weeks of Standard Protocol

Intervention and his rate of improvement (ROI) is 1.25 wcpm in reading compared to 2.0 wcpm in reading and he is still only at the 15th percentile in reading compared to grade level peers?

Generic Problem Solving Model

• Problem Identification• Problem Analysis• Problem Definition• Goal Setting• Intervention Selection• Intervention Implementation• Intervention Evaluation

Problem Identification

• Problems in general area: (e.g., reading, math, writing, spelling)

• Compare to same grade peers• If discrepant from peers then consider

problem analysis.

Problem Analysis

• Probe/assessment to determine basic skills that demonstrate rate and/or accuracy issues

• Specificity• Consider RIOT and ICEL and IPF

Problem Definition

• Must contain:– Measurable and observable language

(e.g., number problems per minute)– Specify conditions (e.g. worksheet, verbal?)– Specify level

(e.g., percent, rate, compared to peers)– Specify date

Writing a Hypothesis• Provide the discrepancy statement• Add because… at the end of the discrepancy statement and

insert your hypothesis.• The hypothesis should be specific, observable, and

measurable.

– Example: Beth is on-task for 35% of intervals while peers are on-task 87% of

intervals during a 20-minute observation during direct instruction in Math class, because she is escaping the Math work which is above her instructional level.

Goal Setting

• Often overlooked• Should be based on problem definition• May include norms, teacher input• Should specify date deadline• Should specify level cutoff• Should be done before considering

interventions

Intervention Selection

• Should be linked to the assessment• Likely to focus on basic learning/behavioral

principles• Should have empirical support and/or be

strongly influenced by research• Keystone intervention, effort, time, difficulty,

acceptability

Intervention Implementation

• Integrity is a key issue

Intervention Evaluation

• Did you meet your goal?*If yes, then intervention effective*If no, then not educationally significantReturn to intervention selection

• Frequent progress monitoring is critical

A Systems-Level Problem

A team at‘Cardinal School’ noticed that approximately twice the number of referrals for Special Education consideration for difficulty in reading had come from the 3rd grade during the first four months of school than in the prior three years.

Step 1: Problem IdentificationQuestion: What is the discrepancy between what is expected and

what is occurring?A. List problem behavior(s) and prioritize.B. Collect baseline data on primary area of concern (target

student and peer). • Record Review• Interview• Observation• Testing

C. State discrepancy between target student performance and peer performance.

sam

plin

g of

stu

den

tsal

l stu

den

ts in

clu

ded

Student TeacherFall WRC

Winter WRC

Winter Percentile

Rank ClassificationRate of Progress

Average Rate of Progress

S, A Smith 209 208 1.00 Well Above Average -0.1 1.3K, D Jones 159 170 0.93 Well Above Average 0.6 1.3F, M Smith 134 156 0.90 Above Average 1.2 1.3H, A Smith 130 148 0.81 Above Average 1.0 1.3E, S Smith 115 145 0.75 Average 1.7 1.3P, A Jones 96 133 0.68 Average 2.1 1.3K, C Jones 109 114 0.51 Average 0.3 1.3S, D Armstrong 66 112 0.46 Average 2.6 1.3B, C Armstrong 92 94 0.36 Average 0.1 1.3E, A Armstrong 61 80 0.25 Average 1.1 1.3A, B Smith 39 65 0.24 Below Average 1.4 1.3R, P Armstrong 42 63 0.22 Below Average 1.2 1.3M, W Jones 50 60 0.20 Below Average 0.6 1.3G, S Jones 28 58 0.19 Below Average 1.7 1.3J, J Smith 20 54 0.17 Below Average 1.9 1.3M, A Smith 38 51 0.15 Below Average 0.7 1.3B, J Jones 47 48 0.14 Below Average 0.1 1.3P, M Smith 47 45 0.10 Below Average -0.1 1.3A, D Armstrong 38 45 0.10 Below Average 0.4 1.3M, T Jones 42 41 0.08 Well Below Average -0.1 1.3D, Z Armstrong 31 39 0.07 Well Below Average 0.4 1.3M, M Smith 30 38 0.03 Well Below Average 0.4 1.3D, A Jones 18 38 0.03 Well Below Average 1.1 1.3K, A Armstrong 8 21 0.02 Well Below Average 0.7 1.3A, J Jones 7 18 0.00 Well Below Average 0.6 1.3

Problem Identification Data

0

20

40

60

80

100

120

140

160

THIRD

S ECOND

FIRST

Box PlotSplit By: Grade

Winter WRC Grade 1

Winter WRC Grade 2

Winter WRC Grade 3

Problem Identification

Statement of Discrepancy: 15 of the 3rd grade students fall below the 25th percentile in reading fluency. Of those, 12 are also not making adequate rates of progress.

Problem AnalysisThrough the Years: Class of 2012

0

20

40

60

80

100

120

140

160

THIRD

S ECOND

FIRST

Box PlotSplit By: Grade

Winter WRC Grade 1

Winter WRC Grade 2

Winter WRC Grade 3

Problem AnalysisCurrent Grade Level Data

0

20

40

60

80

100

120

140

160

THIRD

S ECOND

FIRST

Box PlotSplit By: Grade

Winter WRC Grade 1

Winter WRC Grade 2

Winter WRC Grade 3

Investigate WHY the problem exists:

- Did the referred students exhibit reading difficulties before the 3rd-grade?- Yes, review of CBM data indicate that the

referred students were roughly the same lowest performing group in 1st and 2nd grade.

- Do the current 1st and 2nd graders show a similar pattern?- Yes, CBM data from the current year indicate

groups of 1st and 2nd grade students not making adequate rates of progress.

Problem Analysis

A significant portion of 3rd grade students are not making adequate rates of progress in reading BECAUSE…..

Not all students established satisfactory reading trajectories during Kindergarten and 1st grade BECAUSE?

Kindergarten Instructional Planning Form

Activity Materials Arrangement TimeMotivational

Strategy

•Language Exposure

•Books•Whole Group

•Teacher Led

•50 min / wk

•Praise for attention

•Letter Naming

•Manipulatives•Books

•Worksheets

•Whole Group

•Small Group•Independent

•30 min / wk

•Reminding

•Independent Reading

•Books •Individual•20 min /

wk

•Praise for appropriate

behavior

First Grade Instructional Planning Form

Activity Materials Arrangement TimeMotivational

Strategy

•Silent Reading

•Books •Independent •10 min / day•Praise for appropriate

behavior

•Choral Reading

•1st Grade Teacher

•Whole Group •10 min / day•Verbal

Feedback

•Word Walls •Word Cards •Whole Group •10 min / day•Praise for

participating

A significant proportion of 3rd grade students are not making adequate rates of progress in reading BECAUSE……..

• Not all students established satisfactory reading trajectories during Kindergarten & 1st grade BECAUSE…….

• Current early elementary reading curriculum places little focus on systematic pre-literacy skill instruction (i.e. phonemic awareness and phonics).

To change trajectories, we must intervene systematically, strategically, and early.