Data Driven Decisions Moving from 3D to D 3. Data Driven Decisions Moving from 3D to D 3 Malcolm...

Data Driven DecisionsData Driven DecisionsMoving from 3D to DMoving from 3D to D33

Data Driven Decisions

Data Driven Decisions

Moving from 3D to DMoving from 3D to D33

Malcolm ThomasMalcolm Thomas

Director, Evaluation ServicesDirector, Evaluation Services

Escambia School DistrictEscambia School District

Presentation AgendaPresentation Agenda

3D Data Driven Decisions - ReviewPower of Data AnalysisMoving to D3 - Formative AssessmentsReview of 3D Resources

Malcolm Thomas

mthomas@escambia.k12.fl.us

(850) 469-5386

www.escambia.k12.fl.us/adminoff/eval

How much water is impacted?

“Insanity is doing the same thing over and over expecting a different result.”Albert Einstein

One of the first steps:

Develop a common vocabulary that will lead to common understanding.

Digging In the Data

NRT

CRT

Scale Score

DVSS NCE

Stanine Raw Score

Percentile

NPR

FCAT

1.1. Norm Referenced Test (NRT)Norm Referenced Test (NRT)

2.2.Criterion Referenced Test (SSS)Criterion Referenced Test (SSS)

Administered to all students in Administered to all students in

Grades 3 - 10Grades 3 - 10

NRT Score Types• Scale Scores

– Use to compare year to year and to interpret trends in performance

• National Percentile Rank (NPR) – % in national sample scoring at or above student (ranks)

NOT equal interval

• Normal Curve Equivalent (NCE)– Equal interval scale from 1 to 99 with average of 50

IS equal interval

• Stanine– Scores divided into 9 equal groups– Scores of 4 – 6 are considered average

Normal Distribution Curve“A Band of Scores”

68%

0-1 1-2-3 2 3

95%

99%

MEAN

= Standard Deviation

what test takers can do and

what to others

report how well students are

doing relative to a pre-

determined performance level

on a specified set of educational

goals or outcomes

FCAT CRT Scores (SSS)

Scale Scores 100 - 500

0

50100

150200

250

300350

400450

500

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade10

• Scale scores have no meaning in isolation

• Scale Scores are used to determine achievement levels

• FCAT CRT (SSS) Scale score range is 100 - 500

FCAT SSS Scale Scores

FCAT Developmental Scale Scores (DVSS)

• Created to help understand students' year-to-year progress.

• DVSS range from 86 to 3008. • Students should receive higher scores as they move

from grade-to-grade according to their increased achievement.

• Use DVSS to monitor academic progress each year.

DVSS Linear Progression

0

500

1000

1500

2000

2500

3000

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade10

Let’s Review Scores ByTypes by Test

NRT• Raw Score• Scale Score• NPR• NCE• Stanine

CRT (SSS)• Raw Score• Scale Score• DVSS• Achievement Level

Learning Gains

• Measure individual student progress from year to year

• Uses the Developmental Scale Score and Achievement Level

• Compares current scores to scores of previous year

Making a Learning Gain…1. Improve FCAT at least one

achievement level (example move from Level 1 to Level 2 or Level 3 to Level 5)

2. Maintain satisfactory achievement levels (Levels 3, 4, 5)

3. Demonstrate more than one year’s growth within Level 1 or Level 2 (using DVSS Gains Table)

Retained students MUST increase 1 achievement level or maintain satisfactory achievement level (cannot use DVSS method for retained students)

Developmental Scale GainGrade Reading Math

3 to 4 230 162

4 to 5 166 119

5 to 6 133 95

6 to 7 110 78

7 to 8 92 64

8 to 9 77 54

9 to 10 77 48

Developmental Scale Scores must be at least one point greater than the above cut-off scores for a student to make a Learning Gain. This method of Learning Gain calculation only applies to promoted students maintaining achievement 1 or 2.

Developmental Scale Scores (DVSS) for consecutive Years

NRT NCE scores for consecutive years

Learning Gains for consecutive years

Monitoring Student Monitoring Student Progress …Progress …

Content Clusters

The way the FCAT SSS breaks down the subject area tests.

The number of items varies across disciplines and grade levels.

SSS Scores are used in School Grade Calculations

Changes are coming to Changes are coming to

School Grade calculations in School Grade calculations in

20072007

School grade will consist of a total of eight components in 2007

• Four (4) Proficiency components: – Reading– Math– Writing– Science **NEW**

• Four (4) Learning Gain Components: – Reading Learning Gain – Lowest 25% Reading Learning Gain– Math Learning Gain– Lowest 25% Math Learning Gain **NEW**

Revised School Grading Scale Grade 2006

Points Required

(Points possible = 600)

2007Points Required

(Points possible = 800)

A 410+ 525+

B 380-409 495-524

C 320-379 435-494

D 280-319 395-434

F <280 <395

Assess the current and future needs of students

Decide what to change

Determine if goals are being met

Engage in continuous school improvement

Identify root causes of problems

Promote accountability

Leadership is the key …Leadership is the key …

Source: Brandeis University

BARRIERS TO DATA ANALYSIS

• Lack of training in data use

• No uniform data collection

• Lack of leadership at the school and district level

• Outdated technology

• Unclear priorities• Distrust of data

use

Elements for Effective Use of Data• Create collaborative / trusting

relationships• Engage in data driven dialogue and

collaborative inquiry• Learn what you can from standardized

tests• Use multiple measures• Examine data in variety of ways

Collaborative Inquiry

Questions usually lead to ….

. . . more questions and more data collection.

When analyzing dataA Picture is Worth …

Collect Multiples Measures• Example:

– Grade 3 Reading FCAT Scale Scores– Grade 3 Reading National Percentile

Ranks

– Or Grade 3 Reading FCAT Achievement Level Percentages for multiple years

Multiple Years of Data

Escambia Reading Level 1 Student Count Grade 3

0

250

500

750

1000

AL 1

AL 1 879 899 758 882 738 492

2001 2002 2003 2004 2005 2006

Escambia Reading Level 1 Percentages Grade 3

0

25

50

75

100

AL 1

AL 1 26 27 23 24 23 15

2001 2002 2003 2004 2005 2006

Comparing FCAT SSS Reading to Reading NRT

Approximate 2005 FCAT Reading NRT National Percentiles for Achievement Level Cut Points

AL1

AL2

AL3

AL4

AL5

0

25

50

75

100

NP

R

AL1 21 22 30 28 35 41 49 52

AL2 32 33 48 48 53 73 75 79

AL3 62 65 74 74 78 94 93 91

AL4 91 90 92 94 94 98 96 95

AL5 100 100 100 100 100 100 100 100

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

DVSS 1926 = NPR 60

Other Examples of Multiple MeasuresCompare Student Reading or Math Scores with:

• Averages of State, District or School

• Attendance Data

• Achievement Scores And Discipline Referrals

• GPA

• Content Areas By Grade Level

FCAT Myths to Avoid

• Performance for individual student assessments should remain the same or improve

• FCAT Proficiency equals grade level

• FCAT Proficiency levels are the same for each grade level

• All data are accurate

Moving ...

From 3D . . . Data Driven From 3D . . . Data Driven

Decisions to . . . Decisions to . . .

DD33 Data Driven Decisions Data Driven Decisions

Summative Assessments

• Summative assessments attempt to summarize student learning at a specific point in time – Most standardized tests are summative– Excellent tool for shaping district / school or

course goals• Major disadvantages

– time lag between assessment and results– lack of specific information that can be used by

teachers for instruction

Formative Assessments

• Occurs when teachers provide information to students in ways that enable the student to learn better, or when students can engage in a similar, self- reflective process

• Primary purpose support high quality learning• Most effective for lower performing students;

although all students benefit

Formative Assessments

• Major Advantages– Can occur in real time– Maximize teachable moments in the

classroom– Both teachers and students receive

information necessary for improvement

Profound Mistake

• Students NOT included as active participants in the assessment process in order for achievement to improve

Remedies

Create a balanced system of summative assessments of learning and formative assessments for learning

A Balanced Approach

Assessment of Learning

How much have students learned as of a particular point in time?

Assessment for Learning

How can we use assessment to help students learn more?

High Quality Classroom Assessment

Teachers should be asking:

• Why am I assessing?

• What am I assessing?

• What is the best assessment method?

• How do I communicate the assessment results?

Assessments Tools, strategies, and techniques to analyze each student’s demonstration of accomplishments of specific

goals and criteria.

• Paper-pencil tests• Exhibits• Interviews• Surveys• Observations• Other measures

Good assessment requires variety to ensure opportunities for success for all students.

Would you rather be at the beach?

Traditional Assessment Methods

• Selected response

• Extended written response

• Performance assessment

• Personal oral communication

Selected Response

Students select an answer from a list or generate a very brief answer.

Examples:Multiple ChoiceTrue / FalseMatchingFill in the blankLabel a diagram

Advantages of Multiple Choice

When there is only one correct answer with several plausible alternatives to the correct answer, MC is an effective tool to:

• Cover a variety of material effectively

• Be scored easily

• Provide formative information

Multiple Choice – D3

• Provide one, and only one, correct answer

• Include plausible options that demonstrate a student’s level of understanding

• Maintain a homogeneous feel in style, length, and visual display

• Colleague review

School Bus - Item

A teacher is planning a field trip and will need school buses to transport students. A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?

A. 31 C. 32

B. 31.33 D. 36

School Bus - Item

A teacher is planning a field trip and will need school buses to transport students. A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed? Explain your answer.

A. 31 C. 32B. 31.33 D. 36

Begin with the end in mind . . .

• MC assessment could be designed so that each stem presents a different level or skill set

• Subsequent assignments could be generated based on the students response to the MC item

Formative Assessments

• Cautions– Must be as rigorous as summative

assessment (FCAT) – Data must be collected and analyzed– Data must be used to create future

instruction

Which Type of assessment has the greatest potential to alter classroom instruction?

Which assessment will produce data driven decisions in the classroom?

D3 = Effective Formative Assessment System

Moving to D3: LESSONS LEARNED

• It takes time.• It has to start at the top.• Progress has to be measurable. • Summative models are starting points.

• Data–driven decision making can be a powerful tool in changing student outcomes and promoting continuous improvement.

70 Years ago ….

H. L. Menchen said:

“ For every complex problemthere is an answerthat is clear,simple and …….

wrong.”

Tools for Data Analysis

• Evaluation Services Web Site– Tools– Documents– Summative data and charts– Student level disaggregation

- secure access (password required)

Malcolm Thomas

mthomas@escambia.k12.fl.us

(850) 469-5386

www.escambia.k12.fl.us/adminoff/eval