Embed Size (px)
Citation preview
Data Driven DecisionsData Driven DecisionsMoving from 3D to DMoving from 3D to D33
Data Driven Decisions
Data Driven Decisions
Moving from 3D to DMoving from 3D to D33
Malcolm ThomasMalcolm Thomas
Director, Evaluation ServicesDirector, Evaluation Services
Escambia School DistrictEscambia School District
Presentation AgendaPresentation Agenda
3D Data Driven Decisions - ReviewPower of Data AnalysisMoving to D3 - Formative AssessmentsReview of 3D Resources
How much water is impacted?
“Insanity is doing the same thing over and over expecting a different result.”Albert Einstein
One of the first steps:
Develop a common vocabulary that will lead to common understanding.
Digging In the Data
NRT
CRT
Scale Score
DVSS NCE
Stanine Raw Score
Percentile
NPR
FCAT
1.1. Norm Referenced Test (NRT)Norm Referenced Test (NRT)
2.2.Criterion Referenced Test (SSS)Criterion Referenced Test (SSS)
Administered to all students in Administered to all students in
Grades 3 - 10Grades 3 - 10
NRT Score Types• Scale Scores
– Use to compare year to year and to interpret trends in performance
• National Percentile Rank (NPR) – % in national sample scoring at or above student (ranks)
NOT equal interval
• Normal Curve Equivalent (NCE)– Equal interval scale from 1 to 99 with average of 50
IS equal interval
• Stanine– Scores divided into 9 equal groups– Scores of 4 – 6 are considered average
Normal Distribution Curve“A Band of Scores”
68%
0-1 1-2-3 2 3
95%
99%
MEAN
= Standard Deviation
what test takers can do and
what to others
report how well students are
doing relative to a pre-
determined performance level
on a specified set of educational
goals or outcomes
FCAT CRT Scores (SSS)
Scale Scores 100 - 500
0
50100
150200
250
300350
400450
500
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade10
• Scale scores have no meaning in isolation
• Scale Scores are used to determine achievement levels
• FCAT CRT (SSS) Scale score range is 100 - 500
FCAT SSS Scale Scores
FCAT Developmental Scale Scores (DVSS)
• Created to help understand students' year-to-year progress.
• DVSS range from 86 to 3008. • Students should receive higher scores as they move
from grade-to-grade according to their increased achievement.
• Use DVSS to monitor academic progress each year.
DVSS Linear Progression
0
500
1000
1500
2000
2500
3000
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade10
Let’s Review Scores ByTypes by Test
NRT• Raw Score• Scale Score• NPR• NCE• Stanine
CRT (SSS)• Raw Score• Scale Score• DVSS• Achievement Level
Learning Gains
• Measure individual student progress from year to year
• Uses the Developmental Scale Score and Achievement Level
• Compares current scores to scores of previous year
Making a Learning Gain…1. Improve FCAT at least one
achievement level (example move from Level 1 to Level 2 or Level 3 to Level 5)
2. Maintain satisfactory achievement levels (Levels 3, 4, 5)
3. Demonstrate more than one year’s growth within Level 1 or Level 2 (using DVSS Gains Table)
Retained students MUST increase 1 achievement level or maintain satisfactory achievement level (cannot use DVSS method for retained students)
Developmental Scale GainGrade Reading Math
3 to 4 230 162
4 to 5 166 119
5 to 6 133 95
6 to 7 110 78
7 to 8 92 64
8 to 9 77 54
9 to 10 77 48
Developmental Scale Scores must be at least one point greater than the above cut-off scores for a student to make a Learning Gain. This method of Learning Gain calculation only applies to promoted students maintaining achievement 1 or 2.
Developmental Scale Scores (DVSS) for consecutive Years
NRT NCE scores for consecutive years
Learning Gains for consecutive years
Monitoring Student Monitoring Student Progress …Progress …
Content Clusters
The way the FCAT SSS breaks down the subject area tests.
The number of items varies across disciplines and grade levels.
SSS Scores are used in School Grade Calculations
Changes are coming to Changes are coming to
School Grade calculations in School Grade calculations in
20072007
School grade will consist of a total of eight components in 2007
• Four (4) Proficiency components: – Reading– Math– Writing– Science **NEW**
• Four (4) Learning Gain Components: – Reading Learning Gain – Lowest 25% Reading Learning Gain– Math Learning Gain– Lowest 25% Math Learning Gain **NEW**
Revised School Grading Scale Grade 2006
Points Required
(Points possible = 600)
2007Points Required
(Points possible = 800)
A 410+ 525+
B 380-409 495-524
C 320-379 435-494
D 280-319 395-434
F <280 <395
Assess the current and future needs of students
Decide what to change
Determine if goals are being met
Engage in continuous school improvement
Identify root causes of problems
Promote accountability
Leadership is the key …Leadership is the key …
Source: Brandeis University
BARRIERS TO DATA ANALYSIS
• Lack of training in data use
• No uniform data collection
• Lack of leadership at the school and district level
• Outdated technology
• Unclear priorities• Distrust of data
use
Elements for Effective Use of Data• Create collaborative / trusting
relationships• Engage in data driven dialogue and
collaborative inquiry• Learn what you can from standardized
tests• Use multiple measures• Examine data in variety of ways
Collaborative Inquiry
Questions usually lead to ….
. . . more questions and more data collection.
When analyzing dataA Picture is Worth …
Collect Multiples Measures• Example:
– Grade 3 Reading FCAT Scale Scores– Grade 3 Reading National Percentile
Ranks
– Or Grade 3 Reading FCAT Achievement Level Percentages for multiple years
Multiple Years of Data
Escambia Reading Level 1 Student Count Grade 3
0
250
500
750
1000
AL 1
AL 1 879 899 758 882 738 492
2001 2002 2003 2004 2005 2006
Escambia Reading Level 1 Percentages Grade 3
0
25
50
75
100
AL 1
AL 1 26 27 23 24 23 15
2001 2002 2003 2004 2005 2006
Comparing FCAT SSS Reading to Reading NRT
Approximate 2005 FCAT Reading NRT National Percentiles for Achievement Level Cut Points
AL1
AL2
AL3
AL4
AL5
0
25
50
75
100
NP
R
AL1 21 22 30 28 35 41 49 52
AL2 32 33 48 48 53 73 75 79
AL3 62 65 74 74 78 94 93 91
AL4 91 90 92 94 94 98 96 95
AL5 100 100 100 100 100 100 100 100
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
DVSS 1926 = NPR 60
Other Examples of Multiple MeasuresCompare Student Reading or Math Scores with:
• Averages of State, District or School
• Attendance Data
• Achievement Scores And Discipline Referrals
• GPA
• Content Areas By Grade Level
FCAT Myths to Avoid
• Performance for individual student assessments should remain the same or improve
• FCAT Proficiency equals grade level
• FCAT Proficiency levels are the same for each grade level
• All data are accurate
Moving ...
From 3D . . . Data Driven From 3D . . . Data Driven
Decisions to . . . Decisions to . . .
DD33 Data Driven Decisions Data Driven Decisions
Summative Assessments
• Summative assessments attempt to summarize student learning at a specific point in time – Most standardized tests are summative– Excellent tool for shaping district / school or
course goals• Major disadvantages
– time lag between assessment and results– lack of specific information that can be used by
teachers for instruction
Formative Assessments
• Occurs when teachers provide information to students in ways that enable the student to learn better, or when students can engage in a similar, self- reflective process
• Primary purpose support high quality learning• Most effective for lower performing students;
although all students benefit
Formative Assessments
• Major Advantages– Can occur in real time– Maximize teachable moments in the
classroom– Both teachers and students receive
information necessary for improvement
Profound Mistake
• Students NOT included as active participants in the assessment process in order for achievement to improve
Remedies
Create a balanced system of summative assessments of learning and formative assessments for learning
A Balanced Approach
Assessment of Learning
How much have students learned as of a particular point in time?
Assessment for Learning
How can we use assessment to help students learn more?
High Quality Classroom Assessment
Teachers should be asking:
• Why am I assessing?
• What am I assessing?
• What is the best assessment method?
• How do I communicate the assessment results?
Assessments Tools, strategies, and techniques to analyze each student’s demonstration of accomplishments of specific
goals and criteria.
• Paper-pencil tests• Exhibits• Interviews• Surveys• Observations• Other measures
Good assessment requires variety to ensure opportunities for success for all students.
Would you rather be at the beach?
Traditional Assessment Methods
• Selected response
• Extended written response
• Performance assessment
• Personal oral communication
Selected Response
Students select an answer from a list or generate a very brief answer.
Examples:Multiple ChoiceTrue / FalseMatchingFill in the blankLabel a diagram
Advantages of Multiple Choice
When there is only one correct answer with several plausible alternatives to the correct answer, MC is an effective tool to:
• Cover a variety of material effectively
• Be scored easily
• Provide formative information
Multiple Choice – D3
• Provide one, and only one, correct answer
• Include plausible options that demonstrate a student’s level of understanding
• Maintain a homogeneous feel in style, length, and visual display
• Colleague review
School Bus - Item
A teacher is planning a field trip and will need school buses to transport students. A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed?
A. 31 C. 32
B. 31.33 D. 36
School Bus - Item
A teacher is planning a field trip and will need school buses to transport students. A school bus holds 36 students. If 1,128 students will be transported, how many buses are needed? Explain your answer.
A. 31 C. 32B. 31.33 D. 36
Begin with the end in mind . . .
• MC assessment could be designed so that each stem presents a different level or skill set
• Subsequent assignments could be generated based on the students response to the MC item
Formative Assessments
• Cautions– Must be as rigorous as summative
assessment (FCAT) – Data must be collected and analyzed– Data must be used to create future
instruction
Which Type of assessment has the greatest potential to alter classroom instruction?
Which assessment will produce data driven decisions in the classroom?
D3 = Effective Formative Assessment System
Moving to D3: LESSONS LEARNED
• It takes time.• It has to start at the top.• Progress has to be measurable. • Summative models are starting points.
• Data–driven decision making can be a powerful tool in changing student outcomes and promoting continuous improvement.
70 Years ago ….
H. L. Menchen said:
“ For every complex problemthere is an answerthat is clear,simple and …….
wrong.”
Tools for Data Analysis
• Evaluation Services Web Site– Tools– Documents– Summative data and charts– Student level disaggregation
- secure access (password required)
