1National Center on Educational
Outcomes
National Center on Educational Outcomes (NCEO)
August 27, 2012
Using Assessment Data as Part of a Results-Driven Accountability System
Input from the NCEO Core Teamand
Sample Approaches
2National Center on Educational
Outcomes
1. Summary of NCEO Core Team Report2. Clarifying Questions3. Overview of Sample Approaches4. Discussion
.
Overview
3National Center on Educational
Outcomes
Core Team
• Peggy Carr, National Center for Education Statistics
• Alan Coulter, Data Accountability Center
• Candace Cortiella, The Advocacy Institute
• David Egnor, Office of Special Education Programs
• Jack Fletcher, University of Houston
• Lynn Fuchs, Vanderbilt University
• Brian Gong, Center for Assessment
• Colleen Riley, Kansas Department of Education
4National Center on Educational
Outcomes
Resource Group
•Rolf Blank, Council of Chief State School Officers
•Anne Chartrand, Sourtheast RRC
•Karen Denbroeder, Florida Department of Education
• Judy Elliott, Consultant
•David Francis, University of Houston
•Michael Kolen, University of Iowa
•Elizabeth Kozleski, University of Kansas
•Rachel Quenemoen, Nat’l Center State Collaborative
5National Center on Educational
Outcomes
Three major sections:
Input of the NCEO Core Team
1. Framing Considerations2. Core Team Suggestions3. Example Reporting Format
6National Center on Educational
Outcomes
1. Public transparency and understandability are critical features of a results-driven accountability system and must be reflected in measures used to review states on student performance.
2. Multiple measures must be included. No single measure should be used in making decisions about student performance results.
3. The use of measures of student performance should provide appropriate incentives to states, particularly in relation to identified values (e.g., inclusion in the general assessment).
Framing Considerations
7National Center on Educational
Outcomes
4. The measures should provide a flag to look deeper into areas that need improvement.
5. A plan should be developed and steps taken to monitor, validate, and improve the use of measures by OSEP and others; additional variables may be appropriate to enhance the measures in the future.
Framing Considerations
8National Center on Educational
Outcomes
6. Variables that may be related to student performance but that have inconsistent interpretations and reliability should not be included in measures that are used for reviewing states on the performance of their students with disabilities.
Framing Considerations
9National Center on Educational
Outcomes
7. No increased burden on states to collect additional data should result from the shift to reviewing student performance results. The developed measures need to fit within what states are doing as they review districts, and should be compatible with and reflective of the state’s overall accountability system used for school improvement.
Framing Considerations
10National Center on Educational
Outcomes
1.Use a reporting format that ensures that multiple measures are considered for students with disabilities receiving special education services.2.Provide data for reading and mathematics separately.3.Include participation of students with disabilities in state assessments.
Core Team Suggestions
11National Center on Educational
Outcomes
4.Include participation of students with disabilities in the general state assessment.5.Include performance of students with disabilities on the general state assessment.6.Include the relative difficulty of state assessments.7.Include the gap in general assessment performance between students with disabilities and students without disabilities.
Core Team Suggestions
12National Center on Educational
Outcomes
8.Include improvement in performance over time.
Core Team Suggestions
These 8 suggestions guided the development of a set of 6 tables to display the data.
13National Center on Educational
Outcomes
Table 1: Reading General State Assessment
Example Reporting Format
14National Center on Educational
Outcomes
Table 2: Mathematics General State Assessment
Example Reporting Format
15National Center on Educational
Outcomes
Table 3: Reading and Math Overall Performance and Targets
Example Reporting Format
16National Center on Educational
Outcomes
Table 4: Reading Alternate Assessments
Example Reporting Format
17National Center on Educational
Outcomes
Table 5: Mathematics Alternate Assessments
Example Reporting Format
18National Center on Educational
Outcomes
Table 6: Participation Rates for Students with Disabilities in Reading and Mathematics Assessments
Example Reporting Format
19National Center on Educational
Outcomes
Two Sample Approaches:
Sample Approaches for Using Assessment Data
1. Decision Matrix (with 3 options)
2. Decision-Making Steps
20National Center on Educational
Outcomes
Important Note
The sample approaches include possible thresholds for deciding whether a state exceeds, meets, or does not meet expectations. OSEP and stakeholders should discuss/consider whether adjustments to these example thresholds are needed.
Stakeholders, experts, and OSEP will need to be involved in determining appropriate thresholds for any elements that are used in reviewing assessment results.
21National Center on Educational
Outcomes
Decision Matrix (Includes State Proficiency Target)
Sample Approach 1a
• Reading and math combined• Data from Core Team Tables 1-3
• Element 1: Participation in general assessment• Element 2: Improvement in percent proficient• Element 3: Gap in proficiency between students
with disabilities and students without disabilities• Element 4: Percent proficient or above• Element 5: Gap in proficiency target and actual
22National Center on Educational
Outcomes
Sample Approach 1a
23National Center on Educational
Outcomes
Sample Approach 1a
24National Center on Educational
Outcomes
Benefits (Pros) and Challenges (Cons)
Sample Approach 1a
Benefits (Pros) Challenges (Cons)• Combines variables that are
difficult to look at separately• Easy to see where state falls
across two content areas
• Complex and may lack transparency
• Each element has issues that need to be considered
25National Center on Educational
Outcomes
Decision Matrix (Without State Proficiency Target)
Sample Approach 1b
• Reading and math combined• Data from Core Team Tables 1-2
• Element 1: Participation in general assessment• Element 2: Improvement in percent proficient• Element 3: Gap in proficiency between students
with disabilities and students without disabilities• Element 4: Percent proficient or above
26National Center on Educational
Outcomes
Sample Approach 1b
27National Center on Educational
Outcomes
28National Center on Educational
Outcomes
Benefits (Pros) and Challenges (Cons)
Sample Approach 1b
Benefits (Pros) Challenges (Cons)• Combines variables that are
difficult to look at separately• Easy to see where state falls
across two content areas
• Complex and may lack transparency
• Each element has issues that need to be considered
29National Center on Educational
Outcomes
Decision Matrix (Without State Proficiency Target)
Sample Approach 1c
• Reading and math combined• Data from Core Team Tables 1-2, with
additional alternate assessment data• Elements 1-4 same as for Approaches 1a and 1b• Element 5: Gap between percent proficient on
general state assessment and percent proficient on AA-AAS
30National Center on Educational
Outcomes
Detailed information on this approach is not included because additional data are needed to make calculations.
A decision matrix approach similar to that in Sample Approaches 1a and 1b would be used.
Sample Approach 1c
31National Center on Educational
Outcomes
Benefits (Pros) and Challenges (Cons)
Sample Approach 1c
Benefits (Pros) Challenges (Cons)• Combines variables that are
difficult to look at separately• Easy to see where state falls
across two content areas• Explicitly includes students in
the AA-AAS
• Complex and may lack transparency
• Each element has issues that need to be considered
32National Center on Educational
Outcomes
Decision-Making Steps
Sample Approach 2
• Reading and math separate• Four steps
• Participation in general assessment• Gap in performance between students with
disabilities and students without disabilities• Proficient rates on general assessment in
relation to the difficulty of the state’s assessment• Alternate assessment participation and
performance
33National Center on Educational
Outcomes
Sample Approach 2
Decision-Making Recording Sheet
34National Center on Educational
Outcomes
Sample Approach 2
35National Center on Educational
Outcomes
Sample Approach 2
36National Center on Educational
Outcomes
Benefits (Pros) and Challenges (Cons)
Sample Approach 2
Benefits (Pros) Challenges (Cons)• Considers all examples of
reporting tables by Core Team
• Provides greater transparency than some approaches
• Allows for adjustment of steps and variable for policy shifts
• Must be completed one state at a time
• More subjective than some approaches
• Comparability is a challenge