2% Modified Achievement Assessments

Overview(C-5)

Ohio Department of EducationOhio Department of EducationOffice of AssessmentOffice of Assessment

September 2008September 200809-05-0809-05-08

1

Federal Rule on Modified Achievement Standards

• Known as 2% Modified Assessment

• Allowances o Modified assessment of academic content standardso Proficient and above scores from such an assessment

may be used in AYP accountability calculations capped at 2% of total test population

• Similar to the 1% alternate achievement standards for students with cognitive disabilities, but with some significant differences

2

Final Federal Rule (April 2007)• The intended population for this assessment is the

students on IEPs who are persistently low performing.

• Modified academic achievement standards must be based on a state’s grade-level academic content standards for the grade in which an eligible student with disabilities is enrolled.

• A state’s academic content standards are not what is modified.

The expectations for whether a student has mastered those standards, however, may be less difficult than grade-level academic achievement standards. (pp. 17748, 17749)

3

Who the Students Are The student’s progress to date in response to

appropriate instruction…, is such that, even if significant growth occurs, the IEP team is reasonably certain that the student will not achieve grade-level proficiency within the year covered by the student’s IEP.

Section 200.1(e)(2)(ii)

4

Cook and Elliot, CCSSO Large-Scale Session, 2007

5

Cook and Elliot, CCSSO Large-Scale Session, 2007

6

Three-State Consortium

Minnesota, Ohio and Oregon are working with AIR on the awarded

three-year GSEG grant.

7

The Goals for the Consortium• Identify the appropriate IEP student group

• Identify the appropriate test question design, the test format and an administration procedure

• Develop 2% Modified Assessments for grades 5–8 in reading and mathematics

• Set modified achievement standards

• Provide professional development for the assessment

• Define eligibility guidelines for participation in the assessment

• Monitor implementation

8

• Each state takes away modifications that will work in that state.

• Each state will move forward with the AA-MAS, using its own item-review process.

• Each state will determine its own blueprint for the AA-MAS.

• Each state will conduct its own standard setting.

• Each state will conduct its own alignment study.

State Outcomes from the Consortium

9

Consortium Outcome for Ohio

• Modify existing and write new items for the general education assessment for use on the AA-MAS

• Use the general education assessment blueprints for the AA-MAS

• Use the currently established review process for the items used on the AA-MAS

• Conduct alignment studies for each AA-MAS• Conduct a standard setting resulting in modified

achievement standards based on grade-level academic standards

Ohio Actions:

10

Task 1. Conduct initial and subsequent meetings

Task 2. Define the student population, conduct data analysis, conduct focus groups, develop student profiles

Task 3. Draft Performance Level Descriptors

Task 4. Develop Eligibility Guidelines and Decision Guide

Task 5. Begin professional development plans for o Eligibility guidelineso Standards-based IEPs

Task 6. Select modification strategies, conduct literature reviews, define modification strategies, modify items and forms

Task 7. Conduct pilot tests for item modifications

First Year Tasks

11

Second Year TasksTask 1. Conduct subsequent meetings

Task 3. Create initial definitions of modified achievement standards, refine PLD, draft definitions of modified achievement standards

Task 5. Implement professional development for

o Eligibility guidelines

o Standards-based IEPs

Task 7. Conduct pilot test of item modifications

Task 9. Produce standard setting plans

12

Third Year TasksTask 1. Conduct subsequent meetings

Task 5. Implement professional development plans for

o Instructional strategies

o Monitoring

Task 7. Field test modifications to items, forms and administration as well as validation of field tests

Task 8. Produce item maps for operational test

13

ResearchMarion, Scott. A Technical Design and Documentation Workbook for Assessments

Based on Modified Achievement Standards. KS, SD, OK, and MD Department of Education Web sites.Higgins, J., Russell, M., & Hoffmann, T. (2004). Examining the Effect of Computer-

Based Passage Presentation on Reading Test Performance. Miranda, H., Russell, M., & Hoffmann, T. (2004). Examining the Feasibility and

Effect of a Computer-Based Read-Aloud Accommodation on Mathematics Test Performance.

Famularo, L. and Russell, M. (2007). Examining the utility of a prototype assessment for assessing students in the gaps.

Dolan, R., Murray, E., & Burling, K. (2007). Providing students with choice: An exploratory study on the application of universal design principles to large-scale assessment of students with learning disabilities and English language learners.

The Assessment and Accountability Comprehensive Center: Assessments based on modified academic achievement standards: critical considerations and implications for implementation.

National Centers on Educational Outcomes. (2007). The assessment and accountability comprehensive center: special populations strand.

National Centers on Educational Outcomes. (2007). A Seven-Step Process to Creating Standards-based IEPs.

14

Initial Data Mining

• Help define the student population• Examine questions that work with the

target population (still undefined)

Analyze the data to help with the following:

15

Initial Data Mining Procedures

• Persistently low performing• Persistently low performing with

IEPs• Other students with IEPs• General education students

Data from the three states’ general education administration were divided into four sets:

16

Initial Data Mining—Research Questions

• What items discriminate well for both IEP populations?

• What are the characteristics of the items that discriminate well for the low-performing group?

• What is the breakdown by race/ethnicity/gender/SES for the low-performing group?

17

Initial Data Mining—Research Questions

• What are the disability classifications in the persistently low-performing groups?

• What happens when we include those students who top out of the AA-SWD?

• What are the performances of students allowed different amounts of time in the regular classroom rather than disability classifications?

18

Stakeholder Input• Focus groups

o Collect input from the special education community who work with the target student population

o Aggregate information across states

• Public surveyo Eligibility guidelines o Decision flowchart

19

Goals for the Focus Group• Begin defining the target population

o Brief review of what the data show about low-performing students

o Professional judgment about who these students are • Review and suggest modification to test questions

o Review questions that work well with the persistently low-performing students

o Suggest modifications to test questions• Review and suggest formats for test booklets• Suggest modifications to administration procedures

20

Requirements for Targeted Population

• Have IEPs• Identified as persistently low performing

on the regular statewide assessments• Receive on-grade-level instruction

Students must conform to the following minimum requirements:

21

Eligibility

Decision

Flowchart

(Handout)

22

Stakeholder Input—Online Survey

Results of the survey:• Who responded• Requirements for eligibility• Training needs

23

• Purpose: lower barriers that students with disabilities face in demonstrating achievement of grade-level skills and knowledge

• Two strategies combine to meet this goal o The assessment incorporates modifications

targeting specific characteristics commonly found in the target population

o Second, performance on the assessment will be judged against modified achievement standards

Ohio Fall 2008 Pilot Goal

24

Grade 7 Fall 2008 Pilot (Cont.)• Two groups tested: target and general

education students

• Each student takes Reading and Mathematics tests

• One consumable test booklet

• Seven forms, each with 42 multiple-choice questions

• Week of October 20, 2008, for Reading

• Week of October 27, 2008, for Mathematics

25

Item modifications:

1. Simplify language where appropriate

2. Inserted question within a box at appropriate place in passage

3. Present summary questions at the beginning and end of passages

4. Use bold print for key words or phrases

Grade 7 Fall 2008 Reading Pilot

26

Item modifications:1. Simplify language where appropriate2. Include relevant pictures, tables and graphics to

replace text (this is different from enlarging or shrinking graphics)

3. Enlarge and/or shrink graphics4. Bold key words5. Use scaffolding: Break multi-step items into

individual steps, each with questions

Grade 7 Fall 2008 Mathematics Pilot

27

Caveats• Not every item will be modified in every

way.• In cases where the modification is not

possible, the unmodified version of the item for the block will be used.

• Examples might include:o An item with a one-step process for the

solution may not be modified using scaffolding.

o An item with no original graphic may not be modified by using enlarged or simplified graphics.

28

Analysis Questions for Fall 2008 Pilot

• Target group will omit fewer modified items

• Target group will omit more unmodified items

Do the modifications improve access?

29

Analysis Questions for Grade 7 Fall 2008 Reading and Mathematics Pilot

• Target group will guess less on modified items

• Target group will guess more on unmodified items

Does performance exceed chance?

30

Analysis Questions for Fall 2008 Pilot • Does performance improve and is discrepancy in

performance reduced?

• Target group will perform better on modified items than on the unmodified items.

• Regular education students will perform about the same on both modified and unmodified items.

• Marginal maximum likelihood will be used for group comparisons rather than aggregating individual maximum likelihood estimates for each student.

• Multi-group IRT model that allows calibration around multiple groups that is a partial credit extension of the Bock & Zimowski, 1997.

31

Grade 7 Spring 2009 Field Test

• Will be similar to the fall 2008 pilot• Interested in participating?• Contact Barry Lowry:

barry.lowry@ode.state.oh.us• District name, IRN, school, and

contact information

32

1. Include relevant picture

2. Scaffolding: priming—asking understanding questions

3. Scaffolding: cueing with focusing questions

4. Scaffolding: graphic organizers

5. Scaffolding: provide thoughtful questions through passages focusing on summative items

Spring 2009 Pilot—Reading Modifications

33

Item modifications:

Item modifications:

1. Include relevant pictures, tables, graphics that replace text

2. Scaffolding: priming—asking understanding questions

3. Group questions by content strand

Spring 2009 Pilot—Mathematics Modifications

34

Thank You.

Questions?Questions?

35