1

Maryland’s Alternate Assessments

based on Grade-Level Content Standards &

Modified Academic Achievement Standards

Trinell Bowman, MSDE

Carolyn Wentzel, Lora Monfils, and Lisa Baumann, ETS

MAG Conference, 2008

2

ETS and MSDE staff will give an overview and answer all your questions about the new Modified High School Assessment – the “Mod-HSA.”

• What is the Mod-HSA? • How was it developed? • How can it benefit schools and students? • When and how will it be administered? • How will it be reported?

Overview

3

Requirements for Testing Students with Disabilities

• Students with disabilities will participate in general state and district wide assessments, with appropriate accommodations, where necessary.Sec. 612(a)(17)(A)

• For students with disabilities who are determined to be unable to participate in general assessments, alternate assessments will be used. Sec. 612(a)(17)(A)(ii)

IDEA REQUIREMENTS

4

Requirements for Testing Students with Disabilities

• Beginning in 2005-06, all students in grades 3-8, including students with disabilities (with reasonable adaptations and accommodations, as necessary) will participate in annual assessments in, at a minimum, mathematics and reading/language arts, and, beginning no later than 2007-08, in science. Sec. 1111(b)(3)(C)

• Accommodations, guidelines, and alternate assessments must be provided in the same manner as under Sec. 612(a)(17) of the IDEA. Sec. 1111(b)(2)(I)

NCLB REQUIREMENTS

5

Goals for Inclusion of Students with Disabilities in the Maryland Assessment Program

• Provide ALL students access to the general curriculum

• Provide ALL students with assessments aligned with grade-level content standards

• Ensure both special and general educators set high expectations for students with disabilities

6

• A student’s disability must not determine which assessment the student will take.

• There is no limit on the number of students TAKING any particular assessment.

• A student’s IEP team will make the decision as to which assessment a student will take.

Goals for Inclusion of Students with Disabilities in the Maryland Assessment Program

7

Maryland State Assessments

• Maryland Model for School Readiness (MMSR)

• Alternate MSA (Alt-MSA)

• Maryland School Assessment (MSA)

• Modified MSA (Mod-MSA)

• High School Assessment (HSA)

• Modified HSA (Mod-HSA)

7

8

• A student’s disability must not determine which assessment the student will take.

• There is no limit on the number of students TAKING any particular assessment.

• A student’s IEP team will make the decision as to which assessment a student will take.

Mod-HSA Considerations

9

•Provide a measure of students with disabilities performance compared with grade-level academic content standards

•Designed to meet the needs of student’s specific instructional characteristics, such as the need for:

Shorter reading passages More visual cues Scaffolding questioning Few items presented on a page

What are Modified Achievement Standards?

10

• Provide measure of students with disabilities compared with grade-level academic content standards

• Based on the same academic content standards as the state’s regular assessment (MSA)

• Provide teachers and parents with information that will help guide access to the general education curriculum and instructional strategies

What are Modified Achievement Standards?

11

What have we learned so far?

WHO ARE THE

STUDENTS?

12

Who Are The Students? What have we learned so far?

Students with all categorical disabilities Students who have difficulty processing

complex information Students who are easily distracted Students who have difficulty completing

written tasks Students who require multiple

accommodations such as extended time, verbatim reading, calculator, graphic organizers, multiple or frequent breaks

13

• A student with an IEP working on grade level content and working towards completion of the high school diploma

• A student making progress on grade-level academic IEP goals, but not a year for a year

• Has had multiple interventions over consecutive years

• Does not qualify for Alt-MSA eligibility (not a student with significant cognitive disabilities)

Who is a Mod-Eligible Student?

14

Modified Assessments and General Requirements

• Students may be eligible to take the Mod-HSA in one content area and take the HSA in another content area.

15

What was the Process for Test Development of the Mod-HSAs?

• Teams of General Educators and Special Educators received Initial training: overview and a public release SR, SPR and BCR were modified and then another similar item allowed the teams to “practice’, discuss and share their modifications.

• This process was done in several stages that allowed the teams to look at specific indicators and their assessment limits and skill statements to ensure continued alignment with the Core Learning Goals, Expectations, and Indicators.

16

What was the Process for Test Development of the Mod-HSAs?

Alterations of test items were based on: Adhering to the CLG assessment limits and

skill statements

Simplified language while not compromising the content vocabulary

Lowered the cognitive demand while not compromising the content such as using “friendlier” numbers in presentation of algebra or biology data charts

17

What was the Process for Test Development of the Mod-HSAs?

• Converted all items to SRs with 3 answer choices• Used data to delete an answer choice from

SRs• Stem:

• Extra line space between stimulus and answer choices

• Chunked information when appropriate. • Extra line space added before test

question.

18

What was the Process for Test Development of the Mod-HSAs?

• Passages must often be presented in full for a student to be able to respond to certain questions (e.g., main idea); but when an item is based on only part of the passage, present just the relevant section of the passage again immediately before the item

• Avoid using “A”, “B”, or “C” to label information in an item stem…in the stem when the answers are numbers or something else. BUT if the question asks the student to select a method or company… then it is appropriate to use “A”, “B”, “C” as labels.

19

What was the Process for Test Development of the Mod-HSAs?

• Selected Responses only; no Brief Constructed Responses (BCRs), no Extended Constructed Responses (ECRs)

• Deleted extraneous information when possible

• Use pictures to support passage

• Online Assessment using Pearson’s Test Nav system (currently used for the MSA Science and Mod-HSAs)

20

Mod-HSA Algebra/Analysis Test Format

Sample Test ItemLook at the equation

below.

125 x N = 375 What value of N makes

this equation correct?

1. 2 2. 3 3. 4 4. 5

21

What are the Benefits for Students?

• Eligible to take the Mod-HSA in one subject and take the HSA in another subject

• Test is more accessible and students with IEPs will do better

• All accommodations available

• Students do not need to “pass” to benefit from participation in assessment

• Assist in increasing points towards composite score of 1602

22

Michelle DaleyDivision of Special Education/Early Intervention Services

mdaley@msde.state.md.us410-767-0743

Trinell BowmanDivision of Accountability & Assessment

tbowman@msde.state.md.us410-767-2498

Contact Information

23

Psychometric Analyses to Support the Modified-HSAs

(Mod-HSAs)

Carolyn Wentzel and Lora Monfils, ETS

MAG Conference, 2008

24

Introduction

Overview Psychometric Goals Test Structure & Student Groups Psychometric Analyses Comparability of Mod-HSAs & HSAs Confirmation of Cut Scores Test Score Summary Results

25

Overview Mod-HSAs are modified versions of the

HSAs comprised of SR items only

Designed to be taken in place of one or more of the HSAs

Two administration options: online & paper and pencil

First administered in May 2008

26

Psychometric Goals for May 2008 Administration

Build Mod-HSA operational forms to the HSA blueprints using 50 modified HSA SR items

Link Mod-HSA scores to the HSA reporting scale (240-650)

Confirm use of HSA cut scores for Mod-HSAs

27

Test Structure

• 2 forms per content area with 70-75 modified HSA items + 24 unmodified HSA items, common to both forms

• 50 items per form would be selected and retained as operational items

28

HSA Linking Items• 24 HSA items, in each content area, used

to link the Mod-HSA to the HSA scale

• Items selected to be representative of the blueprint so as to comprise a “mini-test”

• Items selected across a range of difficulties

• Linking items did not count toward final scores

29

Student Groups Two groups of students took May

2008 Mod-HSAs:

1) Target population: Special education students eligible to take the Mod-HSA based on IEP and progress in learning course content (2,000 – 2,700 students)

1) Linking samples: Subset of students who took the May HSA in the same content area (1,900 – 3,000 students)

30

Psychometric Analyses: IA & DIF Classical item analysis (IA)

Obtain item raw score statistics Percent of students obtaining correct response Correlation of correct and incorrect responses with test

scores Omit rates

IA statistics obtained for both groups

Differential item functioning analysis (DIF) Compare item performance between two groups

matched on the total score of a test DIF statistics obtained for both groups

31

Psychometric Analyses: IRT Scaling

• Data from linking samples used for Item Response Theory (IRT) scaling

• Items calibrated concurrently using three parameter logistical (3PL) item response model

• HSA linking items had two sets of parameters: Parameters estimated during the current calibration Banked parameters expressed on the HSA reporting

scale

• Stocking and Lord (1983) procedure used to align the TCCs based on the two sets of linking item parameters

• Resulting constants used to transform the Mod-HSA parameters and express them on the HSA score scale

32

Criteria to Select Operational Items

Blueprint coverage maintained IA statistics within recommended range

0.10 < P-values < 0.90 Item-total correlations ≥ 0.10 for correct

responses Item-total correlation weak or negative for

incorrect responses Percent of students omitting an item ≤ 5%

DIF: No evidence of differential item performance

IRT: Well-estimated during scaling

33

Comparability of Mod-HSAs & HSAs for Linking Samples

Correlations ranged from 0.70 (English) to 0.76 (Biology)

Average performance similar

Variability greater for Mod-HSAs

34

Scatterplot of Mod-HSA & HSA Scale Scores for Algebra

240

270

300

330

360

390

420

450

480

510

540

570

600

630

240 270 300 330 360 390 420 450 480 510 540 570 600 630

HSA SS

MO

D S

S

35

Confirmation of Cut Scores Standard settings conducted.

Panels consisting of general educators and special educators.

Verified that cut scores established for HSAs could be used.

Common scale and cut scores permits Mod-HSA and HSA to be used to create composite scores.

36

Results: Score Summary Statistics by Group

Target N PassMean SD %

ALG 2,720 360 50 10BIO 2,012 360 48 16ENG 2,532 353 42 10GOV 2,270 357 50 20

Linking N PassMean SD %

ALG 2,513 420 37 66BIO 3,144 420 40 75ENG 1,947 408 40 67GOV 2,249 424 50 81

Scale Score

Scale Score

37

Additional Information

Mod-HSA technical report available early 2009

Posted to:http://hsaexam.org/

38

Test Administration

Mod-HSA follows the same testing calendar as the HSA

Mod-HSA test takers are to be included in the same pretest file as the HSA test takers

Mod-HSA is available for online testing or paper and pencil testing

Mod-HSA has an online practice test

39

Test Administration

Mod-HSA has the Large Print, Braille and Kurzweil versions of the test forms and an audio test form is also available for online

Mod-HSA has a set of Examiner’s Manuals

40

Questions & Reflections

41

ETS Contact Information

Lisa Baumann lbaumann@ets.org

Carolyn Wentzel cwentzel@ets.org

Lora Monfils lmonfils@ets.org