Identifying Students in Need of Modified Achievement Standards and Developing Valid Assessments

Identifying Students in Need of Modified Achievement Standards

and Developing Valid Assessments

Who are the students needing modified achievement

standards?

….and thoughts for developing eligibility criteria

Sue Bechard and Judy Snow

January 16, 2008

Washington D.C.

01.16.08 MT GSEG: Bechard & Snow 3

Who are the students needing Modified Achievement Standards (MAS)?

Can include:

□ 2% of the total student population who can be counted as proficient on MAS,

□ students with disabilities, from any of the 13 disability categories,

□ students who are addressing grade level content standards on their IEPs, but are not expected to meet grade level achievement standards in the current year

□ students who need less difficult test items, covering the same breadth of content.

US Dept of Ed., 2007


Determination of eligibility for MAS….

Must consider….□ objective evidence□ multiple measures of progress over time□ IEP goals that are based on grade level content standards□ providing students the opportunity to show what they know

and can do on an assessment that is based on grade-level academic achievement standards.

Must not consider….□ a specific disability category□ racial or economic background

US Dept of Ed., 2007


Population Identification Issues

□ On average, students with disabilities comprise approximately 10% of the total student population.

□ If 10% - 1% = 9%, what characteristics should be used to distinguish the students appropriate for the 2% option within this group?□ Are there test performance distinctions?□ Are there specific learning characteristics?□ Are there specific learning needs for access to the

general curriculum?□ What are the “modified” expectations relative to grade

level content which distinguishes this population?


Population Identification Issues (cont.)

□What is the purpose of the AA-MAS?□To allow students to do better (AYP

improvement)?□To provide better information for instructional

planning (data on what students know)?□To increase student self-esteem?□To provide greater alignment between

cognition, expectations, instruction and assessment?

□Is an AA-MAS needed at every grade level/content area?


Selected results from prior research….

□Colorado: Report from the HB 05-1246 Study Committee, December, 2005

□New England Compact (RI) Enhanced Assessment Grant, 2004-2006

□Montana General Supervision Enhancement Grant, 2005-2007


CO report: Low Performers Who Score in the Bottom 1/3 of Scale Scores

□ Students who score in the bottom one third of scale scores on CSAP are almost twice as likely to be Black or Hispanic as students of other ethnicities.

□ Only 60% of students with IEPs scoring at lowest possible scale scores were able to be matched with a test the following year; thus, they may be more mobile than their counterparts who score at higher levels.

□ For those students scoring in the bottom one-third of scale scores, and where a match the following year was able to be made, it was found that these students did make substantial longitudinal growth.


CO report: Students with IEPs Who Do Not Make Longitudinal Growth

□ On the Colorado CSAP Reading Test, there were 250 students (of 444,407) across grade levels that were determined to be “Students in the Gap”.

□ On the CSAP Math Test, there were 658 students (of 444,910) that were determined to be “Students in the Gap”.

□ The CSAP as currently administered may not reflect their academic achievements; however, if appropriate accommodations and more intensive instruction were provided, these students too may make more gains.


10

Georgia EAG

Also looked at snapshot vs. longitudinal growth:

□Low Performing: lowest performance level in at least one assessment

□Persistently Low: lowest performance level for three consecutive years

Melissa Fincher, July 2007


Rhode Island (New England Compact - NEC) EAG

Teacher judgments of class work were compared to test performances and revealed two gaps of students performing below proficient:

Performance gap □ The test may not reflect classroom performance. Teachers see

students performing proficiently in class, but test results are below proficient.

Information gap □ The test may not be helpful for instructional planning. Teachers

rate students’ class work as low as possible and test results are at “chance” level. No information is generated on what students can do.

Parker & Saxon, 2007 Bechard & Godin, 2007


NEC EAG data sources□ State assessment data – grade 8 mathematics

results from two systems□ General large-scale test results□ Demographics (special programs, ethnicity, gender)□ Student questionnaires completed at time of test□ Accommodations used at time of test

□ State special education data □ Disability classification□ Free/reduced lunch□ Attendance

□ Classroom teacher data□ Individual interviews□ Judgments of all students’ classroom work


NEC EAG findingsThe Information Gap in grade 8 mathematics comprised 2.3-

4.3% of the total population

□ Included non-disabled students. □ Test performance:

□ Students mostly guessed on the test items. □ Most used multiple accommodations.

□ Teacher perceptions: □ These students operate below grade level in class. □ Teachers are not surprised by their low test results.□ There is a disconnect in what is tested vs. what is taught.□ These students need more supports in the classroom.

□ Student perceptions: □ Think the test is harder than their classroom work. □ They try hard on the test.


NEC EAG: Special program status of students in the

Information GapBreakdown of Sub-Populations within Information Gap and Comparison Groups

Within Group IEP only ELL only IEP&ELL General Ed Information gap (n=185)

80.0% 6.5% 2.7% 10.8%-

Non-gap comparison (n=369)

69.4% 9.8% 1.6% 19.2%

Overall Population

15.1% 1.9% 0.2% 82.8%

Non-gap comparison = students who performed at “chance,” on the test but higher in the classroom.

The majority of students performing at “chance’ were students with IEPs.


NEC EAG: Disability designations of students in the Information Gap

Learning disabilities: □ Lower percentages of students with SLD

were in the information gap than in the general population.

Other disabilities:□ deaf/blind□ multiple disabilities□ hearing impairments□ mild to moderate cognitive disabilities□ combinations of disabilities.


Montana GSEG, 2005 Students in the Sample□Grade 5 students statewide □Census sample of:

□Students with an IEP□Who took Spring 2006 Grade 4 math CRT□A few (13) who scored well on the Alternate

Assessment also included, selected by scores and recommendations from IEP teams

□ CRT-M = 672 students, CRT = 199 studentsMontana Office of Public Instruction and Measured Progress


MT GSEG, 2005 data sources

□Pilot test results□Student survey□Test administrator survey□Standard setting results□Recorded discussions of standard

setting panelists□Interviews with standard setting

panelists


MT GSEG 05: Test Information Functions

Test Information

0

5

10

15

20

25

30

35

40

45

50

-4 -2 0 2 4

Theta

Info

rma

tio

n

CRT

CRT_M


MT GSEG 05: Performance Level Comparisons, CRT-M vs. CRT

Up Three Levels

7

1% All 7 went to Advanced

Up Two Levels

120

18% All 120 went from below proficient to proficient or advanced

Up One Level282

42% 88 went from nearing to proficient, 96 went to advanced

No Change 234 35%

Down One 23 3% 11 went from proficient to nearing proficiency

Down Two 5 .7%

Down Three 1 .1%


MT GSEG 05 student and teacher surveys: Difficulty

• Students taking the CRT-M found the test slightly less difficult when compared to classroom content than students taking the regular CRT.

• Teachers felt the modified test should be modified more to reach the students having with the greatest challenges.


MT GSEG 05 feasibility question: Is the CRT-M a better measure?

□Students answered more items right□Student scores went up□Students moved up to another proficiency

level□Validity indicators improved

□More data analysis and study to determine which CRT-M students benefited most


Selected considerations from current research….

□Montana (+ NEC) EAG, 2007-2009: Adapting Test Items to Increase Validity of Alternate Assessments Based on Modified Achievement Standards

□Montana GSEG, 2007-2010: Identifying Students in Need of Modified Achievement Standards and Developing Valid Assessments


MT EAG, 2007

Focus on high school reading comprehension to:□determine the processing requirements

of test passages and items (use coding strategy)

□describe the cognitive abilities and challenges of the target population

□conduct cognitive labs□develop item modifications based on

cognitive variables


Preliminary feedback from Expert Panel (01-11-08)

□ Some noted cognitive variables:□ Abstract reasoning that relies on information from entire

passage□ Long passages that require sustained attention□ Limited experience with multiple meanings of

vocabulary words□ Dense passages that require large amounts of working

memory□ Location in the passage where necessary information is

found to answer the question□ Irrelevant information in passage makes mapping and

sorting difficult□ Emotional content difficult for students with ED to

process□ Answers to questions not found in passage (e.g. reliance

on prior knowledge)


Montana GSEG, 2007

Focus on middle school reading and mathematics to:

□ Identify students in need of modified achievement standards (MAS).

□ Determine what content knowledge the student is lacking to achieve proficiency

□ Develop dynamic online assessment that provides scaffolding based on distractor selection .


Students who will be included in the MT GSEG study samples

Middle school reading and mathematics□Sample for analyzing items and

distractors: All students who took the tests of interest, disaggregated

□Sample for cognitive labs: convenience sample of 48 students (24 per content area)

□Sample for pilot test: Approximately 5% of the total population


Cumulative%

P.L.%

Cumulative%

P.L.%

-4.00 0 0.0% 0.0%

-2.99 1 0.0% 0.0%

-2.57 2 0.0% 0.0%

-2.31 3 0.0% 0.0%

-2.12 4 0.0% 0.2%

-1.97 5 0.0% 0.2%

-1.85 6 0.0% 0.2%

-1.74 7 0.1% 0.3%

-1.64 8 0.1% 0.5%

-1.56 9 0.2% 1.1%

-1.48 10 0.2% 1.1%

-1.40 11 0.3% 9.4% 1.6% 42.5%

-1.33 12 0.3% 1.8%

-1.27 13 0.5% 3.0% 1

-1.21 14 0.7% 4.2%

-1.15 15 0.9% 5.5%

-1.09 16 1.2% 8.0%

-1.03 17 1.5% 10.5%

-0.98 18 1.8% 12.0%

-0.93 19 2.2% 14.5%

-0.88 20 2.7% 17.1%

-0.83 21 3.2% 19.8%

-0.78 22 3.7% 21.9%

-0.74 203 23 4.3% 24.7%

-0.69 206 24 5.1% 26.9%

-0.65 209 25 5.8% 30.0%

-0.60 213 26 6.6% 33.3%

-0.56 216 27 7.4% 36.4%

-0.51 219 28 8.4% 39.4% Research Population-0.47 222 29 9.4% 42.5% N = 581

-0.42 225 30 10.6% 46.5% 54.9%

-0.38 228 31 11.9% 50.0% 5.2%

-0.34 232 32 13.2% 52.6%

-0.29 235 33 14.6% 55.7% 2

-0.25 238 34 16.0% 58.9%

-0.21 241 35 17.8% 61.9%

-0.16 244 36 19.6% 65.4%

-0.12 247 37 21.5% 68.7%

-0.07 251 38 23.5% 71.0%-0.03 254 39 25.8% 73.4%0.02 257 40 28.3% 75.9%0.06 260 41 31.3% 79.4%0.11 263 42 34.2% 82.2%0.16 266 43 37.6% 83.9%0.21 270 44 40.9% 85.7%0.26 273 45 44.5% 87.6%0.31 276 46 48.2% 88.9%0.37 279 47 52.2% 90.5%0.42 282 48 56.1% 92.3%0.48 285 49 60.4% 94.2%0.54 288 50 64.6% 95.2%

% of IEP Population =% of Full Population =

raw score chance

chance + error

scale score floor (RI EAG)

floor + error

26.5%

200

12.0%

43.1%

PerformanceLevel

FullPopulation

(N = 11,174)

IEPPopulation

(N = 1,058)*

ThetaScaleScore

RawScore

3

26.2%

MT CRT

Grade 10 Reading Example, 2007


Implications of research for identification of target population

Use of performance data :□Longitudinal performance data□Students who are so low performing,

nothing is known about them□Match between classroom

performance and test performance□Distractor analyses


Implications of research for identification of target population (cont.)

Use of other data:

□Teacher judgment data□Opportunity to learn variables

□Mobility□Attendance□Program placement

□Performance data analyzed by cognitive modeling information

□Data from standards-driven IEPs


Data collected for “The Whole IEP Process” (C.

Massanari)

What is the desired outcome for this student? Three to four years from

now Student’s desired post-

school outcome

What are the skills and knowledge essential to meeting the desired outcome?

What are the expectations of the general curriculum relative to the student’s age/grade? Content Expectations for learning and

demonstration of learning Extracurricular activities or

events available


How do skills and knowledge essential to meeting the desired outcome compare with the general curriculum, including content and expectations for learning? Where are the similarities/connections?Where are the differences?Where within the general curriculum,

including extracurricular, are the opportunities for learning the needed skills and knowledge?

What are the student’s present levels of performance? What skills and knowledge does

the student already possess? What other strengths does the

student present? What are the areas of challenge? What accommodations,

modifications, or other supports have proven beneficial for this student?

Given all the information we have discussed thus far, what do we think are reasonable goals for this year? What are the objectives for each

goal? What instructional accommodations

are needed? What modifications to the general

curriculum are needed? How will progress be reported and

how often?

Given the information we have discussed thus far, how will the student participate in state and district-wide assessments? With peers as given With peers and with

accommodations or modifications

Alternate assessment


Also, consider how model of 1% eligibility guidelines might apply

For example: Montana's eligibility questions for the CRT-Alt. The student MUST:

□ Program: Have an active IEP□ Learning characteristics: Have cognitive abilities

and adaptive behaviors which require substantial adjustments to general curriculum

□ Learning objectives and expected outcomes: Focus on functional application of skills

□ Delivery of instruction: Requires direct and extensive instruction


Consider how model of 1% eligibility guidelines might apply (cont.)

Montana's eligibility questions for the CRT-Alt. Decisions must NOT be based on:

□ Excessive or extended absence □ Disability category□ Social, cultural, or economic differences □ Amount of time receiving special education

services□ Expectation of failure on general test


So…2% eligibility considerations might address:

Learning characteristics: □What are the cognitive abilities and adaptive

behaviors of the target population? □What adjustments are needed for the student

to participate in the general curriculum (e.g., accommodations/modifications)

Learning objectives and expected outcomes: □How does the student demonstrate application

of learned knowledge, skills, and abilities?


2% eligibility considerations (cont.)

Delivery of instruction: □What are the deconstructions of constructs

necessary for the student to master the grade level content?

□What adjustments must be made to simplify the materials used in instruction?

Academic achievement:□How is the progress of this student different

from the pattern of progress typical for all students at the targeted grade level?


References□ Bechard, S. and Godin, K. (2007). Identifying and Describing

Students in the Gaps in Large-Scale Assessment Systems. Paper submitted for publication.

□ Colorado Department of Education. (2005, December). Assessing “students in the gap” in Colorado: Report from the HB 05-1246 Study Committee. Denver: Author.

□ Montana Office of Public Instruction and Measured Progress. (2007, May). Determining the Feasibility of an Alternate Assessment Based on Modified Achievement Standards: A Planning Project and Pilot Test [Final Report for Montana’s General Supervision Enhancement Grant CFDA 84.373X – Priority B]. Helena: MT.

□ Parker, C. E., & Saxon, S. (2007). “They Come to the Test, and There is Nothing to Fold”: Teacher Views of Students in the Gaps and Large-Scale Assessments. Paper submitted for publication.

□ Title I—Improving the Academic Achievement of the Disadvantaged; Individuals With Disabilities Education Act (IDEA). Final rule. 72 Fed. Reg. 17748–17781, pts. 200 and 300 (2007, April 9).

Documents

Identifying Students in Need of Modified Achievement Standards and Developing Valid Assessments