N C E O National Center on Educational Outcomes In Search of Consistency: Guidelines for the Administration of Accommodations by Third Parties

National Center on Educational Outcomes

N C

E O

In Search of Consistency:

Guidelines for the Administration of

Accommodations by Third Parties


N C

E O

Presenters

Ann Clapper, National Center on Educational Outcomes

Amanda Blount Morse, National Center on Educational Outcomes

Janet Filbin, Colorado Department of Education

Scott Trimble, Kentucky Department of Education


N C

E O

Session Overview

1. What are third parties?

2. Why are guidelines important?

3. What is the current status of state policies on third parties?

4. How are states addressing the issue of third parties?

5. What are key elements of the process and the product?


N C

E O

Who are we talking about?

Reader• Reads test directions, items, passages, and/or prompts

orally to students who are unable to decode text visually.Scribe• Writes down what a student dictates through speech,

sign language, pointing or by using an assistive communication device.

Sign Language Interpreter• Translates test directions and/or items from spoken

English into American Sign Language or other method of sign language. The sign language interpreter also allows students to respond to written questions in sign language.


N C

E O

The Importance of Guidelines

• Standardization is an essential feature of educational assessments designed to support comparisons among participants.

• Strict adherence to guidelines detailing instructions and procedures for the administration of accommodations is necessary to ensure that the test results reflect actual student learning.

• If accommodations are administered inconsistently the results will not be comparable across examinees.


N C

E O

The Number of States With Existing Guidelines

Scribe Reader Sign Language

Interpreter

26 20 20

Student Assessment Program

Janet FilbinColorado Department of [email protected](303) 866-6703

mailto:[email protected]

The Colorado Student Assessment Program (CSAP)

Legislated in 1993 through HB 1313 Criterion-referenced assessment originally

designed to obtain a point in time picture of 3rd -4th -7th and 10th graders in the area of reading and writing 5th -8th -10th grade students in math8th grade students in science

Provides state, district and building level information about how students from year to year were progressing toward state standards

Requires all students have opportunity to learn state standards

Accommodations are changes made to the assessment procedures in order to provide a student with access to information and an equal opportunity to demonstrate knowledge and skills without affecting the reliability or validity of the assessment.

CSAP Accommodations

Eligibility for CSAP Accommodations

Any student may receive an accommodation that is allowed on the assessment if that student has received the same accommodation during daily classroom instruction for at least three months prior to the assessment.

Only accommodations the student requires for the specific content area may be selected.

The possible effects on the student of allowing the accommodations must be considered.

A student may be allowed more than one accommodation.

A student who is an English Language Learner is entitled to the same instructional and assessment accommodations as his or her English-speaking peers.

Development of CSAP Accommodations

Support from NCEO

Analysis of existing large scale assessment accommodations

Special Populations Advisory Cross CDE Representatives District Superintendents District Assessment Coordinators District Director’s or Specialized Services Parent Centers

Standard CSAP Accommodations

extended time

large print

Braille

extended time

large print

Braille

use of number line

use of manipulatives

Reading and Writing Mathematics

reading/signing directions in the student’s native language

use of scribe signing/pointing use of augmentative device reading of the writing prompts

only

reading/signing directions, word problems, designated answers to problems, and glossary words and definitions in the student’s native language

use of scribe

signing/pointing

use of augmentative device

oral presentation or word problems

Early Emerging Issues

Discrepancies in the use of accommodations from grade level to grade level, district to district and across populations

Variability in the approaches to administering accommodations

Assumed “creative license” in applying non-standard, non-approved modification to testing situations

Focus on compensatory skill development in the classroom for many students

Charge Ensure access of all populations

Lack of previous large scale assessment history for ELL/special education

Reduce the potential for misadministration Discrepancies between buildings/districts in use of

accommodations Increase clarification about the nature of the

accommodation, potential users, and administration guidelines Ongoing issue of accommodations vs. modifications Confusion regarding state vs. district assessment accommodations Low teacher expectations

Provide direction to districts regarding proctor qualifications Certification/training of translators Issues with signing

Stakeholders Cross Unit Collaboration

Assessment Title 1 English Language Acquisition Special Education

Association of Colorado Education Evaluators (ACEE)

CSAP Accommodations Expert Panel Review District-level Administrators Title I, ELL, Content, Special Education Teachers Building Principals CDE personnel Parents

CSAP Bias/Content Review Committee

Dynamic Process

Analysis of data student performance

aggregated special populations

participation rates use of accommodations

special populations by district/school

interviews with educators and administrators in the field

Review of existing literature and research on accommodations (Tindal (1996); Tindal and Fuchs (1999);NCEO Studies-Thurlow, et.al. (1999 2000, 2001); Thompson, et.al. (2002), NAEP Validity Studies-Weston (2002), etc.)

Examination of developed training materials

Recommendations

Increased guidance around specific accommodations Scribe Oral/signed/translated presentation Technology

Relevant training/materials Standard large print test Non-standard accommodation application

process Language simplification

Implementation Process

Statewide CSAP Administration Training Procedures Manual Tools for Increasing the Performance of

Students/CSAP Accommodations Video Fast Facts Website Guidelines For Documenting Student

Need/Teacher or Proctor Training Considerations for Making Decisions

0

10

20

30

40

50

60

2001 2003

0

5

10

15

20

25

30

2001 2003

IEP Title 1 504 bilingual ESL

4th Grade Reading

10th Grade Reading

0

2

4

6

8

10

12

14

16

1998 1999 2000 2001 2002 2003

4th Grade 7th Grade 10th Grade

% o

f st

ud

ents

Changes In The Use Of Reading Accommodations Over Time

Lessons Learned

What you don’t know can hurt you

The road to standardization should not be less traveled

Just when you think your done, your not

The TAO of inservice: Once trained may not retain

Colorado’s Accommodation Guidelines and Training Information

Homepage for the Colorado Department of Education

http://www.cde.state.co.us-homepage

Procedures manual

http://www.cde.state.co.us/cdeassess/csap/2004/proc_man.pdf

Accommodations Video/training materials

http://www.cde.state.co.us/cdeassess/csap/acc/index.htm

http://www.cde.state.co.us-homepage/

Two Basic Principles

• All students can learn at high levels

• Accountability must be inclusive of all students, including those at risk of failure.

Define third-party providers of accommodations

(who can provide the services)

• Third-party providers of accommodations are trained by local districts, are either employees of or volunteers in service to the local district, and are not students.

• These are typically the same persons who provide accommodations during the normal delivery of instruction.

Participation in Assessment

Students with disabilities participate in state- required assessments:

– With no accommodations,– With accommodations consistent with delivery

of instruction, or– In the Alternate Portfolio Assessment.

Accountability Environment

Kentucky’s accountability environment is considered:

– High stakes for schools.

– Not high stakes for students.

Why Accommodations?

• Kentucky’s program is intended to evaluate the effectiveness of the instructional delivery system.

• Students with legally identified disabilities through the ARC may have been provided special accommodations in order to access the general curriculum.

• In such cases, it is critical to assess such students consistent with the instructional accommodations provided for in the IEP.

Questions

• Why is it important to have guidelines for third parties?

• What prompted your state to write guidelines for third parties?

2003 Reading & Mathematics High School

0

20

40

60

80

100

120

140

Reading Mathematics

Content Areas

Aca

dem

ic In

dex

(0-

140)

All Students

Female

Male

White

African American

Hispanic

Asian

Other

Title I

Migrant

LEP

ESS

G&T

F&R Lunch

SWD

Process

Describe the process that Kentucky followed to develop the guidelines for third party providers of accommodations:

stakeholders collaboration research?

Questions

• How was this implemented?

• How is it working?

• Suggestions?


N C

E O

Development Process - Current

• Reviewed state guidelines

• Identified key elements (process and product)

• Drafted composite list of considerations

• Sent draft to 4 states and conducted telephone interviews

• Revised draft


N C

E O

Key Features - Process

• Stake-holder involvement

• Collaborative effort

• Continually evolving process


N C

E O

Key Features - Product

• Comprehensive

• Aligned

• Clear

• Specific

• Coherent

• Accessible


N C

E O

Major Components

• The need for guidelines (i.e., access, test validity, standardization)

• Discussion and clarification of relevant terms (e.g., scribe, reader, etc.)

• Qualifications and characteristics of “third party administrators” (e.g., certified staff members, be familiar with the student, can’t be the parent of the student receiving the accommodation, have completed training)

• Detailed directions and procedures for implementing specific accommodations (read aloud, dictation to scribe, sign language interpretation) during testing

• An explanation of inappropriate administration practices • Implications for test security/confidentiality


N C

E OSpecific Considerations

• Are there guidelines for when and where third party test administrators can review test materials?

• Are there procedures for third party administrators to follow if they need clarification of test items or materials prior to test day?

• Are students trained on the use of a scribe, reader, and sign language interpreter during testing?

• Are there guidelines specific to the type of question?• Are there guidelines specific to content areas?• Are procedures defined for addressing third-parties’

questions or concerns that arise during testing?


N C

E OSpecific Considerations

• Are there requirements for how familiar the person providing the accommodation must be with the student?

• Are there suggestions to guide the pre-discussion between test-taker and third-party administrator?

• Are provisions for the settings in which accommodations can be administered described?

• Are there procedures that need to be followed in order to ensure that all test administrators followed the guidelines established to protect standardization and the validity of the test?


N C

E O

Resources

Allman, C.B. (2003). Making tests accessible for students with visual impairments: A guide for test publishers, test developers, and state assessment personnel. Louisville, KY: American Printing House for the Blind.

Educational Testing Service. (2004). ETS guidelines for a test reader. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/readguid.html

Educational Testing Service. (2004). ETS guidelines for a test writer recorder. Retrieved June. 1, 2004, from ETS Web site: http://www.ets.org/disability/writguid.html


N C

E O

Development Process - Planned

• Conduct more interviews

• Revise and disseminate for additional comment

• Write brief and develop tool


N C

E O

Contact Information

If you have comments or questions please contact:

Amanda Blount Morse

[email protected]

Ann Clapper

[email protected]


N C

E O

Visit: http://www.nceo.info

or Search for NCEO

Documents

N C E O National Center on Educational Outcomes In Search of Consistency: Guidelines for the Administration of Accommodations by Third Parties