Welcome Review TeamsWelcome Review TeamsVision Teachers and Content Area Teachers

Oregon Department of Education

December 7 and 8, 2010

Objectives Introductions Housekeeping Workshop Norms Workshop Evaluation Agenda Vocabulary Process Highlights with “Released”

Sample Test Items

Individuals understand that this workshop is an initial step

Initial “quick” examination of ODE’s test item pools (about 18,000 test items) to discern items accessible as-is by Students Using Braille via• 18-cell refreshable Braille reader or• Braille embosser (“on the fly”)

Gather your written input • Comment forms(yellow forms – any time)• Evaluation (end of workshop)

Parking

Confidentiality Agreement: sign and submit

Security today and tomorrow: no cell phones, no email

Restrooms and Exits

Temporary Employees • Conditions of Temporary Employment• ODE Temporary Application• Form I-9 (Identification Required)• Form W-4• Criminal History Verification of Applicants• Employee Questionnaire for Self-Identification of

Race/Ethnicity (Optional)• Timesheet

Everyone Mileage that you drive: by date, include

departure/arrival time, from & to “towns” Parking or Bus: enter what you pay for

parking or bus daily (receipts not needed) Initial to verify that you will not get any

other travel awards Name, SSN, Signature, and Mailing

Address

Those traveling 70 miles or more

Record Meal Costs and Lodging by Date

Lodging: Maximum State Rate in Salem is $70 plus tax. Enter the tax as Misc.

You will need to attach a zero balance lodging receipt.

Those traveling 70 miles or more (cont.)

Meals • Breakfasts: If your hotel provides a breakfast,

you will not be reimbursed for restaurant breakfast.

Otherwise, attach receipt and reimbursement will be up to $11.50.

• Lunches: Lunch is provided at this meeting.

• Dinners: Attach receipt. Reimbursement will be for up to $23.00. (Remember: No alcohol)

Meeting RulesMeeting Rules No electronic device turned on during the

review sessions (This includes laptops, cameras, cellular phones, PDA)

DO NOT replicate or send secure test items electronically

DO NOT make or keep copies of any test item materials

DO NOT disclose confidential testing materials for any actual or potential test items to anyone outside ODE

DO return any and all of the materials entrusted to you by ODE or its contractors in their entirety after your work has been completed

Meeting NormsMeeting Norms

Each person quietly focuses on workshop tasks– quiet so others can concentrate

Work quickly and consistently (4 to 7 items/minute)

Concisely ask questions if needed Remember if something is said: please do

not repeat Use the Comment Forms to communicate

concerns

Grade Level Content TeachersThere is one Content Teacher per Item Pool (Subject/Grade).In your breakout room, you will have a laptop. In consultation with your vision teacher, quickly go from test item to test item to discern your first impression of “best fit category” Carefully record the category for each test item. Vision TeachersThere is one Vision Teacher on each High School Group (HS Math for example). There is one Vision Teacher on each Elementary and Middle School Group (3rd and 4th grade math group, 5th and 6th grade math group).

ODE SpecialistContent Area Facilitators:Jim Leigh (Math), Ken Hermens (Reading), Leslie Phillips (Science and Social Sciences)

Team RolesTeam Roles

Category (1 to 5)1 – Definitely this item works for students using Braille2 – Yes with simple Transcriber Notes3 – Yes with Braille Embosser4– Yes with Braille Embosser and simple Transcriber Notes5 – Omit for Now

Is there a table in this object? Yes Or leave blank

Is there a graphic that is not substantive (“eye candy”)? Yes Or leave blank

Team Roles:Team Roles:3 Tasks Per Test Object3 Tasks Per Test Object

Mark:• I have a question• I have a concern

Concisely and professionally print your question or concern

Name optional

The daily schedule The facilities The quality and usefulness of the

materials and resources The quality and information in the

training session The quality and productiveness of the

item categorization sessions

Today (9:00 – 3:30)• Overview, Training, Item Pool Review, Recap• Breaks: 10:30 – 10:45 and 1:45-2:00• Working Lunch: Noon – 12:30

Tomorrow (9:00 – 3:30)• Debrief, Item Pool Review, Reflection,

Evaluation• Breaks: 10:00 – 10:15 and 1:45-2:00• Working Lunch: 11:30 – 12:30

In your folder, you have a vocabulary sheet

If it is blank, then you will be looking for definitions

If it has definitions, then you will share definitions with those seeking – and initial that person’s sheet for def. given

Goal: 10 minutes – fill out as much of your sheet or give out as many definitions as possible

10:00 – 10:15

Braille will be delivered to the student via an 18-cell refreshable Braille display (Literary Braille only)

Include items that need verbatim text only (includes items with graphics that are non-substantive “eye candy”, not needed for student understanding)

Exclude items that need Nemeth Braille, have a graphic needed to understand or respond to the question, or have complicated formatting that would require tactile display

Braille will be delivered to the student via an 18-cell refreshable Braille display (Literary Braille only)

Include items that need verbatim text plus simple, non-interpretive transcriber notes (e.g., inserting a one-word label for a simple graphic needed to understand or respond to the question)

Exclude items that need Nemeth Braille, have a complicated graphic needed to understand or respond to the question, or have complicated formatting that require tactile display

Braille will be delivered to the student via a Braille Embosser (both Literary and Nemeth Braille)

Include items that need verbatim text only but require either Nemeth Braille or tactile display

Exclude items that have complicated formatting / graphics or graphics that require transcriber notes (e.g., labels)

Braille will be delivered to the student via a Braille Embosser (supports both Literary and Nemeth Braille)

Include items that need verbatim text and requires either tactile display or Nemeth Braille simple, non-interpretive transcriber notes (e.g., inserting a one-word label for a graphic needed to understand or respond to the question)

Exclude items that have complicated formatting/ graphics

Include items in that have complicated formatting or graphics

If you spend 15 seconds on an item, it is a “5”

ODE will develop new items that cover the same content standard and item difficulty, but that is formatted in a manner appropriate for Braille users

OAKS Online Braille Item Categorization Criteria

Output Device 18-cell Refreshable Braille Display Braille Embosser

Categories 1 2 3 4 5

Category Descriptions

Definitely Yes Yes, with Simple Transcriber Notes

Yes, with Braille Embosser

Yes, with Braille Embosser and Simple Transcriber Notes

Omit for Now

Criteria for Inclusion

verbatim text only includes “eye

candy” graphics

verbatim text plus simple, non-

interpretive transcriber notes (e.g., label)

verbatim text plus needs tactile

display or needs Nemeth

Braille

verbatim text plus needs tactile display

plus simple, non-interpretive transcriber notes (e.g., label) or

needs Nemeth Braille

includes necessary complicated graphic or formatting

When in doubt, it’s a Category 5

Criteria for Exclusion

includes necessary graphic

needs tactile display or

needs Nemeth Braille

includes necessary complicated graphic

needs tactile display or

needs Nemeth Braille

includes necessary complicated graphic or formatting

needs transcriber notes (e.g., labels)

includes necessary complicated graphic or formatting

Type of Braille Supported Literary Braille only Literary and Nemeth Braille

If in doubt, assign the item to a higher category →→→ (15 second review limit per item)

Conservatism Rules!!!15-Second Rule