Embed Size (px)
Citation preview
NCELA is operated under contract ED-04-CO-0094/0002 from the US Department of Education to The George Washington University. Our mission is to provide technical assistance information to state education agencies,
local education agencies, and others regarding the education of English language learners.
Test Development & Item Writing for Teachers of ELLs
March 31, 2010
Ms. Lauren Kennedy, Director of Test Development, SLTI
Ms. Catherine Casteel, Test Developer and Literacy Specialist, SLTI
If you are having any technical difficulties at this time, please call 1-866-239-3239. Thank you
• Welcome to the webinar on “Test Development and Item Writing for Teachers of ELLs.” Today’s webinar is hosted by the National Clearinghouse for English Language Acquisition, NCELA, located at the Graduate School of Education and Human Development at The George Washington University, funded through a contract with the U.S. Department of Education's Office of English Language Acquisition.
• NCELA's mission is to provide technical assistance information to state and local educational agencies on issues pertaining to English language learners.
• My name is Kathia Flemens, Ph.D., a Research Associate at NCELA and your Webinar facilitator.
Today our presenters are:Ms. Lauren Kennedy, Director of Test Development at Second Language Testing Inc.
and Ms. Catherine Casteel, Test Developer and Literacy Specialist at Second Language Testing Inc.
BLUEPRINTS AND ITEM WRITING
Test Design for Teachers
Validity
Degree to which a test and the resulting scores are meaningful for an intended purpose.
Scores will reflect progress toward learning goals when items/tasks are clear to students.
test content matches content standards/objectives.
items/tasks are represented in the same proportions as in the standards and objectives.
Content Validity
Degree to which test reflects the content domain.
Essential to standards-based assessment.
Content validity can be evaluated comparing the alignment of test tasks to content standards.
State assessments are based on content standards; ensuring classroom assessments are based on the same standards provides content validity for classroom tests.
Blueprint
Organizes and displays the mix of content and tasks Example
Concept Knowledge level
Application level
ComprehensionLevel
SynthesisLevel
Total Percent
Student will discriminate the subtraction sign from the addition sign
3 3 15%
Student will discriminate correctly solved subtraction problems from incorrectly solved subtraction problems
3 3 15%
Student will solve correctly single-digit subtraction problems 6 6 30%Student will solve correctly subtraction problems with double-digit numerators and single-digit denominators
6 6 30%
Student will explain their solution process for a double-digit subtraction problem.
2 2 10%
Total 3 12 3 2 20Percent 15% 60% 15% 10% 100%
Testing Vocabulary
Item – a test question Includes stem/prompt and any options
Stem – the stimulus (question, sentence) Open – incomplete statement complete statement or question
Closed – complete statement or question
Option – choices given to test taker to answer the stem Distractor – an incorrect answer
Key – a correct answer
Prompt – item that asks the student to create a constructed response
Multiple Choice: Good Practices in Writing Test Stems
Present a single, well defined problem
Base the item on a standard or objective
Write the item at the reading level of the students
Phrase the stem positively, not negatively
Use language that is clear, simple and precise
Stems: Clear, Precise Language
Poor: In carrying out scientific research, the type of hypothesis
which indicates the direction in which the experimenter expects the results to occur once the data has been analyzed is known as a(n) ...
Better: Closed
What is a hypothesis that indicates the expected result of a study called?
Open A hypothesis that indicates the expected result of a study is
called…
What to Avoid in Writing MC Stems
Questions that are based on the content of another item.
Frequency adverbs, such as “Always” and “Never”
Humor
Trick questions that intentionally mislead students
Items that can be answered by the content of another item.
Writing Multiple-Choice Options
Make comparable in content, length, & logic
Include the central idea in the stem, not the options
Use parallel language
Logically or numerically order the options
Avoid “All of the above” and “None of the above”
Create plausible distractors
Vary the location of the correct answer
Present vertically not horizontally
Research suggests that 3 options are adequate
MC Options: Avoid Grammatical Clues
What causes night and day?
A. The earth spins on its axis. [correct]B. The earth moved around the sun.C. Clouds have blocked the sun's light.
MC Options: Avoid Association Clues
Who are the “Magnificent Seven?”
A. a pro basketball teamB. a group of seven fictional Western figures [correct]C. a protest music group popular during the 1960sD. a set of nutritional foods
Option B includes the word “seven.”
MC Options: Order Numerically
A book sells for $9.99. If it is discounted
25%, how much does it cost?
WrongA. $5.00B. $7.50C. $2.50D. $6.66
Right
A. $2.50
B. $5.00
C. $6.66
D. $7.50
MC Options: Order Logically
The speaker’s tone in the passage can best be described as which of the following?
WrongA. contemptuousB. whimsicalC. pretentiousD. satirical
RightA. contemptuousB. pretentiousC. satiricalD. whimsical
Good Practices for Essay Prompt Writing
Topics should be meaningful, relevant and motivating
Prompt should be written in language that is clear and precise
Text prompts, which use reading as a basis for writing, may penalize poor readers regardless of their writing ability
Good Practices for Essay Prompt Writing
Prompt can be written as a statement or a question Specify and define what mental process you want the
students to perform Use verbs that define the writing task
Contextualize the writing Specify audience, purpose, genre, form
Structure the prompt to minimize subjective interpretations
Good Practices for Essay Prompt Writing
Provide complete instructions before the promptThe criteria for gradingDescribe where they are to write the essayDescribe how long the essay should be, or how
long a student has to write Include advice on organizing ideas and proofing
the essay
Things to Avoid in Essay Prompt Writing
Prompts that inadequately specify the task Example: Capital punishment. Discuss.
Prompts that rely on factual knowledge
Vague, ambiguous, or non-specific verbs
Providing a choice of essay questions Students may waste time deciding on an option.
Students may start-over, or mis-understand and write to all.
Some questions are likely to be harder to answer than others.
Improving a Test
Review – before administration Review a copy of the test. Evaluate it from a student perspective. Check for spelling, grammar, punctuation, capitalization. Review the standards/objectives covered. Ask a knowledgeable colleague to critically review the test. Revise the test based on feedback. Have revised version reviewed by someone else. Revise & finalize.
Review – after administration Examine student responses
Identify problematic questions—is the problem due to content or wording of the item or prompt? If due to content, that material needs to be re-taught. If due to construction, the item stem, prompt, or options need to be
revised.
Questions?
Thank you for having participated in today’s webinar on “Test Development & Item Writing for Teachers of ELLs” presented by Ms. Lauren Kennedy and Ms. Catherine Casteel; hosted by the National Clearinghouse for English Language Acquisition, NCELA, located at the Graduate School of Education and Human Development at The George Washington University.
• For more information or if you have additional questions regarding today's webinar topic contact:Ms. Lauren Kennedy at [email protected]. Catherine Casteel at [email protected]
or• If you have additional questions regarding the webinar contact:Kathia Flemens at [email protected] webinar will be archived on NCELA’s website. To view archived webinars, please visit http://www.ncela.gwu.edu/webinars/
