Assessment Literacy for the Middle Level Educator
Jennifer BorgioliLearner-Centered Initiatives,
Ltd.
Organizational Focus
Assessment to produce learning…
and not just measure learning.
Do you honestly want to know what X exactly is? Is your life going to be improved by momentarily knowing what x is? No. Absolutely not. This whole problem is a conspiracy against hardworking American students. Let me tell you, solving for X right now is not going to stop the recession. It fact, it’s not going to do anything. And another thing. When have you ever had to know what is X is in your long esteemed professional career? Exactly. This is a futile attempt for “educators” in this district to boast of their student’s success rate. I am going to go the rest of my life not knowing what X is. Because what is X when you really think about it? A letter, the spot, two lines crossing each other. I don’t think anyone will ever really know what X truly is because the essence of X is beyond our brain potential. In conclusion, Harry S. Truman’s middle name was just the letter S, not an actual name. Now that is a letter that’s actually being utilized. See, you learned something, and it was not because of this logarithm. The End.
Talking about the science of
our profession does not
discredit the art.
“Less than 20% of teacher preparation programs contain higher level or advanced courses in psychometrics (assessment design) or instructional data analysis.”
Inside Higher Education, April 2009
ImplicationsMinimize
interruptions.Make them worthy.
To be assessment savvy….
Essential Element #2
2f. The degree to which the middle-level educational program includes ongoing Standards-based assessments
Assessment• Definition: The strategic collection of
evidence of student learning. (Martin-Kniep, 2009)
Assessment: test as dogs: pitbull
• A thing and a process
“And while the exams may be a thoroughly vetted, sophisticated means of measurement, they are an inadequate, constricted form of expression.”
March 12, 2013SEATTLE’S LOW-STAKES TESTING TRAPPosted by Michael Guerriero
Read more: http://www.newyorker.com/online/blogs/newsdesk/2013/03/seattles-low-stakes-testing-trap.html#ixzz2NWx8bv00
What are the implications of chasing the pineapple?
“Standards-Based Assessment”
RL.05.06a: Recognize and describe how an author’s background and culture affect his or her perspective. (NYS)
Assessment considerations
1999 APA Testing Standards
“The higher the stakes of an assessment’s results, the higher the expectation for the documentation
supporting the assessment design and the decisions made based on the assessment results.” (Section 13)
Traditional Assessme
nt
Performance-
Based Assessme
nt
Performance-Based Assessments (PBAs)
A performance task is an assessment that requires students to demonstrate achievement by producing an extended written or spoken answer, by engaging in group or individual activities, or by creating a specific product. (Nitko, 2001)
PBA’s versus TraditionalLiskin-Gasparro (1997) and Mueller (2008)
Attribute Traditional Performance
Assessment activity
Selecting a response
Performing a task
Nature of activity Contrived Emulates real life
Cognitive levelKnowledge/
comprehensionApplication/
analysis/synthesis
Development of solution
Teacher-structured
Student-structured
Objectivity of scoring
Easily achieved Difficult to achieve
Evidence of mastery
Indirect Direct
PBA’s versus TraditionalLiskin-Gasparro (1997) and Mueller (2008)
Attribute Traditional Performance
Assessment activity
Selecting a response
Performing a task
Nature of activity Contrived Emulates real life
Cognitive levelKnowledge/
comprehensionApplication/
analysis/synthesis
Development of solution
Teacher-structured
Student-structured
Objectivity of scoring
Easily achievedDifficult to
achieve
Evidence of mastery
Indirect Direct
Validity = Accuracy
How do we ensure alignment and validity in assessment?Degrees of Alignment
Goal is Best Fit
If you want to assess your students’ ability to perform, design, apply, interpret. . .
. . . then assess them with a performance or product task that requires them to perform, design, apply, or interpret.
I cannot claim my
assessment is valid if I do not have
some type of blueprint
Minimum
Basic
Articulated
New York State Learning Standard: Read to collect and interpret data, facts, and ideas from unfamiliar texts (4 items, 15% of test)
23
The student chose a response that completes the sentence
with an inference that is related to another element in the
passage but not to the specified detail
The student chose a response that completes the sentence
with an inference that is related to the main idea of the
passage but not to the specified detail
Correct Response: The student chose the correct response,
demonstrating that the student can infer a detail from passage
text
The student chose a response that completes the sentence
with an inference that may be based on prior knowledge and not supported by the passage
24
The student chose a response that describes a point of view
that is mentioned in the passage, but that is not the
author or narrator's point of view
The student chose a response that describes a point of view
that is related to passage content, but that is not stated
or implied in the passage
Correct Response: The student chose the correct response,
demonstrating that the student can infer an author or
narrator's point of view
The student chose a response that describes a point of view that is contradicted by details
in the passage
How many?3-5
3 – 5 standards in a PBA (reflected in rows in the rubric)
3 – 5 items per standard on a traditional test
Reliability = Consistency
I cannot claim my assessment is
reliable if I do not have statistics to support my claim
Reliability
Indication of how consistently an assessment measures its intended target and the extent to which scores are relatively free of error. Low reliability means that scores cannot be trusted for decision making. Necessary but not sufficient condition to ensure validity.
Three Types of Measurement Error
• Subject effect• Test effect• Environmental effects
Subject Effects
Others…
• Fatigue• Sleep deprivation• Illness• Disability
Testing Fatigue
Test Familiarity Bias
Score
Score
In what ways do we knowingly
(or unknowingl
y) contribute
to measureme
nt error?
Test Effects
Examples
• Not enough space for a response• Confusing items• Typos• Misleading (or lacking)
directions• Scorer inconsistencies
10. Format the item vertically instead of horizontally.
From A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment by Haladyna, Downing, and Rodriguez
21. Place choices in logical or numerical order. Students should not have to hunt to find an answer. Answers should be
provided in a logical, predictable pattern.
Compare with . . .
Final Eyes isn’t about editing
rather “is this what you want the students to
see/read?”
From Haladyna:26. Avoid All-of-the-above.28. Avoid giving clues to the right answer, such as specific determiners including always, never, completely, and absolutely
Test from
Period 1
Test from Period 2
Engage in peer review “Final Eyes”
– Is each item aligned to a standard?*– Is each item rigorous?– Is each item fair?– Does each item have one, unambiguous
correct key?*– Are all plausible/text based?– Are all tasks meaningful and build upon
student comprehension?
*Very hard to answer without a test map
3. Develop Context-Dependent Item Sets for
Content Areas
Develop Test Maps and Item Analysis Procedures
• The higher the stakes of an assessment, the more we need to play by the rules
• If it’s a mid-term or final exam, there should be a test map.
• Consider also:– Item analysis– Using choice E (primarily for pre-
assessments)
Environmental Effects
In what ways might we knowingly (or
unknowingly) contribute to
measurement error?
Reliability = Consistency
3 general ways to collect evidence of reliability
• Stability: How consistent are the results of an assessment when given at two time-separated occasions?
• Alternate Form: How consistent are the results of an assessment when given in two different forms?
• Internal Consistency: How consistently do the test’s items function?
• Inter-rater reliability: How consistently do scorer’s use the scoring rubric?
Cronbach’s Alpha
“In statistics, Cronbach's (alpha) is a coefficient of reliability. It is commonly used as a measure of the internal consistency or reliability of a psychometric test score for a sample of examinees. Alpha is not robust against missing data.” (Nitko, 2012)
Item Analysis
“This is unfamiliar to me”
Percent of Students Selecting Choice “E”
One assessment does not an assessment
system make.
WHEN DESIGNING A PRE/POST PERFORMANCE
TASK •the standards and thinking demands must stay the same.
•the modality that students express their thinking through must also stay the same.
•the content of the baseline and post must be different. •the rubrics for the pre/post will be the same in terms of thinking and modality, but the content dimension will be different.
MS General Music ExamplePre Assessment Post Assessment
Students listen to 3 songs from Western African
Your mother has just returned from a business trip and is playing a song from the country she visited. (Students listen to song) Explain in a blog post how you know what country or region she went to based on the song you heard.
Students listen to 3 songs from Spain
Your task as a member of your school newspaper is to write an introduction to the new exchange student. Write a brief paragraph that introduces people to the music of her country.
Music 4a. Identify the cultural contexts of a performance or recording and perform (with movement, where culturally appropriate) a varied repertoire of folk, art, andcontemporary selections from the basic cultures that represent the peoples of the world
W.6-8.4: Produce clear and coherent writing in which the development, organization, and style are appropriate to task, purpose, and audience.W.6-8.2d: Use precise language and domain-specific vocabulary to inform about or explain the topic.
HS PE ExamplePre Assessment Post Assessment
The superintendent of a neighboring school is proposing cutting PE to 40 minutes once a month. Explain to the superintendent why her idea is a good or bad one.
Our State Congresswoman is advocating for increasing PE to 40 minutes a day for all students who are not playing a formal sport in school. Explain to the Congresswoman why you agree or disagree with her idea.
PE 2. iii. Accept physical activity as an important part of life. Self-renewal, productivity as a worker, energy for family activities, fitness, weight control, stress management, and reduction in health-care costs are understood as benefits of physical activity
W.6-8.4: Produce clear and coherent writing in which the development, organization, and style are appropriate to task, purpose, and audience.W.6-8.2d: Use precise language and domain-specific vocabulary to inform about or explain the topic.
HS Art ExampleStudents will create their own artwork that communicates the theme of personal identity as an artist through the purposeful use of the elements and principles of art. (NYSVALS Commencement 1.b & 1.c) Students will respond critically to their artwork through dictation or writing by using the language of art criticism to describe, analyze, interpret, and make a judgment regarding that work of art. (NYSVALS Commencement 3a & WHST. 9-10. 2.d)
Students will create their own artwork that communicates the theme of personal identity as a learner through the purposeful use of the elements and principles of art. (NYSVALS Commencement 1.b & 1.c) Students will respond critically to their artwork through dictation or writing by using the language of art criticism to describe, analyze, interpret, and make a judgment regarding that work of art. (NYSVALS Commencement 3a & WHST. 9-10. 2.d)
Grade 8 ELA (Local Assessment)
Pre Assessment Post Assessment
Does setting influence character?Select a character from a book
you read last year. Write a brief scene (with dialogue, an established setting, and plot) that places your character in NYC on September 10, 2001.
Write a second scene that places your character in NYC
on September 12, 2001.
Select a character from a book you read this year. Write a
brief scene (with dialogue, an established setting, and plot) that places your character in
Philly on July 3, 1776.
Write a second scene that places your character in Philly
on July 5, 1776.
It’s 2063. Write a first draft of a Wikipedia entry for these two
photos. Use what you
learned in SS 8 to
compare, contrast, and
make connections
to US history,
culture, laws, and society.
http://www.youtube.com/lciltd
Recommended
