10-21-14 Review: Responding to Student Writing Understanding Rubrics Finish TAW (Assessment) Agenda: Starter: Discussion of Responding to Student Work

10-21-14

•Review: Responding to Student Writing

•Understanding Rubrics

•Finish TAW (Assessment)

Agenda:

Starter: Discussion of Responding to Student Work

Teacher Comments on Successful and Unsuccessful Assignments…

Traits of Successful Assignments Students have a degree of choiceStudents are interested in the workStudents have a personal connection Work is relevant to student goalsAssignment is concrete & specific:

*clear instructions*clear expectations

Teacher provides tools (scaffolding)and feedback along the way

Models of successful & unsuccessfulwork are provided

Includes low stakes elements beforehigh stakes performance

Is appropriate for students’ ability leveland confidence level

Traits of Unsuccessful Assignments High stakes without adequate practiceTask is artificial (i.e., meaningless) Format is unclearSome terms are undefinedWork has no clear value to students Work is beyond students’ capabilitiesWork is outside students’ comfort zoneWork is outside students’ trust zoneWork is overwhelming

ProofreaderFixes errors (and assumes responsibility for finding errors)

EditorPolishes the text; tells writers what to do to improve (and assumes some responsibility for the quality of the revised draft)

CriticEvaluates quality; points out positives and negatives (and assumes responsibility for “correctness” of evaluations)

CheerleaderPraises the “good stuff” (and, like the critic, assumes responsibility for “correctness” of evaluations)

Roles Readers Can Play When Responding

FacilitatorHelps writers make their own decisions; points out potential problem areas and suggests options (and makes the writer responsible for his or her decisions)

AllyTries to help authors get by the gatekeeper (and shares some responsibility with the writer, at least for offering good advice)

GatekeeperStops sub-standard material from passing (and is responsible to the institution for “quality control”)

Quality of ideas

Appropriateness of the material

Accuracy of content presented

Organization of ideas

Depth/development of ideas

Likely audience reactions

Stylistic issues

Grammar/mechanics issues

Aspects on Which to Comment:

From Straub & Lunsford, 12 Readers Reading

Ways to Respond:

From Straub & Lunsford, 12 Readers Reading

More Options for RespondingNo Responding: Sharing

Sometimes it’s enough simply to let the writer read aloud

Descriptive Responding

•Sayback – Tell the reader what the text says to you

•Pointing – Point to, or identify, key words or phrases

•What’s Almost Said – Identify what the writing implies

•Structure / Voice / Point of View / Level of Abstraction / Attitude toward Reader / Language / Diction / Syntax – Identify these aspects for the writer

•Metaphorical Description – Describe the shape or some other feature of the text

Analytical Responding

•Skeleton feedback – Identify reasons & support, assumptions, and the implied audience

•Believing – Accept everything & offer additional ideas to help build the case

•Doubting – Challenge everything & offer counter arguments that are not addressed

•Descriptive outline – Explain what the text says and does

Reader-Based Responding: Movies of the Reader’s Mind

Criteria-Based Responding: Compare Text to a Rubric

From Elbow & Belanoff, Sharing & Responding

Teacher Comments on Effective and Ineffective Responses…

Traits of Effective Responses Specific goals/questions/tasksFocused/targetedPositive (i.e., praise)Evaluations included suggestionsNot too numerous: just a few comments per paperMaybe combine with a short conference to be sure students understand commentsTrain students how to read our particular comments

Traits of Ineffective Responses Generic (“nice,””good,””awk”)Not fully explainedToo many comments per paperIncomplete thoughtsIllegible

Respondingcould be, but is not necessarily, the same as

Assessingwhich could be, but is not necessarily, the same as

Grading

What does a grade mean?

Does an A in an Advanced Placement class mean the same as an A in a “regular” class?

Does a B in Mrs. Smith’s classmean the same as a B in Mr. Jones’ class?

What’s the difference between an 83 and an 85?

Does a C mean “OK” or “almost failing”?

Grades are not inherently meaningful; they can serve different purposes and even have different meanings in different situations. We therefore need to explain grades to invest them with meaning.

What IS a rubric?

Rubric = measuring stick

Fail Pass

Fail Pass

Fail Pass

Fail Pass?

A rubric should…

…list the criteria being evaluated

…describe a performance at each rating level

Ideally, the rubric should use similar language for each level within a category, so raters compare “apples to apples” when making judgments about that particular feature.

Also ideally, students should have models of each performance level for each category.

Kinds of Rubrics

Analytical Holistic

Score = sum of scores for individual traits

Score = reader’s impression of overall quality

Usually Formative:

intended to help thewriter identify specificareas to work on

Usually Summative:

intended only tomeasure the qualityof the writing

Yields more information Quicker & easier

BOTH can be useful

SCORE CONTENT/DEVELOPMENT ORGANIZATION VOICE CONVENTIONS

4

•Presents a clear central idea about the topic•Fully develops the central idea with specific, relevant details •Sustains focus on central idea throughout the writing

•Has a clear introduction, body, and conclusion.•Provides a smooth progression of ideas throughout the writing.

•Minor errors in standard written English may be present.

3

Presents a central idea about the topicDevelops the central idea but details are general, or the elaboration may be unevenFocus may shift slightly, but is generally sustained

Has an introduction, body, and conclusion.Provides a logical progression of ideas throughout the writing.

Uses precise and/or vivid vocabulary appropriate for the topicPhrasing is effective, not predictable or obviousVaries sentence structure to promote rhythmic readingStrongly aware of audience and task; tone is consistent and appropriate

Errors in standard written English may be present; however, these errors do not interfere with the writer’s meaning.

2

•Central idea may be unclear•Details may be sparse; more information is needed to clarify the central idea•Focus may shift or be lost causing confusion for the reader

Attempts an introduction, body, and conclusion; however, one or more of these components could be weak or ineffective.Provides a simplistic, repetitious, or somewhat random progression of ideas throughout the writing.

Uses both general and precise vocabularyPhrasing may not be effective, and may be predictable or obviousSome sentence variety results in reading that is somewhat rhythmic; may be mechanicalAware of audience and task; tone is appropriate

A pattern of errors in more than one category (e.g., capitalization, spelling, punctuation, sentence formation) of standard written English is present; these errors interfere somewhat with the writer’s meaning.

1

•There is no clear central idea•Details are absent or confusing•There is no sense of focus

Attempts an introduction, body, and conclusion; however, one or more of these components could be absent or confusing.Presents information in a random or illogical order throughout the writing.

Uses simple vocabularyPhrasing repetitive or confusingThere is little sentence variety; reading is monotonousThere is little awareness of audience and task; tone may be inappropriate

Frequent and serious errors in more than one category (e.g., capitalization, spelling, punctuation, sentence formation) of standard written English are present; these errors severely interfere with the writer’s meaning.

B BlankOff TopicInsufficient amount of original writing to evaluateUnreadable or illegible

OT

IS

UR

Why USE rubrics?

Identify the target

Save time

Avoid bias

Be transparent about grades

Prepare students for statewide tests

Principles for Using Rubrics Responsibly

Use a rubric that matches your instructional goals.

Acknowledge the limitations of rubrics.

Remember that rubrics don’t simply measure quality;rather, they define quality.

Distribute rubrics to studentsat the BEGINNING of the assignment.

Use a variety of rubrics. (One size DOES NOT fit all!)

Use a rubric that matches your instructional goals.

(Measure what you teach.)

___/60: Ideas

___/40: Organization

___/100 Total Points

Acknowledge the limitations of rubrics.

Irony?Irony?

Wit?Wit?

Using last week’s Using last week’s vocabulary words?vocabulary words?

Creative

Creative

subheadings?

subheadings?Avoiding a problem

Avoiding a problem encountered last time?

encountered last time?

Following the

Following the teacher’s advice?

teacher’s advice?

Remember that rubrics don’t simply measure quality; rather, they define quality.

“How sweet of her to write! What lovely handwriting! ”

“Did Bobby really say that???

“Organization? Diction? Development? Conventions?”

Distribute rubrics to students at the BEGINNING

of the assignment.

Use a variety of rubrics. (One size DOES NOT fit all!)

BeginningMiddleEnd

_____ Word count

_____ Number of examples

_____Creativity

_____Showing connections

_____Fourteen lines

_____Iambic pentameter

_____Rhyme scheme

_____Coherent stanzas

_____Final couplet

____Audience awareness

____Format

____Conventions

DONEDONE / NOT DONE

NOT DONE

Last week’s sample paper got grades from D+ to A. Let’s try scoring it with a “standard” rubric.

(Yes, this is a “top-down” rubric imposed on us by the administration, so the rubric is not directly related to the assignment. Still, rubrics do get imposed on teachers this way, so you might face such a situation in your classroom.)

Assignment for sample paper

Select a “hot topic” in education – an issue about which there is currently discussion and disagreement – and write a paper of 800-1200 words in which you explain the topic and the different views, cite at least four published articles (or editorials or book chapters) relevant to the discussion, and state and support your position on the issue. Use MLA format for citations, and include a Works Cited page.

Questions or comments so far?

Let’s look more closely at a high-stakes rubric for high school students…

SCORE OF 6An essay in this category demonstrates clear and consistent mastery, although it may have a few minor errors. A typical essay

•effectively and insightfully develops a point of view on the issue and demonstrates outstanding critical thinking, using clearly appropriate examples, reasons, and other evidence to support its position

•is well organized and clearly focused, demonstrating clear coherence and smooth progression of ideas

•exhibits skillful use of language, using a varied, accurate, and apt vocabulary

•demonstrates meaningful variety in sentence structure

•is free of most errors in grammar, usage, and mechanics

Scoring Guide for the SAT Essayhttp://www.collegeboard.com/student/testing/sat/about/sat/essay_scoring.html

SCORE OF 5An essay in this category demonstrates reasonably consistent mastery, although it will have occasional errors or lapses in quality. A typical essay

•effectively develops a point of view on the issue and demonstrates strong critical thinking, generally using appropriate examples, reasons, and other evidence to support its position

•is well organized and focused, demonstrating coherence and progression of ideas

•exhibits facility in the use of language, using appropriate vocabulary

•demonstrates variety in sentence structure

•is generally free of most errors in grammar, usage, and mechanics


SCORE OF 4An essay in this category demonstrates adequate mastery, although it will have lapses in quality. A typical essay

•develops a point of view on the issue and demonstrates competent critical thinking, using adequate examples, reasons, and other evidence to support its position

•is generally organized and focused, demonstrating some coherence and progression of ideas

•exhibits adequate but inconsistent facility in the use of language, using generally appropriate vocabulary

•demonstrates some variety in sentence structure

•has some errors in grammar, usage, and mechanics


SCORE OF 3An essay in this category demonstrates developing mastery, and is marked by ONE OR MORE of the following weaknesses:

•develops a point of view on the issue, demonstrating some critical thinking, but may do so inconsistently or use inadequate examples, reasons, or other evidence to support its position

•is limited in its organization or focus, or may demonstrate some lapses in coherence or progression of ideas

•displays developing facility in the use of language, but sometimes uses weak vocabulary or inappropriate word choice

•lacks variety or demonstrates problems in sentence structure

•contains an accumulation of errors in grammar, usage, and mechanics


SCORE OF 2An essay in this category demonstrates little mastery, and is flawed by ONE OR MORE of the following weaknesses:

•develops a point of view on the issue that is vague or seriously limited, and demonstrates weak critical thinking, providing inappropriate or insufficient examples, reasons, or other evidence to support its position •is poorly organized and/or focused, or demonstrates serious problems with coherence or progression of ideas

•displays very little facility in the use of language, using very limited vocabulary or incorrect word choice

•demonstrates frequent problems in sentence structure

•contains errors in grammar, usage, and mechanics so serious that meaning is somewhat obscured


SCORE OF 1An essay in this category demonstrates very little or no mastery, and is severely flawed by ONE OR MORE of the following weaknesses:

•develops no viable point of view on the issue, or provides little or no evidence to support its position

•is disorganized or unfocused, resulting in a disjointed or incoherent essay

•displays fundamental errors in vocabulary

•demonstrates severe flaws in sentence structure

•contains pervasive errors in grammar, usage, or mechanics that persistently interfere with meaning

Essays not written on the essay assignment will receive a score of zero.


•point of view on the issue; critical thinking

•organization, focus, coherence, and progression of ideas

•use of language (vocabulary)

•variety in sentence structure

•grammar, usage, and mechanics

What SAT Readers are Attending To


6 - demonstrates meaningful variety in sentence structure

5 - demonstrates variety in sentence structure

4 - demonstrates some variety in sentence structure

3 - lacks variety or demonstrates problems in sentence structure

2 - demonstrates frequent problems in sentence structure

1 - demonstrates severe flaws in sentence structure

Sample Trait at Difference Performance Levels


6 - demonstrates meaningful varietymeaningful variety in sentence structure

5 - demonstrates varietyvariety in sentence structure

4 - demonstrates some varietysome variety in sentence structure

3 - lacks variety or demonstrates problems in sentence structure

2 - demonstrates frequent problems in sentence structure

1 - demonstrates severe flaws in sentence structure



6 - is well organized and clearly focused

5 - is well organized and focused

4 - is generally organized and focused

3 - is limited in its organization or focus

2 - is poorly organized and/or focused

1 - is disorganized or unfocused



6 - is well organizedwell organized and clearly focusedclearly focused

5 - is well organizedwell organized and focusedfocused

4 - is generally organizedgenerally organized and focusedfocused

3 - is limited in its organization or focus

2 - is poorly organized and/or focused

1 - is disorganized or unfocused


POV & Critical Thinking Organization & Focus Use of Language Sentence Structure Grammar, Usage, Mechanics

6

5

4

3

2

1

Analytically, the score would be 5+5+6+3+3, or 22. (ave = 4.4)

Holistically, should it be 4.4? 4? 5?

In mass scoring sessions (such as SAT or AP), readers “calibrate” their scores using “benchmark” papers – papers that experienced readers have agreed demonstrate a particular score. Readers discuss these benchmarks papers and “recalibrate” periodically throughout the scoring session.

In such readings, papers are measured against other papers in that group: that is, some papers should get the top score and some should get the bottom score, as all the scores are relative. (Hence, a given paper could get one score one year and a different score another year.)

The moral:

A rubric designed for someone else’s purposes might not be of much use

in your classroom.

So make your own rubrics!

Trait 1________

Trait 2________

Trait 3________

Trait 4________

Trait 5________

10 pts

9 pts

8 pts

7 pts

6 pts

5 pts

4 pts

3 pts

2 pts

1 pt

YOU decide what traits

to score

YOU decide the value of each trait

YOU create “descriptors” that describe performances at each level

Trait 1title

Trait 2hook

Trait 3intro

Trait 4conclusion

7 pts main idea & preview are artful, clever, or

intriguing to me

6 pts intro includes

main idea & preview

5 ptstitle grabs my

interest & makes me want to read more

hook grabs my interest & makes me want to read more

4 ptstitle reveals some,but not too much,

about topic

intro includesmain idea or

preview, but not both

3 pts hook is mildly

interesting to me

conclusion artfully ties up the piece

2 ptstitle is dull

or summarizesthe topic

has a separate introduction

conclusion gives me a sense of closure

1 pt has a titleopening line seems designed to hook

the reader

opening paragraph is both intro & body

final paragraph is separate from body

paragraphs

0 pts no title no hook no intro no conclusion

Suppose, for example, that you’ve been working on writing introductions and conclusions…

… so you create your own 20-point rubric for scoring only the intro and conclusion:

Trait 1________

Trait 2________

Trait 3________

Trait 4________

Trait 5________

10 pts

9 pts

8 pts

7 pts

6 pts

5 pts

4 pts

3 pts

2 pts

1 pt

YOU AND YOUR STUDENTS decide

what traitsto score

YOU AND YOUR STUDENTS decide the

value of each trait

YOU AND YOUR STUDENTS create “descriptors” that

describe performances at each level

YOUR STUDENTS learn to score

their own drafts

•WHY am I including these particular traits (and not those other ones)?

•WHY am I valuing this trait at X points and that one at Y points?

•HOW WELL do my descriptors match what I’m actually teaching?

•HOW WELL can my students evaluate their own work using this rubric?

Creating your own rubric forces you to REFLECT on your practice:

Key questions:

Analytical or holistic?

How many traits?

How many points (total or per trait)?

Who will be using the rubric?

Reality Check

Assessing does not always equal grading

Rubrics are no panacea for assessment

Brief interview with Maja Wilson

Maja Wilson, Alfie Kohn, and others have written about some of the problems with – and even potential dangers of – using rubrics.

http://www.youtube.com/watch?v=kH3PSjh2G20

http://www.youtube.com/watch?v=kH3PSjh2G20

A few concerns about rubrics:

• They create the illusion of objectivity. (The decision to include to exclude any particular trait is a subjective one.)

• They force readers to set aside their professional experience and read “as if” they were somebody else. (That is, rubrics create the fiction of a single reader, although in the real world, no two readers are exactly alike.)

• They can’t capture any feature not specifically targeted.

• They can stifle students’ creativity, especially if students write for the rubric.

• A rubric can be a substitute for “meaningful response.”

• The standardization of rubrics could lead to the standardization of writing.

Wilson’s principles for assessment (from RR, 64-65):

• Honor rhetorical purpose and effect, acknowledging that different readers read differently.

• Encourage new insights; allow readers to identify values in response to specific pieces of writing.

• Understand (and value) disagreement by using it to explore the writer’s purposes and the reader’s responses.

• Encourage readers to articulate their positions.

• Teach students how to extract clarity from disagreements.

(Short version: Acknowledge the messiness of writing.)

Gallagher’s List of Purposes

•Express & reflect•Inquire & explore•Inform & explain•Analyze & interpret•Take a stand•Evaluate & judge•Propose a solution•Seek common ground

You read the text differently, depending on its purpose.

You should therefore also assess the text differently, depending on its purpose.

Subject

Text

Writer

CONTEXT

Reader

What clues in the TEXTtell us what the WRITERthinks about the READER?

What clues

in the T

EXT

tell us th

e write

r’s PURPOSE?

What clues in the TEXTtell us about the CONTEXT?

Gallagher on Assessment

Assessment should improve writing

Adopt the stance of a reader, not a grader

Read through a craft lens

Gallagher on Assessment

Assessment should improve writing

Adopt the stance of a reader, not a grader

Read through a craft lens

Assess during process, not just at the end(Surgery is too late when the patient is already dead)

Gallagher’s Tenets for Teaching Editing

Address needs as they arise

Teach to individuals more than to the class

Don’t use grammar books; make them

Keep the focus narrow

Teach the “big” conventions (p 150)

Don’t correct too much per page

Student editing is generally ineffective

Track the “spelling demons”

Use conferences

“I am not Superman…”

This draft looks good. I like the way you bring in an extra source to comment on the topic of grade inflation today – years after the Staples article was published. The thesis and topic sentences are generally clear (see green highlights), and you offer a fair amount of text-based evidence. Most of the sentences make sense and are grammatically sound. I marked a few errors (blue) and a few questionable lines (yellow), so you can see there’s still room to make your work clearer and cleaner – but for this point in the semester, this draft is solid. B

If you can break out of the five-paragraph theme model, you might have some good papers. You make some good points, and your writing is delightfully error-free. Further, you make sense at the paragraph level. Citation note: I marked two pairs of sentences in gray. In each case, the quote illustrates some point in the sentence that precedes it. To show this connection, you could end the first sentence with a colon; then you wouldn’t have to say “he says” or “he exclaims” at the end. (Remember, a colon says, “Here’s some additional material to illustrate what I just now said.) It makes for smoother, tighter writing. Aside from the novel/short story confusion and the accompanying format errors, my only complaint is with the format: three unrelated features strung together to “support” a fairly obvious claim (that the story “illustrates the life as a parent”). That’s not really a purpose. (WHY is he illustrating parenthood? Is he maybe trying to make a point, or perhaps show that it’s not always as simple as some folks think? I think you could probably come up with a much better thesis, but that you bailed to the easy format so you wouldn’t have to think too hard. (In short, I’m saying that I think you could turn out a really good paper if you’d get away from this middle school format.) C+

The good news is that most of your sentences are solid, and a couple of the paragraphs actually engage in analysis. The bad news, however, is that the whole first third of this draft is merely a summary of the article – useful, but WAYYY too long – and you completely fail to cite any of the quotes, most of which are too long anyway. Starting out with a factual error doesn’t help. (The scene, not the whole book, takes place at a pool.) And using long quotes suggests that you don’t have much of your own to say. When you do have something to say, it’s pretty good. The paragraph that begins with “Another important lesion” (which, of course, should be “lesson,” not “lesion”) is actually pretty solid. Your writing is smooth and easy to follow, and you have something reasonable to say. If the whole paper were that good, it would get a much higher grade. I recommend that you do enough invention to give you enough to say to allow you to cut the extended summary and the overly long quotes; once you do that, you ought to have some good work. But this draft isn’t there yet. C-

End Comments from papers I recently graded

Classroom Realities

Students need to see lots of models

Good models > weak models

You need to write with your students

Gallagher’s Commandments

Remember: writers are fragile

Set “improvement” as your major goal

Focus on the writer, not just the paper

Intervene early

Conference as often as possible

Give students some voice with the rubric

Remember, it’s ENGLISH class, not just LITERATURE class.

(Yes, they need knowledge, but they also need skills.)

For next week, read and respond to ETC, ch. 5.

Philosophy statement due.

Documents

10-21-14 Review: Responding to Student Writing Understanding Rubrics Finish TAW (Assessment) Agenda: Starter: Discussion of Responding to Student Work