STUDENT-LED ASSESSMENT: RUBRIC DEVELOPMENT AND PEER GRADING IN WRITING

ASSIGNMENTS

Presented by Brice Particelli, PhD English Department, New York City

2017 Faculty Institute, Pace University

LIST

What do you expect from a good Faculty Institute session?

What are the central elements or expectations?

A small study on student responses to a writing assignment where student-led assessment was a central element, and a look into pedagogical implications.

Students developed the rubric for the assignment, and they graded each other based on that rubric.

STUDENT-LED ASSESSMENT: RUBRIC DEVELOPMENT AND PEER GRADING IN WRITING

ASSIGNMENTS

PAIRS

What strategies do you use to make expectations explicit for your writing assignments?

Do you use rubrics to communicate this?

(Perhaps use a recent writing assignment as example.)

GUIDING QUESTIONS FOR THE DAYHow can we make more aspects of assessment more formative than summative? (assignment sheet, rubrics, peer review, grades, etc.)

How do we make expectations more transparent (for the benefit of long-term student learning)?

What are the pedagogical implications of letting students develop the assignment’s rubric?

What are the pedagogical implications of letting students grade each other?

ENG 201: WRITING IN THE DISCIPLINES• “…writing effective essays and research papers in disciplinary modes

and in students' field of interest. It may include interviews, analysis of journal articles, and appropriate documentation style formats…”

• Focus on writing about their disciplines: • first a profile of an individual within their discipline (6 weeks) • then a genre study of a genre within their discipline (6 weeks) • and a presentation of how a genre works in their discipline (2 wks)

• Goals: • think critically about their field • think critically about how their discipline communicates • write more effectively within the language of their field • write more effectively across disciplines by better understanding

why and how each communicates differently

PROFILE ESSAY A “public” genre vs. “occluded” genre

Familiar to anyone who reads magazines, blogs, or websites that feature individuals

Scaffolded learning: from comfortable to complex. Genre study of the public, moving to self-guided genre study of the occluded

Read 20+ broad exemplar texts: from Sports Illustrated to Business Insider to New York Magazine to Foreign Policy to Elle

Six-week assignment

Genre Study: “Genre as a typified social action that responds to a recurring situation,” as Irene Clark writes, and one that can better be “understood as originating from suitability and appropriateness, rather than from arbitrary formal conventions” (2005).

DEVELOPING CLASS CULTURE• Readings that established metacognitive questions: How do writers learn to write

with authority? How do genres function within a discourse community? (i.e. Bartholomae’s “Inventing the University”). Goal: expose “authority,” help students access it.

• Read exemplar texts: Without a formal assignment sheet, we began reading profile essays (across fields, styles, and subjects)

• Write to question: Weekly Blackboard writing focused on genre study - comparative and focused analyses on audience, structure, intent, etc

• Write for emulation/creation: In-class writing to test the waters for their own profile subject, constantly questioning why writers make the choices they do (depending on audience, subject matter, experience, time, etc)

• Surveys (3) to broaden discussion: i.e. Survey 1: faith in grades despite stated lack of transparency; reliance on peers to understand assignment; benefits of peer review

• Discussions on grading: Questioned how writing is assessed and who determines the goals, professionally and in class. (Asked: Most frequently awarded grade at Pace?)

Rojstaczer & Healy - research on the average grades of hundreds of universities

Of all letter grades given in 4-year colleges, 45% are As; 34% are Bs; 14% are Cs.

ASSESSMENTMore problem with grades… • Often cited study by Brookhart (1991) that concluded that instructors offer a

“hodgepodge grade of attitude, effort, and achievement”

Why use a rubric? (To make assessment more meaningful and helpful) • In a 2010 review of rubric use in higher education, Reddy and Andrade found that

“One striking difference between students’ and instructors’ perceptions of rubric use is related to their perceptions of the purposes of rubrics. Students frequently referred to them as serving the purposes of learning and achievement, while instructors focused almost exclusively on the role of a rubric in quickly, objectively and accurately assigning grades” (439). This is a problem because research shows “higher achievement and deeper learning by students who have rubrics to guide their work” (439).

• Similarly, Margaret Heritage wrote, “When used by students as part of a formative assessment of their works in progress, rubrics can teach as well as evaluate. Used as part of a student-centered approach to assessment, rubrics have the potential to help students understand the targets for their learning and the standards of quality for a particular assignment, as well as make dependable judgments about their own work” (p. 437, 2010).

ASSESSMENTWhy get students involved in assessment? • Benefits to writer: Peer-review is frequently used: more eyes on their work,

offering more feedback. Students read each other with an assessment goal in mind (i.e. “look at the evidence used, specifically focused on xxxx.”)

• Benefits to reviewer: pushes reviewer to better understand their own writing through the critique of others (Lundstrom & Baker 2009)

Why get students involved in the grading piece of assessment? • It’s reliable: In a 2006 study of 708 online students across 16 different

courses at 4 universities, Cho, et al, found that when at least four peers grade a piece of writing, the aggregate grade is as reliable and valid as a single instructor. And in less that one-third of the courses, did students offer low ratings of reliability and validity for the peer assessment process (2006).

• It empowers students: It offers the same benefits as peer-review, but with higher stakes and greater responsibility. Students must embrace becoming experts (with authority) on our version of the genre.

TIC TOC• Rubric Development (3wks out): “popcorn” to create consensus;

assignment sheet and rubric sent out after that process (yes…late)

• “Rater Training” (1-2 wks out): Applied our rubric as a class to an anonymous student-written profile; Applied it as part of peer review to a peer’s profile; Applied it to their own

• Grading Day: Students brought in two printed copies. Each graded two, offering written feedback as well as the filled-out numeric rubric. • Rewrites Reminder: To push them toward formative-focused

responses (and remove social pressures) I reminded them that I allow for rewrites. Only 3 took me up on rewrites (few rarely do), but it helped ease the worry.

• Control: I graded independently, putting my grades in a spreadsheet before reviewing the peer-graded copies.

RESULTSAverage grade offered by students: 87.4% Average grade offered by instructor: 87.9% I sided with the student average in all but three cases, where a peer-grader was particularly harsh, offering a D when the other peer-grader (and I) offered a B- or C+. That resulted in an uptick to a final 88.3% average.

Inflated? They’d been informally graded by a peer, informally graded themselves, done two guided peer-review sessions, and gotten direct feedback from me. Plus they’d picked the subject. If they didn’t get a B, they’d probably missed multiple classes (and were lost)…

RESULTS

Survey 2: one week before due date (and peer grading):

• 89% Agreed or Strongly Agreed that “I understand the basic elements of this genre,” while 11% answered Neutral/No Opinion. No one Disagreed nor Strong Disagreed.

• The results for “I understand the requirements of this writing assignment” were even more positive, at 97% Agreed or Strongly Agreed, and 3% Neutral/No Opinion

Survey Survey QuestionStrongly Disagree Disagree

Neutral/No Opinion Agree

Strongly Agree

Initial:It is possible for students to grade each other's

written assignments fairly and competently2 9 18 8 1

Pre-Grading:I am confident my peers will grade my work fairly

and competently6 17 11 5 0

Post-Grading: My peers graded my work fairly and competently 1 3 8 15 3

Initial Survey Pre-Grading Survey Post-Grading Survey

Surveys over time: Students' confidence in their peers' ability to grade their work fairly and competently.

2

6

1

9

17

3

18

11

88

5

15

10

3

0

2

4

6

8

10

12

14

16

18

20

StronglyDisagree Disagree Neutral/NoOpinion Agree StronglyAgree

RESULTS

Student responses to the Post-grading survey statement "Grading my peers' writing, I learned (check all that apply)"

15 14 13 12 11 9 9

3

How to better evaluate peers

thoughtfully and effectively

A better understanding of how others

evaluate my writing

More about my own approach to this writing

assignment

About the subject-matter

within the essay

Styles and techniques that I can incorporate

into future writing

How to read similar texts

more critically

It reinforced things I already

know about writing and

reading

I did not learn anything of

value

0

2

4

6

8

10

12

14

16

RESULTS

A PROGRESSIVE JOURNEYAs Stadler writes, instructor expertise does not means that a “unidirectional practice of teacher-as-assessor is itself always justified or best. A strong case can be made that students should be taught how to change their pattern of thinking so that they know not only how to respond to and solve (externally sourced) problems but also how to frame problems themselves. They need this partly to guide their learning in between, or to prepare for, teacher assessments, but equally as part of their progressive journey into self assessment, and at more advanced levels, as a key skill for professional life” (2006)

QUESTIONS

Can this be used for occluded genres that are less familiar to students?

Can this be used in non-composition classrooms?

Is the metacognitive focus taking away time from content, or should the questions on discourse community and genre be considered as part of the content required of students (in any classroom)?

Was the central benefit the class-led rubric development, the peer-grading, or all of the above?