IS Peer Grading a Valid Assessment for MOOC?HENG “PATRICK” LUO & ANTHONY C. ROBINSON

JOHN A. DUTTON E-EDUCATION INSTITUTE, PENN STATE UNIVERSITY

The Problem of Assessment Why is assessment needed in MOOCs?

◦ Differs a MOOC from an educational website◦ Closes the loop

Why is assessment difficult in MOOCs? ◦ Massive - workload◦ Open – heterogeneous student population◦ Online – technology affordances◦ Course – reliability and validity emphasized

Why peer grading might be the solution

Automated Grading◦ Pros: immediate, easy-to-setup◦ Cons: no feedback, can’t assess complex

assignments

Teaching Assistant Grading◦ Pros: accurate, meaningful feedback (maybe)

can assess complex assignments◦ Cons: expensive, takes time, “unsustainable”

Local Learning Center (blended)◦ Pros: accurate, meaningful feedback◦ Cons: difficult to build collaboration, only

accessible to certain students.

Pros ??

Peer Grading

Findings from Literature Review Peer Grading Reliability

◦ Measured by the agreement among peer graders◦ Generally positive findings regarding reliability ◦ Calculated by Pearson r, Proportion of Variance, or Intraclass Correlation◦ Sometimes mistaken for validity, much smaller body of research

Peer Grading Validity◦ Measured by the similarity between peer grading and instructor grading◦ Positive findings reported in various contexts

Peer Grading in MOOCs◦ Few empirical studies◦ Mixed experiences reported from both instructors and students

Coursera’s Peer Grading System Each submitted assignment is randomly graded by 3-5 peers.

Instructor provides the grading rubric and builds it into the Peer Grading System.

Evaluation can be quantitative, qualitative, or both.

Students can grade their own assignment.

Median Score is used as the final peer grading score.

The Context of Research (MGR)

Research Questions: RQ1: Can Coursera’s peer review system provide a reliable and valid assessment of student assignments in a MOOC?

RQ2: Does the use of median score provide a more valid assessment than the use of mean score when calculating the final peer grading scores?

RQ3: What are the perceived effects of peer grading on students’ MOOC learning experience?

Data Source

The structure of peer grading assignment data in Coursera’s database

Data Source

Attributes of a submitted peer grading assignment

Finding - Reliability

  Intraclass Correlation

95% Confidence Interval F Test with True Value 0

Lower bound Upper bound Value df1 df2 Sig

Single Measures  .262 .240 .284 2.774 1824 7300 .000

Average Measures  .640 .613 .665 2.774 1824 7300 .000

Intraclass Correlation Coefficient (Case 1) for Peer Grading Scores (N=1825)

Finding- Reliability

  Clarity of the Presentation

Convincingness of the Story

Quality of the Cartography

Aesthetics of the Design

Single Measures .216 .215 .176 .210

Average Measures .579 .578 .516 .571

Intraclass Correlation Coefficients (Case 1) for the Four Criterion Scores (N=1825)

Finding-Validity

instructor_grading peer_grading_median peer_grading_mean self_grading

instructor_grading 1 .619** .662** .341**

peer_grading_median 1 .952** .279**

peer_grading_mean . 1 .464**

self_grading 1

Pearson’s Correlation Coefficient between Instructor, Peer, and Self Grading Scores (N=93)

**. Correlation is significant at the 0.01 level (2-tailed)

Finding-Perceived Effects

Question Questions (1-5 from strongly disagree to strongly agree) N Positive* 

Mean

The peer review process helped me to grow and improve as a spatial thinker. 2121 63% 3.67

I received fair grades on my Lesson 5 Mapping Assignment from my peers. 1694 62% 3.75

The feedback my peers gave me on my Lesson 5 Mapping Assignment was useful. 1719 61% 3.69

The peer grading activity in this course made me feel more engaged in the course. 2027 63% 3.70

The peer grading activity in this course made me feel more connected with other students. 2039 57% 3.57

The peer grading activity in this course provided me with opportunities to review and/or reflect on the course content.

2044 72% 3.88

I would recommend keeping the peer grading assignment in future offerings of this course. 2185 70% 3.90

*. Agreed or strongly agreed responses from the students are considered as positive responses

Students’ Ratings of the Seven Statements Regarding the Peer Grading Activity in MGR

Conclusion The inter-rater reliability of peer grading is low.

The use of multiple student graders can significantly increase the peer grading reliability.

Peer grading seems to be a valid assessment of student assignment (both median-based or mean-based).

Self-grading is less valid and tends to produce inflated scores.

General perception towards the peer grading activity in MOOC is positive. Students think it is fair, useful, and beneficial to their MOOC learning experience.