Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work

Aug. 9, 2007 Gehringer: Improving Course Materials … Through Peer Review … [email protected]

Expertiza: Improving Course Materials Expertiza: Improving Course Materials and Learning Outcomes through Peer and Learning Outcomes through Peer

Review of Student WorkReview of Student Work Edward F. GehringerDept. of Computer Science

North Carolina State University

Supported by NSF DUE under a CCLI grantNCSU Center for Teaching and Learning

NCSU LITRE (Learning in a Technology-Rich Env.)


Outline Introduction Expertiza Rationale Demo Experiment 1: Improving a Textbook Experiment 2: Review Wiki Submissions


Introduction Electronic peer review is students reviewing

other students’ work over the Web. Building resources through electronic peer

review gets students working together to improve others’

learning experiences, helps them learn, by performing tasks that are similar

to real-world responsibilities, gives them experience in writing their ideas up for an

audience of their peers, allows each cohort to “stand on the shoulders” of

students in earlier classes.


Homework, traditionally … For students to demonstrate mastery of

the subject. Every student does the same thing—

redundant effort. Work is graded and thrown away, never

benefiting anyone but the student who did it. Now the best work can be published, to

help others learn.


Outline Introduction Expertiza Rationale Demo Experiment 1: Improving a Textbook Experiment 2: Review Wiki Submissions Conclusion


The Rationale Improve student learning Improve teaching Better utilize resources


Improving student learning The Expertiza platform improves student

learning in these ways: Integrates active and cooperative learning

Active learning allows students to take responsibility for their own learning.

Extends active learning to out-of-classroom activities and distance education

DE has been a roadblock to the use of active learning … students viewing lectures remotely can work only by themselves.

Discourages plagiarism Multiple deadlines and milestones make it impossible

to submit a finished product obtained from an external source.


Improving teaching The Expertiza platform improves teaching in

these ways: Increases the supply of examples/homework

problems/test questions Students are assigned to make up such

examples/questions, and these are peer reviewed.

Focuses students on explaining/understanding the concepts that are hardest to master

“Write an example to clarify the hardest concept in lecture k.”


Improving resource utilization Some work is peer-graded, so teaching

assistants can spend more time working with students and less time grading. Having inadequate TA support no longer limits the

amount and kinds (e.g., design problems) of homework that can be assigned.

Students rely more on their peers for help, less on the course staff.

Makes teaching large classes an advantage! Large classes can produce more and better resources.







Improving an OOD Text Our class

Master’s level course CS & ECE Substantial DE enrollment

In Fall 2005, we used a new object-oriented design text for the first time, Dale Skrien’s, An Intrduction to Object-

Oriented Design and Design Patterns Using Java


Three Homework Assignments

Improve an explanation of a topic covered in the text.

Create a new example of a concept covered in the text.

Write a new exercise for a chapter in the text.

All students did not do these exercises in the same order.


The Review Process: How It Begins

Classroom discussion of the most difficult topics

Students select a topic from a list. Several students are allowed to select the

same topic, But the number of slots is limited.


Signing up for a First-Round Topic


Student Choices


The Review Rubric Students review a submission based on the following

questions (for examples). Does the example fully illustrate the concept being

explained? Is the example easy to understand, i.e., as clear as it

could be and still illustrate the concept? Does the example model the real world, i.e., could it be

implemented in practice? Is the example code elegant? Does the example use up-to-date Java code or UML? Does the text that accompanies the example explain it

well? Is the example more useful than the examples in this

section of the book? Does the example seem to be original? Other: How would you rate the submission on factors

not reflected in the scores on other questions?


After the Initial Review

Resubmission phase. 2–7 days to revise work in response to reviewer comments.

Grading phase. 3–7 days to make final comments and assign scores.

Review of review phase. Students review each other’s reviews.


Results

17 student submissions selected for text!

I learned a lot from doing the peer-reviewed

assignments related to the text.

0

5

10

15

20

Stron

gly A

gree

Agre

e

Neut

ral

Disa

gree

Stron

gly D

isagr

ee

I enjoyed doing the peer-reviewed assignment

related to the text.

0

5

10

15

20

Stron

gly A

gree

Agre

e

Neut

ral

Disa

gree

Stron

gly D

isagr

ee




Teaching with Wikis A wiki is, essentially, a Web site that can

be edited by any user. Homework done on wikis promotes

collaboration between students. Problem: How to assess so much writing. Solution: Peer review.


Our Approach Students select from a set of topics for a weekly

wiki assignment. The students are given class time to work on

the topic in groups. Then they take a couple of days to finish up

their submission. Their submission is peer-reviewed by other

students. They have a chance to revise their submission. Students rate the contributions of their

partners. Instructor considers the reviews, revisions, and

partner evals in assigning grades.


Students Seem to Benefit Wiki seems to be preferred as a submission

mechanism. “The wiki was really fantastic for creating pages

because it guarantees a uniform style for everything, putting the major focus on the content created, rather than formatting issues. It also helped format content so that it was more understandable.”

Contributions are more extensive than we have seen when files are submitted and resubmitted.

Peer review is the only scalable solution. Expertiza gives a framework for review.


Other Uses Lecture annotations Code reviews: Contributions to OSS

projects Class “proceedings” in a research course

Survey papers or research papers FAQ in a service-learning course


Future Work “Peer-assisted” review Total Quality Management Automated reviews of reviews


Conclusion Students can do good work—

work that will help their classmates learn the material

work that will help the instructor improve the class

work that allows each class to “stand on the shoulders” of earlier classes.

The Expertiza approach enables faculty to introduce these exercises into their courses.


For more info … http://research.csc.ncsu.edu/efg/expertiza Ed Gehringer, [email protected]

Documents

Expertiza: Improving Course Materials and Learning Outcomes through Peer Review of Student Work