Rubrics and Peer-Review in Canvas: Examples from FAS and HKS

Daniel Jamous, Senior Instructional Technologist, Academic Technology Services, Harvard University Information Technology

Alice Heath, Master in Public Policy Candidate, Harvard Kennedy School

Kelly Miller, Lecturer on Applied Physics, Harvard John A. Paulson School of Engineering and Applied Sciences

Salimah Samji, Manager of Building State Capability Program, Center for International Development, Harvard Kennedy School

Allison Pingree, Director of Professional Pedagogy, Strengthening Learning and Teaching Excellence (SLATE), Harvard Kennedy School

IT Summit June 2 2016

UsingCanvasforPeerReviewatHKS:PolicyAnalysis&Design(API-505)

AllisonPingree &AliceHeathHarvardITSummit

Thursday,June2,2016- 2:20– 3:15pm

CourseContext:API-505,PolicyAnalysis&Design• Required forall1st-yearMastersofPublicPolicyStudents

• 4sections,eachtaughtbyadifferent faculty

• Integratesothercorecourses, leadingtodeliverables thatreflectthepracticeofreal-worldpublicpolicy:memos,oralbriefings, slidedecks

• Onesection(coursehead,Prof.JohnDonahue,chairofMPPProgram)mademostextensiveuseofpeerreviewandrubricsinCanvas

• Twoothersectionsusedpeerreview inCanvaswithsimplifiedrubric;4thusedpeerreviewviae-mail

WhyUsePeerReviewwithaRubric?WhyinCanvas?

• WeknewfromSpringExercise[experiential capstone]thatthedraft-feedback-rewritecycleisagreattoolforlearning.

• Wealsoknewfrompriorversionsofthiscoursethatthefacultysimplycouldn’tgivedetailedfeedbackoneverydraft,especiallynotintimeforittobeuseful.

• We’dlearned fromobserving theNegotiationcoursethattheMPPs—especiallyincohorts—wereskilledatdealingwitheachothercandidlybutthoughtfully.

• Coursefocusitselfistheevaluationofoptionsalongvariouscriteria - rubricallowedustobemoretransparent aboutquality,andpracticewhatwepreach.

• Weneededawaytomanagethelogistical complexityofpairingupstudentsandtrackingtheirexchanges.

Overall,HowDiditGo?In-classpollofstudents:

• verystrongsupportfortheusefulness ofthepeerreviews (~40%extremelyuseful,30%definitelyuseful,20%somewhatuseful,and10%orlessnotuseful)

End-of-courseevaluations:mentioned5xasoneof“2-3mostvaluableaspects”ofcourse:

• “Requiringustoprovidepeerfeedbackonthefirstmemoassignmentwasveryhelpful--forcedustodosomethingthatwewouldnothavedoneotherwise,butwasreallyvaluable.”

• “Thepeerfeedbackmechanismwasexcellent.Ourgroupfounditsovaluablethatweorganizedpeerfeedbackforthelastassignment eventhoughitwasn'trequired.”

Faculty:

• “Ithinkitwasoneofthebestfeaturesofthecoursethisyear.... Canvas,whilenotwithoutitsglitches, isahugeleapforward.”

ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde

thequalityofthepeerreview.

2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.

• Lineedits• Attachfiles• Usearubric• Writeageneral comment

3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.

ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde

thequalityofthepeerreview.

2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.

• Lineedits• Attachfiles• Usearubric• Writeageneral comment

3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.

ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde

thequalityofthepeerreview.

2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review

• Lineedits• Attachfiles• Usearubric• Writeageneral comment

3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.

ButNotWithoutItsChallenges...1) Canvasdoesnotallowyoutosetadeadlinewithinpeerreview,ortograde

thequalityofthepeerreview.

2) Thereareseveral differentwaystogivefeedbackonCanvaspeerreview,someofwhichmissthemainpointofdoingpeer review.

• Lineedits• Attachfiles• Usearubric• Writeageneral comment

3) Fromastudent’sperspective, findingtheirpeers’ reviewcommentsisnotintuitive-- thereareseveral routestoseedifferent typesoffeedback.

LessonsLearned• Strategies

• Ifyouwanttosetadeadlineorgradethepeerreview,createanewassignment.• Createadocumentwithinstructionsonhowtogiveandviewpeerfeedback.

• Usescreenshots.• Testtheinstructionsfromastudent’sperspective.

• Benefits• Tostudents:

• Peersofferedvaluableinsights• Givingfeedbacktoothersfocusedthemonassessment criteria,whichclarifiedhowtoimprovetheirownwork

• ToFaculty:• Logisticsfarmorestreamlinedandsimplified (vs.manual/email)• Easiertocheckcompletion• Requiredfewerdocuments– rubricintegrated• Abilitytouseanonymousfeedback

AAP 50 and Canvas

AP50lectures group work exams projects

• Physics as a Foundation for Science & Engineering

• new class in SEAS Sept. 2012

• 30 students first year, now 75 students

• undergraduate engineering students

• project and team based

• “Flipped classroom”

Applied Physics 50

Team & Project Based Learning

AP50lectures group work exams projects

• 3 projects per semester

• 3 groupings per semester

• final report for each project

• teaching staff provides feedback on reports via rubrics uploaded on Canvas

• teams resubmit reports (via Canvas) for a final grade

Canvas for Groups and Reports

Our$challenge:$How$to$diffuse$our$ideas$and$create$a$community$of$practice?

• Huge%demand%for%PDIA,%short%supply%of%faculty• Change%agents%in%developing%countries%cannot%afford%to%attend%an%executive%education%course– Sustainable%change%requires%teaching%teams%who%work%together

– There%is%a%spacing%effect%of%learning%in%one’s%own%environment%

• Why%not%experiment%with%a%MOOC?

Our$answer:$Canvas$Catalog

• We%already%had:– Blog:%International%following%(155%countries)– 34%high%quality%3P5%min%videos%%(why?)

• Canvas%Catalog– No%Harvard%ID%required%to%sign%up– Easy%to%use– Do%It%Yourself%(DIY)

What$did$we$do?

• Launched%our%first%MOOC%on%Nov.%8,%2015– 6%weeks–Weekly%assignments• Written%answers%(no%quiz)• 4%conceptual%and%2%openPended

–Weekly%reflection%exercises– Average%of%3P5%hours%of%work– Peer%review%and%rubric%tool%• to%grade%and%to%learn%from%each%other

Example$of$assignment$rubric

241$people$from$53$countries$completed$Part$I

Top$10$Countries

# ofStudents

United%States 34Nigeria 31Peru 21Great%Britain 15India 8Kenya 7Tanzania 7Pakistan 7Nepal 6TimorPLeste 6

What$did$we$learn?

79%$rated$the$Peer$reviewsgood$or$higher

“The$peer$reviews$were$truly$amazing."A"huge"part"of"it"was"helpful"and"usually"highlighted"areas"I"may"not"have"considered.”

“Very"useful"and"have"never$had$it$so$good$in$all$the$online$programs$i have"undertaken"in"the"past”

“I"think"the"peer"review"was"a"very"good"opportunity"to"engage"with"like=minded"persons"that"provided$competent$reflection$and$criticism.”

“Generally"good."In"fact"I"think"the"act"of"peer"reviewing"is"more"valuable"than"the"act"of"being"peer"reviewed."Doing"the"reviewing"forces"me"to"think"seriously"about"another"person's"take"on"the"question.”

“It"is$informative$to"read"others"reflections,"and"adds$to$the$community$feel$of"the"course.”

Student$Feedback$on$Peer$Reviews

“A"little"disappointing."A"couple"offered"interesting"comments"that"were"worth"thinking"about"but"others$were$delayed$or$not$completed.”

“Peer"review"assignments"were"very"useful"except"that"it"was"very"difficult"to"find"course"colleagues"to"comment"on"work"consistently."It"looks"like"people"were$very$busy$and$could$not$find$much$time$to$peer9review$work.”

“Sometimes,"a"reviewer"can"score"a"student"low,"because"of"his/her"limited$understanding$of$the$concepts.”

“Some"reviewers"do"not"simply"understand"the"concept"or"the"problem,"hence"they"provide"minimal"contribution."

Student$Feedback$(cont’d)

Valuable$feedback$and$peer$interaction

Generic$no$value$add

Limited$understanding$of$the$concepts

Limited$understanding$of$concept/language

Where$to$next?

• More%specific%questions/prompts%for%reviewer–What%did%you%like?–What%would%you%suggest%for%improvement?– How%does%this%differ%from%what%you%learnt?

• Assignments– Hired%graders%

• Flipped%classroom%Fall%2016– Opportunity%to%test%peer%review%in%residential%course%(calibrated%reviews)