Session begins at 1PM ET/12PM CT/11AM MT/10AM PT. Please configure your audio by running the Audio Set Up Wizard: Tools>Audio>Audio Set Up Wizard. Using Calibrated Peer-Reviewed Writing in the STEM Classroom What is your experience with scientific writing? Respond below by using the textbox tool (the 4 th icon down on the vertical toolbar to the left) Chad Wayne Associate Instructional Professor, Department of Biology & Biochemistry, University of Houston Todays Speaker: Welcome To Todays CIRTLCast! Learn more about CIRTL at EVENTS JOURNAL CLUB COURSES WORKSHOPS SUMMER INSTITUTES Using Calibrated Peer-Reviewed Writing in the STEM Classroom Chad Wayne, Ph.D. Dept. Biology & Biochemistry, University of Houston Where does a STEM student learn to write? (Discuss; you can respond below by using the textbox tool (the 4th icon down on the vertical toolbar to the left)) Science Writing Wasteland Where does a student learn to write? Writing Basics class through an English department Formulated science lab reports Some will experience one science course that has a large writing assignment Little to No exposure to writing in the sciences Very few see the actual literature Even fewer have used Pubmed or Web of Science or other search tools Even fewer still have done research and reported those findings Never visited a library much less used any of the search tools Dont understand the difference between good and bad sources Why is a.com website bad? Why is the primary literature good? Dont understand citations Why we use them When to use them Why writing in the sciences? Pros and Cons of Writing Assignment Pros Allows students to think beyond the course-presented material Develops communication skills Hones critical thinking skills Prepares them for their professional fields Cons Coming up with relevant assignments Plagiarism issues Standardized rubrics Having to read them all Its a lot of work! Overview Have the right tools Peer review & distribution system Plagiarism Use clear instructions Assignment instructions Peer review questions Be clear on your goals Explain the purpose of the assignment Prepare a good rubric Explain everything Teach and train How to write a paper How to review a paper (calibration) How to respond to critiques Be ready for grumpy students Recruit partners to help Library Writing Services The advantage of peer review Why? Solves most of the cons of writing assignment Allows students to see good and bad writing Sharpens their own writing skills Forces students to confront actual performance Hones critical thinking skills Helps students to understand how to assess their own writing using a standard metric/rubric If using multiple topics, exposes them to more subject material Prepares students for the professional world Why not? Students dont know how to grade Students dont understand what they are grading Students dont understand rubrics Students dont understand the material being presented Students need to be trained Calibration Teaches students the rubric and what the aim of the assignment might be Teaches the students how to generally grade a paper using a rubric Teaches the students how to grade like you. Tools needed Well-thought-out writing assignment Well-defined rubric Plagiarism checking/tracking mechanism Paper distribution mechanism Rubric-based, peer review mechanism Partnerships The Writing Assignment Should be level appropriate Freshman example: writing an abstract or a structured lab report Senior example: writing a 5000 word literature review; writing a research report documenting lab assignment or other research Instructions should clearly delineate the assignment parameters Should be accompanied with good examples of what the assignment should look like Use examples from former courses Convert a published paper into a word document in the format of your assignment Characteristics of bad assignment instructions Assignment is not clearly defined or ambiguous. Write a 10 page paper on ________. Explain how ______ works. General or broad topics Describe the physiology of the kidney. Explain parturition. Focused on course lecture material One goal is to get students beyond the classroom and explore. Examples of assignment topics Immunity. Describe the stepwise development of the B- lymphocyte into a functional plasma cell. Include in your discussion the B-cell surface markers that differentiate a B-cell from a plasma cell, the molecular pathways and the molecules that promote B-cell differentiation, proliferation and survival as a plasma cell. The functional role of these regulator molecules in maintaining the humoral immune response should also be explored. ipRGCs. What is an intrinsically photosensitive retinal ganglion cell? Describe the ipRGC, its similarities and differences to other retinal cells, its functional role in light detection, and the purpose of the ipRGC. Be sure to describe the molecular mechanisms of phototransduction in the ipRGC. Whats a rubric Scoring standard. Generally 3 to 5 levels of measure Unacceptable: Doesnt meet up to the standards that we expect from our students at this particular level Acceptable: About what we would expect from students at this particular level Exemplary: Exceeds our expectations for students at this level The measuring stick by which the writing assignment is judged So when writing a paper, this is the criteria that the student is trying to meet. What purpose does the rubric serve? Communicates clear expectations Provides feedback Serves as the standard for evaluation Assignment level Learning outcomes across a department, college, or university Aids the student with self- reflection and self-evaluation of their own work Criterion-referenced rather than norm-referenced Where do I get a rubric? Borrow Check your college or department Check what is available online and adapt it Build from templates Developing your own Identify what you want to assess. Identify the characteristics or dimensions (rows). Identify the scale (columns). Create descriptions for each characteristic and scale level. Test rubric. Revise. Plagiarism tracking services Most are subscription-based Check your college to see what tools are available Some do have free access Turnitin 3 tools (Originality, Grademark, Peermark) Compares submitted papers against an expansive database Student submitted works Published articles Web-based content Some Turnitin alternatives Plagtracker Viper (Scanmyessay.com) Plagiarisma Plagscan Advantage of plagiarism checking tool For instructor Checks papers for copied works Allows you to see degree of plagiarism For student Allows student to examine their own work and correct Compare against a standard of acceptability Promotes accountability No fuss, no muss for either instructor or student Reduces plagiarism Peer Review Mechanism Calibrated Peer Review Peermark Part of Turnitin DIY (do-it-yourself) Primary reason, we dont have writing assignments in the sciences. Example: Writing assessment score sheet UH B IOLOGY & B IOCHEMISTRY W RITING ASSESSMENT Scoring Guide 2014 The order in which the Criteria appear is not intended to suggest sequence. not applicable unacceptable Better than unacceptable acceptable Better than acceptable exemplary Criteria:NA Task Responsiveness: the composition establishes purpose and responds to the assignment task Development: the composition develops with regard to audience, genre, and/or conventions Engagement or Evidence-Based Reasoning: the composition engages with sources, evidence, and/or data Organization: the composition is organized to manage flow appropriate to genre Language Control: the composition demonstrates control of academic language Students find this totally useless Converting the rubric Students find a rubric grid difficult to understand For peer review, its easy to convert a rubric into a series of questions applicable to their assignment The grading questions need to be clear Bad: The abstract could stand alone away from the whole paper. (Scale 1 to 5) Good: The abstract could stand alone away from the whole paper. The abstract lacked ambiguity. As the reviewer, you could determine immediately what the paper was about, what the major points were going to be and what the authors conclusions were. (Scale 1 to 5) Questions should be extant, but not too many to overwhelm the student with drudge work 20 to 35 questions Need free response questions for reviewer comments. Need minimum limits Students crave feedback Peer Review example Distributing the paper Papers need to be submitted without identifying marks Ensures that peer review is blind Using Turnitin already tags paper to submitter, so no IDs on it for distribution Students need to be reminded or told to exclude IDs and names off papers. Other systems may force instructor to do extra work to remove IDs Mechanism of distribution for peer review Using Turnitin or CPR will randomly assign papers to each reviewer. Manual, random distribution is not fun Teach along the way Remember that your students have likely never been exposed to scientific writing, so its your job to teach them how Introduce them to writing throughout the assignment period Recommend against doing it in one class period Take 5-10 minutes over several classes, randomly distributed Serves as a constant reminder that the assignment is in progress and that there are stages of development Partnerships Library Usually have programs, tutorials, webinars, courses on how to do research Can help point students in right direction, especially with project knowledge Writing or Auxillary Services Can help student with writing issues Can help develop students writing skills Usually prepaid in student fees How to calibrate Requires an in-class and out-of-class component Out-of-class Student needs to read a paper (at least three reading assignments one each of good, bad, and excellent) Either use previous (anonymous) student papers for examples or prepare them yourself Student scores the paper using the questions from the converted rubric Have students turn in scores prior to in-class discussion for assessment In-class Poll students about paper using the rubric as the standard. Discuss why paper was good, bad, or excellent See what student thinking was Redirect thinking to match your own Uses 10 15 min of class time The carrot and the stick Each calibration needs to be scored to ensure participation Use scaled scoring to provide for flexibility during calibration period Repeat at least on 3 occasions, once for each paper You should see students beginning to align with the rubric Making the grade Student turns in a paper within the allotted time period Student receives n anonymous papers to read and score I recommend 5 papers/student (see Snell RR (2015) Menage a Quoi? Optimal Number of Peer Reviewers. PLoS ONE 10(4): e ) Student scores each paper based on each question from the converted rubric Student provides comments Limit the time for this to 2 5 days At the end of assignment period, student receives n anonymous reviews from classmates Grade is assigned based on reviews To ensure that students do reviews well, you will need to motivate them If reviewer scores are far outside the norm, I penalize the reviewers paper If reviewer fails to do a review, then for each review a fifth of their own paper grade is lost I average the five reviews for a final review score (1 to 5) which is converted into a numerical grade (50 to 100) To make it all work, student just needs to be trained (calibrated) Guaranteed pitfalls Students will not like being graded by peers They fear that their peers lack the ability to assess good vs. bad writing Countered by calibration and explanation Will complain about scores/grades Think their peers were overly harsh Countered by multiple reviews Demonstrate consistency in scoring Explanations go a long way Revisions One purpose of the writing assignment is to improve writing skills Recommend at least two drafts Peer review feedback leads to improvements Allows students to see how the readers see their work and to make appropriate adjustments Also strengthens students ability to critique Second (final) draft also is peer- reviewed Using same rubrics and mechanisms Randomized distribution ensures a broader rather than directed changes to students papers Grading considerations How do the reviews count toward the grade on the assignment Average of all or majority? How does participation in peer review affect grade? Define your scales as scores Consider what each score in the scale represents What is an equivalency of a 1 score? Of a 5 score? Converting a scale to a grade Define your formula before grading. Students need to understand the metrics before writing. How do you penalize? Late assignments Incomplete assignments Failure to follow directions/instructions Plagiarism Failure to participate in calibrations Failure to do peer review(s) Summary Have the right tools Plagiarism Peer review & distribution system Use clear instructions Assignment instructions Peer review questions Be clear on your goals Explain the purpose of the assignment Prepare a good rubric Explain everything Teach and train How to write a paper How to review a paper (calibration) How to respond to critiques Be ready for grumpy students Recruit partners to help Library Writing Services February CIRTLCast Series: Learning in the Active STEM Classroom February 3: Using Calibrated Peer-Reviewed Writing in the STEM Classroom Featuring Chad Wayne, University of Houston February 10: Working with First-Year STEM Students Featuring Pamela Bowen Smith, Bloomsburg University February 17: Flipped Classrooms: What Are They, and Why Should You Use Them? Featuring Martha Dunkelberger, University of Houston February 24: Details TBD