Through the Looking Glass:Insights into VisualizationPedagogy through SentimentAnalysis of Peer Review Text

Zachariah BeasleyUniversity of South Florida

Alon FriedmanUniversity of South Florida

Paul RosenUniversity of South Florida

Abstract—Peer review is a widely utilized feedback mechanism for engaging students. As apedagogical method, it has been shown to improve educational outcomes, but we have foundlimited empirical measurement of peer review in visualization courses. In addition to increasingengagement, peer review provides diverse feedback and reinforces recently-learned courseconcepts through critical evaluation of others’ work. We discuss the construction and applicationof peer review in two visualization courses from different colleges at the University of SouthFlorida. We then analyze student projects and peer review text via sentiment analysis to inferinsights for visualization educators, including the focus of course content, engagement acrossstudent groups, student mastery of concepts, course trends over time, and expert interventioneffectiveness. Finally, we provide suggestions for adapting peer review to other visualizationcourses to engage students and increase instructor understanding of the peer review process.

IN RECENT YEARS, visual communication hasshown growth in many professional fields andmedia, from scientific discovery to humanitiescommunications. It is thus no surprise that weare experiencing dramatic growth in the numberof students enrolling in visual communicationcourses—a growth that makes it difficult to pro-vide the detailed subjective feedback studentsneed to improve the quality of their work [7].The recent proliferation of online education hasfurther increased this pressure.

While visualization education as a wholeremains a work in progress [5], [18], we

have identified two broad visualization educationthemes. The first is the proper construction ofvisualizations—teaching students the algorithmsand visualization principles to use in creatingvisualizations. The second is the subjective evalu-ation of the quality and accuracy of visualizations.In other words, training students to criticallyevaluate others’ visualizations, which is the focusof our work. These necessary skills are com-monly taught through informal methods, suchas group or whole-class discussions. However,informal methods which lack active participa-tion leave students’ skills underdeveloped [19].

1

arX

iv:2

108.

0283

9v1

[cs

.HC

] 5

Aug

202

1

Isenberg et al. expanded upon this understandingby emphasizing “collaborative visualization” [9],further defined as the contributions of differentexperts toward a shared goal of understandingthe visual object, phenomenon, or data underinvestigation [17].

Peer review is a highly-engaging feedbackmechanism often associated with liberal artscourses but also found in professional code re-view and scholarly publication. It is particularlysuited to visualization education [1], [6]. Ratherthan rely solely on instructors for visualizationfeedback, peers collaborate to provide diverse,multi-sourced feedback to one another. Mostimportantly, the evaluation process itself givesstudents an opportunity to reinforce recently-learned course concepts through formal criticalevaluation.

In this article, we take a deeper look at fouryears of insights gained from peer review invisualization education. We begin by describ-ing the two peer review-oriented visualizationcourses we use in our analysis—one from thefield of Computer Science and the other fromthe field of Mass Communications. This includesa discussion on our generic peer review rubric,which was customized to reinforce key conceptsin each course. Next, we describe the collec-tion of peer review text from multiple semestersof each course and the evaluation of the datausing sentiment analysis and aspect extraction.Rather than directly compare the distinct courses,our primary goal is to demonstrate the typeof information that can be obtained from peerreview text, which is highly dependent on thevisualization course context and data collected.This methodology allows an instructor to morethoroughly analyze the peer review process to 1)determine its effectiveness at engaging students,2) understand whether students have masteredcourse concepts, 3) note course trends over time,and 4) identify the effects of expert intervention.Finally, we share insights that others may drawfrom to improve their own visualization coursesthrough the use of peer review.

PEER REVIEW IN VISUALIZATIONIn a review of several major visualization

venues, InfoVis, SciVis, VAST, EuroVis, and Pa-cific Vis, we found no empirical evaluation of

students’ work or engagement in the classroom,aside from our own [1], [6]. The use of peer re-view in the visualization classroom thus remainsan open area of research, despite the fact that cri-tiquing has long been acknowledged as a criticalpart of the visualization design process [10].

Fortunately, other disciplines have reportedextensively on peer review, and many of theirinsights transfer to the visualization educationdomain. In the liberal arts, e.g., numerous re-searchers have investigated whether the ability towrite can be mastered in one context and easilytransferred to another, a claim that remains dis-puted [22]. However, this concern primarily im-pacts the use of summative reviews for assessmentpurposes (i.e., providing a grade). It does notdiminish the value of using formative peer reviewto stimulate learning. In fact, peer review has beenshown to be a highly engaging, active-learningmechanism [23]. Active-learning, in general con-texts, has been shown to improve comprehension,retention, and overall educational outcomes overnon-active-learning approaches [12].

A THEORETICAL FRAMEWORK FORPEER REVIEW IN VISUALIZATION

Education researchers constantly rely on the-oretical frameworks to explain student behavior,experience, and means of success. Many of thoseeducational theories are designed to enhance stu-dents’ learning experience when teaching takesplace. In the field of visualization education,Oliver & Aczel reviewed four major educationtheories, including cognitive theory, cognitive di-mension, Popper-Campbell model, and Vygot-sky’s theory [16]. They evaluated their translationto visualization education under logic learningprocesses, where students acquire logic steps toreview a visualization.

In the field of writing analytics, researchersoften cite Vygotsky’s model for its influence onstudent peer review engagement success. Moxleynotes that Vygotsky’s theory helps instructors toimprove student reading, writing, and collabora-tion skills using an online dashboard dedicatedto peer review [13]. Using Vygotsky’s theory,Moxley built a rubric to accommodate studentengagement. Our previously reported peer reviewmodel for visualization was influenced by Mox-ley’s rubric and Vygotsky’s theory [1], [6].

2

In the field of visualization education, Oliver& Aczel examined Vygotsky’s model of the zoneof proximal development (ZPD) [16]. The ZPDmodel focuses on three important componentswhich aid the learning process: 1) the presence ofan individual with knowledge and skills beyondthat of the learner, 2) social interactions with atutor that allows the learner to practice skills, and3) supportive activities provided by the educatoror more competent peers. Oliver & Aczel reportedthat the ZPD model could provide active andtailored feedback to the learner, supporting thestudent’s learning experience. Using peer reviewin visualization courses supports the social andpeer components of the ZPD model.

A PEER REVIEW RUBRIC FORVISUALIZATION

Rubrics are used by instructors in a varietyof disciplines to provide feedback or to gradestudent products, e.g., writings, presentations, andportfolios [3]. Generally speaking, a rubric is apedagogical tool that articulates the expectationsfor an assignment. It lists the most importantcriteria and describes levels of competency frompoor to excellent.

However, effective rubrics are difficult to syn-thesize from scratch—they must consider coreconcepts of the course, project requirements, andoverall educational goals. In our Pedagogy ofVisualization 2017 workshop paper, we addressedthe limited availability of rubrics for peer re-view in the visualization classroom by provid-ing a generic rubric that can be customized tothe needs of the individual instructor, course,and project [6]. We later evaluated the effi-cacy of the rubric in a computer science vi-sualization course [1]. A LATEX version of therubric can be retrieved at https://github.com/USFDataVisualization/VisPeerReview.

The rubric was built by reviewing the contentof multiple visualization courses and extractingkey concepts necessary for demonstrating profi-ciency in learning their objectives. The structuredivides the rubric into five major assessmentcategories: algorithm design, interaction design,visual design, design consideration, and visual-ization narrative. The algorithm design categoryfocuses on algorithm selection and implemen-tation. Interaction design concentrates on how

the user interacts with the visualization. Visualdesign relates to the technical aspects of howdata are placed in the visualization (e.g., theexpressiveness and effectiveness of visual encod-ing channels). Design consideration correspondsto the composition and aesthetic aspects of thevisualization, e.g., embellishments. Visualizationnarrative is the final category, which is used insituations where the story is as important as thevisualization itself.

Each of the five major categories has a varietyof assessments associated with it. For scoringpurposes, each assessment is affixed to a 5-point scale. Furthermore, the intent goes beyondnumeric scores—providing a text-based commentis encouraged for every assessment category. Onefinal key element of our original rubric designwas the intention for a high-level of customizationbased upon the content of a course or project. Assuch, the rubric is versatile enough to work withcourses that have very different focuses, whichwe demonstrate in this article.

COURSES OVERVIEW

Data Visualization CourseOur first course entitled “Data Visualization”

was taught in the Computer Science departmentin the College of Engineering at the University ofSouth Florida. The course was offered as a mixedundergraduate/graduate elective.

Course Content. The course was taught usingMunzner’s Visualization Analysis & Design [15]with additional research papers and outside vi-sualization content (e.g., Vis Lies, New YorkTimes Graphics, etc.) to generate discussions. Thecourse materials were primarily presented usinglecture, research paper presentations by students,and discussions (e.g., small group and whole classcritiques).

The main learning objectives for the coursewere that students would demonstrate the abil-ity to: evaluate data and user requirements andprogram an effective visualization to match thoserequirements; associate elements of visualizationswith the basic components, e.g., data abstractionsand visual encodings, that are used in their con-struction; and critique interactive visualizationswith respect to their effectiveness for selectedtasks, visual encodings, and interaction design

3

and implementation.Project Design & Peer Review. We pursued

structured assessment (i.e., well-defined projects),consisting of eight projects, one in Tableau andseven in Processing, totaling 50% of the students’final grade. Projects deadlines were every 10-14days, except for Project 6, which was assignedover Spring Break and allowed approximately30 days. When designing projects, we desiredthat students gain visualization skills by demon-strating proficiency in using software engineeringproblem-solving techniques [4]. Many projectsincluded direct reuse of components and associ-ated feedback from previous projects.

The project design included a dual goal ofteaching visualization practice and software de-sign. Therefore, the projects were divided intofour categories: Familiarization: One project em-phasized familiarizing students with using stan-dard visualization types. Foundations: These threeprojects emphasized the implementation of thebasic mechanisms of visualization, e.g., data ab-straction, visual encoding, and interaction. Trans-ferability: In two projects, students applied theirfoundational skills to develop more complex visu-alization types. Software Engineering: These twoprojects use software engineering skills to buildand enhance a visualization dashboard.

Projects were set up to maximally build uponand reuse components from previous projectswhile still challenging students with new projectrequirements. Therefore, the peer reviews serveas an integral feedback mechanism by allow-ing students to use feedback to refine the workthey submit. To build the peer review rubric,we customized the rubric per project. To dothis, we simply extracted the relevant componentsfrom the full rubric template described earlier.For example, Project 1 was the only projectthat included a narrative component. Projects 4-8 required interaction, while Projects 1-3 didnot. Finally, the sub-assessments included forearly projects reflected only topics that had beencovered in class to that point.

After each project, students provided reviewsto three randomly selected peers’ work using theprovided rubric within 5-7 days. The peer reviewform consisted of a series of assessments, eachscored on a 5-point Likert scale. Each assessmentalso had an optional input box for text-based

feedback. Peer feedback was the primary form ofqualitative feedback given to students. In addition,students could request additional feedback fromthe instructor. Students received small amountsof (spot checked) completion-based credit, whichwas approximately 10% of the final course grade.Projects were graded by the instructor and teach-ing assistants primarily using objective require-ments, as well as some subjective judgment. Peerreview scores did not influence the project grade.

Visual Literacy CourseThe second course, “Introduction to Visual

Communication,” later renamed “Visual Liter-acy,” was taught in the Mass Communicationsdepartment in the College of Arts and Sciences atthe University of South Florida. The course wasoriginally designed for mass communication andjournalism undergraduate students. With a grow-ing number of students enrolled in the course,it eventually became a required course for allundergraduate students enrolled in the College ofArts and Sciences, and the course content wasexpanded to provide freshmen with a strongerfoundation for visual communication.

Course Content. Visual Literacy was a sur-vey course designed to give students a basicknowledge of the many forms visual communi-cation can take and provide practical experiencein developing projects. It spanned a wide range ofareas, including visual persuasion, data visualiza-tion, and visual storytelling. Under visual literacycourse learning objectives, students would notonly demonstrate the ability to use and generategraphic designs with design principles covered inthe class but would also review their peers’ workbased on the designated rubric. Furthermore,continued moves toward social media analyticspushed our curriculum toward exploring variousforms of visual content found on social and digitalmedia.

Project Design & Peer Review. To gainvisual communication skills, the course was basedon applying the aesthetics associated with mediaproductions and data findings. Assignments weredivided into fourteen different modules, whereeach module addressed a different visual tool orsoftware from using mobile phones as camerasto generating a logo with Adobe Creative Cloudto finding data and converting it to a visual map

4

Figure 1: Left: Submission for Visual Literacy visual map narrative. Right: Visualization of thequalitative feedback received from four peers.

narrative. In the data visualization module, thestudents learned about data collection and utilizeda combination of Plot.ly and Adobe CreativeCloud to generate infographics (see Figure 1).

Students’ performance was assessed viaweekly assignments with peer reviews. In the datavisualization module, the rubric that was utilizedhighlighted five major topics that matched thecurriculum: clear detailed labeling, the lie factor,the data-ink ratio, chart junk data density, andGestalt principles. The first category asked thestudent to search his/her peer’s visualization forclear and detailed labeling of every aspect of thedata represented in the graph or chart. The nextthree categories were based on Tufte’s principlesof design [20]. The lie factor directs the relation-ship between the size of the effect shown in thegraphic and the size of the effect shown in thedata. The data-ink ratio, the proportion of ink (orpixels) used to present data compared to the totalink used in the display, refers to the non-erasableink used for the presentation of data. If data-ink were removed from the image, the graphicwould lose its content. Accordingly, non-data-

ink is used for scales, labels, and edges foundin the visual work. The next category, known aschart junk, refers to all of the visual elements incharts that are not needed for comprehending theinformation on the chart or that distract from thisinformation. Finally, Gestalt principles describehow the human eye perceives visual elements—inparticular, they tell us that complex images reduceto simpler shapes.

In the data visualization assignment, eachstudent was asked to create a map based on datafound using leading open-source data repositoriesand visual dashboards. In the second stage of thisassignment, each student was asked to evaluateeach other’s work using a peer review rubric. Thepeer review form consisted of a series of assess-ments, each scored on a 4-point Likert scale. Inaddition, a single input box was provided for text-based feedback. Each week students submittedtheir assignments and conducted peer reviewson five peers’ assignments. At the same time,highly engaged students could optionally requestdetailed instructor feedback beyond peer reviews.The overall grades in the class were set by the

5

instructor, with the student’s peer review gradescounting for 10% of the overall grade assigned.

DATA COLLECTIONTo analyze both visualization courses, we

collected significant quantities of data. We uti-lized quantitative analysis via natural languageprocessing (specifically, sentiment analysis), cou-pled with qualitative analysis from representativestudent works.

Data DescriptionIn the Data Visualization course, the open-

ended review comments were gathered from eightprojects in 2017 and seven projects in 2018 and2019. The rubric contained multiple commentsections, each of which was concatenated intoa single string to produce 3,116 reviews. Ap-proximately 27% of the comments (or 846 outof 3,116) had no sentiment keywords and couldnot be scored but still contributed to the part-of-speech analysis. Students were incentivized witha completion point (spot-checked by the teachingassistant) to submit a high-quality review. Thenumeric score from the rubric was not included inthe dataset for comparison with the text. Finally,we manually reviewed student projects to findthose which, in combination with peer comments,exemplified the value of peer review.

In the Visual Literacy course, we collecteddata from Spring 2017 through Fall 2020 (eightsemesters). Open-ended peer review comments,with an associated numeric score, were collectedfrom a single project. Students received a com-pletion score for filling out a quality peer review.There were 1,571 reviews after those with nocomment, no score, or an out-of-bounds scorewere excluded. Of the remaining reviews, 29%(or 455 out of 1,571) had no sentiment keywordsand could not be scored but still contributed tothe part-of-speech analysis.

Analysis MethodologyRather than analyzing how a visualization

corresponds to an instructor’s grade, we focusedon peer review feedback that students gave toone another. In a prior paper, we found that stu-dents perceived the most benefit from reviewingothers and providing feedback, rather than self-reflecting or receiving feedback from others [1].

Many students explicitly mentioned this aspectas most helpful in a post-course survey, and itis consistent with other researcher’s findings [8].By analyzing the reviews a student gave, wecould better determine student engagement andthe relationship between a student’s commentsand their mastery of concepts in the course.

Textual feedback from both courses was an-alyzed with a dictionary-based natural languageprocessing algorithm [2]. The algorithm matchedpositive and negative sentiment-bearing keywordsto produce metrics that included overall sentimentof the review (positive or negative), counts ofparts-of-speech (i.e., noun, adjective, adverb), andthe average length of comments. The algorithmincluded an aspect extractor that scanned text ina sliding window and produced a list of importantaspects (nouns) in close proximity to sentimentwords (adjectives). The aspects were frequentlycommented upon words associated with eitherpositive or negative sentiment that indicatingtheir importance to reviewers. We compared thesewords to the rubric to determine whether studentsstayed on topic or commented on unrelated top-ics. Figure 2 shows, highlighted in teal, the adjec-tive (“colorful”) and noun (“images”) matchingwithin the sliding window. While “colorful” hasa positive inferred sentiment, it is modified by thecomparative adjective, “more,” which reverses thesentiment to negative. It is worth noting that thedevelopment of the algorithm targets peer reviewin engineering courses. It was not explicitly tunedfor visualization courses or the particular rubricused. Thus, the algorithm did not “look” forvisualization keywords, only for general nounswith associated sentiment-bearing words.

The field of writing composition provides auseful reference for our non-experimental eval-uation methodology. Mulder et al., e.g., usedsimilar qualitative and quantitative methods byperforming content analysis on peer review com-ments and student questionnaires [14]. Analyzingthe words, both number and variety, used inwritten comments has also been used for mea-suring both learning outcomes [11] and student

some more colorful images would have helped

Figure 2: Aspect Extractor example where a nounappears in close proximity to an adjective.

6

engagement [21].

PEER REVIEW AND STUDENTOUTCOMES

Peer review textual feedback, though ripe withinformation, may largely be ignored in largecourses. However, the unstructured responses canprovide insights into course content, student en-gagement, and the peer review process as a wholeto indicate students’ visual literacy and the effec-tiveness of visualization education.

Reinforce Course Content with Peer ReviewWe began by analyzing the aspects (nouns)

in student peer review comments to determinewhether peer review reinforces course content inthe visualization classroom. If students mentionedkey concepts learned in the course in their rubricresponses, we interpreted it to mean that theyidentified course content in context.

For the Data Visualization course, the top20 aspects included: visualization, legend, color,ink, data, type (of data), graph, information, ratio(of data), use (of color, encoding), scale, amount(of ink, data), interaction, chart, lie, and den-sity. Perhaps unsurprisingly, these words matchedthose of the rubric. Figure 3a provides context.It displays the aspects in the center column withthe number of connected positive and negative (ornegated positive) sentiment words to the left andright columns, respectively. Saturation denotesthe strength of sentiment, and the height of thebars indicates the number of occurrences.

In the Visual Literacy course, the top 20aspects included: work, map, title, design, graph-ics, headline, element, infographic, object, back-ground (of visualization), understand (the visu-alization), color, shape, and axes. Interestingly,these aspects (Figure 3b) form a very different setthan those in Data Visualization, which hints at adistinction between students and course content.Students did not often mention the language ofthe rubric, which included “Detailed label,” “Liefactor,” “Data/color ink ratio,” and “Chart Junk.”Students provided similar ratios of positive andnegative feedback on their aspects to those inData Visualization. Students also used positiveand negative modifiers more consistently, show-ing more agreement in phrasing. When combinedwith other indicators of engagement, these stu-

(a) Data Visualization aspects

(b) Visual Literacy aspects

Figure 3: Comparison of peer review sentiment.Aspects (center columns) are connected to pos-itive (left) and negative (right) sentiment wordsthat they appear by in text.

dents appear less likely to stay on topic and givethorough reviews.

While we cannot directly measure if studentsunderstand concepts better through the use of peerreview, we can observe the approach providing re-peated exposure to important concepts, both giv-ing and receiving on-topic feedback. In addition,students are, at the very least, frequently repeatingterminology in context, thereby increasing theirvisual literacy and critical analysis skills.

Understand Student Engagement with PeerReview

We next considered whether students engagein the peer review process. If not, the advan-tages of reinforcing course content are lost. Asrecommended by [21], we analyzed the number

7

Table 1: Analytic evaluation of peer comments in Data Visualization (DV) and Visual Literacy (VL)

ReviewsAvg Avg Positive Avg Negative Avg Words Avg Words Avg Avg Avg

Keywords Keywords Keywords Per Review Per Sentence Adjectives Adverbs NounsDV Graduate 1543 7.75 4.99 2.76 100.31 7.85 9.78 4.91 30.50DV Undergraduate 1573 9.26 5.78 3.48 135.61 10.45 12.42 8.37 36.86VL Freshman 111 2.50 1.54 0.96 34.84 12.10 3.48 2.03 8.78VL Sophomore 109 2.59 1.72 0.87 38.26 14.58 3.74 2.28 9.28VL Junior 17 2.82 1.65 1.18 36.88 15.29 3.35 2.12 9.35VL Senior 879 1.89 1.28 0.61 22.96 11.14 2.50 1.16 6.50

and variety of words to quantitatively evaluatestudent engagement. We specifically identifiednouns (aspects), adjectives (aspect modifiers), andadverbs (sentiment enhancers) in the peer reviewcomments. The relevant summary statistics forboth Data Visualization and Visual Literacy areshown in Table 1.

We noticed that undergraduate students inData Visualization wrote more than graduate stu-dents. In addition, they used more words persentence, and they used a greater variety of taggedparts of speech, especially adverbs. Interestingly,undergraduates and graduates had a similar ratioof negative to total keywords, 38% and 36%,respectively. This is an important measure ofengagement because critically evaluating a vi-sualization requires more investment than just acursory review—it requires explaining why some-thing is wrong using learned concepts (e.g., ana-lyzing for “lie factor”). Considering factors, suchas length, variety of parts of speech, and the ratioof negative keywords indicates that undergraduatestudents may be slightly more invested in the peerreview process than their graduate peers.

In contrast to the Data Visualization course,peer review comments in Visual Literacy werefrom undergraduate only and were coded by classlevel. This enabled a more fine-grained analysis.It should be noted that juniors comprised a signifi-cantly reduced dataset (19) compared to freshmen(149), sophomores (141), and seniors (1262), so itis difficult to infer statistically significant resultsfor those students.

For Visual Literacy students, the ratio of nega-tive to total keywords was 33%, which was lowerthan the Data Visualization students. When bro-ken down by class level, we found juniors (42%)were the most critical, followed by freshmen(37%), sophomores (34%), and finally seniors(32%). Ultimately, senior students seem the leastengaged through their comparative lack of use of

sentiment-bearing keywords (Figure 4).The Visual Literacy student review comments

were shorter than those of the Data Visualizationstudents (average words per review), and theyalso exhibited less sentiment (average positiveor negative keywords) and part-of-speech variety(average adjectives, adverbs, and nouns). VisualLiteracy students did, however, write longer sen-tences, although they wrote fewer. The combina-tion of this information can perhaps be attributedto the level of detail in the rubrics utilized. InData Visualization, the rubric provided multipletextboxes for students to write a detailed analysison each rubric item, rather than the single textboxof the Visual Literacy rubric.

The Data Visualization course provides anadditional piece of evidence: a qualitative per-spective on the engagement of students in the peerreview process. It highlights the difference be-tween submissions to demonstrate the effect of astudent receiving and implementing peer feedbackon their iterative project. In Figure 5, the initialbar and line charts received the comment: “the

Figure 4: Average Sentiment Keyword Usage PerClass Level in Visual Literacy

8

“the text size used for labels could have been morelarger [sic] for clear vision. [...] in bar and line chart,the line points could have been brought into center"

“There wasn't any color present until the combo chartseparated the two chart types […]. However, the dataseemed bland when presented in black and white”

“I believe using at least oneother color would make thecharts even more appealing”

“somehow line graphs and scatter plots are shifted"

Figure 5: Example progression of a single student’s three projects with associated feedback receivedand utilized to improve the design in Data Visualization.

text size used for labels could have been morelarger [sic] for clear vision. [...] in bar and linechart, the line points could have been brought intocenter” and “There wasn’t any color present untilthe combo chart separated the two chart typeswith a red color. However, the data seemed blandwhen presented in black and white.” To respondto the comments, the student changed the barchart color, centered the line chart points to thelabel (actually displaying the points themselves)and made the axis titles slightly larger, as shownin the center charts Figure 5. The projects in thecenter column of Figure 5 received comments,including: “somehow line graphs and scatter plotsare shifted” and “I believe using at least oneother color would make the charts even moreappealing.” The student added another color tothe dashboard (right chart in Figure 5) and shiftedthe appropriate graphs to not overlap in response.Therefore, in addition to the benefit of reviewingothers’ work, the student appeared to considerand implement a fair portion of the feedback theyreceived.

Mastery of Concepts and Peer ReviewNext, we considered whether student senti-

ment in comments provided to peers was relatedto performance on the project, thereby establish-ing a link between engagement in peer review(sentiment or part-of-speech metrics) and visualliteracy (grade on assignment). Although studentgrade information was not available in Data Vi-sualization, students perceived benefit from peer

review in general (and in particular, through pro-viding feedback to their peers), despite the extratime it took [1]. Over three semesters, 82% ofpost-course survey respondents reported learningat least somewhat more because of peer review(score of 3 out of 5 or more; mean = 3.6) and75% recommended continuing peer review (scoreof 4 out of 5 or more; mean = 4.1) [1].

Grade information was available in VisualLiteracy, and Figure 6 shows the average numberof keywords in students’ reviews by their projectgrade. In total, 755 students earned an ‘A’, 216earned a ‘B’, 78 earned a ‘C’, 63 earned a‘D’, and 4 earned an ‘F’. Although we cannotdetermine that those who write short reviews

Figure 6: Average Sentiment Keyword Usage PerProject Grade in Visual Literacy

9

will receive an ‘F’ due to the small sample size,it is interesting to note that students earning a‘C’ wrote the most. This is perhaps because ‘A’students felt comfortable that they would do wellin the course without devoting additional effort,while borderline students felt the need to improvetheir performance by devoting additional effort tothe reviews.

When comparing the sentiment used in re-views, there is one group of students that standsout in Figure 7. Those earning a D on theirproject grade were the only group that wrotemore negatively on average. Perhaps these criticalstudents hoped to learn by carefully analyzingtheir peers’ work for mistakes, or perhaps they(more pessimistically) desired to make their worklook better in comparison.

One final qualitative observation from review-ing student work provides more evidence for theconnection between student feedback and per-formance in the visualization classroom. In theData Visualization course, we found that studentsoften comment on others’ work in ways that theyhave already implemented in their visualizations.Figure 8 shows both a student’s projects andthe feedback that the student gave to peers. InProject 2, for example, the student mentions axislabels and ticks—something they had carefullyimplemented in their bar and line charts. Thestudent pointed out the correct use of color, andthey directly referenced materials learned in class

Figure 7: Average Overall Sentiment and Ab-solute Value of Sentiment Per Project Grade inVisual Literacy

in their Project 3 feedback. In their Project 6 feed-back, the student referenced a specific techniquefor avoiding clutter in a parallel coordinates plot.Finally, in Project 7 feedback, they included atip to delineate charts on the dashboard. In allsituations, the student offers advice that directlycorresponded to a technique they implemented.This example corroborates our finding that peerreview reinforces course content by providing stu-dents an avenue to communicate recently learnedconcepts. This is an option they might not other-wise have. Thus, the mere opportunity of peerreview may be a significant factor for studentsuccess.

Measure Intervention EffectivenessThe previous analysis has suggested, from

student data, how sentiment analysis of peer re-view text can benefit visualization students andincrease visual literacy. However, peer reviewdata can also provide insights into course trendsover time, establishing baselines and providing ametric for class success.

For example, Figure 9 shows the variance inpart-of-speech utilized by Visual Literacy stu-dents per semester. The trend lines highlightseveral interesting phenomena over the lifetimeof the course. First, student part-of-speech varietyhas decreased on average from Fall 2017 toFall 2020. As time progressed and students tookmultiple classes with the same instructors, theymay have become more comfortable with theminimum level of expected effort. Since springsemesters are typically lower than fall semesters,it might also indicate burnout.

The major decrease from Fall 2018 to Spring2019 corresponded to two factors in Visual Liter-acy: 1) a re-positioning of the course from senior-level to freshman-level and 2) a transition inmodality from face-to-face to online in responseto Covid-19. The combination of these factorsmay have contributed to the rapid decrease in thevariety of part-of-speech usage.

When the instructors of the Visual Literacycourse were questioned about factors related tothe large increase from Spring 2019 to Fall 2019,they mentioned that in Fall 2019, USF providedexpert support for the development of coursematerials and the improvement of assignments. Inparticular, the instructors were coached to employ

10

Distorted because thereare no labels or an axisto use as a reference.

I would possibly addadditional tick-marks tomatch the amount forthe y-axis as there arex-axis.

I think if you add the "border" (whitespace) that Dr. Rosen showed us inclass so that way the data can bebetter contained inside the x/y boxthat would help.

Only thing I would change is the number ofcolor hues in scatterplot matrix, I thinkthere are just too many for us perceptually(refer to 'color' slides where Dr. Rosenmentioned we can only take in a handfulof colors at a time).

For the PCP graph, I wouldconsider trying to make thelines more translucent oropaque. I think processinghas a built in transparencythat maybe you can try out.

I also think the "box" around thescatterplot could be used for theline graph and histogram forsymmetry. Along those same lines, Ithink some kind of "cutoff" lineswould help declutter the dashboardby showing distinctive graphs andvisualizations in a single dashboard.

Project 2

Proj

ect 3

Proj

ect 6

Project 7

Figure 8: Illustrative example of a student’s projects and the feedback they gave to their peers, reflectingapplied concepts in Data Visualization.

verbs in their project and rubric descriptions.While this indicates the effectiveness of suchexpert intervention and underscores the impor-tance of well-designed materials in a large, onlinecourse, ultimately, it appears the students quicklyadjusted the next semester (Spring 2020) to areduced level of engagement.

Instructor Perspective: Observed BenefitsWe have observed several additional benefits

from an instructor’s perspective that we infor-mally evaluate.

The teaching assistant for the 2018 and 2019Data Visualization course pointed out that one ad-

Figure 9: Average Part-of-Speech Usage Per Re-view By Semester in Visual Literacy

vantage of peer review: formalizing existing col-laboration. Many students discuss projects withone another and seek input without instructorintervention. However, peer review also initiatedinteraction for some students who would oth-erwise not interact with and offer constructivecriticism to others. To them, it offered a lowerrisk environment for sharing their opinions thanin class discussions. Requiring feedback in thisway is important for the educational process bypracticing critical evaluation skills, as well as tothe visualization design process by providing agreater variety of perspectives.

Secondly, since many visualization coursescover specific programming languages required togenerate visualizations, the instructor can includea code review phase that adds a new dimension tostudent engagement. This helps develop the stu-dents’ ability to appraise their work from multipleperspectives (i.e., from peer review comments onthe visualization’s ‘front-end’ to self-reflection onthe code ‘back-end’). A student employing visualpeer review in this manner is adding inherentlyasymmetrical analysis skills.

Finally, peer review feedback provides theinstructor not only with a diverse set of opinionsfrom which to review the visualization work butalso provides insight into student learning fromthe quality of their review comments. This is yetanother tool by which an instructor can identifysuccessful students and is a tool for increasingtransparency in the grading process.

11

LIMITATIONSWhile the results analyzed above are highly

specific to our courses, the approaches are byand large generalizable to other design-orientedvisualization courses. If an instructor already usesa semester-long continuous design project, theymay incorporate one or more peer review stagesusing our rubric or one of their own design.Beyond reinforcing concepts, a continuous im-provement process is supported by the feedbackgiven in peer review. If projects are discrete, thefeedback cycle cannot be utilized, but many of theimportant peer review benefits will be retained,such as the reinforcement of course concepts orincreased engagement.

Instructor Effort

On the surface, peer review appears to resultin substantial time savings for instructors whono longer need to provide subjective feedback.For example, 60 students with eight assignmentsper semester and 10 minutes of feedback perassignment should result in over 80 hours saved.However, much of that time is diverted to otheractivities. Additional administrative tasks arise—peer review requires additional effort to set upand administer, peer reviews need to be monitoredfor quality, and grades need to be assigned. At thesame time, highly engaged students might requestmore detailed instructor feedback beyond peerreviews. Overall, peer review is a compliment, nota replacement, to existing educational approachesthat requires the instructor, as the expert, toremain involved.

Risks

When considering the use of peer review,the risk for collusion, malice, and cheating mustbe considered, even with double-blind reviews(e.g., the review of a friend may be overly opti-mistic). Instructors should be careful to random-ize peer reviewers. Keeping submissions anony-mous is helpful, but it is challenging to maintainanonymity, particularly in a smaller educationalsetting where students will talk and discover theyare evaluating each other’s work. In conjunction,grades should only be loosely based upon theresults of the peer review.

CONCLUSIONIn this work, we contrast peer review in two

visualization courses with distinct cultures anddomains to identify the variety of informationthat can be obtained from the analysis of peer re-view text. We discovered differences between thetwo domains regarding the taxonomy of studentcomments to their peers. However, we also foundsimilarities regarding student engagement and re-inforcement of the core concepts each instructorpresented in class. Sentiment analysis and aspectextraction of peer review comments augment theinstructor’s ability to draw insights on studentengagement and success in the classroom, thefocus of course content, course baselines, and theeffectiveness of instructor intervention.

Peer review in the visualization classroomoffers two significant benefits to students. First,it provides a mechanism for students to activelyengage through critical evaluation of visualiza-tions, increasing visual literacy and awareness.Second, it enables providing students with diverseand timely feedback on their projects and othercoursework. Ultimately, we encourage the visu-alization community to adopt peer review, withits rubrics and associated analysis, to improve in-structor understanding of student engagement andpromote visual literacy through critical analysis.

In the future, we are interested in the cus-tomization of the rubric by reducing constraints.Assuming students’ critical thinking skills areunderdeveloped, particularly in the domain ofvisualization education, the rubric serves as animportant tool to improve those skills. At thebeginning of a course, the rubric can includeall relevant scoring categories, and as the courseprogresses, related categories can be combined orremoved. This way, students will progress fromhighly structured evaluation to entirely free-formevaluation.

Secondly, we noted a lack of high-qualityreviews. Approximately 30% of reviews in eachclass did not contain enough information to re-ceive a fine-grained sentiment score from thenatural language processing algorithm. While lesscritical for part-of-speech analysis of engagement,the addition of a fine-grained sentiment score onpeer review text can provide a complementarycomponent to the numerical score from the review

12

form. Measuring and improving the ‘helpfulness’of peer reviews with real-time suggestions forstudents with weak responses is an active areaof research that we would like to incorporate intoour future courses. This can be done by gamifyingpeer review to include a top reviewers scoreboard.

Finally, it is important to remember that thereis both art and science to visualization, withno single optimal design. Using the wisdom ofthe crowd to build machine learning models thatleverage peer feedback to semi-automatically as-sign grades is an exciting direction for futurestudy.

DisclosureWhile the first author (Beasley) was a graduatestudent in the 2017 Data Visualization course,his participation in the research began after thecourse was completed.

ACKNOWLEDGMENTWe thank our anonymous reviewers for their

helpful feedback on our paper. We also thankCoby O’Brien and Kevin Hawley for support-ing our effort to promote visual peer reviewin their classrooms and Ghulam Quadri for hisperspective as a teaching assistant in the DataVisualization course. This project was supportedin part by the National Science Foundation (IIS-1845204).

REFERENCES1. Z. Beasley, A. Friedman, L. Piegl, and P. Rosen. Lever-

aging peer feedback to improve visualization education.

In IEEE Pacific Visualization Symposium (PacificVis),

pp. 146–155, 2020. doi: 10.1109/PacificVis48177.2020.1261

2. Z. J. Beasley and L. A. Piegl. A domain-dependent

lexicon to augment cad peer review. Computer-Aided

Design and Applications, 18(1):186–198, 2021. doi: 10.

14733/cadaps.2021.186-198

3. Z. J. Beasley, L. A. Piegl, and P. Rosen. Designing

intelligent review forms for peer assessment: A data-

driven approach. In ASEE Annual Conf. & Exposition,

2019. doi: 10.18260/1-2--32337

4. W. S. Cleveland. Visualizing data. Hobart Press, 1993.

5. G. Domik. Do we need formal education in visual-

ization? IEEE Computer Graphics and Applications,

20(4):16–19, 2000. doi: 10.1109/38.851744

6. A. Friedman and P. Rosen. Leveraging peer review in

visualization education: A proposal for a new model.

In Pedagogy Data Visualization, IEEE VIS Workshop,

2017.

7. A. Friedman and E. Schneider. Developing a visu-

alization education curriculum in the age of big data

using the dick and carey model. Visual Communication

Quarterly, 25(4):250–256, 2018. doi: 10.1080/15551393.

2018.1530115

8. V. Garousi. Applying peer reviews in software engi-

neering education: An experiment and lessons learned.

IEEE Trans. on Education, 53(2):182–193, 2010. doi: 10

.1109/TE.2008.2010994

9. P. Isenberg, N. Elmqvist, J. Scholtz, D. Cernea, K.-L.

Ma, and H. Hagen. Collaborative visualization: Defi-

nition, challenges, and research agenda. Information

Visualization, 10(4):310–326, 2011.

10. R. Kosara. Visualization criticism-the missing link be-

tween information visualization and art. In Intl. Conf.

on Information Visualization, pp. 631–636, 2007. doi: 10

.1109/IV.2007.130

11. J. McGourty, P. Dominick, and R. Reilly. Incorporating

student peer review and feedback into the assessment

process. Frontiers in Education Conf., 1, 1988. doi: 10.

1109/FIE.1998.736790

12. J. Michael. Where’s the evidence that active learning

works? Advances in Physiology Education, 30(4):159–

167, 2006. doi: 10.1152/advan.00053.2006

13. J. M. Moxley. Afterword: Disciplinarity and the future of

creative writing studies. In D. Donnelly, ed., Does the

writing workshop still work? Multilingual Matters, 2010.

doi: 10.21832/9781847692702-021

14. R. Mulder, C. Baik, R. Naylor, and J. Pearce. How does

student peer review influence perceptions, engagement

and academic outcomes? a case study. Assessment &

Evaluation in Higher Education, 39, 2014. doi: 10.1080/

02602938.2013.860421

15. T. Munzner. Visualization analysis and design. AK

Peters/CRC Press, 2014. doi: 10.1201/b17511

16. M. Oliver and J. Aczel. Theoretical models of the role

of visualisation in learning formal reasoning. Journal of

Interactive Media in Education, 2002(2), 2002. doi: 10.

5334/2002-3

17. R. R. Raje, M. Boyles, and S. Fang. Cev: collaborative

environment for visualization using java rmi. Concur-

rency: Practice and Experience, 10(11-13):1079–1085,

1998.

18. H. Rushmeier, J. Dykes, J. Dill, and P. Yoon. Revisiting

the need for formal education in visualization. IEEE

Comp. Graph. and Appl., 27(6), 2007. doi: 10.1109/MCG.

2007.156

19. B. S. Santos, S. Silva, and P. Dias. Heuristic evalu-

13

ation in visualization: An empirical study: Position pa-

per. In IEEE Evaluation and Beyond-Methodological

Approaches for Visualization (BELIV), pp. 78–85, 2018.

doi: 10.1109/BELIV.2018.8634108

20. E. R. Tufte, N. H. Goeler, and R. Benson. Envisioning

information, vol. 2. Graphics press Cheshire, CT, 1990.

21. M. Van Zundert, D. Sluijsmans, and J. Van Merrienboer.

Effective peer assessment processes: Research find-

ings and future directions. Learning and Instruction,

20(4):270–279, 2010. doi: 10.1016/j.learninstruc.2009.08.004

22. E. Wardle and K. Roozen. Addressing the complexity

of writing development: Toward an ecological model of

assessment. Assessing Writing, 17(2):106–119, 2012.

doi: 10.1016/j.asw.2012.01.001

23. D. Weaver and A. Esposto. Peer assessment as a

method of improving student engagement. Assessment

& Evaluation in Higher Education, 37(7):805–816, 2012.

doi: 10.1080/02602938.2011.576309

Zachariah J. Beasley is an Instructor 1 at the Univer-sity of South Florida in the Department of ComputerScience, where he received his Ph.D. with a focuson sentiment analysis in peer review. Dr. Beasley hasreceived the ASEE State of Engineering Education in25 Years Award, the USF Spirit of Innovation Award,and is a USF STEER STEM Scholar. Contact him atzjb@usf.edu.

Alon Friedman is an Associate Professor at the Uni-versity of South Florida in the School of Information.He received his Ph.D. from the Palmer school of li-brary & information science at Long Island University.His recent research interests include using differenttheoretical perspectives to organize data through avisualization lens. His current projects include MarkLombardi’s design, Peircean semiotics theory throughvisualization lens, and Visual Peer review. Contacthim at alonfriedman@usf.edu.

Paul Rosen is an Assistant Professor of ComputerScience at the University of South Florida. His Ph.D.is from Purdue University. His recent research inter-ests include geometry- and topology-based problem-solving in visualization. He and his collaboratorshave received several best paper awards and hon-orable mentions, and he received a National ScienceFoundation CAREER Award in 2019. Contact him atprosen@usf.edu.

14