Correlation Between Student Collaboration Network Centrality and AcademicPerformance

David L. Vargas, Ariel M. Bridgeman, David R. Schmidt, Patrick B. Kohl, Bethany R. Wilcox, and Lincoln D. CarrDepartment of Physics, Colorado School of Mines, Golden, CO 80401, USA

(Dated: August 2, 2018)

We compute nodal centrality measures on the collaboration networks of students enrolled in threeupper-division physics courses, usually taken sequentially, at the Colorado School of Mines. Theseare complex networks in which links between students indicate assistance with homework. Thecourses included in the study are intermediate Classical Mechanics, introductory Quantum Me-chanics, and intermediate Electromagnetism. By correlating these nodal centrality measures withstudents’ scores on homework and exams, we find four centrality measures that correlate signif-icantly with students’ homework scores in all three courses: in-strength, out-strength, closenesscentrality, and harmonic centrality. These correlations suggest that students who not only collab-orate often, but also collaborate significantly with many different people tend to achieve highergrades. Centrality measures between simultaneous collaboration networks (analytical vs. numericalhomework collaboration) composed of the same students also correlate with each other, suggest-ing that students’ collaboration strategies remain relatively stable when presented with homeworkassignments targeting different skills. Additionally, we correlate centrality measures between collab-oration networks from different courses and find that the four centrality measures with the strongestrelationship to students’ homework scores are also the most stable measures across networks involv-ing different courses. Correlations of centrality measures with exam scores were generally smallerthan the correlations with homework scores, though this finding varied across courses.

I. INTRODUCTION

Physics education research has enjoyed a great dealof success in identifying and clarifying students’ difficul-ties with physics concepts, developing problem solvingmethods, and structuring the knowledge that is taughtto students [1–5]. Such studies have allowed researchersto make quantitative statements about the presence andpersistence of students’ difficulties, in contrast to histor-ical physics education that relied on anecdotal informa-tion [2]. In recent years, physics education researchershave begun taking advantage of another powerful toolfor quantitative analysis in the social sciences known ascomplex network theory [6, 7], the use of which is oftenreferred to as social network analysis (SNA). SNA hasa significant history of use in educational research gen-erally [7–9]; however, the use of SNA has only recentlybegun to gain traction within the physics education re-search community (e.g., Refs. [10, 11]).

In this paper, we use SNA to study self-reported stu-dent collaboration and its potential relation to studentperformance. We examine how collaboration betweenstudents evolves between semesters and how nodal cen-trality measures correlate with homework vs. examgrades. Furthermore, we compute the differences in cor-relation strengths between our measures, allowing us toquantify which measures are most strongly related to stu-dent grades. Finally, we also compare the network cen-trality of students between the collaboration networks ofdifferent types of homework assignments within a sin-gle course, allowing us to assess the similarity of rolesadopted by students in response to assignments of differ-ing nature.

Complex network measures provide succinct sum-

maries of the order present in complex networks. Of-ten such measures are aggregate summaries of the en-tire structure of a network and are useful because thestructure of connectivity can determine the efficiency ofprocesses taking place in the network as observed insocial, neural, communication, and transportation net-works [7, 12]. Furthermore, the nodal centrality measureswe review in Sec. IV tell us how well connected studentsare in the context of their homework collaboration net-works [13, 14]. For example, a simple measure of howwell a student is connected to other students in the net-work is encapsulated by their out-strength. Out-strengthis simply the number of peers a student helps with home-work and, thus, is one measure of the influence of stu-dent in a collaboration network. Parallel to out-strengthis in-strength, the number of students that help a partic-ular student with homework. While out-strength can bethought of as a coarse measure of the influence of a stu-dent on the collaboration network, in-strength is a mea-sure of how a student gathers information from differentparts of the network. Other more subtle measures of astudent’s connections within the network include close-ness centrality, which looks at the “distance” betweenstudents (i.e., if information is to pass from student i tostudent j, how many other students must it go throughfirst), as well as betweenness centrality, which looks atthe degree to which a particular student can control theflow of information between other students. These cen-trality measures are discussed in detail in Sec. IV. Thevariety of complex network measures we consider provideus with different perspectives of the students forming ourcollaboration networks. The aggregate correlation of thecentrality measures with student grades provide insightinto how strongly and in what manner different collabo-

arX

iv:1

808.

0031

5v1

[ph

ysic

s.ed

-ph]

1 A

ug 2

018

2

ration patterns are central to the educational process.

While the application of complex network-based meth-ods to student networks formed within the physics class-room are rare, there is a growing body of work in thisarea. In particular, network analysis has been consis-tently applied to investigations of student persistence[15–18]. The use of network analysis in the context ofunderstanding student persistence is motivated by theidea that a student’s decision to persist (or not) within aparticular major is impacted by their integration within,and interaction with, their academic and social commu-nities [16]. Forsman et al. reported on students’ in-classsocial and academic interactions. They showed that thesesocial and academic networks had distinct connectionpatterns indicating that the underlying processes gov-erning the formation of these networks is different. Theythen argued that both networks need to be consideredwhen addressing student persistence. Zwolak et al. [16]created networks based on students’ in-class interactionswithin a highly interactive introductory course and foundthat certain network centraility measures correlated sig-nificantly with students’ persistence into the next coursein the sequence. This correlation held even when con-trolling for the impact of a student’s grade. Zwolak etal. [17] later built on this work by incorporating an addi-tional network based on students’ interactions outside ofclass. They found that for middle-performing students,out-of-class centrality measures dominated in terms ofpredicting students’ persistence to the next course in thesequence.

Network analysis has also been utilized in investiga-tions not directly tied to student persistence. Dou etal. [19] reported correlations between network centralitymeasures and changes in students’ self-efficacy over thecourse of an introductory physics course taught usingmodeling instruction. Brunn and Brewe [11] used net-work centrality measures calculated based on students’interactions in an introductory physics course to predicttheir grades in a future course. The use of network anal-ysis in physics education research has also been extendedbeyond social networks. For example, Bodin [10] appliednetwork analysis to visualize connections between stu-dents’ epistemic ideas when solving physics problems in-volving simulations and modeling tasks, and Brewe etal. applied network analysis to characterize students’ re-sponses to an introductory conceptual assessment.

The current study is distinct from the work describedabove in several ways. The majority of the prior workfocuses on students’ in-class interactions in the contextof an introductory physics course utilizing a highly in-teractive curriculum. The current work focuses on threeupper-division courses all taught with a mixture of tradi-tional lecture punctuated by the use of interactive tech-niques. Additionally, the current work deals with stu-dents’ out-of-class collaborations on homework assign-ments; thus, these interactions are driven almost entirelyby the students without the significant pressure to col-laborate usually associated with an interactive classroom

environment. Using a wide variety of complex networkmeasures, we obtain detailed information about the roleof different collaboration strategies in different types ofproblem sets and on exams. This study is a step to-wards responding to multiple calls to take advantage ofthe analysis power of complexity science within physicseducation research [17, 18].

Here, we build off the work described above by ad-dressing the following set of questions. Do well-connectedstudents have good grades? Does access to the reason-ing of many of their peers better equip students to com-plete homework assignments, or does excessive partici-pation in a collaboration network stifle the ability of astudent to perform well on their own work? Do the ben-efits of collaboration extend to exams, where a studentdoes not have access to their collaborators? How sta-ble are these measures of collaboration in different con-texts? That is, do students tend to take on different rolesin response to different types of assignments or differentsubject-matters, or are students’ collaboration strategiesstatic?

II. DATA COLLECTION

The Colorado School of Mines (Mines) is a public re-search university in Golden, Colorado. The university,which has close to six thousand undergraduate and grad-uate students, focuses on engineering and the applied sci-ences. Additionally, it is one of very few institutions thatawards more than 50 Physics Bachelors per year, placingit in the top ten of all Ph.D. granting departments in theU.S. [20]. The physics department has research focus ar-eas in condensed matter, subatomic, optical, renewableenergy, theoretical, and computational physics.

The data for our networks was collected over twosemesters: Fall 2012 and Spring 2013. Students in theirjunior year in engineering physics at Mines take Classi-cal Mechanics during the Fall semester and both Quan-tum Mechanics and Electromagnetism during the Springsemester. The course in Classical Mechanics covers La-grangian and Hamiltonian mechanics. The Electromag-netism course is the first course in a two-course sequence,and covers electrostatics and magnetostatics, includingthe appropriate Maxwell equations, boundary conditions,and treatments of free and bound sources. The course inQuantum Mechanics introduces the formalism of Quan-tum Mechanics (e.g., the solutions of a particle in a box,scattering from a potential well, etc.). We summarizethe homework, exam, and enrollment information for thethree courses in Fig. 1. Note that the longitudinal na-ture of our data means that many students in our data setappear in all three courses. Prior to their junior year atMines, physics majors are encouraged to collaborate in aphysics studio setting, a setting in which students work ingroups of three to complete homework-like assignmentsand labs. Additionally, the summer before their junioryear, Mines physics majors participate in a physics field

3

Classical Mechanics

75 students

10 Analytical HWs

5 Numerical HWs

3 Written ExamsElectromagnetism

76 students

12 Analytical HWs

2 Written Exams

1 Oral Exam

Quantum Mechanics

75 students

14 Analytical HWs

4 Written Exams

Fall 2012 Spring 2013

70 Students67 Students

in common

FIG. 1: Course information for Classical Mechanics,Quantum Mechanics, and Electromagnetism. Linesconnecting two courses indicate the number of studentscommon to both courses. There were 67 studentsenrolled in all three courses.

session in which groups of ten students move betweensections on computing, vacuum systems, machining, andlasers. In all of these sections students are encouraged tocollaborate, and in some of them students are split intogroups of three to complete assignments. Thus, collabo-ration is an explicit part of Mines’ lower-division programalready, and is strongly encouraged.

There were two forms of data collection: paper formsduring Fall 2012 and electronic spreadsheets duringSpring 2013. In the paper form of data collection, stu-dents were provided with a form for each assignmentin which they were to list any students they helped orreceived help from for each assigned problem. In theelectronic form of data collection, students provided thesame data by entering the names of their collaboratorsinto question/answer boxes on the learning managementsystem Blackboard [21]. In both cases, students wererequired to complete the surveys described above in or-der to receive credit for their homework assignments.This policy incentivized survey completion and ensureda nearly complete set of data. Student names were thenreplaced with a set of randomly generated three lettercodes to anonymize the data prior to analysis.

The data from the surveys above was compared withstudent grades in the three courses (Fig. 1). The course inClassical Mechanics had ten analytical homework assign-ments, five numerical homework assignments, and threewritten exams. For numerical assignments, students wereasked to simulate various physical scenarios using Math-ematica. For the course in Classical Mechanics, we com-puted three measures of a students’ performance: thesum of their analytical homework grades, the sum of theirnumerical homework grades, and the sum of their examgrades. For the courses in Quantum Mechanics and Elec-tromagnetism, all homework assignments were analytical(see Fig. 1), and we measured a students’ performance bythe sum of their homework grades and the sum of theirexam grades. It is important to note that in Classical Me-chanics, the teaching assistants graded exams with sub-sequent review by the instructor. In Quantum Mechanics

and Electromagnetism, the instructors graded all examsthemselves. Finally, the instructors of Quantum Mechan-ics and Classical Mechanics both applied curves to examgrades, whereas the instructor of Electromagnetism didnot.

III. METHODOLOGY FOR CONVERTINGDATA INTO NETWORKS

From the data collected in the surveys above, we con-structed directed and weighted networks for each courseand type of homework assignment using the followingprocedure. A network is a collection of nodes and links.Nodes are any object that can be connected to any otherobject by some relation and links are the connectionsbetween nodes. In our networks, nodes correspond tostudents, and a link corresponds to an interaction be-tween the students consisting of providing or receivingassistance. Note that this is not a symmetric relation; ifi helped j, it does not imply that j helped i. This is thedefining feature of a directed network; its connections areasymmetrical. For directed networks, one says that a linkgoes from node i to node j to indicate the direction ofthe link. For the kth network a link is placed from nodei to node j if and only if (iff) student i helped student jwith homework assignment k. Summarizing our networkin terms of the entries of an adjacency matrix,

Akij =

1 iff student i helped student j

with homework assignment k

0 otherwise.

(1)

However, we found that it was necessary to resolvediscrepancies in the reports provided by students. Forexample, student i may claim that they helped studentj, but student j’s survey indicates that they did not re-ceive help from student i. These discrepancies may bedue to forgetfulness or conflicting perceptions of interac-tions [11]. To resolve the discrepancies in student reports,we employed a Maximal discrepancy resolving technique[22, 23]. Applying an element-wise logical OR to the ad-jacency matrices created from each student’s survey wemade a final network. That is, every reported interactionis considered to have happened even if one student doesnot report it. Discrepancies can occur in either directionof an interaction, and Aij is resolved separately fromAji. Other discrepancy sorting cases were investigatedbut yielded quite sparse adjacency matrices [22, 23]. Foreach course, we then compute a weighted adjacency ma-trix by summing the adjacency matrices correspondingto the homework assignments in that course,

Aij =

NHW∑k=1

Akij . (2)

Where NHW is the number of homework assignments inthe relevant course. Thus if two students i and j col-laborated frequently on homework assignments they will

4

have a heavily weighted connection in one of the weightednetworks depicted in Fig. 2. In Fig. 2, nodes are indi-cated by circles and the links connecting nodes are in-dicated by the arrows between nodes. The direction ofthe arrow indicates the direction of assistance. For thecourse in Classical Mechanics, we construct networks forthe numerical homework assignments and the analyti-cal homework assignments separately. This procedureresults in two networks for the course in Classical Me-chanics: a network constructed from the collaborationnetworks on analytical assignments ACM

A , and a networkconstructed from the collaboration networks on numer-ical assignments ACM

N . We denote the network for thecourse in Quantum Mechanics by AQM, and the networkfor the course in Electromagnetism as AEM.

IV. COMPLEX NETWORK ANALYSIS

Using the NetworkX network analysis software [24], aswell as some of our own independently developed net-work analysis code, we study the networks described inthe previous section by computing various nodal cen-trality measures and other measures of the structureof a node’s connections. We then compute the cor-relation between these measures and different estima-tors of student performance. Nodal centrality measuresare measures of a node’s importance to the structureof the network. As they are quantitative measures ofeach student’s role in the collaboration network, nodalcentrality measures are ideal for our study. Our se-lection of standard complex network measures includesout-strength, in-strength, out-disparity, in-disparity, lo-cal clustering, closeness centrality, harmonic centrality,and betweenness centrality. The local clustering coeffi-cient is only defined on undirected networks. Before com-puting the local clustering coefficient we first convert ourdirected networks into undirected networks as such that,Aundirected

ij = max (Aij , Aji).The out-strength of a node is the sum of its outgoing

connections to other nodes and is defined as

souti =

L∑j=1

Aij , (3)

where L represents the number of nodes (i.e., students)in the network. A node can have high out-strength if ithas outgoing connections to many other nodes, or if ithas strong connections to only a few other nodes. Statedsimply, students who help many of their peers and stu-dents who frequently help a smaller set of peers both canhave a high out-strength. The in-strength is similarly de-fined and distinguishes a node by the number of incomingconnections

sini =

L∑j=1

ATij , (4)

or the number of instances in which a student receivedhelp. We also study the net out-strength

sneti = souti − sini . (5)

Students with high net out-strength correspond to stu-dents that help many other students but are not helpedby many students.

The out-disparity of a node’s connections is a mea-sure of the non-uniformity of the outgoing connectionstrengths. If a node has a single strong connection in ad-dition to other, much weaker connections, the node hashigh out-disparity. If the connection strengths of a nodeare all approximately equal strength, then it has a lowout-disparity. Out-disparity is defined as [25, 26]

Y outi ≡ 1

(souti )2

L∑j=1

(Aij)2

=

∑Lj=1 (Aij)

2(∑Lj=1Aij

)2 . (6)

Nodes with high disparity correspond to students thatcollaborate with certain nearest neighbors much moreoften than they collaborate with other nearest neigh-bors. Nodes with low disparity correspond to studentsthat collaborate equally with all students that they col-laborate with. Analogously, in-disparity measures thenon-uniformity of the incoming connection strengths. Tocompute Y in

i one makes the substitution A → AT inEq. (6), resulting in

Y ini ≡

1(sini)2 L∑

j=1

(AT

ij

)2=

∑Lj=1

(AT

ij

)2(∑Lj=1A

Tij

)2 . (7)

Note that out-disparity can only be defined for student’swith souti > 0; thus students with souti = 0 are not in-cluded in correlations involving out-disparity. The sameholds for in-disparity for student with sini = 0.

The local clustering coefficient is a measure of the tran-sitivity of connections of individual nodes, that is, thelikelihood that a is connected to c, given that a is con-nected to b and b is connected to c. The local clusteringcoefficient is defined as

cLi ≡T (i)

ki(ki − 1), (8)

where T (i) is the number of existing triangles in whichnode i is a vertex, and ki is the degree of node i. Effec-tively, this provides the fraction of all possible trianglesthrough node i that actually exist. Nodes with low localclustering correspond to students whose collaborators donot tend to collaborate with each other. Nodes with highlocal clustering correspond to students whose collabora-tors frequently collaborate with each other, such as intight-knit study groups.

In a weighted network one can define a distance be-tween any pair of nearest-neighbor nodes. For our anal-ysis, we define the distance between nearest neighbors i

5

(a) Analytical Assignments Network for Classical Mechanics (b) Numerical Assignments Network for Classical Mechanics

(c) Assignments Network for Quantum Mechanics (d) Assignments Network for Electromagnetism

Grade

100%

0%

25%

50%

75%

FIG. 2: Student collaboration networks for three upper level physics courses. Weighted student collaborationnetworks constructed from surveys given to students in three upper-division courses: Classical Mechanics, QuantumMechanics, and Electromagnetism. Nodes correspond to students and the direction of each arrow indicates thedirection of assistance on homework assignments. The color of a node indicates the grade of a student on homeworkassignments. Although we do not normalize grades in our analysis we present grades as a percentage here toillustrate multiple courses simultaneously.

and j to be the inverse of the weight connecting them

Dij =1

Aij. (9)

If nodes i and j are not directly connected by a linkthen Dij = ∞. This definition of the distance be-tween nearest-neighbor nodes is then used to define the

shortest-path distance between any two nodes dij . Apath connecting node i to node j is a sequence of linksalong which one may walk to traverse the network fromnode i to node j when one walks along links in the di-rection of the link. The shortest-path distance betweentwo nodes is the sum of the nearest-neighbor weights Dij

6

along the shortest path connecting two nodes, that is,

dij = minP

∑(l,k)∈P

Dlk , (10)

where P is a path connecting node i to node j.Closeness centrality is a measure of how close a node

is on average to other nodes when one must travel alongdirected links in the direction of the link. Closeness cen-trality is defined as

cCi =n− 1

|A| − 1

1∑j 6=i dij

, (11)

where n is the number of nodes reachable from node i,and |A| is the number of nodes in the network definedby the adjacency matrix A [14, 24]. Reachable meansthat one can travel from node i to node j by walkingalong links in the direction of the link. Any nodes thatare not reachable from node i are neglected in the sumof Eq. (11). In the context of social networks, closenesscentrality can be thought of as a measure of independenceas described in [14]. This is because a node with a largecloseness centrality does not have to rely on any one ortwo other nodes to transmit messages across the network[14]. In the context of weighted student collaborationnetworks, closeness centrality is a measure of both thefrequency with which a student assists others and howwidely a student collaborates.

Harmonic centrality is also a measure of how close anode is to other nodes in the network when one musttravel along directed links in the direction of the link.Harmonic centrality is defined as

cHi =∑j 6=i

1

dij, (12)

where dij is the shortest path distance from node i to j[24, 27]. Harmonic centrality has a similar definition tocloseness centrality, both being defined in terms of theinverse distances between nodes. The intuition for thetwo measures is the same. Nodes that are close to othernodes are more central as measured by closeness cen-trality and harmonic centrality. However, when comput-ing harmonic centrality, if node j is not reachable fromnode i, then the distance between the two nodes is set todij =∞. The corresponding term in the sum is then setto zero, 1/dij = 1/∞ ≡ 0. This may be preferable to theprocedure used to calculate closeness centrality as thisprocedure has been shown to introduce a bias towardsnodes in small components because it does not take intoaccount nodes that are not reachable by the node of in-terest [27].

Betweenness centrality is measure of how important anode is as a go-between for message transmission betweennodes in a network, assuming that information travelsalong paths of shortest distance [28]. Betweenness cen-trality is defined as

cBi =∑

j,k∈V

σ(j, k|i)σ(j, k)

, (13)

where σ(j, k|i) is the number of shortest paths from nodej to node k that pass through node i and where σ(j, k)is the number of shortest paths from node j to node k[13, 24]. Nodes with high betweenness centrality corre-spond to students with the most control over informationtransfer throughout the network. Therefore, the weightof the links in our networks do not modify betweennesscentrality directly, but do indirectly contribute throughthe path lengths.

V. RESULTS

We now correlate each of the nodal centrality measuredescribed in the previous section with students’ home-work assignment or exam scores. In Fig. 3, we display theresults of these calculations. Statistical significance of thecorrelation coefficients for each course and assignmenttype was determined at the p < 0.05 level using Holm-Bonferroni corrected p-values calculated via a bootstrapre-sampling with 10,000 re-samplings of each correlationcoefficient [29]. The use of bootstrap re-sampling wasmotivated by the fact that centrality measures from com-plex network analysis are inherently interdependent and,thus, violate the assumption of independence fundamen-tal to standard parametric statistics [15].

With respect to our first question of interest — do wellconnected students get good grades? — Fig. 3 suggeststhat this depends both on the type of centrality measureand the course in question. Four centrality measures havestatistically significant correlations to homework grades(both numerical and analytical) for all three courses:closeness centrality, harmonic centrality, in-strength, andout-strength. Recall that in- and out-strength measurehow often a student collaborates by receiving or givinghelp respectively, while closeness and harmonic centralityare both measures of how “far” a student is from otherstudents. Fig. 3 shows that in- and out-disparity have anegative correlation to homework scores in all classes andthat correlation is statistically significant for at least onetype of homework (analytical or numerical) in two of thethree courses. In- and out-disparity measure a student’stendency to collaborate often with only a small number ofstudents. This result, combined with the significant pos-itive correlations with the four centrality measures abovesuggest that students who not only collaborate often, butalso collaborate significantly with many different peopletend to achieve higher grades.

Fig. 3 also shows three measures that tend to havesmaller, and less often statistically significant, correla-tions with homework scores; these are betweenness cen-trality, local clustering, and net out-strength. Recall thatbetweenness centrality measures the extent to which astudent can control the flow of information between otherstudents, local clustering determines the extent to whicha student may be part of a close knit study group, andnet out-strength determines whether a student helps oth-ers more than they are helped. That these measures are

7

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pe

ars

on

R

Num erical Hom ework

Analyt ical Hom ework

Exam s

(a) Classical Mechanics Analytical Network

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pe

ars

on

R

Num erical Hom ework

Analyt ical Hom ework

Exam s

(b) Classical Mechanics Numerical Network

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pe

ars

on

R

Hom ework

Exam s

(c) Quantum Mechanics Network

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pe

ars

on

R

Hom ework

Exam s

(d) Electromagnetism Network

FIG. 3: Correlation of nodal centrality measures with student grades in three upper level physics courses. Correlationof complex network measures with student grades for three courses: Classical Mechanics, Quantum Mechanics, andElectromagnetism. Filled markers indicate correlation coefficients that are statistically significantly different from acorrelation coefficient of 0 (Holm-Bonferroni corrected p < 0.05). Statistical significance was determined using abootstrap re-sampling with 10,000 re-samplings of each correlation coefficient.

less often significant suggests that collaborating often andwidely has a greater relation to homework grades thanbeing part of a single close knit study group or beinga link between groups or students. The small correla-tion of homework scores with net out-strength relativeto the correlations for both in-strength and out-strengthalso suggests that the quantity of collaborations mattersmore than the type of collaboration (i.e., helping oth-ers vs. being helped). This finding is perhaps surprisinggiven that one might expect that students who more of-ten give help than receive it would be the naturally higherperforming students.

Comparing the correlations in Fig. 3 across the threecourses shows similar trends across most of the measures.One significant exception to this is the local clustering co-efficient which is larger (and statistically significant) inboth Classical Mechanics networks than in the networks

for the other two courses. This suggests that for the Clas-sical Mechanics course, integration into well-establishedstudy groups had a stronger relation to students’ scoreson both the analytical and numerical homework. To bet-ter understand possible sources of this difference, we lookat differences between the courses themselves. ClassicalMechanics is the first “hard” upper division course mostphysics students take; in fact, it is generally consideredby students to be one of the hardest courses in the Minesphysics curriculum. Thus, students may still be modi-fying their lower-division study habits to accommodatethis additional challenge. Thus, we may be seeing theeffect of tight knit study groups in a way that dissipatesas students get to know each other in the ensuing upper-division courses. The numerical component of the Classi-cal Mechanics course, which was not present in the othercourses and is unique to the curriculum, also represents

8

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pearson R

Classical Mechanics Analytical

(a)

c_C c_B c_H s_in s_out Y_in Y_out c_L s_net-1.0

-0.75

-0.5

-0.25

0.0

0.25

0.5

0.75

1.0

Pearson R

Quantum MechanicsElectromagnetism

(b)

FIG. 4: Correlation of nodal centrality measures from the Classical Mechanics Analytical network with (a) thecorresponding nodal centrality measure from the Classical Mechanics Numerical Network, and (b) the correspondingnodal centrality measure from the other two courses: Quantum Mechanics, and Electromagnetism. Filled markersindicate correlation coefficients that are statistically significantly different from a correlation coefficient of 0(Holm-Bonferroni corrected p < 0.05). Statistical significance was determined using bootstrap re-sampling with10,000 re-samplings of each correlation coefficient.

a possible contributing factor.

Another question of interest for this study pertainsto whether the benefits of collaboration extend beyondhomework scores to students’ performance on exams. Itmay be that the individual nature of exams suppressesthe impact of collaboration on students’ performance.Here again, Fig. 3 suggests that the answer depends onwhat course we look at. For both Quantum Mechanicsand Electromagnetism, correlations between all central-ity measures and exam scores are lower than correlationsof the same measure with homework scores. In fact, onlyout-strength correlates significantly with exam scores andthen only in Quantum Mechanics. This suggests that forthese courses, the potential benefit of centrality withinthe collaboration network does not extend to exams. Onthe other hand, in the Classical Mechanics course, thecorrelations of centrality measures with exam scores iscomparable to the corresponding correlations for home-work scores for nearly all measures. Beyond the inclusionof a fairly significant conceptual component in the Clas-sical Mechanics exams (∼1/4 of the questions), it is notobvious why the Classical Mechanics networks appear tocorrelate better with exam scores. However, these find-ings suggest that whether the benefits of collaborationextend to exam scores as well as homework scores de-pends on the structure of the course and/or content of theexams. Application of this analysis to other courses willbe necessary to pinpoint the course and exam featuresthat best realize the benefits of students’ collaboration.It is worth noting that, depending on the instructionalgoals of the course, exams may not be designed to real-ize the benefits of student collaboration. Moreover, all

exams in this study were traditional, individual exams;these findings would likely shift if the format of the examwas less traditional (e.g., group exams).

In our study, we have two simultaneous networks com-posed of exactly the same students: the analytical andnumerical networks for Classical Mechanics. This allowsus to investigate another question of interest – how sta-ble are these centrality measures across different typesof assignments? Comparing across the two Classical Me-chanics networks (analytical and numerical), we see inFig. 3 nearly identical patterns in the correlations of thedifferent centrality measures with homework scores bothin terms of magnitude of the correlations and which cor-relations are statistically significant. We also see verysimilar correlations when correlating network centralitymeasures created using information from the analyticalnetworks to the scores on the numerical homework andvice a versa. Together these findings suggest that thesecentrality measures are quite stable across different typesof homework. We can also use these two simultaneousnetworks to quantify the stability of the roles taken bystudents in response to different types of homework as-signments. To do this, we correlated centrality measurescalculated using the analytical network with those calcu-lated using the numerical network. We found large corre-lations (in this case r > 0.5) for eight of the nine central-ity measures (see Fig. 4a). Only the local clustering coef-ficient had only a moderate correlation (r = 0.33) acrossnetworks. All correlations were statistically significant(Bootstrap re-sampling and Holm-Bonferroni correctedp < 0.05). This result suggests that students’ collabo-ration strategies remain relatively stable when presented

9

with different types of homework assignments.In our study, we also have a large subset of students

who took all three courses (N = 67, see Fig. 1). Focusingspecifically on these students, we can also investigate thestability of network centrality measures across time asthe students advance from Classical Mechanics to Quan-tum Mechanics and Electromagnetism. Since all of thesecourses are upper-division (typically junior-level) courses,these students have already had much of their undergrad-uate career to develop collaboration strategies that theybelieve work for them; thus, we might anticipate thattheir strategies would be relatively stable over time andacross courses. In Fig. 4b, we correlate each centralitymeasure, student-by-student, between Classical Mechan-ics and Quantum Mechanics and also between ClassicalMechanics and Electromagnetism. Since neither Quan-tum Mechanics nor Electromagnetism included numeri-cal homework, we utilize the Classical Mechanics analyti-cal network for the purposes of investigating stability be-tween these courses. Of the nine network centrality mea-sures, four showed large correlations (in this case r > 0.6,see Fig. 4b) between the Classical Mechanics network andboth the networks of the other two courses. Interestingly,these are the same four measures with consistent signifi-cant correlations with the grades in all three courses (i.e.,closeness centrality, harmonic centrality, in-strength, andout-strength). The remaining centrality measures in Fig.4b show smaller correlations suggesting less stability instudents’ network positions between different course net-works. Local clustering coefficient once again highlightsas having a particularly small correlation, which in thiscase, is not statistically significant. Thus, the networkcentrality measures that have the strongest relationshipto students’ homework scores are also the network cen-trality measures that appear to be the most stable acrossnetworks involving different courses or assignment types.

VI. SUMMARY AND CONCLUSIONS

We utilized the tools of complex network analysis toform social networks based on students’ self-reported col-laborations when completing regular homework assign-ments in three upper-division physics courses. Fromthese networks, we then calculated multiple measures de-scribing the centrality and role of each student within thenetwork. By correlating these nodal centrality measureswith students’ scores on both homework and exams, wefound that four of the nine centrality measures (close-ness centrality, harmonic centrality, in-strength, and out-strength) correlated significantly with students’ home-work scores in all three classes. Together, the signifi-cance of these four measures suggest that students whonot only collaborate often but also collaborate signifi-cantly with many different people tend to achieve higherhomework grades. We also found that students’ collab-oration strategies are relatively stable when presentedwith different types of homework assignments (e.g., ana-

lytical vs. numerical) within the same class. Finally, wefound that while some centrality measures appear to shiftsignificantly when students move into a different course,the four centrality measures most strongly related to stu-dents’ homework scores are also the most stable betweennetworks from different courses. Correlations of central-ity measures with exam scores were generally smallerthan the correlations with homework scores, though thisfinding varied across courses. Note that this finding doesnot necessarily suggest that exams more accurately repre-sent a student’s individual understanding; while the cor-relation with collaboration is generally smaller, the highstakes nature of exams introduces a number of factorsbesides ability that can impact a student’s exam scores(e.g., stereotype threat).

This work helps provide insight into whether andhow students’ collaboration impacts their success inthe course as measured by course exam and homeworkgrades. It also contributes to a growing body of re-search utilizing complex network theory to better under-stand the role of social networks within the undergrad-uate classroom. There are several important limitationsto the study. The findings reported here are correlationaland thus cannot clearly establish that broad collabora-tion improves students’ performance, only that studentswho collaborate broadly tend to have higher scores. Ad-ditionally, these data come from a single institution andhave relatively low-N (N < 100). Replicating these anal-yses in additional courses at additional institutions willbe important for establishing the generalizability of thesefindings. For instructors, these results suggest that en-couraging students not only to collaborate, but to collab-orate with multiple other students may be an effectivestrategy towards improving students’ homework scores.Moreover, they suggest that the benefits of collaborationare not automatically transferred to exam performance,but rather the structure of the course and exams canenhance or suppress the relation between student collab-oration and exam scores. Finally, we point out that thesame kind of study could be performed on collaborationsamongst researchers at the graduate level and beyond,using not only in-class studies in graduate school analo-gous to those considered here, but also collaboration net-works on the arXiv. A general hypothesis to be examinedis whether an overall broader collaboration strategy leadsto higher outcomes, in for example h-index, total numberof citations, and grant funding.

VII. ACKNOWLEDGMENTS

This material is based in part upon work supported bythe US National Science Foundation under grant num-bers PHY-1520915, OAC-1740130, and DMR-1407962.The authors acknowledge the efforts of Esteban Chavezfor developing the code that sorted reporting discrepan-cies in the survey data we collected. Esteban Chavez wasfunded under the NSF grant DUE-0836937. We would

10

like to thank Mark Lusk for sharing course data with us. Lastly we would like to thank all of the students thatparticipated in our study.

[1] L. C. McDermott and E. F. Redish, “Resource letter:PER-1: Physics education research,” American Journalof Physics 67, 755 (1999).

[2] L. C. McDermott, “Research on conceptual understand-ing in mechanics,” Physics Today 37, 24 (1984).

[3] A. Van Heuvelen, “Learning to think like a physicist,”American Journal of Physics 59, 891 (1991).

[4] F. Reif, “Scientific approaches to science education,”Physics Today 39, 48 (1986).

[5] E. F. Redish and R. N. Steinberg, “Teaching physics:figuring out what works,” Physics Today 52, 24 (1999).

[6] S. P. Borgatti, A. Mehra, D. J. Brass, and G. Labianca,“Network analysis in the social sciences,” Science 323,892 (2009).

[7] M. E. J. Newman, “The structure and function of com-plex networks,” SIAM Review 45, 167 (2003).

[8] Stanley Wasserman and Katherine Faust, Social networkanalysis: Methods and applications, Vol. 8 (Cambridgeuniversity press, 1994).

[9] Daniel Z Grunspan, Benjamin L Wiggins, and Steven MGoodreau, “Understanding classrooms through socialnetwork analysis: A primer for social network analysisin education research,” CBE-Life Sciences Education 13,167–178 (2014).

[10] M. Bodin, “Mapping university students’ epistemic fram-ing of computational physics using network analysis,”Physical Review Special Topics - Physics Education Re-search 8, 010115 (2012).

[11] J. Bruun and E. Brewe, “Talking and learning physics:Predicting future grades from network measures andForce Concept Inventory pretest scores,” Physical ReviewSpecial Topics - Physics Education Research 9, 020109(2013).

[12] V. Latora and M. Marchiori, “Efficient behavior of small-world networks,” Physical Review Letters 87, 198701(2001).

[13] L. C. Freeman, “A set of measures of centrality based onbetweenness,” Sociometry 40, 35 (1977).

[14] L. C. Freeman, “Centrality in social networks conceptualclarification,” Social Networks 1, 215 (1979).

[15] Eric Brewe, Laird Kramer, and Vashti Sawtelle, “Inves-tigating student communities with network analysis ofinteractions in a physics learning center,” Phys. Rev. STPhys. Educ. Res. 8, 010101 (2012).

[16] Justyna P. Zwolak, Remy Dou, Eric A. Williams, andEric Brewe, “Students’ network integration as a predic-tor of persistence in introductory physics courses,” Phys.Rev. Phys. Educ. Res. 13, 010113 (2017).

[17] Justyna P. Zwolak, Michael Zwolak, and Eric Brewe,“Educational commitment and social networking: Thepower of informal networks,” Phys. Rev. Phys. Educ.Res. 14, 010131 (2018).

[18] Jonas Forsman, Rachel Moll, and Cedric Linder, “Ex-tending the theoretical framing for physics educationresearch: An illustrative application of complexity sci-ence,” Phys. Rev. ST Phys. Educ. Res. 10, 020122 (2014).

[19] Remy Dou, Eric Brewe, Justyna P. Zwolak, Geoff Potvin,Eric A. Williams, and Laird H. Kramer, “Beyond per-formance metrics: Examining a decrease in students’physics self-efficacy through a social networks lens,”Phys. Rev. Phys. Educ. Res. 12, 020124 (2016).

[20] AIP Statistical Research Center. http://www.aip.org/statistics.

[21] Blackboard Inc., Blackboard. http://www.blackboard.com, (2013).

[22] D. Schmidt, A. Bridgeman, and P. Kohl, “Student col-laborative networks and academic performance,” in APSMeeting Abstracts (2013).

[23] D. Schmidt, A. Bridgeman, and P. Kohl, “Studentcollaborative networks and academic performance inphysics,” in PERC 2013 (2013).

[24] A. A. Hagberg, D. A. Schult, and P. J. Swart, “Explor-ing network structure, dynamics, and function using Net-workX,” in Proceedings of the 7th Python in Science Con-ference (SciPy2008) (Pasadena, CA USA, 2008) p. 11.

[25] S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, andD. Hwang, “Complex networks: Structure and dynam-ics,” Physics Reports-Review Section of Physics Letters424, 175 (2006).

[26] E. Almaas, B. Kovacs, T. Vicsek, Z. N. Oltvai, and A.-L.Barabasi, “Global organization of metabolic fluxes in thebacterium Escherichia coli.” Nature 427, 839 (2004).

[27] P. Boldi and S. Vigna, “Axioms for centrality,” InternetMathematics 10, 222 (2014).

[28] M. E. J. Newman, Networks: An Introduction (OxfordUniversity Press, Inc., New York, 2010).

[29] L. Wasserman, All of Statistics: A Concise Course inStatistical Inference (Springer, New York, 2004).