Cognitive distance between expert panels and units of assessment

A. I. M. Jakaria Rahman, Raf GunsLoet Leydesdorff and Tim C.E. Engels

ECOOM day 20 November 2015

2

Discipline-specific research evaluation exercises - Improve the quality of scientific research - Research quality of individuals and groups (Nedeva et al., 1996; Rons, et al.,

2008; Butler & McAllister, 2011; Lawrenz et al., 2012)

- Research proposals (Li & Agha, 2015)

Expert Panel composition method

Trustworthiness of the assessment (Engels, et al. 2013);

Publications are key indicators of expertise (Rybak, Balog, & Nørvåg, 2014)

Barycenter method can measure and quantify (Rahman, Guns, Rousseau &

Engels 2015). Bibliographic coupling (Wang and Sandström, 2015)

3

Data

9 research groups of Physics department, and Biology department

Expert Panels – 6 members each – Entire career publications until the year of assessment

SCI-Expanded of Web of Science indexed citable items

Panel members had no prior involvement with the research groups

MethodThe cognitive distance between entities decreases as they have more publications in the same or similar Web of Science subject categories and/or journals

4

Global map of ScienceWeb of Science Subject Categories map from web-of-science data

Leydesdorff, L., Carley, S., & Rafols, I. (2013). Global maps of science based on the new Web-of-Science categories. Scientometrics, 94(2), 589–593

5

Loet Leydesdorff, Ismael Rafols, and Chaomei Chen (2013). Interactive Overlays of Journals and the Measurement of Interdisciplinarity on the basis of Aggregated Journal-Journal Citations, Journal of the American Society of Information Science and Technology 64(12)

Global map of ScienceJournal map from web-of-science data

6

Physics panel members’ publication overlay map

Physics groups’ publication overlay map

Global map of Science based on Web of Science Subject Categories

7

Groups publications overlay Panel publications overlay

Biology Department

Global map of Science based on Journals

8

Barycenter = Center of weight

Here, Barycenter locations determined by a weighted average of the vertices’ coordinates (Rousseau, 1989; Jin & Rousseau, 2001) according to the number of publications in each journals.

A B

N-Dimension/similarity matrix

Two dimensions

Barycenter calculation

𝐶𝑘=∑𝑗=1

𝑁

𝑚 𝑗 𝐿 𝑗 ,𝑘

𝑀

𝐶1=∑𝑗=1

𝑁

𝑚 𝑗𝐿 𝑗 ,1

𝑀;𝐶2=

∑𝑗=1

𝑁

𝑚 𝑗𝐿 𝑗 ,2

𝑀

A

B

C

D

J2

J1

J3

J4

J5

J6

J7

J9

J8

Panel

RG-B

RG-A

RG-C

RG-D

RG-E

Barycenter Example

Panel

RG-B

RG-A

RG-C

RG-E

RG-D

Barycenter Example

d

d

d

d d

Euclidean distance

12

Barycenter overlay map of Physics panel, panel members (PM) and research groups based on Web of Science Subject Categories

13

Barycenter overlay map of Physics panel, panel members (PM) and research groups based on Journals

14

Barycenter distances between Physics groups, panel and panel members in the similarity matrix of Journals.

Group PHYS-A PHYS-B PHYS-C PHYS-D PHYS-E PHYS-F PHYS-G PHYS-H PHYS-I

Panel 1.360 7.234 1.983 2.527 6.208 2.184 4.012 8.456 2.205 5.380

PM1 6.153 6.648 7.963 8.294 0.976 5.111 5.193 11.527 7.957 6.369

PM2 3.761 9.154 1.072 4.667 7.629 4.475 5.415 10.561 1.059 7.531

PM3 4.411 5.803 5.895 5.518 6.313 4.538 5.144 9.918 5.989 4.641

PM4 4.868 7.492 6.698 4.040 8.181 4.954 6.559 5.722 6.911 5.924

PM5 3.169 8.509 2.795 2.437 8.035 4.063 5.811 8.536 3.047 6.547

PM6 2.857 8.538 0.764 3.902 6.975 3.503 4.508 9.483 0.845 6.789 Shortest distances between a group and a panel member are underlined and printed in bold.

Barycenter distances between Physics groups, panel and panel members in the similarity matrix of Web of Science Subject Categories.

Groups PHYS-A PHYS- B PHYS-C PHYS-D PHYS-E PHYS- F PHYS-G PHYS-H PHYS-I

Panel 0.422 2.594 0.319 0.631 2.038 0.743 1.726 1.736 0.335 1.998

PM 1 1.876 2.161 2.243 2.624 0.150 1.416 1.083 2.544 2.238 2.026

PM 2 1.060 3.154 0.501 0.674 2.645 1.413 2.306 2.220 0.513 2.575

PM 3 1.039 2.297 1.220 1.592 1.767 1.035 1.368 2.063 1.171 1.682

PM 4 1.015 2.655 1.230 0.787 2.526 1.278 2.295 1.190 1.266 2.032

PM 5 0.726 2.881 0.303 0.471 2.350 1.077 2.034 1.910 0.352 2.277

PM 6 0.715 2.870 0.229 0.633 2.243 1.006 1.907 1.970 0.271 2.293 Shortest distances between a group and a panel member are underlined and printed in bold.

15

SummaryEuclidean distances – can be calculated both in two and more dimensions and the results are very similar.

Overlay maps visualize the position of panel and group publications

Barycenter method allows to - identify the Euclidean distances between the panel and units of assessment

- calculate average distances, comparison of distance and visual exploration of the barycenters on the map

- identity the relevance of the expertise in a coherent way.

Cognitive distance between expert panels and units of assessment

Thank you