Bibliometric assessment of research performance in

social sciences and humanities

Henk F. Moed

Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands

2

Assessment of research performance: Basic assumptions - 1

• The concept of research quality does have a meaning in all fields of science and scholarship

• An assessment methodology should take into account the nature of the field

3

Assessment of research performance: Basic assumptions - 2

• Important contributions to scholarly progress are sooner or later communicated in scholarly publications

4

Research quality: Dimensions

• Contribution to scholarly progress

• Enlightenment of the general public

5

Science versus humanities (Price) - 1

Science Humanities

Orientation Quantitative Qualitative

Status of knowledge

Short term permanence

Permanent significance

Organisation Groups interacting at the research front

Individuals

6

Science versus humanities (Price) - 2

• Different substantive contents ask for different types of social organisation and information exchange

• Science: Citation Index

• Humanities: Normal archival library

7

Social sciences

• Social sciences constitute a heterogeneous domain, with both ‘science-like’ and ‘humanities-like’ orientations

• Even sub-disciplines may be heterogeneous (e.g., sociology)

8

Overall ISI coverage by main field

EXCELLENT (> 80%) VERY GOOD (60-80%) GOOD(40-60%)

Biochem & Mol Biol Appl Phys & Chem Mathematics

Biol Sci ~ Humans Biol Sci ~ Anim & Plants

Economics

Chemistry Psychol & Psychiat Engineering

Clin Medicine Geosciences MODERATE (<40 %)

Physics & Astron Soc Sci ~ Medicine & health

Other Soc Sci

Humanities & Arts

9

Sub-disciplines (non-exhaustive list)

Social sciences related to medicine and health

Other social sciences

Humanities

Public environm & occupat health

Nursing

Sport sciences

Substance abuse

Sociology

Anthropology

Educational sciences

Political science

Law

Literature

Linguistics

Historical sc

Philosophy

11

‘Other’ social sciences and humanities

• ‘National publication model’ is dominant

• Books play an important role

• Basic assumptions of a journal citation index are less valid

• ISI Indexes have inadequate coverage

• They also contain national journals

12

Indicator of a journal’s national orientation (INO)

• The share of the papers from the country most frequently publishing in a journal

• A purely national journal would have an INO value of 100 per cent

13

National orientation of journals (INO) for 4 fields

Discipline No. Journals

Median INO % Journals with

INO>90%

Physics & astronomy

260 37 10

Mol Biol & biochem

530 41 4

Humanities & arts

1,110 71 24

Other social sci

879 72 22

14

Policy assumption for social sciences and humanities

The extent to which research findings

• reach beyond a purely national or local viewpoint

• and are exposed to criticisms from a wide international scholarly audience

is a relevant criterion of research quality

15

The problem: ISI ≠ ‘International’

• Works exposed to an international audience are not necessarily included in the ISI Indexes

• Works included in the ISI Indexes are not necessarily exposed to an international audience

16

This book argues:

• It cannot be taken for granted that the ISI Citation Indexes provide such indicators in all subfields of these domains of scholarship

• A challenge would be to systematically explore alternative data sources and methodologies

17

4 Types of bibliometric studies

Cited/Target Citing/Source ISI coverage Field/Study

1 ISI ISI Excellent – Very Good

Astronomy

2 ISI+non ISI ISI Very Good – Good

Mathematics

3 ISI+non ISI ISI+non ISI Good – Moderate

Computer Science

4 No citation analysis at all Moderate NL Law

18

Alternative approaches

1. Expand the WoS with additional sources

2. Classification of publications and sources based on scholars’ quality perceptions

3. Library collection analysis

19

1. Expand the WoS with additional sources

• University of Granada (Spain):

• Creation of a citation index with 200 Spanish social science source journals not covered by the WoS

• 50,000 uses per year

20

2. Classifications based on scholars’ quality perceptions

• Case study on Flemish Law

• Questionnaires; No citation analysis

• Publications in Dutch: 81 %

• Publications in journals: 59 %

21

Classification of (national) journals based on a questionnaire into:

• Scholarly vs. non-scholarly

• Outstanding (A), good (B) and less good (C)

22

The ‘best’ indicator of scholarly research performance (in Flemish Law):

Count the number of

• Single and multi-authored books (first editions only)

• PhD theses

• Publications with a length > 5 pages

23

3. Library collection analysis

• Focuses on books

• Determine the number of academic library copies per book title

• Example: Use Worldcat (Linmans, CWTS, 2007)

END