A systematic empirical comparison
of different approaches for
normalizing citation impact
indicators
Ludo Waltman and Nees Jan van Eck
Centre for Science and Technology Studies (CWTS), Leiden University
14th ISSI conference, Vienna, Austria
July 16, 2013
2
Introduction
• Citation-based indicators need to be normalized for
differences in citation practices between fields
• Traditional normalization based on WoS subject
categories is problematic because many subject
categories are heterogeneous in terms of citation
practices
3
Clinical Neurology: Citation density
Visualization produced using VOSviewer
(Van Eck et al., PLoS ONE, 2012)
4
Clinical Neurology: Reference density
Density of references instead of citations
Notice the similar patterns in the two visualizations!
5
Normalization approaches
• Normalization based on a classification system (‘cited-
side normalization’)
• Source normalization (‘citing-side normalization’)
6
Normalization based on classification system
• Requires a field classification system in which
publications are assigned to fields
• Following common practice, we use the WoS journal
subject categories
• Citations are compared to the field average
e
cNCS
Number of citations of a publication
Average number of citations of all
publications in the same field
7
Source normalization (1)
• No field classification system is needed
• Citations are weighted differently depending on the
number of references in the citing publication or the
citing journal
• Instead of giving equal weight to each citation, equal
weight is given to each citing publication
8
Source normalization (2)
• Three source normalization variants:
– Audience factor (Zitt & Small, JASIST, 2008):
– Fractional citation counting (Leydesdorff and colleagues):
– Revised SNIP (Waltman et al., JOI, 2013):
• Only ‘active references’ should be considered!
c
i ia1
)1( 1SNCS
c
i ir1
)2( 1SNCS
c
i ii rp1
)3( 1SNCS
Average number of references per publication
in citing journal
Number of references in citing publication
Proportion of publications in citing journal
without references
9
Evaluation
• To what degree does each normalization approach:
– correct for field differences?
– correct for differences in the age of publications?
• Using the same classification system both in the
implementation and in the evaluation of a normalization
approach gives biased results (Sirtes, JOI, 2012)
• For evaluation purposes, we use four classification
systems:
– WoS journal subject categories
– Algorithmically constructed classification systems A, B, and C
• 3.8 million WoS publications from the period 2007–2010
• Classification systems constructed using large-scale
clustering approach (Waltman &Van Eck, JASIST, 2012)
• Clusters defined at the level of individual publications
rather than at the journal level
• Number of clusters (research areas) per classification
system:
– Classification system A: 21
– Classification system B: 161
– Classification system C: 1334
Algorithmically constructed classification
systems
10
Average score per normalization
approach and per publication year
11
0
2
4
6
8
10
12
CS NCS SNCS(1) SNCS(2) SNCS(3)
Ave
rage
sco
re
2007
2008
2009
2010
Similarity of normalized citation
distributions
• Let’s now look beyond averages
• To what degree do fields have identical normalized
citation distributions?
12
Results based on WoS subject categories (1)
• Similarity of normalized citation distributions in different
fields
13
235 WoS subject categories; publication year 2007
Inequality index
• How to summarize the degree to which citation
distributions coincide?
• We use the methodological framework of Crespo et al.
(PLoS ONE, 2013)
• Citation distributions are compared percentile-by-
percentile using the Theil inequality index
14
Results based on WoS subject categories (2)
15
Results based on classification system A
16
Results based on classification system B
17
Results based on classification system C
18
19
Citation density before
normalization
Citation density after
normalization using SNCS(3)
20
Citation density before normalization
Citation density after normalization using SNCS(3)
Conclusions
• Using the same classification system both in the
implementation and in the evaluation of a normalization
approach should be avoided
• NCS (subject-category-based normalization) and SNCS(2)
(‘fractional citation counting’) do not perform so well
• SNCS(1) (‘audience factor’) and SNCS(3) (‘revised
SNIP’) have a good performance
• Need for more practical experience with SNCS(1),
SNCS(3), and alternative normalization approaches, in
particular percentile-rank normalization
21
22 22
Thank you for your attention!
Problem of using the same classification
system both in the implementation and in the
evaluation of the NCS approach
Publication Field No. of citations NCS
1 A 1 0.50
2 A 3 1.50
3 B 2 0.50
4 B 6 1.50
23
Publication Field No. of citations NCS
1 X 1 0.67
2 Y 3 0.67
3 X 2 1.33
4 Y 6 1.33
• Results based on the correct assignment of publications to fields
• Results based on an incorrect assignment of publications to fields
24
Practical issues
• World average
• Document types
• Citation windows
• PPtop indicators
• ‘Trade journal problem’
25
A related problem (1)
26
A related problem (2)
298 articles in 2008
40 citations
27
A related problem (3)
Resp. 266 and 486 articles in 2008
Resp. 6% and 15% of all articles and reviews in WoS categories Business and Business, finance
Resp. 29 and 8 citations
28
National journals
Effect of excluding national journals on MNCS
See Waltman & Van Eck (2012)
29
National journals
Effect of excluding national journals on MSNCS3