Upload
lucas-boyd
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
June 2006
How good is our research?
New approaches toresearch indicators
June 2006
Average is a metric; distribution is a picture
• ‘Average impact’ is a good bibliometric index but not sufficient– A tool for reporting but not for action
• Data are skewed, so average is not central– Many papers are uncited and a few papers are very highly cited
• New approach looks at where the spread of performance falls– Activity is located within distribution by more than a single metric
– Thresholds help in describing peak of performance.
• This improves descriptive power, information content and management value
June 2006
Traditional impact indicatorsRebased average impact (world = 1.0) for G8 nations
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Year
Reb
ased
Imp
act
UK
USA
CANADA
FRANCE
GERMANY
ITALY
JAPAN
RUSSIA
EUROPEAN UNION
Data: Thomson ISI® National Science Indicators
June 2006
Distribution of “research performance”
Performance measure of 'research quality'
Fre
qu
ency
of
ach
ieve
me
nt
Benchmark e.g. 'world average'
Threshold of 'excellence'
June 2006
Distribution of “research performance”
Performance measure of 'research quality'
Fre
qu
ency
of
ach
ieve
me
nt
Benchmark e.g. 'world average'
Threshold of 'excellence'
A
B
Overall volume A = B
Average quality A = B
Activity above threshold B > AA good ‘indicator’ should capture and reflect this in some meaningful way
Do current metrics do this?
June 2006
Distribution of data values - income
0
5
10
15
20
Income category
Fre
qu
ency
Income per FTE Gross income
RAE2001 - research income for units in UoA14 Biology
£10m per unit
£250k per FTE
MaximumMinimum
June 2006
0
100
200
300
400
Impact category (normalised to world average)
Fre
qu
ency
UK Physics papers for 1995 = 2323
World average
Maximum0
Distribution of data values - impact
The variables for which we have metrics are skewed and therefore difficult to picture in a simple way
June 2006
Simplifying the data picture
• Scale data relative to a benchmark, then categorise– Could do this for any data set
• All journal articles– Uncited articles (take out the zeroes)
– Cited articles• Cited less often than benchmark• Cited more often than benchmark
– Cited more often but less than twice as often– Cited more than twice as often
» Cited less than four times as often» Cited more than four times as often
June 2006
Categorising the impact data
All papers
Uncited papers
Cited papers .
Papers cited less often than
benchmark Papers cited more often than benchmark
Papers cited more than
benchmark, but less than four times as often
Papers cited more than four times as
often as benchmark
= 0 >0 >0.125 >0.25 0.5 < 1 1 < 2 2 < 4 4 < 8 > 8
This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.
June 2006
UK ten-year profile 680,000 papers
0
5
10
15
20
25
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
ce
nta
ge
of
ou
tpu
t 1
995
-200
4
% of UK output over decade
AVERAGERBI = 1.24
MODE (cited)
MEDIAN
THRESHOLD OF EXCELLENCE?
MODE
June 2006
Implications
• Is the UK as good as we thought?– YES - the average is unchanged– What lies beneath just became apparent
• The effective peak is very concentrated
– Other countries would probably look similar
• New metrics are needed– Average impact not indicative of distribution– Need to add median, mode– Proportion of activity at thresholds of excellence
• Above world average, More than 4 x world average, etc
• Evaluate methodology– Does it work by year and by subject?– How can we apply it?
June 2006
Time profile
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Group
Per
cen
tag
e o
f o
utp
ut
for
per
iod
1995-1997 1998-2000 2001-2003
June 2006
Subject based curvesUK - Biology & Biochemistry
0
5
10
15
20
25
RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
% of UK for Year and Subject
UK - Chemistry
0
5
10
15
20
25
RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
% of UK for Year and Subject
Median RBI - 0.627
UK - Clinical Medicine
0
5
10
15
20
25
RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
% of UK for Year and Subject
Median RBI - 0.450
UK - Social Sciences, general
0
5
10
15
20
25
30
35
40
RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
% of UK for Year and Subject
Median RBI - 0.372
June 2006
Subject & site profiles – molecular biology
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Pe
rce
nta
ge
of
ou
tpu
t 1
99
5-2
00
4
% of UK for subject and time period shown as a
smoothed line
UK average shown as red
symbol
June 2006
HEIs – 10 year totals - 1
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
Leading research university Big civic 'Robbins' type university Former Polytechnic
June 2006
HEIs – 10 year totals - 2
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
ce
nta
ge
of
ou
tpu
t 1
995
-200
4
Leading research university Big civic 'Robbins' type university Former Polytechnic
June 2006
HEIs – 10 year totals – 4.1
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
Leading research university Big civic 'Robbins' type university Former Polytechnic
Smoothing the lines would reveal the shape
of the profile
June 2006
HEIs – 10 year totals – 4.2
Absolute volume would add a further element
for comparisons
0
5
10
15
20
25
30
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Per
cen
tag
e o
f o
utp
ut
1995
-200
4
Leading research university Big civic 'Robbins' type university Former Polytechnic
June 2006
HEIs – 10 year totals – 4.3
0
2000
4000
6000
8000
10000
RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8
Nu
mb
ers
of
arti
cles
199
5-20
04
June 2006
What next?
• Profiles– Create a view of the distribution of performance– Provide more information useful to management– Require a change in metrics
• Applications– Disaggregate the components of the research base– Track institutional profiles against benchmark– Evaluate the link between platform and peak– Track papers through time: e.g. leaders vs. climbers
June 2006
How good is our research?
New approaches toresearch indicators