WHAT CAN BIBLIOMETRICSTELL US ABOUT MEDICINE & BIOMEDICINE?

Philip Purnell

LeidenMay 2010

HOW DO WE EVALUATE RESEARCH?• Research grants

– Number and value

• Prestigious awards– Nobel Prizes

• Patents– Demonstrating innovative research

• Faculty– Number of post-graduate researchers

• Citation analysis– Publication and citation counts– Normalised by benchmarks

• Peer Evaluation– Expensive, time consuming and subjective

2

BRIEF HISTORY OF THE CITATION INDEX• Concept first developed by Dr Eugene Garfield

– Science, 1955

• The Science Citation Index (1963)– SCI print (1960’s)– On-line with SciSearch in the 1970’s– CD-ROM in the 1980’s– Web interface (1997) Web of Science

• Content enhanced:– Social Sciences Citation Index (SSCI)– Arts & Humanities Citation Index (AHCI)

• The Citation Index– Primarily developed for purposes of information retrieval– Development of electronic media and powerful searching tools

have increased its use and popularity for purposes of ResearchEvaluation 3

WEB OF SCIENCEJOURNAL SELECTION POLICY

WHY DO WE SELECTJOURNALS?

THOMSON REUTERSJOURNAL CITATION REPORTS

40% of the journals:

• 80% of the publications

• 92% of cited papers

4% of the journals:

• 30% of the publications

• 51% of cited papers

5

0

20

40

60

80

100

120

0 1000 2000 3000 4000 5000 6000

# of journals

% o

f data

base

Articles Citations

WEB OF SCIENCEJOURNAL SELECTION POLICY• Approx. 2000 journals evaluated annually

– 10-12% accepted

• Thomson Reuters editors– Information professionals– Librarians– Experts in the literature of their subject area

6

Web of Science

Journals under evaluation

Journal ‘quality’

THOMSON REUTERSJOURNAL SELECTION POLICY• Publishing Standards

– Peer review, Editorial conventions

• Editorial content– Addition to knowledge in specific subject field

• Diversity– International, regional influence of authors, editors,

advisors

• Citation analysis– Editors and authors’ prior work

7

GLOBAL RESEARCH REPRESENTATIONWEB OF SCIENCE COVERAGE

8

Region # Journals from Region in Web of ScienceEurope 5,573 49%

North America 4,251 38%Asia-Pacific 965 9%

Latin America 272 2%Middle East/Africa 200 1%

Language # Journals in Web of ScienceEnglish 9114 81%Other 2147 19%

Analyses based on authoritative, consistent data from the world’sleading provider of Research Evaluation solutions

Thomson Reuters has developed a selection policy over the last50 years designed to hand-pick the relevant journals containingthe core content over the full range of scholarly disciplines

This has created a large set of journals containing comparablepapers and citations

Thomson Reuters has always had one consistent editorial policyto index all journals cover-to-cover, index all authors and index alladdresses. This unique consistency makes Web of Science theonly suitable data source for citation analysis

SUMMARYCONSISTENCY IS THE KEY TO VALIDITY

EVALUATINGMEDICINE

RELATIVE IMPACT OF MEDICINE

11

RELATIVE CONTRIBUTION TO WORLD’SLITERATURE

12

RELATIVE PERFORMANCE OFMEDICAL DISCIPLINES

13

EVALUATINGCOUNTRIES

COMPARATIVE PERFORMANCEBY COUNTRY

15

EFFICIENCY COMPARED WITHEUROPEAN AND GLOBAL AVERAGES

16

GOVERNMENTS AND INSTITUTIONSUSING TR DATA FOR EVALUATION (INCL.)

• NWO & KNAW, Netherlands

• France: Min. de la Recherche, OST - Paris, CNRS

• Germany: Max Planck Society, several gov’t labs, DKFZ, MDCUS:National Institutes of Health

• United Kingdom: King’s College London; HEFCE

• European Union: EC’s DGXII(Research Directorate)

• US: NSF: biennial Science & Engineering Indicators report (since 1974)

• Canada: NSERC, FRSQ (Quebec), Alberta Research Council

• Australian Academy of Science, gov’t lab CSIRO

• Japan: Ministry of Education, Ministry of Economy, Trade & Industry

• People’s Republic of China: Chinese Academy of Science

• Times Higher Education: World University Rankings (from 2010)17

18

GLOBAL REACH: >4,000 RESEARCHCENTRES (91 COUNTRIES)

Asia-Pacific

353 Customers in 26 countries

Russia 147 Customers

Europe,Middle

East andAfrica

2,500+ Customers In 50 countries

244 Customers in 12 countries

LatinAmerica

760 customers

NorthAmerica

EVALUATINGINSTITUTIONS

EVALUATING INSTITUTIONS

Source: Thomson Reuters

North America University Science Indicators

COMPARATIVE PERFORMANCE OFGLOBAL MEDICINE RESEARCH

21

BENCHMARK YOUR PAPERS AGAINSTGLOBAL AVERAGES

22

Hematology articles fromthis year have been cited18,83 times

This article is ranked in the12,92nd percentile in itsfield by citations

Articles publishedin ‘Blood’ from2004 have beencited 34,30 times

This paper has received40/34,30=1,17 times theexpected citations forthis journal

This paper has received40/18,83=2,12 times theexpected citations forthis subject category

WHICH COLLABORATIONSARE THE MOST VALUABLE?

23

Collaborations with theseinstitutions have producedhighly cited papers withintheir subject fields

EVALUATING INDIVIDUALS

WHO ARE OURMOST PRODUCTIVE AUTHORS?

25

WHO ARE OURMOST INFLUENTIAL RESEARCHERS?

26

WHICH AUTHORS HAVETHE MOST IMPACT?

27

Normalises citationcounts for quantityof papers…

… but not for age ofpaper, documenttype or subject field!

WHICH AUTHORS’ PAPERS HAVEPERFORMED THE BEST IN THEIR FIELD?

28

Normalises citationaverage for subject fieldand age of papers

Meaning you can nowcompare the geneticistwith the historian

HOW CAN WE COMPARE RESEARCHERS?

29

Author A: 60 papers Author B: 117 papers

EVALUATING JOURNALS

EFFICIENCY JOURNAL IMPACT FACTOR

2008 Impact Factor

200820072006

Source paper – published in 2008

Cited reference – published in 2006 or 2007

Citations

AllPreviousYears

2005 2009

Citations in 2008To items published in 2007 = 273

To items published in 2006 = 463

Sum = 736

Number of itemsPublished in 2007 = 133

Published in 2006 = 173

Sum = 306

736

306

= 2,405

CALCULATING 2008 IMPACT FACTOREPILEPSY RESEARCH

JOURNAL IMPACT FACTORBIOTECHNOLOGY JOURNALS

CITATION BEHAVIOUR VARIESBETWEEN SUBJECT CATEGORIES

JOURNAL 5-YEAR IMPACT FACTORBIOTECHNOLOGY JOURNALS

HOW DO RESEARCH INSTITUTIONSEVALUATE JOURNALS FOR LIBRARIES?• Faculty head request

• Publisher packages

• Budget constraints

• Library recommendation

IN WHICH JOURNALS DO OURBIOLOGISTS PUBLISH?

AND IS THE GLOBAL COMMUNITYINFLUENCED BY YOUR RESEARCH?

USING THE IMPACT FACTOREVALUATING JOURNALS

• Appropriate use– To evaluate journals

• Misuse– Evaluation of individual articles– Evaluation of institution or researcher

USING THE IMPACT FACTORMISUSE: EVALUATING INDIVIDUAL PAPERS

30% of articles inFood Policy werenot cited at all

RESEARCHER IDSCHOLARLY RESEARCH COMMUNITY• Accurate Identification

• Organize and Manage

• Increase Visibility &Recognition

• Measure Performance

• Collaboration

• Security

Science, March 2009

INDIVIDUAL LEVELRESEARCH EVALUATION

View accuratepublication list dueto unique authoridentification See personalized

metrics using Web ofScience citation data

RESEARCHER IDANALYZE COLLABORATION NETWORK

Seek global collaborationopportunities by author,field, institution or country

RESEARCHER IDVISUALIZE CITING ARTICLES NETWORK

RESEARCHER IDELECTRONIC CV RESOURCE

Top Five CountriesTop Five Institutions

• Top Ten Institutions• University College London• The University of Queensland• Monash University• Harvard University• University of Michigan• University of Pennsylvania• ETH Zurich• University of Cambridge• Stanford University• McGill University

RESEARCHER IDGLOBAL PARTICIPATION

RESEARCHER IDUPLOAD SERVICE• Institutions can upload content on behalf of their

researchers– Upload researcher names Obtain a ResearcherID

account– Upload individual publication portfolios Articles are

matched to Web of Science records and ResearcherIDportfolios are generated

RESEARCHER IDDOWNLOAD SERVICE• Download data about the individuals at your

institution– Names and name variants, current and past affiliations

• Download ResearcherID publication portfolios– Bibliographic details of each item in the portfolio– For articles that were successfully matched to Web of

Science records there will be a Times Cited count and UTtag.

• Can be used by the customers for their owninternal systems