THOMSON SCIENTIFIC

Patricia Brennan

Thomson Scientific

January 10, 2008

THOMSON SCIENTIFIC

Research Publication and Performance

- Impact Factor Imperative?

•How do we assess value?

• Current Academic Evaluation Market

• Role of citations, citation metrics

• Possibilities with new metrics – Usage Factors, H Index,

• Irish Research output: Summary Metrics

THOMSON SCIENTIFIC

Stakeholders in Academic Evaluation

THOMSON SCIENTIFIC

Drivers for increased evaluation and assessment

THOMSON SCIENTIFIC

Source: National Citation Report Ireland 2003-2003Scholarship is increasingly Collaborative

Scholarly communications are changing: what is the output ?

THOMSON SCIENTIFIC

Scholarship is Global

0

200

400

600

800

1000

1200

1400

1600

1800

1981 1983 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003

Year

Pape

rs w

ith N

Cou

ntrie

s

> 5 countries > 10 countries > 15 countries

>5 countries

Source: Web of Science®

THOMSON SCIENTIFIC

Mean Authors per Paper

0

0.5

1

1.5

2

2.5

3

3.5

4

81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 0 1 2 3 4 5 6

year

auth

or/p

aper

Source: Web of Science®

Scholarship is increasingly collaborative

THOMSON SCIENTIFIC

Largest Collaboration in 2006: 2512

authors, a “collaboration of collaborations”

Collaborations are getting broader and more complex

THOMSON SCIENTIFIC

What to measure? When?• Total Papers• Total Citations• Citation Impact (cites per paper)• Percent Cited Paper• Impact Relative to Field• Percentile Rank in Field • Collaboration Indicators• Expected Citation Count• Ratio of Citations to Expected citation count• Expected Citation Rate for Category• Mean / Median Citation• H Index• Citation Frequency Distribution• Time Series Trends

9

Authors

Institutions

Nations

Topics

Fields

THOMSON SCIENTIFIC

Metrics: Individual papers1. Cites: Total citation count for selected paper.

2. Cites2: 2nd generation cite count based on total citations received by the citing articles.

3. Expected Citation Rate: An average rate of citation for all the papers of that document type (articles, reviews, letters, etc.), in that journal, for that selected year. This is a metric to evaluate citation counts.

4. Ratio: Ratio of expected cites to actual cites

5. Field: Subject area for the journal in which the paper appeared.

6. %: Percentile position of the paper based on citations in the same field.

THOMSON SCIENTIFIC

Metrics: Individual papers

1. Cites: Total citation count for selected paper.

Cites: 85

THOMSON SCIENTIFIC

Metrics: Individual papers

2. Cites2: 2nd generation cite count based on total citations received by the citing articles.

Cites: 85

Cites2: 574

THOMSON SCIENTIFIC

Metrics: Individual papers3. Expected Citation Rate: An average rate of citation for all the papers of

that document type (articles, reviews, letters, etc.), in that journal, for that selected year. This is a metric to compare peer journal papers.

Cites: 85

Cites2: 574

Expected Citation Rate: 20.8(All Articles from European Journal of

Neuroscience in 2001 received on

average 20.8 cites through year-end 2006.)

THOMSON SCIENTIFIC

Metrics: Individual papers

4. Ratio: Ratio of expected cites to actual cites

Cites: 85

Cites2: 574

Expected Citation Rate: 20.8

Ratio: 4.1

[85: 20.8 = 4.1]

THOMSON SCIENTIFIC

Metrics: Individual papers

5. Field: Subject area for the journal in which the paper appeared.Cites: 85

Cites2: 574

Expected Citation Rate: 20.8

Ratio: 4.1

Field: Neuroscience

[Note: For the multidisciplinary journals Science, Nature and PNAS,

all articles and reviews are reassigned based on the primary

category to which the article’s citing and cited journals are assigned.]

THOMSON SCIENTIFIC

Metrics: Individual papers

6. Percentile: position of the paper based on citations in the same field and year.

Cites: 85

Cites2: 574

Expected Citation Rate: 20.8

Ratio: 4.1

Field: Neuroscience

Percentile: 3.5%[The 85 cites to this Neuroscience

paper places it in the top 3.5% based on the citation distribution to all papers published in this field in

2001. ]

THOMSON SCIENTIFIC

Metrics for groups of papers1. Total # papers and total # cites: combined numbers for the set

2. Mean times cited: Total cites divided by total papers. [average impact]

3. Median times cited: Midpoint for citations

4. H-Index: Number of papers (N) in a given dataset having N or more citations.

5. C-Index: Sum of all actual citations divided by sum of all expected citations.

6. Average Percentile: average of the field percentile measures which are based on field and year of publication

7. Disciplinarity: reflects the level of multidisciplinarity in a set of papers, ranging from 0 to 1, where the lower the number, the greater the multidisciplinarity. (Herfindal Index)

THOMSON SCIENTIFIC

H-Index: Number of papers (N) in a given dataset having N or more citations.

14 papers in this set had

14 or more citations

THOMSON SCIENTIFIC

C-Index: Sum of all actual citations divided by sum of all expected citations.

C index = 1.98

357 180.46

THOMSON SCIENTIFIC

Average Percentile: average of the field percentile measures which are based on field and year of publication

Average Percentile =

31.62

THOMSON SCIENTIFIC

Tools for multiple workflows: Author, Researcher

• Promotion and Tenure– Author Finder– The Distinct Author Identification System (DAIS)– Citation Report– ResearchID.com

21

“.. I would prefer to see something like a unique identifier for

people (so I can still get all of an author's work even if I don't know his professional history) ..”

THOMSON SCIENTIFIC

• Evaluating Research TeamsInitial Views and Exploration– Analyze Tool– Citation ReportProfiling Within Institutions– Journal Use ReportsA Global View– Essential Science IndicatorsDetailed and Precise Analysis– Custom Analysis

22

Tools for multiple workflows: Administrators, Researchers

THOMSON SCIENTIFIC

Tools for multiple workflows: Librarian, Publisher

• Evaluating a journal collection:– Journal Citation Reports, – Journal Use Reports

23

THOMSON SCIENTIFIC

The Impact Factor Imperative

IF Calculation

THOMSON SCIENTIFIC

Irish Research Output: Summary Metrics

Metric Count

Total Papers 18,218

Total Cites 131,826

Mean Times Cited 7.24

Median Times Cited 2

H Index 86

Average Percentile 50.70

C-Index 1.115

Disciplinarity Indicator .01

Source: National Citation Report Ireland 2003-2003

THOMSON SCIENTIFIC

Irish Research Output: Comparative Output

Source: National Science Indicators 06

THOMSON SCIENTIFIC

Irish Research Impact: Comparative Impact

Source: National Science Indicators 06

THOMSON SCIENTIFIC

Irish Research Output: Summary Metrics

• Most Cited Author: Wolf K H– Cited 6143 times

Source: National Citation Report Ireland 2003-2003

THOMSON SCIENTIFIC

Understanding Context: Average Impact per Field

THOMSON SCIENTIFIC

Establishing Best Practices

• Consider whether available data can address the question

• Choose publication types, field definitions, and years of data

• Decide on whole or fractional counting

• Judge whether data require editing to remove “artifacts”

• Ask whether the results are reasonable

• Use relative measures, not just absolute counts

• Obtain multiple measures• Recognize the skewed

nature of citation data• Confirm data collected are

relevant to question• Compare like with like

THOMSON SCIENTIFIC ACADEMIC & GOVERNMENT MARKETS

Thank You