NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Support for Bibliometric/Portfolio Analysis from the NIH Library

Terrie Wheeler, MLSAnne White-Olson, MLS

Brigit Sullivan, MLSTom Moskal, DVM

May 3, 2013

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Establishing linkages: using the literature to identify value

Using the literature to assess science requires:1. clear linkages from

funding to papers to new science, and

2. clean data3. meaningful metrics

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Concerns with standard publication metrics

• Impact Factor– Average # of citations

(within one year) received by all papers in a journal published within the subsequent 2 years

• h-index– g-index– tapered h-index– a-index– etc.

Source: http://en.wikipedia.org/wiki/File:H-index-en.svg

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Percentile Ranking

• Normalizes citation counts• Is not as affected by a

skewed distribution• Provides information about

publication impact• Is calculated fairly cleanly• Over time percentile ranking

in a discipline is anticipated

Percentiles for papers published by field, 1999 - 2009Multidisciplinary 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years

0.01% 493 401 346 584 350 399 198 164 91 72 13 4930.10% 468 305 346 584 350 399 162 163 71 59 13 1961.00% 62 69 79 121 107 65 57 52 40 26 5 65

10.00% 5 5 12 16 14 14 16 16 14 7 2 1020.00% 2 2 5 5 4 5 5 8 7 3 1 450.00% 1 1 1 1 1 1 1 1 1 1 0 1

Neuroscience & Behavior 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1254 1286 1306 954 680 557 345 295 122 69 12 8480.10% 596 564 532 440 310 325 206 140 78 28 7 4031.00% 260 230 219 182 145 116 94 68 36 13 3 151

10.00% 74 71 66 59 49 41 32 22 13 5 1 4220.00% 45 43 40 36 31 27 21 15 9 3 1 2550.00% 18 17 16 15 13 11 9 6 4 1 0 8

Psychiatry/Psychology 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 943 909 630 821 871 335 656 197 87 35 10 6240.10% 409 397 297 282 250 183 135 100 50 17 6 2451.00% 160 144 130 111 99 81 56 40 22 8 3 92

10.00% 47 44 41 35 31 26 19 13 7 3 1 2520.00% 27 26 24 21 19 16 12 8 5 2 0 1450.00% 9 8 8 8 7 6 5 3 2 1 0 4

Clinical Medicine 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1388 1400 1541 1528 1013 817 742 390 244 88 16 10200.10% 573 519 471 434 402 316 252 175 99 34 6 3541.00% 179 172 156 145 127 105 83 56 33 12 3 112

10.00% 48 46 43 39 35 30 24 17 10 4 1 2820.00% 28 27 26 24 21 19 15 10 6 2 0 1650.00% 9 9 9 8 8 7 6 4 2 1 0 5

All Fields 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 All Years0.01% 1216 1210 1139 1030 816 638 499 316 197 71 12 8040.10% 480 452 411 361 308 248 193 129 76 28 6 2961.00% 161 152 139 124 105 88 67 46 27 11 3 97

10.00% 42 40 37 34 30 25 20 14 8 3 1 2420.00% 24 23 22 20 18 15 12 9 5 2 0 1350.00% 8 7 7 7 6 5 4 3 2 1 0 3

Percentiles for papers

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Using the literature to identify value

• NIAID – impact on standards of care

• NIMH – impact of extramural funding

• CSR – seeking which study sections have funded most drug/device discoveries

• NHLBI – papers from funded research over the last ten years

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

NIDCD PIs and IC Collaborators• Who are the NIH PIs that the NIDCD scientists

collaborate with?

• How many publications have they authored during a specific time period (2007-2012)

• Who is first author? Who is last author? What number author are they?

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

How to approach this?

• PubMed (few analytics)• Web of Science (analytics, but fewer bib records

than PubMed)• BiomedExperts Elsevier• NIH IRP Website• NIDB• Reporter

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Capturing the Data

• Web of Science: Author Search and Analyze Results/Author

• Endnote: Each author was individual custom group

• Excel: Each NIDCD author had spreadsheet, where listed all authors and counts of what number author s/he was

• Exported Endnote to Word: Bibliographies for each author.

Visual of Data

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Take Away• Dirty data and numbers vary per system

• Different paths give different numbers. Pick one and use that consistently

• No one system has all the answers

• Value of Librarians: Know the systems, Know data, Ask a lot of questions, Use a lot of sources.

Patterns of electric light on Earth seen from space at night show the digital divide between north and south. Patterns of electric light on Earth seen from space at night show the digital divide between north and south. (UNEP report)

Mapping Science

Views of Earth at Nighthttp://www.nasa.gov/mission_pages/NPP/news/earth-at-night.html

Mapping Science

NIDCD Data SetPortable Digital File format = .pdf

NIDCD Data Set

Scalable Vector Graphics format = .svg

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIDCD Data Set

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

The NIAID question

What is NIAID’s impact on society? How has NIAID’s work influenced standards of care? 

The Standard of Care Prevention and Control of Influenza with VaccinesRecommendations of the Advisory Committee on

Immunization Practices (ACIP), 2010 Morbidity and Mortality Weekly Report www.cdc.gov/mmwr

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

How to Assess Impact• How many intramural

NIAID authors are cited in the references at the end of this standard or care?

• How many papers in the reference listing were funded with NIAID grant funds?

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Logistics• 552 citations in

Scopus

• 524 export

• Lose 7

• Lose 69--not indexed in Medline

• Scopus to EndNote to Medline to WOS to EndNote

• Dynamics of dedupping– Select criteria carefully– Rerun with different criteria

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

NIAID affiliation• Intramural

• Extramural

• Address field

• Grants

• UNIQUE

11

59

10

54

67

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Conclusion

• Out of the 448 references* cited in the document Prevention and Control of Influenza with Vaccines Recommendations of the Advisory Committee on Immunization, NIAID is responsible for or affiliated with 67 or 15%

*the document has 552 references but only 448 made it through the system without getting lost

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Who else is citing the NIAID papers?

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

Can we predict future science with publication data?

• Reliable metrics• Clean data• Establish linkages

– funding– papers– new science

• Evaluate trends

NIH LIBRARY | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH

For more information

Terrie WheelerNational Institutes of Health Libraryterrie.wheeler@nih.gov301.496.1157