Program Level Assessment of Outcomes and Impacts of Research of CentersJuan D. RogersSchool of Public PolicyGeorgia Tech

Project: Assessment of 15 Nanotechnology Science and Engineering Centers’ (NSECs): Outcomes and Impacts: Their Contribution to NNI Objectives and Goals, NSF 0955089.

AEA, Annaheim CA 2

ObjectivesPublication output and impact of the

NSECsCo-authorship patterns change over

timeGeographical spread or concentration of

NSEC research collaborationEvolution of areas of research as

reflected in keyword changesMultidisciplinary footprint of the centersThe case for centers

11/5/2011

AEA, Annaheim CA 3

Methodology OutlineQualitative-quantitative concurrent design

◦Qualitative component Multiple embedded case studies of mechanisms for

center outcomes Follow up field level impact with quantitative analysis

◦Quantitative component Bibliometrics (productivity, citation, co-authors, etc.) Personnel and funding data analysis Intellectual property instances (patents, licenses, etc.) Links with business and commercial applications

◦Methodologically: Final results are generalizations to theory from qualitative analysis We offer analysis of combined quant-qual data Interpretive schemes for quantitative findings on

centers

11/5/2011

AEA, Annaheim CA 4

Data and methods Acquisition of 85 center annual reports from all 15 NSEC

centers Extraction and clean up (duplicate removal) of publication

lists Extraction of NSEC articles from Web of Science (n=3,500) Look up and extraction of articles citing NSEC articles

(n=75,000) Clean up and classification of collaborating organizations Clean up, look up, identification, and matching of NSEC

authors in author listings Analyses of different type, multiple tools:

◦ Growth, shares, and overall trends (tabular)◦ Networks and collaborations (Gephi)◦ Geographical spread, GIS (ArcGIS)◦ Keywords and topics (VantagePoint)◦ Multidisciplinarity and science maps (Pajek)

11/5/2011

AEA, Annaheim CA 5

NSEC publication activity grows in three waves

11/5/2011

Notes: *Publication data not reported by all NSEC centers; last column reports average annual change for rows with change data.Source: ISI-WoS publication data based on NSEC annual reports by center.

2001

2002

2003

2004

2005

2006

2007

2008

2009

-10*

-

100

200

300

400

500

600

700

800

-

5,000

10,000

15,000

20,000

25,000

NSEC publications (all centers)

Citing articles

NS

EC

Pu

blic

ati

on

s

Cit

ing

art

icle

s

2001 2002 2003 2004 2005 2006 2007 2008 2009-10* 2001-10*

NSEC publications (all centers) 66 133 221 262 499 515

715

737

361 3,870 ▪ Annual change 102% 66% 19% 90% 3% 39% 3% -51% 34% Citing articles 48 391 1,164 2,619 4,595 7,415 10,469 15,243 19,149 94,484 ▪ Annual change 715% 198% 125% 75% 61% 41% 46% 26% 12%

Publications

2001-04 2005-062007-08

AEA, Annaheim CA 6

One out of 10 NSEC publications has an industry co-author

11/5/2011

Total unique firms co-authoring articles with NSEC (2001-2010): 146Total unique firms maintaining other types of collaborations with NSEC (as of 2010): 275**

Notes: * Publication data not reported by all NSEC centers; last column reports average annual change for rows with change data. **The type of collaborations are not specified by centers (only number of industry partners was provided).Source: ISI-WoS publication data based on NSEC annual reports by center and lists of industry partners provided by NSEC centers.

2001 2002 2003 2004 2005 2006 2007 20082009-

10*2001-

10*

NSEC centers with publications 3 6 6 13 13 15 15 15 13 15

NSEC publications (all centers) 66 133 221 262 499 515 715 737 361 3,509

NSEC pubs. co-auth. with industry 12 13 16 17 35 52 76 65 34 360

▪ Annual change 8% 23% 6% 106% 49% 46% -14% -48% 22%

▪ Share industry co-auth / all pubs. 18% 10% 7% 6% 7% 10% 11% 9% 9% 10%

Unique co-author firms 11 13 9 16 31 29 50 43 22 146

▪ Annual change 18% -31% 78% 94% -6% 72% -14% -49% 20%

Industry collaborations

AEA, Annaheim CA 7

Comparative Impact of NSEC Papers (Citations Cohort 2001)2001 2002 2003 2004 2005 2006 2007 2008 2009

C_Mean

0.88 6.30 14.44 23.64 35.88 48.48 61.11 75.61 90.17C_Med

0 2 4 6 9.5 11 12.5 14.5 17.5C_Max

12 52 148 297 528 776 1062 1409 1760N_Mean

0.26 2.11 5.13 8.18 11.62 14.78 17.87 21.11 24.04N_Medi

0 1 2 4 5 6 8 9 10N_Max

39 163 376 747 1268 1803 2286 2902 3484

11/5/2011

AEA, Annaheim CA 8

Comparative Impact of NSEC Papers (Citations Cohort 2002)2002 2003 2004 2005 2006 2007 2008 2009

C_Mean

1.20 9.06 21.32 36.92 51.75 67.13 85.15 101.80C_Med

0 3 8.5 16 20.5 25.5 30.5 36.5C_Max

19 88 196 400 585 807 1063 1330N_Mean

0.28 2.32 5.30 8.83 12.10 15.36 18.79 21.92N_Med

0 1 2 4 6 7 8 9N_Max

50 153 340 661 1053 1499 2042 2587

11/5/2011

NSEC papers appear to have higher impact measured by citations: median and mean citations grow faster than the cohort with window length

AEA, Annaheim CA 9

Rank in Cohort of Top 20 NSEC Papers

1 2 3 4 5 6 7 8 9 10

2001 5 29 98 167 179 223 313 411 465 6292002 6 10 17 20 23 56 69 124 127 148

11/5/2011

11 12 13 14 15 16 17 18 19 20

2001 834 1345 1383 2077 2992 3202 3203 3283 3284 37552002 175 192 228 279 322 382 430 513 559 566

NSEC papers rank highly in their cohort of Nano papers. Total Cohort 2001: 30462 papers. NSEC Cohort 2001: 66 papersTotal Cohort 2002: 34971 papers. NSEC Cohort 2002: 128 papers

AEA, Annaheim CA 10

Comparative Impact of NSEC Papers (Citations Cohort 2003)

2003 2004 2005 2006 2007 2008 2009

N_mean 0.30 2.28 5.76 9.17 12.62 16.32 19.77

N_median 0 1 3 4 6 7 9

N_max 46 251 674 1144 1676 2299 2913

C_mean 1.09 7.75 19.29 31.08 43.12 58.35 73.32

C_median 0 4 10 15 21 26 30

C_max 13 80 209 377 573 855 1204

11/5/2011

The distributions parameters show NSEC high position in the field but the toppaper is still not by the centers. N = 40813 CN = 222

1 2 3 4 5 6 7 8 9 10

3 14 15 34 36 39 47 65 83 98

11 12 13 14 15 16 17 18 19 20

119 126 129 132 137 159 198 211 238 245

AEA, Annaheim CA 11

Comparative Impact of NSEC Papers (Citations Cohort 2004)

2004 2005 2006 2007 2008 2009

Xmean 0.28 2.56 5.78 9.21 12.96 16.50

Xmedian 0 1 3 4 6 8

Xmax 46 299 650 1076 1683 2334

NXmean 0.84 7.32 16.14 25.74 37.31 47.79

NXmedian 0 4 10 15 20.5 28

NXmax 19 80 154 225 306 392

11/5/2011

1 2 3 4 5 6 7 8 9 10

42 52 55 71 73 79 100 105 123 147

11 12 13 14 15 16 17 18 19 20

248 261 300 314 472 473 539 557 588 614

N = 48952; CN = 259

AEA, Annaheim CA 12

Comparative Impact of NSEC Papers (Citations Cohort 2005)

2005 2006 2007 2008 2009

N_mean 0.36 2.65 6.04 9.97 13.74

N_median 0 1 3 5 7

N_max 69 303 590 974 1553

C_mean 1.20 9.07 20.51 34.96 49.12

C_median 0 4 10 16 23

C_max 25 145 405 949 1553

11/5/2011

1 2 3 4 5 6 7 8 9 10

1 5 8 17 18 25 29 32 36 38

11 12 13 14 15 16 17 18 19 20

42 50 81 86 100 102 105 129 130 149

N = 55998; CN = 499

AEA, Annaheim CA 13

Comparative Impact of NSEC Papers (Citations Cohort 2006)

2006 2007 2008 2009 2010

N_mean 0.39 2.67 6.37 10.11 10.94

N_median 0 1 3 5 6

N_max 39 176 396 671 756

C_mean 1.06 7.53 18.04 29.43 40.77

C_median 0 4 9 14 18

C_max 20 94 270 574 980

11/5/2011

1 2 3 4 5 6 7 8 9 10

1 3 5 6 21 25 30 48 51 70

11 12 13 14 15 16 17 18 19 20

84 89 95 99 100 103 132 146 148 190

N = 62351; CN = 512

AEA, Annaheim CA 14

Increasing JIF of Target Journals

11/5/2011

AEA, Annaheim CA 15

Increasing JIF of Target Journals

11/5/2011

AEA, Annaheim CA 16

Leveraging Support Reflected in Publication Support Acknowledgment

11/5/2011

AEA, Annaheim CA 17

NSEC co-authorship networks grow and become more widespread

11/5/2011

Co-authorships2001-2006

Notes: Nodes represent authors. Node size represents number of publications for the period. Node colors represent 15 NSEC centers. Line colors are those of the centers that maintain each co-authorship as found in publication databases.Source: ISI-WoS publication data based on NSEC annual reports by center.

Co-authorships

Co-authorships2007-2010

AEA, Annaheim CA 1811/5/2011

Central and wellconnected authors Highly collaborative

projects

Multiple productive andcollaborative authorsNSEC collaboration networks

present diverse patterns of co-authorship

(2001-2010)

Co-authorships

AEA, Annaheim CA 19

Wide geographic spread of NSEC research with concentration in specific locations/regions

11/5/2011

Note: Number of NSEC publications from 2001-2010 = 3509; number of citing publications = 75335. Citing publications, 2001-2010 exclude all NSEC publications.

Collaboration and places

AEA, Annaheim CA 20

Co-authoring extends beyond the NSECs to nearly all US states

11/5/2011

Collaboration and places

Clusters of top keywords(21 cluster solution reported)

More specializedterms

More linkedterms

Research topics

AEA, Annaheim CA 22

Top 20 Terms Across 15 NSECs(relative position of 30 most common, 2001-2006 v. 2007-2010)

11/5/2011

FOUNDATIONAL•FABRICS•THIN-FILM•SELF-ASSEMBLED MONOLAYER•SPECTROSCOPY•CARBON NANOTUBE

STABLE•NANOCRYSTAL•OPTICAL PROPERTY•MOLECULAR-DYNAMICS

GROWTH•FIELD-EFFECT TRANSISTOR•LITHOGRAPHY•GOLD NANOPARTICLE•QUANTUM DOT•SENSOR•WALL CARBON NANOTUBE•IN-VITRO•SEMICONDUCTOR•MOLECULAR-DYNAMICS SIMULATION•BIOSENSOR•IN-VIVO•THIN-FILM-TRANSISTOR•GRAPHITE

MATURE•DNA•SCATTERERS•TRANSISTOR•DIP PEN NANOLITHOGRAPHY•SILVER NANOPARTICLE•AQUEOUS-SOLUTION•ATOMIC FORCE MICROSCOPY•WAVE-GUIDE•HETEROSTRUCTURE

Research topics

AEA, Annaheim CA 23

Neurosciences

Agriculture

Chemistry

Physics

Engineering

Environ Sci

Matls SciClinical Med

Computer Sci.

Geosciences

Ecology

General Med

Map of Science: Subject Categories within all Nano (2009)

Source: Rafols, I. and Meyer, M. (2009) Diversity and Network Coherence as indicators of interdisciplinarity: case studies in bionanoscience. /Scientometrics/, 81(2), in print; Leydesdorff, L. and Rafols, I. (2009) A Global Map of Science Based on the ISI Subject Categories. Journal of the American Society for Information Science and Technology, 60(2), 348-362.

11/5/2011

AEA, Annaheim CA 24

Neurosciences

Agriculture

Biomed Sci

Chemistry

Physics

Engineering

Environ Sci

Matls Sci

Infectious Diseases

Clinical Med

Computer Sci.

Geosciences

Ecology

General Med

NSEC research is multidisciplinary with focus areas in materials science, chemistry and biomedical sciences

Map source: Rafols, I., Meyer, M. (2009) Diversity and Network Coherence as indicators of interdisciplinarity: case studies in bionanoscience. Scientometrics, 81(2), in print; Leydesdorff, L., Rafols, I. (2009) A Global Map of Science Based on the ISI Subject Categories. Journal of the American Society for Information Science and Technology, 60(2), 348-362.

Multidisciplinarity

Global Map of Science, 2006175 SCI Subject Categories

11/5/2011

AEA, Annaheim CA 25

Neurosciences

Agriculture

Biomed Sci

Chemistry

Physics

Engineering

Environ Sci

Matls Sci

Infectious Diseases

Clinical Med

Computer Sci.

Geosciences

Ecology

General Med

A range of disciplines beyond focal areas cite NSEC works

Multidisciplinarity

Global Map of Science, 2006175 SCI Subject Categories

11/5/2011

AEA, Annaheim CA 26

Implications for Center Policy

NSEC publication growth rate indicates rapid take-off by new centers.

NSEC mechanism allows for greater involvement of authors over time in diverse locations.

NSEC research involves some foundational  as well as some emerging (and some maturing) topical areas.

NSECs work in multiple disciplines and their citation influence involves many more disciplines (especially in Biotechnology related fields).

11/5/2011

Conclusions

AEA, Annaheim CA 27

The Case for CentersReporting by PIs on crucial role of centers:

◦ Unique incentive to go deeply across disciplinary boundaries Significantly reduced cross-disciplinary transaction costs

◦ Unique research experience for graduate students Rich and diverse research infrastructure enables more risk taking Exposure to greater number of high quality scientific contacts Exposure to unique industry contacts

◦ Accelerator of promising young researcher development Center as recruiting tool of top talent Diverse mentoring opportunities for rapid career development

◦ Unique infrastructure possibilities Leverage of resources for shared new facilities Design of new unique instruments and experimental

arrangements

11/5/2011

AEA, Annaheim CA 28

Some Program ChallengesSome scientific contributions are difficult to

explain to the lay public ◦ Important for long term support of the enterprise

The special benefits for development of faculty and graduate students should be scaled up to reach the rest of the university community

The pre- and extra- university education efforts are dependent on the center program and are difficult to institutionalize without it. ◦ Their sustainability should be a program concern

Similar sustainability issue is raised by specialized infrastructure that may go to waste if centers are discontinued

11/5/2011