Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song€¦ · ‒ Introduction to the Analytic Hi erarchy Process (T. L. Satty, 1990) ‒ Propose a reliable AHP method using Delphi technique,

PICMET ’13 Conference

Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song

Korea Institute of S&T Evaluation and Planning (KISTEP)


The objective of this study is to examine the relative importance of the attributes of national R&D output index. Relative importance data were collected from AHP(Analytic Hierarchy Process) survey of experts who have responsibility of R&D output management in universities and institutes located in Seoul and Dae-jeon cities in Korea. Five attributes were basic research output, application development research output, economic outcome, social outcome, and human-resource outcome. We found that economic outcome was the attribute that affected researcher preference. The findings of this study could be applied as a basic information in terms of decision making process for the efficient management of national R&D performance.

1872

2013 Proceedings of PICMET '13: Technology Management for Emerging Technologies.


Contents

Problem Description

Solution Procedure: AHP

Computational Results

3/27


[Negative Balance of Technology Trade]

1995 1999 2004 2009

1,835 million $

2,493 million $2,731 million $

4,856 million $

[Research Productivity & Sum of Tech. Transfer Revenue]

1.48%

4.32%

151million $

2,280million $

: Sum of Technology Transfer Revenue

:Research Productivity

2010 2010 2009

• Lower Application of R&D Performance in Korea

Utilizations and diffusions of research performance are

emerged as important issues

Research Background

SolutionProblem Result

• Policy of Technology Transfer and Commercialization

-『 Acton theperformanceevaluationandmanagementofnational researchanddevelopmentprojects』

-『 2nd Basic Research Plan of Research Performance management and Application (2011~2015) 』

4/27

1873



Performance Management

System

Research for R&D Performance Management System

Research for R&D Performance Analysis Method

Need for Analyzing Relative Importance of R&D Performance

Index

Expansion of R&D Performance Transfer &

Commercialization

Maximize Effective of R&D


Performance Evaluation

Management& Application

of R&D Performance

Existing Proposed Research

Necessities of Research

Necessities of Research

Improvement Strategy Policy IssueImprovement Strategy

Improvement Strategy PolicyPolicy IssueIssue


R&D Performance Management System

Response Strategy System Centered to Issue

Understand Level from Analyzation

Policy Environments

TechnologySupply

Technology Demand

Transfer Infra

5/27

PICMET ’13 Conference SolutionProblem Result

Efficiency

Effectiveness

Enforceability

Political Acceptability

Equity

Compatibility

FlexibilityMAUT

Multi-Attribute Utility Theory

AHPAnalytic Hierarchy Process

PROMETHEEPreference Ranking Organization

METHod for Enrichment Evaluation

Reason for Choosing AHP

Criteria for Policy Assessment

Objective of This Research

• Structuralize and analyze the problems

• Measure relative importance or preference of

each factors

• Gather the opinions of the hands-on workers

Multi-criteria Decision Making

6/27

1874



Literature Review

AHP Survey ‒ Introduction to the Analytic Hierarchy Process (T. L. Satty, 1990)‒ Propose a reliable AHP method using Delphi technique, which aims to incorporate the relative

expertise of panel and to increase their consensus (Song, et al. 1999)

Definition of R&D Performance‒ Propose main types of evaluation studies (A. Airaghi, et al. 1999)‒ Cross tabulation of important impact dimensions of public research and technological

development spending is suggested that depicts different classes of effects with regard to time and worlds (L. Georghiou, et al. 2001)

Application to other areas‒ Development of Evaluation Model for Knowledge Management Performance (Jung, et al, 2010)‒ Evaluation Items in Technology Transfer (Lee, et al. 2011) ‒ Information Security (Sung, et al. 2010)


7/27


Contents

Problem Description



8/27

1875



A

A

B

B

C

C

9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9

Increasing Importance to the Left Increasing Importance to the Right


To Investigate the relative importance between the decision - making factors‒ Possible to analyze the multi-criteria decision-making‒ Applicable to all areas which need to decision-making

AHP survey process‒ Setting Decision-making Hierarchy ‒ Collecting data through Pair-wise Comparison‒ Estimating of Relative Importance between the decision - making factors‒ Driving overall ranking by summing weight between the decision - making factors

9/27


Conceptual Model

Step 1Step 1

Step 2Step 2

Step 3Step 3

Step 4Step 4

Exploratory

Solid

Evaluation studies Time

ShortMedium

LongWorld

ScienceEconomy

Society

Research Performance

Economical Effect Ripple EffectEconomical

Effect

Output Impact

Research Performance

Output

Impact

Economical Impact

Workforce Impact

Basic Research Performance

Application Research Performance

Social Impact


10/27

1876



– Georghiou, L., Rigby, J. and Cameron, H., “Assessing the Socio-economic Impacts of the Framework Programme, University of Manchester, pp. 169-172, 2001.

Definition Example

Solid Impact Studies

More characteristic evaluation approaches, take place ex post and are concerned with a full range of effects

Legal impact, Economic Impact, et al.

Exploratory Impact Studies

Examine potential effects ex ante but have a more economic and quantitative focus

Paper, Patent, Industrial property, et al.

- Airaghi A., Busch N.E., Georghiou L., Kuhlmann S, Ledoux M.J., van Raan A.F.J. and Viana Baptista J., “Options and Limits for Assessing the Socio-Economic Impact of European RTD Programmes,” ETAN, Commission of the European Communities, 1999.

Outcome Impact

Short-term Medium-term Short-term Medium-term

Scientific and Technological

New/improved techno logical know-how Increased productivity Increased

competitiveness

Economic New/improved products, services and processes

Increased sales and market share

New markets; development of SMEs

Societal Job creation better infrastructure, healthcare, new regulation

Increased prosperity satisfaction equality

Stable, cohesive society with highly participative

citizen

11/27


제 1세대(~1995)

제 1세대(~1995)


Effe

ct o

f Im

pact

Range of Performance

Direct Research Performance

Direct Economical Effect

Indirect Economical Effect

Indirect Ripple Effect

Smaller Bigger

Expl

orat

ory

Solid

Performance from R&D(ex. Patent, Paper)

Economical Profit from Technology Transfer & Commercialization

Economical Ripple Effect (ex. Value-added of Enterprise, Positive Balance of Technology Trade)

Scientific, Technological, Social, and Cultural Ripple Effect(Without Economical Ripple Effect)

12/27

1877



Index for Research


Direct Effect

Ripple Effect

Economical Impact

WorkforceImpact


Applicable Research

Performance

Social Impact


13/27


14/27

1878



Direct Effect

Ripple Effect

Economical Impact

WorkforceImpact


Application Research

Performance

SocialImpact

Level 1 Level 2

Science Citation Index

Highly cited scientific

articles(top 1%)

Nature science cell

The number of Highly Cited Researchers

Nobel prizes

Patent applications per thousand total

researchers

Number of "triadic" patent

famillies

Highly cited patent applications to the U.S. PO(top 1%)

Highly cited patent applications to the

EPO(top 1%)

Number of patent with co-

applicants

Value added of innovation-intensive

manufacturing sectors

High-tech exports(% of manufactured exports)

Share of high technology in total exports (%)

Technology balance of payments

The World Top 200 universities

Global 1000 leading companies

Venture capital investment

GERD as a percentage of GDP

Graduation rates at doctorate level

Graduation rates at doctorate level

Science and engineering graduates at doctorate

level

Science and engineering graduates at doctorate

level

Proportion of the national field of science and technology degree

Proportion of the national field of science and technology degree

Scientific and technical personnel ratio of

business professionals

Scientific and technical personnel ratio of

business professionals


15/27


Level 1Level 2

Factor Reference


Science Citation Index COSTII, Web of Science

Highly cited scientific articles(top 1%) ESI DB

Nature science cell Web of Science

The number of Highly Cited Researchers Thomson Reuters ISIHIGHLYCITED.COM

Nobel prizes IMD World Competitiveness

ApplicationResearch

Performance

Patent applications per thousand total researchers WIPO(World Intellectual Property Organization)

Number of "triadic" patent famillies OECD Main Science and Technology Indicators

Highly cited patent applications to the U.S. PO(top 1%) National R & D performance analysis reports

Highly cited patent applications to the EPO(top 1%) OECD STI Scoreboard

Number of patent with co-applicants COSTII, OCED iLibrary

Economical Impact

Value added of innovation-intensive manufacturing sectors EU Innovation Scoreboard, OECD STI Scoreboard

High-tech exports(% of manufactured exports) COSTII, World Bank Science & Technology

Share of high technology in total exports (%) COSTII, OECD Science, Technology and Industry Outlook

Technology balance of payments COSTII, OECD Main Science and Technology Indicators

SocialImpact

The World Top 200 universities COSTII, Times Higher Education-QS World University Rankings

global 1000 leading companies COSTII, Forbes, Global 2000 Leading Companies

Venture capital investment COSTII, OECD Science, Technology and Industry Outlook

GERD as a percentage of GDP COSTII, OECD STI Scoreboard

WorkforceImpact

Graduation rates at doctorate level Science and Engineering Index, OECD STI Scoreboard

Science and engineering graduates at doctorate levelEU Innovation Scoreboard, Science and Engineering Index,

OECD STI Scoreboard

Proportion of the national field of science and technology degree COSTII, Scienceand Engineering Index, OECDSTI Scoreboard

Scientific and technical personnel ratio of business professionals EU Innovation Scoreboard, COSTII, OECD STI Scoreboard


16/27

1879



Pair-wise Comparison

Questionnaire Configuration


AHP survey for research performance management indicators- Literature review and exports' advice related in research performance management

and application

- Respondents represent points as relative importance for each index

Period: 2011. 10. 15. ~ 2011. 12. 1. (for 6 weeks) Interview R&D Performance management representatives in Korean

research institutions

Develop matrix of pairwise ratios to estimate of relative weights

SurveyBelong Number

University 6

Government Research Institute 13

Enterprise Institute 2

Techno Park 4

Sum 25

- These members are experts of Korea

R&D performance management at each

research centers

17/27


Contents

Problem Description



18/27

1880



17.3

14.6

29.5

18.2

20.4

0 5 10 15 20 25 30 35

Workforce Impact

Social Impact

Economical Impact

Application Research

Performance


Relative Importance(%)


An Indicator of the Consistency of the Respondents( Consistency Ratio <0.1 ) → Can be trusted

19/27


11.5

11.8

20.3

27.1

29.3

0 5 10 15 20 25 30 35


Nobel prizes


The number of Highly Cited

Researchers

Highly cited scientific articles(top

1%)




20/27

1881



13.8

18.1

19.8

23.6

24.7

0 5 10 15 20 25 30

Number of patent with co-

applicants

Patent applications per thousand

total researchers

Highly cited patent applications to

the USPO(top 1%)

Highly cited patent applications to

the EPO(top 1%)

Number of "triadic" patent families




21/27


22.4

23.3

27.0

27.3

0 5 10 15 20 25 30

Value added of innovation-

intensive manufacturing sectors

Technology balance of payments

Share of high technology in total

exports (%)

High-tech exports(% of

manufactured exports)




22/27

1882



17.9

21.1

23.2

37.8

0 5 10 15 20 25 30 35 40

The World Top 200 universities

Venture capital investment

global 1000 leading companies

GERD as a percentage of GDP




23/27


11.9

16.0

34.8

37.3

0 5 10 15 20 25 30 35 40

Graduation rates at doctorate

level

Proportion of the national field of

science and technology degree

Science and engineering

graduates at doctorate level

Scientific and technical personnel

ratio of business professionals




24/27

1883



Level 1 Level 2 Priority

Factor Importance(%) Factor Importance(%) Importance(%) Ranking

Basic

Research

Performance

20.4

Science Citation Index 11.5 2.35 21

Highly cited scientific articles(top 1%) 29.3 5.98 7

Nature science cell 20.3 4.14 12

The number of Highly Cited Researchers 27.1 5.53 8

Nobel prizes 11.8 2.41 20

Application

Research

Performance

18.2

Patent applications per thousand total researchers 18.1 3.29 15

Number of "triadic" patent famillies 24.7 4.50 10

Highly cited patent applications to the U.S. PO(top 1%) 23.6 4.30 11

Highly cited patent applications to the EPO(top 1%) 19.8 3.60 13

Number of patent with co-applicants 13.8 2.51 19

Economical

Impact29.5

Value added of innovation-intensive manufacturing sectors 22.4 6.61 4

High-tech exports(% of manufactured exports) 27.3 8.05 1

Share of high technology in total exports (%) 27.0 7.97 2

Technology balance of payments 23.3 6.87 3

Social Impact 14.6

The World Top 200 universities 17.9 2.61 18

global 1000 leading companies 23.2 3.39 14

Venture capital investment 21.1 3.08 16

GERD as a percentage of GDP 37.8 5.52 9

Workforce

Impact17.3

Graduation rates at doctorate level 11.9 2.06 22

Science and engineering graduates at doctorate level 34.8 6.02 6

Proportion of the national field of science and technology degree 16.0 2.77 17

Scientific and technical personnel ratio of business professionals 37.3 6.45 5


25/27


Result Summary

Further Research

Conclude


26/27

1884



Reference

27/27


Thank you!

Email : [email protected]

1885


Documents

Kil-Woo Lee, Yu-Cheong Chon, Jung-Min Song€¦ · ‒ Introduction to the Analytic Hi erarchy Process (T. L. Satty, 1990) ‒ Propose a reliable AHP method using Delphi technique,