Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea Chan Goo Yi (Pukyong National University, Korea ; [email protected])[email protected]

Meta-evaluation of the Performance Evaluation System of Public Research

Institutes in Korea

Chan Goo Yi(Pukyong National University, Korea ; [email protected])

Jang Jae Lee(Korea Institute of Science and Technology Evaluation and

Planning (KISTEP) ; [email protected])

Yong Soo Hwang(Korea Science and Technology Policy Institute (STEPI) ;

[email protected])

AEA Evaluation 2009 Conference Nov. 11-14, Orlando, Florida

2

Contents

1. Introduction 2. Research Method and Framework 3. Korean Research Institute Evaluation 4. Current Practise & Meta-evaluation 5. Discussion for Future Development 6. Conclusion 7. References

3

1. Introduction

� Background - In Korea, evaluation system of research

institutes introduced in 1999, and transferred into the performance evaluation system in 2005

- Arguments among various stakeholders such as CEO, researchers and evaluation panel, whether evaluation can contribute

• the quality enhancement of R&D results • the development of management

system

4

1. Introduction(con.)

� Research Purpose

- To meta-evaluate (1) the rationality of evaluation system itself and (2) the appropriateness of its current practise

- To discuss policy alternatives for development of the evaluation system itself and its implementation

5

2. Research Method & Framework

� Research Method - In-depth interview with 109 stakeholders • conducted between March and May 2008 • 99 internal stakeholders : CEOs(10), managers in administrative dept.(27), principal investigators(30), researchers(32)

• 10 external stakeholders : evaluation panels from university(4), industry(3) and public research institute(3) - Meta- evaluation approach : Evaluation of

evaluation system and practical process

6

2. Research Method & Framework(con.)

� Research Framework : Components of In-depth Interview & Meta-evaluation

Implementation(4)

Paradigm(2)

Purpose Object

Utilization(2)

Impact Type

Panel Interval

Method Indicator

7

3. Korean Research Institute Evaluation

� Brief History of Institute Evaluation System - Introduction period (1999-2002) • similar system operated among research councils • research achievements < management achievements - Diversification period (2002-2005) • improved representing characteristics of individual

member research institutes • research achievements ≒ management achievements - Development period (2006-current) • transferred performance evaluation system • focusing rather outcome or impact than output • research achievements > management achievements

8

3. Korean Research Institute Evaluation(con.)

� R&D Governance in Korea : 3 Research Councils System

- Korea Research Council for Fundamental S&T (KRCF) • 13 member research institute • under the Ministry of Education, Science &

Technology - Korea Research Council for Industrial S&T (ISTK) • 13 member research institutes • under the Ministry of Knowledge Economy - National Research Councils for Economic,

Humanities and Social Sciences (NRCS) • 23 member research institutes • under the Office of the Prime Minister

9

3. Korean Research Institute Evaluation(con.)

� Framework of Current Evaluation SystemField Part Item Indicator

ResearchAchieve. (70%)

Individual Performance Goals (50%)

Objective 1 Sub Objective 1.1 Sub Objective 1.2

Objective 5 Sub Objective 5.1 Common Items (7) Elective Items (10)

Indicator 1.1.1 Indicator 1.1.2 Indicator 1.2.1 Indicator 1.2.2

Indicator 5.1.1

Selected by each institutes autonomously

Comprehensive Performance Goals (20%)

Manage.Achieve.(30%)

Responsibility & Innovation

R&D & Resource Management

3 indicators

3 indicators

10

4. Current Practise & Meta-evaluation

(1) Evaluation Paradigm� Evaluation Purpose - Formal : future development strategies,

research performance enhancement, accountability, program/management improvement, knowledge transfer etc.

- Actual : R&D program/project development, internal management system improvement

(Meta-evaluation) - Incompatible between formal purposes and

actual - Focused rather short-perspective evaluation

purposes

11

4. Current Practise & Meta-evalu.(con.)

� Evaluation Object - Research Achieve.(70%)vs.Management A.(30%) • Each field divided into sub parts and items - Covering both ‘basic R&D program’(grant fund)

and ‘national R&D program’(competition fund) (Meta-evaluation) - Too many evaluation objects • Not differentiating among evaluation objects - Absent of essential objects for core evaluation

purposes such as ‘future development strategy’ - Lacks of consensus for core evaluation objects

among stake-holders

12


(2) Evaluation Implementation� Evaluation Panel - Panel from university, research institute and

industry, & comprising all domestic experts • Research : individual panel for each institute • Management : common panel for all institutes (Meta-evaluation) - Panel members’ professionalism limited • Professionals in sub performance goals of R&D

project rather than peer reviewer or upper performance goal in R&D program

- A few lacks of considering international excellency

13


� Evaluation Interval - Until 2007 : evaluating all institutes every year - From 2008 : separating research achievement

and management one • Research A. : 3 years ; Management A. : 1 year (Meta-evaluation) - Too often evaluated and burden to institutions • Main factor negative affecting other components

such as evaluation purpose, object, utilization - Resulted in more focusing the visible and short

term outputs rather than long term and comprehensive outcomes or impact

14


� Evaluation Method - External panel conducts full evaluation process - Evaluating 4/5 member institutes as one group in

a comparative perspective with others - Combination of the ‘review of performance

report’ submitted by each institute and the ‘site visit’ for four or five hours in individual institute

(Meta-evaluation) - Focused on literature review rather than site visit - Evaluation in a comparative/relative perspective

not an absolute one, in particular for the research performance

15


� Evaluation Indicator : Focused on Research - Each Program(50%) vs. Comprehensive R&D(20%) • Performance goals -> Objects -> Indicators - Each institute suggests their own goals/indicators • Evaluating performance level targeted in advance

(Meta-evaluation) - Disconnection of performance goal and core R&D

activities - Indicator pool limited for comprehensive R&D

performance goal - Indicators more fit for research program rather than

development one

16


(3) Evaluation Utilization� Evaluation Impact - Evaluation findings directly fed back to the basic

R&D(grant fund), but indirectly did to national R&D(competition fund)

- Feed back to internal management system (Meta-evaluation) - Evaluation findings more effecting management

system rather than R&D management process - Lacks of consensus of impact among

stakeholders • Insiders : low, Outsiders : relatively high

17


� Type of Evaluation Utilization - Formal : future development strategy, R&D/

management improvement, R&D prioritization and budget allocation, best practise, CEO’s annual pay adjustment, policy suggestion

- Actual : budget reallocation of basic R&D, adjustment of CEO’s annual payment, best practise

(Meta-evaluation) - Limited and confined evaluation utilization - Instrumental utilization is further actual type

than conceptual one

18

5. Discussion for Future Development

� Outline of Discussions for Developments - Institutional Approach (IA) • Development/amendment of evaluation

system itself and related systems at the level of research councils or the government

• Long-term and institution-based perspective - Operational Approach (OA) • Improvement of evaluation practise and

process under the current system • short-term and operation-based perspective

19

5. Discussion for Future Development(con.)

(1) Evaluation Paradigm� Evaluation Purpose - Need to transfer from internal

responsibility to external one (OA) • Because current system has more

contributed internal responsibility/ management rather than external one

- More focusing responsibility for external stakeholders (OA)

• In particular, responsibility for citizen

20

5. Discussion for Future

Development(con.)

� Evaluation Object - Rearrangement of current evaluation object

(IA) • Whether competitive national R&D program

be included or not? - Re-setting the evaluation objects in the longer

perspective, such as; (IA) • Long-term vision/strategy of the institution • Future potentials and R&D infrastructures • Risk management in related public sector

21


Development(con.)

(2) Evaluation Implementation� Evaluation Panel

- Setting up individual/independent panel for each institution, from single panel for all (IA)

- Enhancement of professionalism of panel (OA) • Recruiting more field experts like as industries • Extension of job term from 2 to 4/5 years • Career management system of panel members - Adoption of international experts panel (OA)

22


Development(con.)

� Evaluation Interval

- Extension of evaluation interval from 1 year to 3 to 5 years (IA)

• Linkage with CEO’s term in office • Same or different evaluation interval

between research results and management results

• Consideration of characteristics of research fields; such as emerging technology vs. long-term basic science

23


Development(con.)

� Evaluation Method - Enforcement of evaluation method in the way

of absolute perspective (IA) • In particular, evaluation of research results - Focusing in-depth review in research lab (IA) • Extending evaluation period to 3 to 4 days • Reviewing first-hand materials(research

note) • Interview and discuss with researchers - Introduction of cross-cutting review • Among related institutions/organizations in

public sector (IA)

24


Development(con.)

� Evaluation Indicator - Improvement of method of establishing

performance goals/indicators (OA) • Closer connectivity between performance

goal/indicators and core R&D activities • More changeable and creative

goals/indicators - High linkage between performance goals and

internal performance management system (OA)

• For example, BSC, MBO, ISO 9001, KM etc - Increase of indicator pools for comprehensive

performance goals (OA)

25


Development(con.)

(3) Evaluation Utilization� Evaluation Impact - Enforcement of feed back system of

evaluation findings to R&D management (OA)

• High collaboration with related other agencies for national R&D management

- Extension of scope and target group of evaluation utilization (OA)

• From top manager to all employees

26


Development(con.)

� Type of Evaluation Utilization - Activation of long-term and conceptual

evaluation utilization (OA) • Setting up vision/mission • Planning R&D strategy • Disseminating the best practise • Producing policy information/knowledge

etc - Informing the multiple type of evaluation

utilization to all stakeholders (OA) • In particular, in-site researchers

27

6. Conclusion

� Summary of the research - In basic, both internal and external

stakeholders consider the current system somewhat useful for R&D management and organizational management

- They also suggest policy alternatives for the development of certain components of evaluation system and its current practise

• Some are the system itself, others are the common limits of evaluation system of public sector in general in Korea

28

6. Conclusion (con.)

� Implication - Policy alternatives for

developments of evaluation system for research institutes in terms of system itself and current practise

- Policy knowledge/ information for analysis and re-establishment of the governance of public research institutes

29

6. Conclusion (con.)

� Limitations and Further Works - Discussion of policy alternatives a little

bit lacks specific and detailed matters in some individual components

• Scope and depth of policy alternatives in certain components is too broad

- Need for more detailed action plan in each evaluation component, based on the findings of this meta-evaluation/research

30

7. References � Further Information for Korean S&T and

Evaluation System - National Science and Technology Council (

http://www.nstc.go.kr) - Ministry of Education and Science and

Technology (http://www.mest.go.kr) - Korea Research Council for Fundamental

S&T (http://www.krcf.re.kr) - Korea Research Council for Industrial S&T (

http://www.istk.re.kr) - Korea Institute of S&T Evaluation and

Planning (http://www.kistep.re.kr)

Documents

Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea Chan Goo Yi (Pukyong National University, Korea ; [email protected])[email protected]