Logic Model and Methodology for Knowledge Innovation Program(KIP) Evaluation in Chinese

Academy of Sciences

Tao Dai Nov. 4, 2011

Evaluation 2011: Values and Valuing in Evaluation Anaheim, California

Institute of Policy and Management, CAS

CAS center for management innovation and evaluation research

2

Evaluation 2011, Anaheim, California

Outline

• Background• Logic model for KIP evaluation

• Methods for KIP evaluation

• Main evaluation results

• Conclusion

3

Evaluation 2011, Anaheim, California

Background

• What is KIP?launched and implemented by CAS three phases: 1998~2000,2001~2005,2006~2010 aims: improve the scientific performance of CAS ;make CAS pre-eminent Science and technology (S&T) center

• Why do KIP evaluation ? Chinese government’s requirementCAS’ requirement

4

Evaluation 2011, Anaheim, California

Background • Challenges of KIP evaluation

Systematic evaluation on all the work CAS carried outCover 13 yearsCAS-centered self-evaluation

How to ensure the objectiveness and fairness ?

5

Evaluation 2011, Anaheim, California

Logic model of KIP evaluation

6

Evaluation 2011, Anaheim, California

Objectives of KIP

“By 2010, CAS will have built itself into a knowledge creation center with powerful and continuous innovation capability, aiming at national strategic objectives and international scientific frontier. It will have become a scientific research base with its overall level reaching advanced international standards, and an important base for bringing up high-caliber talents, and a base for promoting the development of high-tech industries in China. It will have developed into a knowledge base, and become the state’s think tank and talent bank with international influences.”

7

Evaluation 2011, Anaheim, California

Evaluation methods

8

Evaluation 2011, Anaheim, California

• progress report to the central government during each

stage

• During the implementation of KIP, hundreds of experts,

managerial professionals and users participated in the

evaluation and presented precious suggestions and

comments

• monitoring mechanism

The bottom layer: basic materials and data collected

9

Evaluation 2011, Anaheim, California

1) Self-evaluation and communicating review of the research

institutes

• 100 institutes conducted self-evaluation firstly

• related materials of self-evaluation were circulated on

internet

• reviewed by 201 experts from CAS headquarters,

bureaus, and institutes

The second layer: thematic evaluation

10

Evaluation 2011, Anaheim, California

2) Consultant review on S&T innovation base

• each innovation base conducted self-evaluation

• 132 experts were invited to discuss and review based on

the self-evaluation, through meeting review,

corresponding review

• consultant review committee made the comprehensive

review

The second layer: thematic evaluation

11

Evaluation 2011, Anaheim, California

3) Self-evaluation and summary of key aspects of management

work

• strategic planning and disciplinary deployment

• construction of academic divisions

• contingent construction

• system reform and construction

• resource allocation and asset management

• Local and international cooperation

• infrastructure construction

• …

The second layer: thematic evaluation

12

Evaluation 2011, Anaheim, California

4) Seminar on deepening the reform of CAS

•3-day seminar

•150 directors from both academy and institute level

•focused on the key issues in CAS developments

The second layer: thematic evaluation

13

Evaluation 2011, Anaheim, California

•carried out simultaneously and interactively with

thematic evaluation (the second layer)

•made full use of the existing evaluating results and

data (the bottom layer)

•the report titled ‘The Evaluation Report on KIP (1998-

2010)’ was formed and submitted to the state council of

China

The top layer: Overall evaluation of KIP

14

Evaluation 2011, Anaheim, California

Evaluation Results

Innovation capability improved significantly

• Number of high quality papers（ Top 1% citation） in

2009 is 11.5-fold increase from 1998

• Undertaking 33%“973 program” and 34% “NSFC Key

Project” in 1998~2009

• Operating 80% large-scale scientific facilities of China

• …

15

Evaluation 2011, Anaheim, California

Evaluation Results

Cultivating and Fostering high-level S&T talents

• Bring in 1292 outstanding talents from overseas

• About 900 scientists held important positions in

international organization or international academic

journals

• Number of Ph.D. graduate from CAS is about 8% of the

total in China, contributing to 18.5% of the “outstanding

doctoral thesis”

• …

16

Evaluation 2011, Anaheim, California

Evaluation Results Effective in S&T achievements transfer

• Number of transfer programs of S&T achievements in

2009 is 5-fold increase from 1998

• The total Business income (1998~2009) of CAS invested

enterprises is about 1,086 billion yuan

• The total profit and tax(1998~2009) of CAS invested

enterprises is about 52 billion yuan

• …

17

Evaluation 2011, Anaheim, California

Evaluation Results

Very active in domestic and international cooperation

• Built 335 technology centers with companies

• Built 10 research institutes with local government

• Built 2 national labs and 6 national key labs with university

• Number of international academic conferences organized

by CAS increased from 57 in 1998 to 354 in 2008

• Number of International cooperation and exchange

visitors increased from 10 000 in 1998 to 2.8 million in

2009

• …

18

Evaluation 2011, Anaheim, California

Conclusions

• feasible to evaluate the performance of CAS in KIP according to the evaluation practice

• based on objective data, facts, and experts judgment

• qualitative evaluation was combined with quantitative evaluation; self-evaluation was combined with experts review

19

Evaluation 2011, Anaheim, California

Thanks for your attention!