Dr. Jim Ryan

Evaluation of Science Foundation Ireland: A Case Evaluation of Science Foundation Ireland: A Case Study of a Programme Designed to Enhance Study of a Programme Designed to Enhance

National Research QualityNational Research Quality

Dr. Jim Ryan Michael Fitzgibbon

www.circa.iewww.circa.ie 2

CIRCA Group EuropeCIRCA Group Europe

CIRCA is an international consulting company, established in 1991 in Dublin, Ireland.

Member of ETAN (European Technology Assessment Network) and the European S&T Observatory (ESTO).

12 Consultants in different areas of S&T, all with significant experience of European S&T issues and systems.

The major Irish consultancy company specializing in S&T policy and management issues.


About Forfás

• Forfás is the national board responsible for providing policy advice to Government on enterprise, trade, science, technology and innovation in Ireland.

• Legal responsibility for the promotion and development of these sectors is vested in Forfás by the State, through the Department of Trade, Enterprise and Employment.

3

[email protected]


Presentation OutlinePresentation Outline

Irish infrastructure – the context

Background to Science Foundation

Ireland

Evaluation Process


Ireland in Summary Population 3.98 million Total at work 1.81 million (was 1.1 million in

1990) GDP per capita €33,919

A GROWING ECONOMY - 2003

GDP €135 bn Exports €113 bn Imports €92 bn Trade Surplus €21 bn - 15.5% of GDP

The average annual rate of increase in Irish GDP has been 9% over the 7 year period 1994 - 2000, the highest in the EU and the OECD


1. Ireland has historically had very low S&T investment 2. Universities & Institutes of Technology are almost the sole public source

of technology & RTD services: Only Public RTD institute is in Food/Agric.3. S&T was a low priority, and the agency structure was under-developed

and staffed, until very recent times

S&T Context: Some notable features


Profile of Enterprise in Ireland

Bulk of our exports aremarketed /sold by organisationsoutside Ireland

Most of the products we manufacture are designed elsewhere


Changing the Capability Profile of Enterprise in Ireland

Need to Build World Class Expertise

International Sales & Marketing

Operations R&D/Technology

Focus on Sectors & Niches


Technology Foresight Ireland: Conducted 1998; published by Forfás 1999

“A world class research capability in selected niches of … enabling technologies is an essential

foundation for future growth. These key technologies require new skills, and in particular the development of a cadre of world class S&T

personnel.”


Ireland –S&T & the Economy 1. Foreign Direct Investment is important to economy

(~130,000 jobs; 14% of GDP)2. Innovative capacity seen as important to the attraction

and retention of FDI firms3. Also vital to the growth of indigenous companies

Investment in RTD was seen as important:

o To provide an incentive for Foreign firms o To develop indigenous high-technology industries

ENTERPRISEIRELAND


“.... There will be a major accelerated increase in Research, Technological Development and Innovation (RTDI) investment with the objectives: to strengthen the capacity of Irish third level institutions and other research establishments to conduct research relevant to the needs of the Irish economy; to strengthen the capacity of Irish firms to assimilate the results of R&D into their products and processes, and; to provide support for sectoral research in agriculture, food, marine and the environment....”


RTDRTD and Innovation Funding in and Innovation Funding in National Development Plan 2000 – 2006National Development Plan 2000 – 2006

Total : €2.5 billionTotal : €2.5 billion

PRTLICapital Facilities &

Research Infrastructure in

Universities

€690m

Science Foundation Ireland.

Quality and Scale of Basic R&D

€650m

Other Sectoral & Industry Supports

€1,160m


Science Foundation Ireland provide

researchers

PRTLI provides infrastructure for

research performance

Research Community view: SFI v PRTLI

Bricks & Mortar Grey Matter


Forced Universities to develop Strategic Plan for RTDBuilt 33 new Research Centres Provided an additional 90,000 m2 research spaceFunded 62 new or expanded research programmes Increased undergraduate Post-grad places by approx 1,500Encouraged inter-disciplinary research and inter-institutional collaborationEncouraged greater interaction between research and teaching

Programme for Research in Third Level InstitutionsProgramme for Research in Third Level Institutions(PRTLI)(PRTLI)

Conway Institute – Dublin Materials & Surface

Science Institute – LimerickNanofabrication

facility – Cork


SFI was established in 2000 and became operational in 2001

‘SFI is investing €646 million in academic researchers who are likely to generate new knowledge, leading edge technologies, and competitive enterprises in the fields underpinning two broad areas:

Biotechnology Information & Communications Technology’

‘Helping Ireland Recruit and Retain Research Groups’

Vision Statement

Through strategic investments in the people, ideas and partnerships essential to outstanding research in strategic areas, SFI will help build research of globally recognised excellence and nationally significant economic importance.


Grants are allocated solely on the criterion of excellence as determined by external, international peer review: relatively unusual in EU

Have established 163 research groups led by principal investigators of whom 34 have come to Ireland from laboratories abroad

SFI groups employ over 1150 research staff and 450 PhD students

Established 6 major R&D Centres in collaboration with industry

Schemes Committed €m2001-2005 (April)

No. of Awards2001-2005

Investigators and Fellows 254 128CSETs 108 6Research Professorships 30 10Research Frontiers 12 107Young Researcher 5 4Other 46.5 182Total 462.5 437


The CSETs are in the following areas :

Adaptive Nanostructures and Nanodevices (CRANN in Trinity College Dublin)

Regenerative Medicine (REMEDI in NUI Galway) Digital Enterprise Research (DERI in the NUI Galway) Alimentary Pharmabiotics (APC in University College

Cork) Telecommunications Value Chain (Trinity College Dublin) Human Proteomics (College of Surgeons, Dublin)

Six Centres for Science, Engineering and Technology (CSETs) have been established in core thematic fields and with considerable industry involvement.


Requested by Department of Enterprise, Trade & Employment at end ‘04

Managed by Forfás.

Department ‘Accept that SFI is a long-term investment ...

…. but also know that the early stages are critical’

it is therefore ‘not too early to form an initial impression about the operational performance of SFI to date’

Review of the performance and impact of SFI to date.


Review of the performance and impact of SFI to date.

Evaluation Methodology:

1. International review panel led the evaluation. Small size of Ireland makes it difficult to find local independent panel-members for such evaluations. External panels are therefore the norm.

2. Consulting groups were commissioned (by public tender) to provide support and information inputs to the panel.

CIRCA Group conducted the Peer Review Assessment and the Bibliometric analysis ( with CWTS, Leiden, NL)

Technopolis (UK) conducted a survey of industry views


Members of International Panel Professor Sir Richard Brook (Chairman)

Director of the Leverhulme Trust (UK) Prof. Karin Markides

Deputy Director General of Vinnova, Swedish agency for Innovation (Sw) Dr. David Clark

Chair of Computer Science & Telecomm Board, US National Academies (US)

Dr. Wilhelm Krull Secretary General of the Volkswagen Foundation (D)

Prof. David Finnegan Institute of Cell & Molecular Biology, Univ. of Edinburgh (UK)

Mr. Pat Toole Retired Senior VP of Corporate Manufacturing & Technology of IBM (US)


Terms of ReferenceAppropriateness or Efficacy:Are the objectives still consistent with the Irish research system and with national

research and innovation policies?Is the investment in SFI appropriate to meet its objectives?Are there sufficient and appropriate complementary measures to SFI which would

allow the desired economic effects to materialise?

Effectiveness:Is SFI effective in meeting its objectives?Are its programmes and activities likely to lead to the desired outcomes?What are the outputs and impacts of its activities?What impact is SFI having on the research system as a whole?

Efficiency:Examine the operational efficiency of SFI in relation to a range of relevant issues (e.g.

application and review procedures, monitoring).


Panel Methodology

Conducted discussions with a representative cross-section of relevant groups:

SFI-Funded Researchers (44) , post-doctoral staff (33) and students (34) funded under SFI awards;

SFI management and administration (5)

Management staff of institutions housing SFI researchers (13)

Other key stakeholders: Public agencies, National Research Funding bodies and private sector (6)

Discussions were conducted in 5 venues over 1 week


Evaluation of the Peer Review systemCIRCA Group

Process Design Peer selection: qualifications, nationality*, experience, gender,

etc Peer process: criteria used, information provided to peers;

ranking and rating processes; Internal SFI Process: mechanisms and speed of decision-making;

rejection rate etc Supports provided to successful and unsuccessful applicants Oversight and monitoring: conflict-of-interest rules; appeals

process; internal audit/evaluations

Process Operation Random Case files were examined to assess the practical

operation of the review process. These were balanced by discipline, university, successful/unsuccessful applicants, and by type of award.

* No Irish reviewers are used


Two studies were commissioned to assess:

Quality of the SFI-funded researchers from non-Irish institutions prior to their coming to Ireland. This was to determine whether the SFI system had selected excellent researchers

The publication record of researchers who had been working in Irish institutions prior to funding. This was to assess whether the receipt of funding had affected their publication output and/or quality

Note that appointments of researchers by SFI commenced in 2001

Bibliometric AnalysisCIRCA Group and CWTS (Leiden)


Percentage of SFI-funded researchers with a higher than average number of papers in the top 10% and Top 5% of

papers in their respective fields

Biotech ICT

Top 10% Top 5% Top 10% Top 5%% %

Foreign* Researchers 82 64 73 80

Resident Researchers 68 64 75 50

* i.e. researchers who were working outside Ireland when receiving SFI award


Industry View of SFITechnopolis (UK)

In-depth face-to-face or telephone interviews with 24 companies were conducted in January/ February 2005. The interviewees were from a mix of large foreign-owned multinationals, large indigenous firms and small high technology start-ups, all operating in business sectors with an interest in biotechnology or information and communications technology research. Generally very positive about SFI approach and activities Some concern about lack of incentives in HE sector re IP

protection etc Recognise that time is required for policy to bear fruit


Overall View of Panel

SFI is a ‘bold and well-designed path to transform Irish research’ There are ‘persuasive indications of the suitability and effectiveness of its process’ The defining aspect of SFI is its ‘demand for excellence in research quality’ Continued success will need ‘ongoing commitment to sustenance of a research culture by government and universities’

Report to be formally published by Forfás in November 2005

Documents

Dr. Jim Ryan