The Making of the FP6 Ex-post Evaluation

Erik ArnoldAmerican Evaluation Association, Orlando12 November 2009

2

The expert group … don’t blame them for what I say!• Ernst Th. Rietschel (Germany) - Chairman• Erik Arnold (United Kingdom) - Rapporteur• Antanas Čenys (Lithuania)• Andrew Dearing (United Kingdom)• Irwin Feller (United States of America)• Sylvie Joussaume (France)• Aris Kaloudis (Greece/Norway)• Lene Lange (Denmark)• Jerzy Langer (Poland)• Victoria Ley (Spain)• Riitta Mustonen (Finland)• Derek Pooley (United Kingdom)• Nicoletta Stame (Italy)

3

What’s changed in Framework Programme goals?• From FP5 “A successful and scientifically strong European industry and

high quality of life for citizens”• To FP6 as an instrument for reaching Lisbon, Barcelona, ERA -

introduction of (shifting) policy goals

4

Structure of the Framework Programme (€20bn 2002-6)• 6FP for RTD and Demonstration (93%)1. Focusing and integrating Community research (76%)

1. Thematic priorities (65%)2. Specific activities covering a wider field of research

(7%)3. Non-nuclear activities of the JRC (4%)

2. Structuring the ERA (15%)3. Strengthening the foundations of the ERA (2%)

• EURATOM (7%) 1. Priority thematic areas (5%)2. Other activities in the field of nuclear

technologies & safety (0.3%)3. Nuclear activities of the JRC (2%)

5

The evaluation

• The first evaluation of a single FP (after 3 ‘Five-Year Assessments’: Davignon; Majo; Ormala)

• The most solidly evidence-based evaluation of the FP yet undertaken

• Hence, broke the mould in moving from high-level discussion to an analysis with empirical ‘bottom’

• Evaluation issues• Rationale• Implementation• Achievements• Recommendations• + Vision

6

The Commission’s FP6 evaluation has focused on the new• New member states • New instruments (In toto, NoEs, ERANETs x2, ETPs, OMC

…)• Activities of DG-ENTR• Behavioural additionality• International standing of FP6 (+China + Bilaterals)• Bibliometric profiling• Network formation in FP6• Gender equality• Ethical review

• Plus the traditional ‘participation survey’ across the board

7

The Member States and others’ evaluations

• Continuation of traditional impact studies in some countries

• Those who need evaluation the most do the least …

• From snapshot to video: Sweden and the ‘Gang of 4’ (Sweden, France, Spain, Czech Republic)

• Associated States: Norway, (Switzerland)• ‘Third Countries’: China

8

Achievements of FP6• High volume of R&D at high quality

• Quality-assured assessment processes• Participation by excellent researchers

• Thematic priorities – two thirds of the FP; appear productive but evidence is general, unsegmented – we’d like to know more

• EURATOM – captured the ITER global fusion facility. Fission?

• Indirect but positive evidence on industrial competitiveness

• Lack of institutional strategies has limited the FP’s strategic influence over the Knowledge Infrastructure

• No ‘great leap forward’ in ‘structuring the ERA’– probably because we haven’t fully agreed what the ERA actually is

• Integrated the New Member States• Gender, Joint Research Centre – unfinished business

9

Expert group recommendations

1. Better, more transparent design2. Bigger role for the MS; avoid the tendency towards a

Commission monopoly of initiative (eg ERC)3. Develop focused strategies for ‘Third’ countries: OECD;

BRICS, poor countries4. Add bottom-up experimentation (compare NEST)5. SMEs matter but stop the silly targeting of lower-

capability firms6. More research infrastructure; more ESFRI7. More women8. More young people and mobility9. Radical overhaul of administration - no more tinkering10.Broader evaluation approach - we know more but far

from enough

10

Vision

• A new, confident and outgoing Europe • A proactive partner in the global knowledge

society• Much stronger global links and collaboration• Top down and bottom up combined

• Grand Challenges• Great Ideas

11

Self-criticism: not enough policy or process

• No discussion of the instrument mix at EU level• Research institutes omitted from the picture• No discussion of the division of labour between the

Member States and the Framework• Did not take on the changed character of the ‘self-

organised’ instruments or their implications for future R&D policy

• In many places, failed adequately to differentiate among the instruments – too much momentum from the old Framework evaluation tradition

• Inadequate treatment of the FP design process and the influence of internal incentives in the Commission (not entirely our fault … )

12

Evaluation, the Commission and the FP

• Massive changes in extent and visibility of evaluation since the 1999/2000 reforms

• Early signs of an ‘evaluation culture’ in places• Use of programme theory an improvement over the

1990s• Input - Output - Results - Impacts a useful

corrective to the old “we can’t measure the impacts so let’s talk about the programme management” approach

• Is there a risk of losing sight of the processes and the political science?

• Still opportunities to improve the link between evaluation and the evolution of the FP

13

Thanks for listening

erik.arnold@technopolis-group.com