ERA Roadmap – Monitoring and Indicators

Luxembourg 6-8 July

Roadmap Monitoring - mandate

• “INVITES ERAC to propose by the end of 2015 a set of core indicators and, where appropriate, qualitative methods allowing to monitor the implementation of the ERA Roadmap. STRESSES that the monitoring of the ERA Roadmap should be put in the context of the monitoring of the ERA progress and CALLS ON the Commission to consider possible integration of the monitoring of the ERA Roadmap into the ERA Progress Report 2016 and the following ERA Progress Report exercises, in close cooperation with the Member States, while avoiding creation of unnecessary administrative burden.”

2

Roadmap Monitoring- approach

• Work closely with Commission, national experts, representatives ERA Groups and Stakeholder Groups

• One short high level indicator per implementation priority ( so eight in all) to provide political focus. This should be linked to the priority area, not necessarily to the individual actions.

• Should draw on information which is already available for all MS/Acs (Eurostat, databases, …)

• In developing the list we need to be pragmatic and realistic; the ideal indicator may well not exist at the moment.

3

Part of a wider monitoring process

• Recognize that a single indicator may not fully capture complex developments

• Therefore ERA Groups are also encouraged to undertake further work to develop additional/more accurate indicators.

• Reminder that this is just part of a wider ERA monitoring process linked to Progress Reports (debate with Com necessary) with much more qualitative and quantitative information

4

Outcomes so far

• Fair degree of consensus on most indicators; in other cases still working on alternatives.

• Some wider political messages as well (e.g. on data availability/collection cycles for Priorities 4, continued data collection in 5b)

• Building on the outcomes there, we do further work with relevant ERA Groups and other experts to resolve the outstanding issues : nominator/denominator, data quality issues, plausibility of results,…

5

Summary

Very advanced• P1 (effect. Res. Syst)• P2a (jointly…)• P3 (open lab market)• P4 (gender)• P5a (kt)• P5b (OA)• P6 (int coop)

More work needed

P2b

Next steps

• Stakeholder meeting : 14 september• The Working Group will meet again on 28

September to agree a final set of indicators and wider political messages

• This will go to the 13 November ERAC Plenary for endorsement.

• Afterwards : further refinement work by JRC/Eurostat needed

Messages and debate

• Are you satisfied so far ? • How does this relate to wider ERA

monitoring process : comments by delegates and COM requested

• How can ERA related groups feed in this wider process ?

• To ERAC : can we debate this in November ?

PRIORITY 1 (Effective national research systems)

Research Excellence Indicator, a composite indicator prepared annually by the Joint Research Centre (JRC) of the European Commission• 4 components: Highly cited publications; PCT patents; ERC

grants; Quality of universities & research organizations

• Pro’s : Captures key aspects of research excellence, no country size bias, data available and tested by JRC

• Con’s : composite indicator (4 numerators + 4 denominators), time lags for some components, number of top institutions often 0 in small countries

Recommendation on possible improvement to be tested by JRC/Eurostat/COM

PRIORITY 2a (Jointly Addressing Grand Challenges)

Share of national GBARD allocated to Europe-wide, bilateral or multilateral transnational public R&D programmes (other denominator ?)• GBARD directed towards : Europe-wide transnational

public R & D programmes and bilateral or multilateral public R & D programmes established between MS or with EFTA and candidate countries

• Pro’s : Reflects cooperation and budgetary efforts; input indicator showing the results of policy decisions

• Con : May understate the "true" figure; no information on alignment to EU grand challenges; potential country size bias (countries with low GBARD will come out high)

PRIORITY 2b - Make optimal use of public investments in Research Infrastructures

Availability of national roadmaps detailing investments in research infrastructure• Pro’s : gives information on the investments at national

level into research infrastructure• Con’s : we need more to measure yearly progress

(ideal indicator would be Share of GBARD allocated to RI investment)

Help needed : can Commission (with the help of ESFRI) provide the data ? or can ESFRI suggest a better alternative ?

PRIORITY 3 (Open Labour Market for Researchers)

Open recruitment: Researcher posts advertised through the EURAXESS Jobs portal per thousand researchers in the public sector per year • Pro’s : directly related to priority action; will encourage

institutions to make use of Euraxess; directly measures a country's (or at least its institutions) commitment;

• Con’s : no absolute reference point (reflects changes in relation to all R&I posts available in country); some MS have national job portals preferred by their institutions;

Question for MS/AC : what portals do national systems use if they do not use Euraxess

ERA PRIORITY 4 (Gender Equality and Gender Mainstreaming in Research)

Proportion of women A grade in Higher Education Sector (HES)• Pro’s : corresponds well with the roadmap top priority;

data are based on a known methodology;• Con’s : covers HES-based researchers only; data are

provided only every three years; what is the optimum here? 50%? How to interpret results ?

Help from Commission needed : explore the possibility of producing the data every two years and the extension to other public sector research organisations

PRIORITY 5 a (Scientific knowledge transfer)

Innovative firms cooperating with (1) universities and HEI’s and/or (2) public research institutes• Pro’s : addresses knowledge transfer; readily available

indicator; it is being part of the European legislation (innovation survey)

• Con’s : some data issues to be resolved

PRIORITY 5b (Promoting Open Access to scientific publications)

Proportion of Open Access papers per country, 2008-2013 The indicator shows the proportion of Open Access papers published in peer reviewed Journals at the European and World levels over the period 2008 -2013 based on data from the Scopus database. • Pro : A relative simple indicator to measure the

proportion of Open Access papers per Country• Con : statistical problems and shortcomings have to be

investigated as well as alternative data sources

Help from Commission needed : a systematic update depends on a contract by the Commission

PRIORITY 6 (International cooperation)

International scientific co-publications with non-EU countries per ‘000 researchers (of public sector, FTE)• Pro’s : proxy for assessing international activity and

cooperation between countries; • Con’s : does not show the positive effects of national

cooperation strategies per se, it just shows how researchers collaborate (which can be the result of many factors); country size bias and linguistic/location bias possible.

Help from Commission needed : depends on a contract with institution that can produce bibliometric indicators (cfr 5b)