Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency

PolicyPolicy REA - Research Executive Agency

REA - Research Executive Agency

Dr Sebastiano FUMERO Head of Unit “FP7 support”

European Commission Research Executive Agency

The evaluation process in the 7th Framework programme

for Research and Technological Development

Chisinau, November 6th, 2012

OverviewOverview

How to apply

The evaluation process: basic facts and figures

Role of Commission/REA staff

Key issues:

• Eligibility check

• Expert selection

• Conflicts of interest

• The criteria

• The observer

• Redress

SEP

Electronic proposal

submission system

Annual

Work

Programme

Year N

Guide for Applicants

Call X

Funding Scheme Y

FP7: how to apply

Participant

portal

Calls for proposals

Budget, deadline, OJ ref., legal documents

Links to

SEP

NCP

The Participant Portal (PP)



7

Rules on submission and evaluation

• The common reference for FP7

• Consistency vs flexibility! Guide for applicants (annexes 1 and 2)


The work programme

• The topics and criteria against which the proposals will be judged (all criteria are important – consider sub-criteria - think as an evaluator)

To consider prior to submissionTo consider prior to submission

Submission

8

Drafting the ProposalDrafting the Proposal

•Respect page limits specified in guide for applicants

•Ensure you meet the minimum eligibility requirements

•Excellent science is a condition but not enough. Consider also:- impact, dissemination and IPS- consider project implementation and management (role of coordinator is essential)

•Be precise, less is sometime more…

•Impartial view…ask your colleagues, friend to read it before…

•Start with SEP asap – a missed deadline implies proposal is not admissible

•administrative data (part A forms) should be consistent with info in part B

Submission

Full Proposal

Proposalforms

Evaluators

Eligibility

Evaluators Evaluators Final rankinglist

PanelSubmission ConsensusIndividual evaluation

Proposals insuggested

priority order

Rejection list

Finalisation

CriteriaCriteria Criteria

COMMISSION COMMISSION

May be “remote”

Experts' role

Evaluation process

May be “remote”

Applicants are informed of the Commission

decisionCommission funding

decision

Quick Information Letter

Proposal

Individual Evaluation

Consensus

Panel reviewHearings

For large projects(optional)

Thresholds

Eligibility

Negotiation

Commission ranking

Commission rejection decision

Remote or in Bruxelles

Evaluation process

Basic facts and figuresBasic facts and figures Funding decisions are based on peer review of

research proposals

• Peer review can also “add value” to projects

High quality evaluators are at the core of the system

Over 118.000 experts registered for FP7 in the old database

Over 15.000 experts registered in the new database (Expert Area in the Participant Portal)

Over 8 300 independent experts engaged in 2011

Approx. 27 000 proposals evaluated in 2011

Evaluation of proposals

Evaluation Process: basic principlesEvaluation Process: basic principles


Three ReferencesThree References

Rules on submission and evaluation


• Consistency vs flexibility!

Guide for applicants (annexes 1 and 2)


The work programme

• The topics and criteria against which the proposals will be judged


Annual

Work

Programme

Guide for Applicants

Call X

Funding

Scheme Y

RULES

Submission & Evaluation

RULES

Submission & Evaluation

Some basic Some basic misconceptions misconceptions

clarifiedclarified

The EU’s peer review system is not a political process

• Lobbying has no influence

Quality of the proposal is the sole criterion for success

• However, “quality” involves a number of factors


Role of Commission/REA staffRole of Commission/REA staff

Check the eligibility of proposals

Oversee work of experts

Conduct briefings

Moderate discussions

Organise the panel and its work

Ensure coherence and consistency

May advise on:

• Background on previously supported or on-going projects

• Relevant supplementary information (directives, regulations, policies, etc.)

(Can even act as experts!)


Eligibility checksEligibility checks

Receipt of proposal before deadline• Firm deadlines (SEP)

Minimum number of eligible, independent partners

• As set out in work programme and the call

Completeness of proposal• Presence of all requested forms

• and readable, accessible and printable

"In scope" vs "Out of scope" Others


Expert selectionExpert selection Based on:

• A high level of expertise• An appropriate range of competences

If the above conditions can be satisfied, then also:• Balance academic/industrial• Gender• Geography• Rotation

But also, of course constrained by:• Availability• Avoidance of conflicts of interest• Uncertainty over number and exact coverage of proposals

Not an easy process…!!!


Number of experts having supported the EC in FP7 by country

Number of experts registered in the EMPP by country

FP7 Expert from this area

Country Number of Experts (participation)

Greece EL 646Romania RO 365Hungary HU 311Bulgaria BG 165Slovenia SI 163Slovakia SK 95Croatia HR 76Serbia RS 62Macedonia MK 8Bosnia and Herzegovina BA 5Moldova MD 3

Country

Migrated Valid

Bosnia and Herzegovina BA 8 8

Bulgaria BG 204 162

Croatia HU 115 89

Greece HR 914 687

Hungary HU 297 204

Macedonia MK 25 21

Moldova MD 16 12

Romania RO 464 361

Serbia RS 101 78

Slovakia SK 121 94

Slovenia SI 166 123

Conflicts of interest (1)Conflicts of interest (1) More clarity in FP7

Types of COI set out in appointment letter Disqualifying COI

• Involved in preparation of proposal

• Stands to benefit directly

• Close family relationship

• Director/trustee/partner

• Employee (but, see exception…)

• Member of advisory group• Any other situation that compromises impartiality

Potential COI• Employed in last 3 years

• Involved in research collaboration in previous 3 years• Any other situation that casts doubt… or that could reasonably appear to do so…

Expert selection

Conflicts of interest (2)Conflicts of interest (2) Experts with a “disqualifying” COI cannot evaluate

• Neither in consensus group considering “problematic” proposal• Nor in final panel• One exception… if:

The expert is employed in same organisation, but different department/lab/institute (e.g. CNRS)

The constituent bodies operate with a high degree of autonomy Justified by the limited pool of qualified experts

• … then the Commission/REA might allow expert to participate in a panel review Should abstain if the specific proposal is discussed

• Exceptionally (very rare!!!), might participate in consensus group Experts with a “potential” COI

• Need to consider circumstances of case

Expert selection

1. S/T quality(in relation to the topics addressed

by the call)

2. Implementation 3. Impact

Sound concept, and quality of objectives

• Appropriateness of the management structure and procedures

• Quality and relevant experience of the individual participants

Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

The evaluation criteria

Some exceptionsSome exceptions

Marie-Curie schemes for training and mobility of researchers• Include, e.g. quality of training

programme, suitability of host institution, etc.

European Research Council (ERC) grants• Scientific quality only criterion

Excellence!!!

The evaluation criteria

Proposal scoring

Each criterion is scored 0-5• Half-scores allowed• Whole range should be considered• Scores must pass thresholds if a proposal is to be

considered for funding Thresholds apply to individual criteria:

• Default threshold is 3 … and to the total score

• Higher than the sum of the individual thresholds• Default threshold is 10

Can vary from call to call!

Interpretation of the scoresInterpretation of the scores0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses.2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses.3 - Good. The proposal addresses the criterion well, although improvements would be necessary.4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible.5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

Proposal scoring

Individual readingIndividual reading

The experts evaluators first carry out individual readings (often done remotely)The experts:Evaluate the proposal individually (without discussing with the other evaluators)Check whether the proposal is “in scope”

second check after the one done by the EC

Complete an Individual Evaluation Report (IER) giving scores and comments on all criteria

• Scores should be in line with comments

Evaluation process

Proposal 1

Proposal 1

Proposal 1

IERIndividual Evaluation

ReportExpert A

IER Expert B

IER Expert C

Consensus meeting

CRConsensus

Report

One proposal can be evaluated by more than 3 experts

May be remote

Consensus: Scores & comments

HEARING(optional)

Evaluation process

ConsensusConsensus

Build on the basis of the individual assessments of all the evaluators

Usually involves a discussion

Moderated by a Commission/ REA representative

Agreement on consensus scores and comments for each of the criteria

One expert acts as rapporteur

Evaluation process

Consensus reports – key pointsConsensus reports – key points

The rapporteur is responsible for drafting the consensus report (CR)• Includes consensus marks and comments

The quality of the CR is paramount The aim is

• A clear assessment of the proposal, with justification• Clear feedback on weaknesses and strengths

To be avoided• Comments that do not correspond with the scores• Recommendations in view of resubmission

A proof reader might be appointed for quality control

Evaluation process

HearingsHearings

Co-ordinators whose proposals have passed the thresholds are invited to Brussels

Intended to clarify any points raised by the experts in advance

Not an occasion to “improve” the proposal

Not an occasion for a multi-media show!

Evaluation process

The final Panel ReviewThe final Panel Review

Key function is to ensure consistency Final marks and comments for each proposal

• Evaluation Summary reports (ESR)• New scores (if necessary)… carefully justified• Clear guidance for contract negotiation

Split proposals with identical consensus scores• Approach is spelled out in WP and GFA

Resolve cases where a minority view was recorded in CR

[Exceptionally] recommendations for combining List of proposals for priority order

Evaluation process

““Initial information” to applicantsInitial information” to applicantsSending of ESR Sending of ESR

The Commission/REA does not change the ESR, except if necessary to:• Improve readability• [Exceptionally] To remove factual errors or inappropriate

comments that may have escaped earlier proof-reading The scores are never changed The ESR is sent to the proposal co-ordinator – no

commitments at this stage regarding funding This is the public face of the evaluation!

Information to proposers

Redress?Redress?

In the past, complaints arrived haphazardly • Handled at different levels• No systematic treatment• No common record

The redress procedure introduced for FP7 does not give a new right of appeal…… but it ensures a consistent and coherent approach to complaints• Establishes “due process”• Uphold principles of transparency and equal treatment

Appeal

Redress: Principles and guidelines Redress: Principles and guidelines

Redress will not “stop the train” • Non-contentious proposals negotiated and selected as

normal

Complaints must relate to shortcomings in the handling of proposal evaluation• Before a Commission decision has been made

The procedure will not call into question the judgement of appropriately qualified experts

Appeal

Independent observers

Provide assurance that the process is fair • And can provide constructive

advice• Not experts in the scientific

area concerned Their reports are made

available to the Programme Committee

IOs are convened annually to a Round Table• What are the common issues?

Monitoring

Commission/REA follow-upCommission/REA follow-up

Evaluation summary reports sent to applicants

Draw up final ranking lists

Information to the Programme Committee

Contract negotiation

Formal consultation of Programme Committee (when required)

Commission decisions

Survey of evaluators

Independent Observers’ reports

Evaluation process

Expert questionnaire

For every call, experts receive a message on returning home

Invited to complete an on-line survey • Personal profile• Evaluation process• Evaluation criteria• Opinion on the task and the other evaluators• Logistics• Comments and recommendations

Early results sent to call co-ordinator after one month Full analysis at end of the year

Survey

96% of the respondents found the quality of the evaluation overall 'satisfactory' to

'excellent'

Survey

91% found the EU evaluation process similar or 91% found the EU evaluation process similar or better than national or international schemesbetter than national or international schemes

Survey



EU research: http://ec.europa.eu/research/

7th Framework Programme: http://ec.europa.eu/research/fp7/

Information on research activity and projects:http://cordis.europa.eu/

Questions? Contact the Research Enquiry Servicehttp://ec.europa.eu/research/index.cfm?pg=enquiries

Links



Thank you for your attention!Thank you for your attention!Thank you for your attention!Thank you for your attention!

Dr. Sebastiano FUMERO

Head of Unit “FP7 Support”European CommissionResearch Executive Agency

Tel: +32-2-296 96 88

[email protected]

Documents

Dr Sebastiano FUMERO Head of Unit “FP7 support” European Commission Research Executive Agency