Upload
rudolph-fowler
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
The Evaluation Partnership
Evaluation of the Commission’s Impact Assessment System
Presented in Brussels on June 28th 2007 by:John P. WatsonJohannes M. WolffMartin H. Kuehnemund
www.evaluationpartnership.com
2TEP-PresentationTEP-Presentation
AGENDA
1. Evaluation Overview & Methodology
2. Achievement of Objectives
3. Key Problem Areas & Options for Change
3TEP-PresentationTEP-Presentation
Evaluation Scope Assess the IA system from 2002 to 2006 with an emphasis on June 2005 onwards.
Review the set-up, its outputs (completed IAs), and the impact of IAs on the policy process.
Examine all IAs completed by Commission Services since 2003, for both legislative and non-
legislative proposals, covering all policy fields.
Evaluation Questions The Terms of Reference defined 31 evaluation questions grouped under 3 headings.
Setup / Inputs• 7 questions on General Approach (applied appropriately, timing, etc.)• 4 questions on Support Structures (SG, training, etc.)• 4 questions on Central overview of the System Implementation (co-ordination, etc.)
Outputs• 11 questions (compliance, duration, quality, etc.)
Impacts• 5 questions (IA and legislative work, quality of EC and MS IAs, etc.)
EVALUATION OVERVIEW
4TEP-PresentationTEP-Presentation
Methodology Overview
EVALUATION OVERVIEW
IA Report Screening(150+ IAs)
Quality Assessments(20 IAs)
CaseStudies(6 IAs)
DocumentReview
Interviews
On-line StakeholderConsultation
Data on individual IAsData on the EC IA system
Interviews
With competentEC, EP andCouncil Staff
Doc. Review
Policy documentsGuidelines
External reports
Evaluation of the Commission’s impact assessment systemComparison with other
IA Systems
EC Survey
Of staff in operational units
DG Info Request
Specific DG measures
5TEP-PresentationTEP-Presentation
IA Screening
Screened 158 IAs available on the europa website by September 2006.
35 criteria related to content and format.
EVALUATION OVERVIEW
6TEP-PresentationTEP-Presentation
IA Quality Assessments
20 IAs selected by type of proposal,
lead DG, and year adopted.
In depth analysis of IA reports,
roadmaps, proposals and 20
interviews with EC officials in charge.
EVALUATION OVERVIEW
DG IA Year
AGRI Review of the Sugar regime 2003
DEV Communication on interconnecting Africa 2006
EAC Decision establishing the programme "Citizens for Europe" 2005
EMPL Review of the European Employment Strategy 2003
ENTR Directive laying down rules on nominal quantities for pre-packed products
2004
Regulation on motor vehicles emissions 2005
ENV Directive establishing an infrastructure for spatial information in the Community (INSPIRE)
2004
Environment & Health Action Plan 2004
Thematic Strategy on Air Pollution 2005
FISH Council Regulation establishing financial measures for the implementation of the CFP and in the area of the Law of the Sea
2005
INFSO Communication on "i2010 - a European Information Society for growth and employment"
2005
JLS Framework Decision on procedural rights in criminal proceedings 2004
Regulation on the applicable law and jurisdiction in divorce matters
2006
General Programme Solidarity and Management of Migration Flows
2005
MARKT Directive on Postal Services 2006
Action plan for the implementation of the legal framework for electronic public procurement
2004
SANCO Regulation on plant protection products 2006
TAXUD Communication on the Community programmes Customs 2013 and Fiscalis 2013
2005
TREN Common rules and standards for ship inspection and survey organisations
2005
Electricity from renewable sources 2005
IA Case Studies
Sample of 6 Case studies selected.
Interviews undertaken to engage those
involved in all phases of the IA work
and subsequent policy process.
IA Screening
Screened 158 IAs available on the
europa website by September 2006.
7TEP-PresentationTEP-Presentation
EC Official’s Survey
Survey of officials in operational units to collect views and opinions about the IA system.
It was available via IPM for a 5 week period and received 89 responses.
PROGRESS TO DATE
Stakeholder Consultation
Stakeholder consultation surveyed via IPM to gather general information about IA system.
Available for an 8 week period and received 118 responses.
8TEP-PresentationTEP-Presentation
Information Request
Information Request was sent to 21 DGs.
Provided quantitative and qualitative information
about specific approaches, structures, provisions
and arrangements in different DGs.
Final Interviews
Held final round of interviews to validate findings
and to discuss the feasibility and usefulness of
options for change.
15 interviews with Commission senior managers,
representatives of the European Parliament and
Council.
EVALUATION OVERVIEW
- PART I -
GENERAL IA INFORMATION AND DOCUMENTATION REQUEST
A) Which unit do you work for? Which DG do you work for? (Please mark on of the following) Unit: __________ P O L I C I E S E X T E R N A L R E L A T I O N S
O Agriculture and Rural Development O Development
O Competition O Enlargement
O Economic and Financial Affairs O EuropeAid - Co-operation Office
O Education and Culture O External Relations
O Employment, Social Affairs and Equal Opportunities O Humanitarian Aid
O Enterprise and Industry O Trade
O Environment G E N E R A L S E R V I C E S
O Fisheries and Maritime Affairs O Communication
O Health and Consumer Protection O European Anti-Fraud Office
O Information Society and Media O Eurostat
O Internal Market and Services O Publications Office
O Joint Research Centre O Secretariat General
O Justice, Freedom and Security I N T E R N A L S E R V I C E S
O Regional Policy O Budget
O Research O Bureau of European Policy Advisers
O Taxation and Customs Union O Informatics
O Transport and Energy O Internal Audit Service
O Legal Service
O Personnel and Administration
O Translation
B) Impact Assessment Guidelines 1) Has your DG developed its own Impact Assessment Guidelines/Manual (such as DG SANCO’s ‘Scoping Paper’ or DG MARKT’s ‘Practical Guide for Impact Assessments’)? O Yes O No 2) Has your DG developed a definition of ‘proportionality’ (i.e. the principle of ‘proportionate analysis’) that is different from / goes beyond the one in the Commission’s Impact Assessment Guidelines? O Yes O No
9TEP-PresentationTEP-Presentation
Evaluation in Figures
Overview of the number of persons contacted during the evaluation.
EVALUATON OVERVIEW
Stakeholder Tools for Contact Number
Interviews
Commission Officials in Operational UnitsIA Quality Assessments 20
IA Case Studies 18
Commission Officials in IA Units
EU Interview Programme I 13
EU Interview Programme II 4
IA Case Studies 4
Commission Senior Managers and members of Cabinets
EU Interview Programme II 5
External experts / consultants IA Case Studies 6
European Parliament and Council Representatives
EU Interview Programme II 8
IA Case Studies 23
External Stakeholders (Civil Society & MS) IA Case Studies 18
Total Interviews 119
Surveys
Commission officials in Operational Units EC Officials’ Survey 89
Commission Officials in IA Units Information Request 21
External stakeholders Stakeholder Consultation 118
Total Survey responses 227
10TEP-PresentationTEP-Presentation
AGENDA
1. Evaluation Overview & Methodology
2. Achievement of Objectives
3. Key Problem Areas & Options for Change
11TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
1. Improve the quality of Commission proposals, in particular by Facilitating a more systematic, coherent, analytical, open, and evidence-based approach to
policy design; Providing a thorough, balanced and comprehensive analysis of likely social, economic and
environmental impacts.
2. Provide an effective aid to decision-making, in particular by Providing policy makers with relevant and comprehensive information on the rationale behind
proposed interventions, and their likely impacts; Enabling policy makers to assess trade-offs and compare different scenarios when deciding
on a specific course of action.
3. Serve as a valuable communication tool, in particular by Fostering internal communication and ensuring early and effective co-ordination within the
Commission; Enhancing external communication by making the policy development process more open
and transparent to external stakeholders.
Ultimately, the achievement of these key objectives should contribute to a better, simpler and more consistent regulatory environment that helps the EU to meet the objectives of the Lisbon and Sustainable Development Strategies.
Objectives
The IA system’s degree of success depends on how much emphasis is placed on each of
its key objectives.
12TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
Objective I: Improve the Quality of Commission Proposals
Results varied: 13 out of the 20 IAs showed some degree of influence on the respective
proposals, but the extent of this influence varied considerably.
• Half of the 20 IAs examined, identified, analysed and compared the most
relevant policy options. The remainder analysed only the preferred option.
• IAs were mostly balanced, but the analysis of economic impacts is often more
developed.
There were significant differences in quality depending on type of proposal (IAs on
legislative proposals were more thorough than non-legislative proposals).
Many IAs suffer from an inappropriate approach, i.e. are launched late in the process,
principle of proportionate analysis not clear, policy decisions already made ex-ante.
Sometimes insufficient tools, expertise, time and resources are available to conduct an IA
capable of significantly improving the quality of the proposal it accompanies.
13TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
Objective II: Provide an Effective Aid to Decision Making
A significant part of Commission staff in Services and at Cabinet level saw shortcomings in
relation to the usefulness of IAs, as they did not feel they were sufficiently detailed or were
too much oriented towards the preferred option.
For Members and staff of the European Parliament and Council the vast majority of IAs
failed to meet their expectations, or met with lack of trust in the “objectivity” of Commission
IAs]
Out of six IA case studies, three played no role in the decision-making process. The
remaining (all on legislative or ‘action plan’ type proposals) played some role, but only one
was genuinely useful as the basis for a better informed decision.
The working culture and capacity inside the institutions is not yet conducive to making full
use of IAs.
• Lack of understanding of IA purpose and role.
• MEPs and members of the Council WPs sometimes lack time and expertise.
14TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
Objective III: Serve as a valuable Communication Tool
Communication with external stakeholders is undertaken in accordance with the
Commission’s minimum standards on consultation.
Survey results showed that the majority of stakeholders agreed that the IA process gives
interested parties adequate opportunities to provide relevant opinions and information.
Nonetheless, the degree to which stakeholders are satisfied with how consultations are
managed, and how their views were taken into account, varies considerably.
ISSGs for the majority of recent IAs and the development of Roadmaps represent progress
in terms of ensuring early and effective co-ordination and communication.
Limits to the success of IAs as a communication tool are their quality and late timing.
15TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
Stakeholder Consultations
In many IA processes, consultation has been carried out to the satisfaction of stakeholders,
and provided relevant and useful input for the Commission.
Some others, however, have been organised in a manner and timeframe that left
stakeholders frustrated, and was unlikely to have had much of an effect on the IA.
Stakeholder opinions on consultations
0%10%
20%30%
40%50%
60%70%
80%90%
100%
Input w as sought atappropriate points
of time during the IAprocess.
There w as enoughtime and
opportunities toprovide input.
The input w assought through
appropriatechannels.
It w as clear w hatinput w as being
sought.
Contributions w ereacknow ledged and
a summarypresented in the IA
report.
Don't know
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
16TEP-PresentationTEP-Presentation
ACHIEVEMENT OF OBJECTIVES
Stakeholder Consultations
Most frequently expressed criticism by stakeholders:
• More time: Sufficient time should be provided for responses (eight weeks as an
absolute minimum).
• Earlier consultation: Stakeholders should be consulted as early as possible in
the IA process in order to ensure that balanced, comprehensive and realistic IAs.
• Limit the use of online consultations and closed questions: Many stakeholders
feel that internet surveys tend to be oversimplified.
• Balance and representativeness: More attention to the weighting of the
responses (e.g. from representative organisations vs. individuals).
• Transparency: The exact purpose and scope of the consultation and the
requested data should be made more explicit.
• Feedback and acknowledgment: Stakeholders often feel that they are not
adequately informed about what happens with their input.
17TEP-PresentationTEP-Presentation
AGENDA
1. Evaluation Overview & Methodology
2. Achievement of Objectives
3. Key Problem Areas & Options for Change
18TEP-PresentationTEP-Presentation
KEY PROBLEM 1
Scope of Application and Proportionate Analysis
IA currently applied to a large number of very diverse initiatives on the CLWP.
Blanket coverage of all CLWP items raises difficulties:
• Important legislative and non-legislative proposals may not be included in the
CLWP, and may thus be exempt from the requirement to undergo an IA.
IA Board con prompt IAs on non-CLWP items.
Several DGs have begun to apply IAs voluntarily to non-CLWP items, but
the current system does not guarantee consistent coverage.
• A significant number of items on the CLWP are not well suited to an IA in its
present form:
Result: some extremely short IA reports that add little value, neither
examine different policy options nor properly assess any impacts.
Seen as a bureaucratic hurdle rather than a tool for better policy-making.
May undermine trust in the value and rigour of Commission IAs overall.
19TEP-PresentationTEP-Presentation
KEY PROBLEM 1
Scope of Application and Proportionate Analysis
The principle of ‘proportionate analysis’ provides an element of flexibility and pragmatism.
This is appreciated, but the principle leaves many open questions, and does not of itself
give drafting officials a clear enough indication of what is expected of individual IAs:
• Perceived to only refer to the depth and scope of the analysis of impacts.
• Not to other aspects that require some flexibility, such as: Which steps should be
emphasised? What types of options and impacts should be examined?
Sco
pe o
f applic
atio
n
Level of differentiation
A
B
C
Wide range of initiative are covered, but no clear
guidance as to which elements are required of
different IAs!
The IA system’s scope of application needs to be
brought in line with the level of differentiation,
understood as flexibility – within well established
parameters – to focus on different elements.
20TEP-PresentationTEP-Presentation
OPTIONS FOR CHANGE
Limit scope of application to proposals with most significant impacts
Reduce the number of IAs, by limiting the scope of application to those proposals that are
likely to have the most significant impacts:
• All legislative initiatives in the CLWP.
• Non-legislative CWLP and other non-CLWP proposals (incl. comitology measures) identified by IA Board.
Differentiate between different types of IAs and approaches
Maintain broad scope of application, but differentiate more between different types of IAs,
and provide clear guidance on what is expected of each type.
Operationalise the principle of proportionate analysis by defining broad categories of IAs,
and providing guidance on areas for differentiation and customisation:
Far-reaching legislative proposals Communications on well-defined future action
Legislative proposals with a narrow scope Broad policy-defining Communications
Expenditure programmes Communications of a purely reporting nature
21TEP-PresentationTEP-Presentation
KEY PROBLEM 2
Timing and Approach
Many IAs are perceived as not fully objective, in the sense that they are focused primarily
on justifying a pre-determined policy choice.
Two main factors have limited the ability of IAs to take a disinterested approach:
• Set mandates or political pressure from the Commission hierarchy and / or from the other EU Institutions.
• Late start of IA work, after key decisions on the nature of the proposed intervention (incl. the preferred option) have already been taken.
IAs that do not start from ‘tabula rasa’ can still fulfil important functions.
However, to maximise their value, IA work should start as early as possible, and
accompany the first phases of the policy development process.
Currently, the IA process is not sufficiently built into the early stages of policy design, when
the problems and objectives are framed and options for their achievement considered.
Risk: IAs that start too late become an artificial exercise with a predetermined result.
Late start to IAs also diminishes the usefulness of (internal and external) consultation.
22TEP-PresentationTEP-Presentation
OPTIONS FOR CHANGE
Better exploit the potential of Roadmaps and early consultation
Remind DGs to launch IAs sufficiently early, especially on far-reaching proposals.
Develop stricter minimum standards and quality control mechanisms for Roadmaps .
Ensure early circulation of Roadmaps to other DGs, and explore possibilities for engaging
external stakeholders and the other EU institutions.
Formalise early IA steps by means of a ‘scoping paper’
Require parts of IAs to be produced from the very beginning of the policy process.
Introduce an obligation to produce a document (scoping paper / initial IA) as soon as the
policy idea is generated, outlining the rationale, policy alternatives and their likely impacts.
De-couple the IA process from the drafting of the proposal
Do not allow proposals to be drafted until the IA has been completed:
• Produce IA before the initiative is included in the CLWP.
• Externalise the entire IA process to an expert body?
23TEP-PresentationTEP-Presentation
KEY PROBLEM 3
Quality Control Mechanisms
IA quality control system operates on three levels:
• Lead DG’s operational units and IA support functions, with input from relevant units in other DGs via the ISSGs and the ISC.
• Secretariat-General (SG): ‘guardian’ of the IA Guidelines.
• IA Board: provides guidance and plays a final quality control role.
IAs don’t always meet the desired and required quality standards in terms of their content,
presentation and procedural issues.
While the Commission has made great strides towards developing its IA capacity, not all
DGs have established dedicated IA support functions capable of providing effective initial
as well as ongoing support and quality control.
The SG can provide assistance on procedural issues, but not on policy specific issues.
More informal mechanisms (incl. ISSGs and stakeholders input from consultations) have
only partly been able to mitigate the lack of systematic quality control.
24TEP-PresentationTEP-Presentation
OPTIONS FOR CHANGE
Encourage DGs to further develop IA capacities
All DGs should assess and, where necessary, enhance their capacity to provide operational staff with support in developing high-quality IAs.
Require and encourage top-level sign-off / buy-in of IAs by Commissioners.
Centralise quality control at SG and IA Board
SG takes on the responsibilities of DG IA support functions, becomes central service providing ongoing quality control before passing IAs on to IA Board for final quality check.
Externalise (parts of) the quality control of IAs
Expand the IA Board to include external experts with full voting rights.
Consult external stakeholders on draft final IA reports.
Involve the European Court of Auditors in ex-post IA quality control.
Create a new independent agency for quality control.
25TEP-PresentationTEP-Presentation
KEY PROBLEM 4
Support and Guidance
Provide more / more relevant / more tailored IA training.
Enhance practical usefulness of the IA Guidelines.
Continue to improve methodologies:
• Lack of appropriate methodologies to assess certain types of impacts,
particularly social impacts.
• Integrate and co-ordinate the efforts of different DGs and Member States;
develop guidance documents for assessing impacts in particular fields.
Address problems with availability of data:
• Comprehensive, reliable and comparable data is often not easily available.
Efforts to collect it are resource-intensive and do not always produce the desired
results.
• Develop a more systematic, integrated approach to data collection, involving
DGs, Member States, stakeholder and expert networks.