Embed Size (px)
DESCRIPTION
Evaluating public RTD interventions: A performance audit perspective from the EU Hendrik Fehr, Gareth Roberts European Court of Auditors American Evaluation Society Portland, 3 November 2006. to discuss. Background the European Court of Auditors the European Union budget - PowerPoint PPT Presentation
Citation preview
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Evaluating public RTD interventions:A performance audit perspective from the EU
Hendrik Fehr, Gareth RobertsEuropean Court of Auditors
American Evaluation SocietyPortland, 3 November 2006
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
to discuss....
• Background
– the European Court of Auditors
– the European Union budget
• What do we mean ‘performance audit’?
• Current audit: The European Commission’s system for
evaluating the impact of RTD interventions
– why this audit, why now?
– audit methodology
– issues arising to date
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
European Court of Auditors
• Independent institution – external auditor of
European Union
• Reports sent to European Parliament and
European Council of Ministers
• 25 Members, based in Luxembourg,
organised into 5 groups, covering:
– agricultural policies
– structural and internal policies
– external actions
– own resources, banking, admin
– co-ordination, quality assurance
• See www.eca.europa.eu
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
European Union Budget (i)
Commitment Appropriations 2005118 billion euro
Source: ECA Annual Report
Agriculture
Structural operations
Internal policies
External action
Aministration
Reserves
Pre-accession strategy
Other
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
European Union Budget (ii)
Internal Policies 200510 440 million euro
Source: ECA Annual Report
Research and TechnologicalDevelopment
Consumers, internal market,industry and netw orks
Training, youth and socialoperations
Energy, EURATOM and theenvironment
Other
Research6bn euro (approx)
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Framework Programmes for RTD (i)
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
FP4
FP5
FP6(avg 5bn pa)
FP7(avg 7bn pa)
Run-in: developing new
programme
Shadow: implementation
to the end
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Framework Programmes for RTD (iii)
• directly managed by the European Commission
– mainly extra-mural
– multi-partner consortia (covering more than 1 EU Member State)
– governed by contracts / grant agreements
• several instruments / types of projects
– networks of excellence
– integrated projects
– specific targeted research projects
– specific support actions
– etc
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
What do we mean by ‘performance audit’?
• Court’s mandate derived from treaty
– provide a statement of assurance as to the reliability of the accounts and the legality and regularity of the underlying transactions,
– examine whether the financial management has been sound
– submit observations on specific questions and deliver opinions
Performance audit:• management of research under FP5• audit of effectiveness of instruments• audit of evaluation system
Financial audits:• annual report / opinion
Opinions:• Rules of Participation for FP7
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
What do we mean by ‘performance audit’?
• According to the Court’s audit manual, a performance
audit is
– an audit of sound financial management, namely of the economy,
efficiency and effectiveness.
– to put it simply, examining
• whether the right things are being done, and
• whether things are being in the right way
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Overall audit question
Does the European Commission have an adequate system for evaluating and assessing the impact of indirect RTD
actions?
Sub question 1:
Are there clear relationships between the policy objectives of the framework programmes and the criteria and indicators used to select, evaluate and assess the impact of RTD
projects?
Sub question 2:
Are evaluation activities carried out in a comprehensive, coherent and
independent manner and are relevant and useful results communicated on a
timely basis, and are these results taken into consideration for future
policy decisions?
The focus of the current audit
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Why this audit (now)?
• because evaluation is relevant in the audit world.....– for transparency and accountability
– for good governance and improving programme management
– for informing policy decisions
• assessed as an issue of particular risk because of...– the inherent difficulty in assessing the impact of public R&D interventions given
the complexity of relationships between inputs, results and impacts, and,
– the increasing levels of European Community funding
– the concerns expressed in the Court’s earlier work about the information available
about the outputs, outcomes and impacts of Community-funded research actions
• to contribute to the development of an improved evaluation system
for the future
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
the audit methodology
Work focused on the Commission:• analysis of existing evaluation system• analysis of objectives and performance criteria• analysis of evaluation reports and methodologies deployed
Work aimed at understanding good practice:• benchmarking visits to USA and Canada• literature review• stakeholder consultation within the Member States
Work to validate credibility of findings• expert / stakeholder focus groups• contradictory procedure with the Commission
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
legal and institutional framework
intervention logic etc
evaluation findings
implementing RTD evaluation
Issues arising from work to date.......
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Legal and institutional framework
• What are the main elements of evaluation governance? and what impact do these have?, eg:
• Legal framework
• Reporting results and link with budgetary processes
• Internal evaluation and other standards
• Quality assurance of the evaluation system
– Who is responsible for monitoring the functioning of the evaluation system?
– What influence do they have in practice?
– Is a distinction between monitoring and evaluation
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Intervention logic etc
• Is intervention logic clear from the outset?
• Are objectives SMART?
• How are performance indicators defined and
targets set? How is performance against
these targets monitored?
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Implementing RTD evaluation
• How far is programme evaluation taken as a important priority by top management?
• Is a robust evaluation strategy in place? If so, how well does it
• consider how to deal with inherent complexity
• explain intervention logic
• take account of the role of evaluation in the policy cycle
• strike a judicious balance between in-house capacities and external expertise
• cover the portfolio of issues to be evaluated , program level challenges
• deploy a mix of quantitative and qualitative methodologies etc
• Are resources adequate for evaluation needs?
– human and financial resources
– data collection / IT systems
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Evaluation findings
• What kind of information does the
evaluation system produce?
• To whom is it disseminated?
• Do stakeholders consider it to be useful
information?
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Comparing ‘performance audit’ and ‘evaluation’
Similarities:• Methodologies and techniques
Differences:• Institutional context• Evaluee relationships• Reports
Do you welcome such an initiative from an audit
institution ?
Evaluating public RTD interventions: A performance audit perspective from the EUEuropean Court of Auditors
American Evaluation Society, Portland, 3 November
Contact details
Gareth RobertsAuditor
Tel: +352 4398 45724
Hendrik FehrHead of Division
Tel: +352 4398 45503
Internal Policies (including Research) Division
European Court of Auditors
12 rue Alcide de Gasperi
L-1615 Luxembourg
http://www.eca.europa.eu
