SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen

7/30/2019 SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen

1/35

SAI Performance Measurement

FrameworkPresented by:

Yngvild H. Arnesen, INTOSAI Development Initiative

ICGFM Winter Conference, Washington, DC, 10-12December 2012

1
http://www.wgvbs.com/Home.aspx


2/35


3/35

1. Introduction

3


4/35

What is a Performance Measurement

Framework? Tool for measuring performance

of an individual, system ororganization

against an agreed performancescale

accommodating localcircumstances

To enable Better management of

performance

Monitoring over time

Evaluation and lesson learning

Contributing to Understanding performance

Improving performance

4

Not everything

that matters can

be measured


5/35

What is the SAI Performance Measurement

Framework (PMF)?

A Performance Measurement Framework tailored to SAIs

Intended to give a high level assessment of SAI performanceagainst established INTOSAI good practice:

The International Standards of Supreme Audit Institutions (ISSAI)

framework The Framework on the Value and Benefits of SAIs

Other INTOSAI guidance material

Combination of objective measurement and qualitative

assessment

Analyzes 7 domains of SAI performance in the country context andaims at assessing the value and benefits of the SAI to society(impact)

5


6/35

SAI PMF How will it be used in Practice?

6

Who decides to initiate a SAI PMF?

Who decides when to do a SAI PMF?

Who decides how to do a SAI PMF?

Head of SAI

Head of SAI Consulting with INTOSAI regions & donors as appropriate

Every 3-5 years recommended

Head of SAIApproach selected to meet the SAIs objectives 3 broad approaches Quality assurance to promote objectivity and credibility

Who decides if, when and how to

publish a SAI PMF?

Head of SAI

1. Using the SAI PMF will be voluntary for the SAIs the tool will be a public

good which SAIs can make use of if it helps them achieve their objectives

2. The SAIs themselves will make the key decisions regarding the assessment

3. The SAI will also decide whether or not to publish results

4. Because it is voluntary, the SAI PMF is not intended to replace any internal

performance measurement procedures


7/35

7

1. SAI SelfAssessment

Trained Team

from withinthe SAI

2. INTOSAI PeerReview

Peer SAI

INTOSAIRegional Body

3. ExternalAssessment

Professional

PFM Body Donor Agency

Consultant

Externalauditor

3 Approaches to Conducting a SAI PMF Assessment

There are three main approaches to conducting an assessment using the

SAI PMF. The PMF will be equally well suited for all three:

The approach will not necessarily affect how the assessment is carried

out

but the SAI may use the results for different purposes depending on

the approach

SAI Involvement


8/35

2. Background and rationale for

developing a SAI Performance

Measurement Framework (PMF)

8


9/35

Background

9

Full set ofInternational

Standards of

Supreme Audit

Institutions (ISSAIs)

adopted by INTOSAI

in 2010

INTOSAI

Framework on the

Value and Benefits

of SAIs adopted

2010

Increased focus onuse of country

public financial

management

systems within the

international donor

community


10/35

Mandate: Johannesburg Accords

(INCOSAI XX, 2010)

10

Working Group on the Value and Benefits of SAIResponsibility

Develop further credible measurement tools to supportevaluations in terms of the framework for Communicating and

Promoting the Value and Benefits of SAIsTask

INTOSAI Capacity Building Committee

INTOSAI-Donor Steering CommitteeCollaborating with

Tools that are already available at the level of internationalorganizationsCognizance of

For consideration at XXI INCOSAI in 2013Timetable


11/35

Improve SAI approaches to capacity development through

use of performance management

Identify opportunities to strengthen SAI performance

Supreme Audit Institutions

Promote transparency and accountability through crediblepublic reporting of SAI performance (ISSAI 20)

Domestic Parliaments and Citizens

Enable better peer learning

INTOSAI Regions and INTOSAI Global Community

Demonstrate SAI performance changes to key stakeholders

International Development Community (Donorsand SAI Providers of Support)

SAI Enhanced SAI integrity andcredibility with domesticstakeholders

Stronger SAI-led capacitydevelopment programs

SAIs become modellearning organizations

Evidence-based policiesand better targeted globaland regional programs

Maintain external supportfor SAI capacitydevelopment

SAI PMF Stakeholders and Purpose Benefit to the SAI

Rationale for INTOSAI to Develop a SAIPerformance Measurement Framework

11


12/35

Why should SAIs use the PMF (1)

As a step towards ISSAI implementation

Increased understanding in the INTOSAI community of what goodpractice for SAIs entails

Individual SAIs can benchmark their performance against good

practice and get an indication of where there is room for

improvement They can use the assessment as a basis for more detailed ISSAI

compliance assessments

To demonstrate progress and value and benefits to society

The framework will be well suited for measuring progress overtime and demonstrating this to external stakeholders

Show to stakeholders how the SAI contributes to strengthening

public financial management

12


13/35

Internal performance measurement / annualreporting

Help SAIs better manage their own performance

ISSAI requirement to assess and report on own performance

lead by example and become a model organization

Inform and secure support for capacity development

initiatives

Identify strengths and weaknesses, increase understanding ofreasons for good and weak performance

Help build a culture for performance improvement

SAIs can use assessment results to get support from donors and

other external stakeholders13

Why should SAIs use the PMF (2)


14/35

Implement SAIDevelopment Action

Plan

High LevelPerformance Overview

Progress Monitoring

Identify Main SAIStrengths andWeaknesses

Investigate UnderlyingCauses

Recommend SAICapacity Building

Measures

Formulate or UpdateSAI Development

Action Plan

(Medium Term)

The Role of SAI PMF Assessments in

SAI Capacity Development

SAI PMF

Assessment

SAI Capacity

Building Needs

Assessment

Formulate orUpdate SAI

Strategic Plan

(Long Term)


15/35

SAI Performance Measurement Framework (SAI PMF)

& ISSAI Compliance Assessment Tool (ICAT)

SAI PMF

ICATs

Founding Principles LimaDeclarationLevel 1

Prerequisites for the

Functioning of SAIsLevel 2

Fundamental

Auditing PrinciplesLevel 3

AuditingGuidelines

Level 4

ISSAI Framework

Auditing

GuidelinesLevel 4

FundamentalAuditing PrinciplesLevel 3

Prerequisites for theFunctioning of SAIsLevel 2

Founding Principles LimaDeclarationLevel 1

ISSAI Framework

2. High level overview or

detailed drill downtool?

3. Assessment

against goodpractices or

against ISSAI

framework?

International

Standards of

Supreme Audit

Institutions

4. Score levels of

performance orassess compliance

with standards and

investigate causes?

ISSAIs

Human

Resources

Strategic

Planning

What needs to bein place to

implement level 4?

1. Monitor change

over time orassess capacity

building needs?

Value &

Benefits

Framework


16/35

3. Responsibilities and reporting

lines

16


17/35

17

Responsibilities and Reporting Lines

INTOSAI Working Group on

the Value and Benefits of SAIsOversight and Direction

WGVBS SAI PMF Task Team

Development of SAI PerformanceMeasurement Framework

Coordinator: INTOSAI-Donor Secretariat @ IDI

SAI PMF Reference Group

Technical Advice, Review andQuality Assurance

INTOSAI CongressApproval

INTOSAI

Capacity Building

Committee

Members

INTOSAI-Donor

Steering Committee

Members

Other Groups:

PEFA Secretariat,

International

Budget

Partnership

INTOSAI

Capacity Building

Committee

Members

INTOSAI-Donor

Steering Committee

Members

InConsultationwith:

INTOSAICapac

ityBuildingCommittee

INTOSAI-DonorSteeringComm

ittee

OtherRelevan

tINTOSAICommittees


18/35

18

Task Team

INTOSAI Regional Organizations:

1. African Organization of English

Speaking SAIs(AFROSAI-E)

2. African Organization of Francophone

SAIs (CREFIAF)

SAIs:

3. Bahrain

4. Brazil

5. Chile

6. Germany

7. India

8. Mexico

9. Russia

10. United KingdomDevelopment Agencies:

11. Inter American Development Bank

Coordinator:

12. INTOSAI-Donor Secretariat @ IDI

Reference Group

INTOSAI Regional Organizations:

1. Pacific Association of SAIs (PASAI)

SAIs:

2. Canada

3. Denmark

4. Estonia

5. European Court of Auditors

6. Japan

7. Netherlands

8. Norway

9. Peru

10. Sweden

Development Agencies:

11. African Development Bank12. OECD

13. European Commission

14. World Bank

Civil Society Organizations:

15. International Budget Partnership

16. Global Reporting Initiative (GRI)17. PEFA Secretariat


19/35

4. Indicative content and design

19


20/35


21/35

SAI PMF Task Team:

Initial Work and Decisions

21

No existing framework fully meets INTOSAI needs

Develop a new tool drawing heavily on existing ones (AFROSAI-E, PEFA, INTOSAI Peer ReviewGuide, IDI Capacity Building Needs Assessment Handbook)

Mapping of 20 Existing SAI Assessment Tools

ISSAI Framework where possiblerecognizing it will continue to evolve over time The SAIs mandate

INTOSAI guides & other generally accepted international standards or good practices

What to Measure Performance Against?

External performance delivery of core audit activities (audit coverage, submission of audit

reports, etc.) Internal enablers of performance SAI strategy, audit processes, SAI support services

SAIs operating environment (i.e. independence and legal framework) recognizing this is largelybeyond the control of the SAI

Reported in the context of the specific country governance and public financial management

environment and its impact on the SAIs performance

HOLISTIC MEASUREMENT OF THE SAI AND ITS ENVIRONMENT

What is performance? What are we going to measure?


22/35

22

Qualitativeassessment

(PerformanceReport)

Measurable

indicators

SAI PMF: Evidence Based

Assessment of Performance

Objective

measurement to

inform qualitative

assessment and track

progress over time

Holistic assessment of

the SAI and its

environment

including factors which

are not covered by the

indicators


23/35

B.

Independence

and Legal

Framework

3 indicators

A. SAIPerformance

3 indicators Contribution

to Public

Financial

Management,Governance

and Tackling

Corruption

SAI Environment SAI Capability Impact

Public Financial Management Environment

SAI

Performance

SAI Performance Measurement Framework Structure

D. Audit Standards and Methodology

3-5 indicators

SAI SUPPORT SERVICES

E. Management

& Support

Structures

2 indicators

F. Human

Resources

3 indicators

G. Communication

and Stakeholder

Management

3 indicators

C. Strategy for Organizational Development

1 indicator

SAI STRATEGY

SAI CORE BUSINESS

23

SAI P f M t F k St t


24/35

B. Independence and

Legal Framework

Independence of the

SAI

Independence of the

Head of the SAI and

its Officials

Mandate of the SAI

A. SAI Performance Financial Audit

Results

Compliance Audit

Results

Performance Audit

Results

SAI Value Added

Services

SAI Annual Report

SAI Contribution to

Aggregate Fiscal

Discipline

Efficient Resource

Allocation

Efficient ServiceDelivery

Good Governance,

Accountability and

Tackling Corruption

SAI Environment SAI Capability Impact

Quality and Timeliness of Annual Financial Statements (PEFA PI-25)

(Including completeness, timeliness and accounting standards used)

Legislative Scrutiny of External Audit Reports (PEFA PI-28)(Including timeliness & extent of hearings, recommendations of the legislative & implementation by the executive)

Public Financial Management EnvironmentInfluences the effectiveness of the SAI but not under direct control of the SAI.

Measured through other assessments but referred to explicitly in the narrative SAI performance report.

Foundational factors set by the

legislative and/or executive, but

which can be influenced by the

SAI in the long term

Under the control of the SAI

SAI Performance

Measured using other assessments.

Achievement of impacts can not be

wholly or directly attributed to SAI

results. However, the way in which

the SAI results are likely tocontribute to high level impacts

should be summarized in the

narrative SAI performance report.

Under the control of the SAI

SAI Performance Measurement Framework StructureNote: all indicators should try to capture both what is written in legislation and procedures (de jure) and what happens in practice (de facto)

D. Audit Standards and Methodology Audit Planning for the SAI

Financial Audit

Compliance Audit

Performance Audit

Quality Control & Quality Assurance of Audit Processes

SAI SUPPORT SERVICES

E. Management &Support Structures

Management

and Internal

Control

Support

Structures &

Infrastructure

F. Human Resources Management of

Personnel

Professional

development

Leadership

G. Communication and

Stakeholder Management

Communications

Strategy, and Internal

Communication

Communication with the

Government Branches

Communication with the

Media, the Citizens and

the Civil Society

C. Strategy for Organizational Development

Strategy for organizational development

SAI STRATEGY

SAI CORE BUSINESS


25/35

SAI Performance Report- end product from an assessment (30-40 pages?)

25

Enable SAI heads and senior management to

Have a high level overview of the SAIs performance

Promote and communicate the value and benefits of the SAI

Identify barriers to improved performance

Track performance changes over time

Explores how SAI external performance is constrained by

Country context and PFM environment

SAI environment

SAI internal capability

Provides a coherent and holistic assessment of SAI performance

Informed by objective and evidence based performance indicators

Analyzed within context Considered as a whole not the sum of its indicator scores


26/35


27/35

Indicator Design Each indicator provides an objective measure of current, not planned,

performance through 1-4 dimensions

Measured on a scale of 0-4

Translated into a single indicator score

27

Activity not Established

or does not Function

LEVEL 0

Founding Level

LEVEL 1

Established Level(Follows ISSAIs)

LEVEL 3

Managed Level

LEVEL 4

Development Level

LEVEL 2

Executiveagents held to

account

Accountabilityfor use of

public funds

SAI enablescontinual

improvement

Accountability forGovernmentperformance

Each levelprovides a basisfor accountability

MeasurementScale


28/35

Quality Assurance

Quality assurance crucial for objectivityand credibility of assessments

Enables a single assessment to meetdifferent stakeholder needs

Most appropriate approach dependson purpose of the assessment

Good practice: SAIs select QA process to meet the

assessment purpose

QA process is transparently disclosed inthe performance report

Separation between those undertakingthe assessment, and those performing QA

QA should cover: Is the report factually correct?

Has the SAI PMF tool been appliedcorrectly?

Is there sufficient evidence to justify theindicator scores?

28

Quality Assurance Statement

Who has prepared the

assessment?

Who has provided quality

assurance of the assessment?

What were their quality

assurance responsibilities?

How have matters raised in the

quality assurance process been

addressed in the final report?


29/35

4. Looking ahead: Responsibilities

and timelines

29


30/35

Potential risks related to having a SAI PMF

Revelations of performance weaknesses

It will be up to the SAI to decide if results are to be shared Openness and commitment to reform likely to encourage external

partners to support capacity development

May help draw attention to constraints outside the SAIs control

Possible negative effect of scoring: Use of performance league tables

Intention is not to aggregate scores to the level of the entire SAI as allindicators may not be of equal importance

Scores partly linked to achievements of SAIs own mandate

Narrative report provides context for scoring

Scoring is already applied in INTOSAI (AFROSAI-E)

Scoring very useful for measuring progress over time within an SAI

Profusion of Assessment Tools

Risk that the SAI PMF becomes an additional tool

Donors and SAIs continue to develop & use other tools

30


31/35

Identified Challenges in the Development

of the SAI PMF (1)

Different SAI models, mandates and country context

make it challenging to develop a global tool ISSAIs and other established good practice is common benchmark

Not measure at a too detailed level Assess against SAIs own mandate

Different SAI models and regions represented in Task Team and

Reference Group

Pilots to identify challenges and address them

Possible to answer N/A and to add local indicators

31


32/35

Identified Challenges in the Development

of the SAI PMF (2)

Scoring / objective measurement not possible Performance report will provide qualitative assessment and put

scoring into context

Scoring useful for measuring progress over time

Challenging timetable Johannesburg accords look for tool to be presented at INCOSAI

XXI, China, 2013

WGVBS looking to approve Exposure Draft for INCOSAI

Global piloting and consultations will extend beyond 2013

32

SAI PMF Timetable


33/35

SAI PMF Timetable

33

2011, Q3 2011, Q4 2012, Q1 2012, Q2 2012, Q3 2012, Q4 2013, Q1 2013, Q2 2013, Q3 2013, Q4

Mapping Existing SAI

Measurement Tools

Concept Note for SAI PMF

Develop SAI PMF Version 2.0

Phase 1 Pilots in 3 SAIs

& Comments from

Stakeholders

Develop

SAI PMF

Exposure

Draft

Phase 2 Piloting and Pre-Consultation

(ongoing into 2014 & 2015)

WGVBS:

Approve

way

forward

WGVBS:

ApproveExposure

Draft

WGVBS:

Set

Direction

Exposure

Draft at

INCOSAI

I l t ti d M i t


34/35

Implementation and Maintenance

(proposed)

Present SAI PMF at INTOSAI and other stakeholder events

Promotion and Awareness Raising of SAI PMF

Development and refinement of training material

Training of trainers within INTOSAI regional bodies

Training of SAI staff

Training of other stakeholders (donors, consultants, CSOs)

Training on Use of SAI PMF Assessments

Host and maintain guidance materials on internet

Provide Guidance and Support to Conduct of SAI PMF Assessments

Maintain and update global summary of the status of SAI PMF assessments, to enhancecoordination

Manage an e-library of published SAI PMF assessments Maintain database on (published) SAI PMF indicator scores for benchmarking, evaluation and

research

Monitor Progress on Use of SAI PMF and Maintain Consolidated Indicator Data

Decision to modify or update the SAI PMF (e.g. as ISSAI framework develops)

Develop and refine the SAI PMF

Technical Development and Refinement of SAI PMF

34

INTOSAI-Donor

Secretariat

INTOSAI-Donor Secretariat



INTOSAI-Donor SecretariatINTOSAI Regional Bodies



Database of SAIs Task Force

Database of SAIs Task Force

INTOSAI WGVBS

SAI PMF Task Team

INTOSAI

Community

Donor

Community

Approved Trainers

Approved Trainers


35/35

6. Further Information

www.idi.no

E-mail: [email protected]

35

Mapping of Tools

for Assessing

Performance of

Supreme AuditInstitutions

IDI, May 2012

Supreme Audit

Institutions

Performance

MeasurementFramework

Concept Note

IDI, May 2012
http://www.idi.no/mailto:[email protected]:[email protected]://www.idi.no/

Documents

SAI Performance Measurement Framework - Rationale, Process and Indicative Content, Yngvild Arnesen