Introduction to Performance

Audit and INTOSAI / Indian

Standards

Jahangir Inamdar

Director (Training & Research)

iCED, Jaipur

Session Overview

Performance of the governments / programmes

Performance Measurement and Program Evaluation

Public Management and Performance Audit

What is Performance Audit

Features of Performance Audit

Difference from other public audits

Basis of Performance Audit

Basic questions of Performance Audit

Approaches to Performance Audit

General Principles of Performance Audit

e.g. Mid Day Meal Programme (Nutritional support to

Elementary Education) of School Education Department

System of Governance

Inputs Process Output Outcome

Feedback

Inputs

Resources:

Human

Financial

Material

Processes

Organisations

Methods

Systems

Environment

Outputs

Products

Services

Outcomes

Fulfilment of Objectives -development

Governments

Mid Day Meal Scheme

Inputs

Resources:

Officers, Staff & Teachers

Money for cooking cost and buying ingredients

Buying supplements

Rice from Food Corporation of India

Processes

Hierarchy of the Department

Schools and Teachers

School Management Committees

System established for procurement / transportation of

rice ingredients, utensils etc.

Outputs

Cooked Meals

Food Supplements

Supply of food to school children

in drought prone areas

Outcomes

Nutritional status of school children

Retention of school children

Enrolment and retention of children of weaker sections

School Education

Department

Performance Measurement

Whether a program has achieved its objectives or requirements,

expressed as measurable performance standards.

Performance measurement serve as an early warning system to

management and as a vehicle for improving accountability to the

public.

It is a matter of internal management and control, not a task

for external auditors

One topic for performance auditing can be whether performance

measurement systems in government programs are efficient and

effective.

e. g. questions could be developed that address whether the

performance indicators measure the right things or whether the

performance measurement systems involved are capable of

providing credible measured results.

Programme Evaluation Program evaluations are individual systematic studies conducted to

assess how well a program is working.

Program evaluations typically examine a broader range of

information on program performance and context than is feasible to

monitor on an ongoing basis

Overall assessment of whether the program works and what can be

done to improve its results.

Program evaluation has objectives identical or similar to those of

performance auditing in that it seeks to analyse the relationship

between the objectives, resources, and results of a policy or

program.

It does not necessarily engage in assessing policy effectiveness

or policy alternatives.

In addition to examining the impact of outputs, program evaluation may

include issues such as whether the stipulated aims are consistent with

general policy.

Program evaluations are one type of study that might also be executed

by a SAI under the general heading of performance audits.

Public Management and Performance Audit

The form of public management employed will necessarily influence

priorities in performance auditing. In countries where public

management is mainly concerned with means and less involved with

ends, audits also tend to focus on whether rules have been observed

and enforced rather than whether the rules serve or are seen to serve

their intended purpose.

Management by objectives (MBO), also known

as management by results (MBR), is a process of

defining objectives within an organization so that management and

employees agree to the objectives and understand what they need to

do in the organization in order to achieve them.

Countries that have acknowledged management by objectives and

results tend to focus more on performance than before.

A common objective of most governments today is to improve the

quality of public services, particularly as people’s expectations

Mandate: Role of SAIs

Lima Declaration (ISSAI 1: Founding Principles):

The traditional task of Supreme Audit Institutions is to

audit the legality and regularity of financial

management and of accounting.

In addition to this type of audit, which retains its

significance, there is another equally important type of

audit--performance audit-- which is oriented towards

examining the performance, economy, efficiency and

effectiveness of public administration.

Performance audit covers not only specific financial

operations, but the full range of government

activity including both organisational and

administrative systems.

Performance Audit : Definition

Performance : A task or operation seen in

terms of how successfully it is performed

Performance auditing is an independent

examination of the efficiency and

effectiveness of government undertakings,

programs or organizations, with due

regard to economy, and the aim of leading

to improvements.

The 3 Es

Economy

Efficiency

Effectiveness

Economy

Judging ‘economy’ in itself implies forming an opinion on the resources

(human, financial and material) deployed.

This requires assessing whether given the context, resources have

been

◦ acquired - in due time, in appropriate quantity and quality at the best

price

◦ held and

◦ used

economically

The performance auditor needs to examine whether the means

chosen represent the

◦ ‘most’ or

◦ at least a ‘reasonable’

economical use of public funds.

Spending less

Efficiency

Relationship between the quality and quantity of goods and services

yielded and the cost of resources used to produce them, in order to

achieve the results.

getting the most from the available resources

◦ output is maximised for any given set of resource inputs,

or

◦ input is minimised for any given quantity and quality of output

resources have been put to optimal or satisfactory use

Methodology:

◦ Comparison with similar activities, with other periods or with a

standard, which the entity has explicitly adopted.

◦ Assessments on efficiency might also be based on the best

practices and available information

◦ Spending well

Efficiency

Auditing Efficiency embraces:

human, financial and other resources are

efficiently used;

public sector programmes, entities and activities

are efficiently managed, regulated, organised and

executed;

services are delivered in a timely manner; and

the objectives of public sector programmes are

met cost-effectively.

Effectiveness

Meeting the objectives set and achieving the intended

results

Whether the programme/activity has achieved its

objectives

The immediate outputs or products and the ultimate

impacts or outcomes. Outcomes are important to the

effectiveness of programmes/activities but may be more

difficult to measure and assess than the inputs and outputs.

Outcomes will often be influenced by external factors and

may require long term rather than short-term assessment.

Identify ways of making programmes work more effectively

Audit of the actual impact of activities compared with the

intended impact’

Spending wisely

Effectiveness: Issues for audit

Assess whether the objectives of and the means provided (legal,

financial, etc.,) for a new or ongoing public sector programme are

proper, consistent, suitable or relevant to the policy;

Assess and establish with evidence whether the observed direct or

indirect social and economic impacts of a policy are due to the policy

or to other causes;

Identify factors inhibiting satisfactory performance or goal-

fulfilment;

Assess whether the programme complements, duplicates, overlaps or

counteracts other related programmes;

Assess the adequacy of the management control system for

measuring, monitoring and reporting a programme's effectiveness;

and

Identify ways of making programmes work more effectively (the

recommendations)

Dilemma of counterfactual

Performance Audit and Programme Evaluation

• In performance audits SAIs may apply conventional auditing guidelines or

a variety of other methodologies, including social science methods.

• Possibilities for SAIs’ becoming more evaluative through the development

of crossover practices.

• SAIs may involve various levels of research, analysis, or evaluation.

• SAIs need no special mandate to conduct program evaluation. Any SAI

who wishes to may perform evaluations separately or embed them within

a mandate for performance audits.

Performance Audit and Program Evaluation

Regularity Audit

Effectiveness

Audit

Efficiency Audit

Economy

Audit

Evaluation

of the

impact of

the policy

Evaluation of

the

consistency of

the policy

Evaluation of

the

effectiveness

of the policy

and analysis of

causality

Performance Audit and Program Evaluation

Performance Audit /

Evaluation

(GAO)

Impact

Process

Cost-Benefit Cost

Effectiveness Evaluation

Outcome

Distinguishing features of regularity and

performance Audit-scope

Regularity audit

Generally covers a financial

period (annual, biennium);

Coverage is for the whole of

the entity for the period; and

Time bound- audit to be

completed by a stipulated

time.

Performance audit

Covers the subject /

programme over a period of

time;

Focused only on a part of the

entity’s activities / programme;

and

Coverage is selective.

Distinguishing features of regularity and

performance Audit- objectives

Regularity audit

Attestation (opinion on) offinancial accountability;

Existence of controls forsafeguarding of assets;

Evaluation of financialrecords and Audit offinancial systems;

Audit of propriety ofadministrative decisions;

Limited to financial matters;Test for assuring complianceto laws, regulations and rules;and

Conclusion / opinion withreference to standardisedrequirements.

Performance audit

Audit of 3E’s;

Assessment of compliance toapplicable laws andregulations in the context ofaudit objectives;

Audit of internal controls thatensures 3 ‘E’s, adherence tomanagement policies andtimely and reliable financialand management information;

Extends to non-financialsubjects also; and

Conclusions related to auditobjectives.

Distinguishing features of regularity and

performance Audit-evidence

Regularity audit

Financial statements per se,

accounting documents,

etc,Transaction documents;

Conclusive nature of

evidence; and

Materiality by amount;

Budgetary assumptions and

appropriation authorisations,

etc

Performance audit

Variety of forms of

evidence;Quite often

qualitative in nature;

Persuasive rather than

conclusive;

Evidence related to pre-

determined audit objectives;

and

Materiality guided more by

the nature or by context

rather than amount alone, etc.

Distinguishing features of regularity and

performance Audit-academic base

Regularity audit

Generally accounting

knowledge.

Performance audit

Always knowledge-

based: of laws, social

sciences, economics,

development studies,

public affairs, science,

and technology.

Distinguishing features of regularity and

performance audit-approach and methodology

Regularity audit

More or less

standardised: manual

generally forms the

entire basis.

Performance audit

Varies widely from subject

to subject; Manual

generally serves as basis

framework.

Distinguishing features of regularity and

performance Audit-assessment criteria

Regularity audit

Standardised (suitable to

all audits) with little

scope for subjectivity.

Performance audit

Widely varying and

subjective with ample

scope for

interpretations;

assessment criteria

support the audit

objectives, which are

unique to the subject of

performance audit.

Distinguishing features of regularity and

performance Audit-report

Regularity audit

Opinion on the financial

statements;

Generally opinion on financial

statements in standardised

format;

Opinion on compliance to laws,

regulations and rules;

Related to specific financial

periods- there is a periodicity

of reporting; and

Specific requirement and

expectations.

Performance audit Report / conclusions on

economy and efficiency withwhich the resources areacquired and used and theeffectiveness with which theobjectives are met;

Generally separatepublications on each subject ofperformance audits; and

Wide-ranging in nature, opento interpretations andsubjective judgement.

Distinguishing features of regularity and

performance Audit-overlap

Regularity audit

Financial audits do not

generally include the

elements of performance

audits.

Performance audit

There can be overlap in

the sense that the

performance audit may

encompass techniques /

methodologies applied

to financial audit.

Are things being done in the

right way?

Whether policy decisions are being carried out

properly.

This question is usually associated with a

assessment vis-à-vis norms i. e. the performance

auditor wants to know whether the executive

has observed the rules or the requirements

consistent with the programme.

Up to this point, performance auditing is mainly

concerned with different aspects of the

economy or the efficiency of operations.

Are the right things being

done? Whether the right things are being done.

In other words, effectiveness of the

operations would be examined by asking

questions whether the adopted policies

have been suitably implemented.

A performance auditor might, for

instance, find a chosen measure ineffective

and inconsistent with the policy

objectives.

Are the right things being

done? Adequate caution should be exercised by

not going beyond the audit mandate by

respecting the roles assigned to executive

and audit.

The correctness of the information or

inputs that were considered while framing

the policy and sufficiency of the

programmes and resources to fulfil the

policy objectives may be assessed and

reported.

Objectives of Performance Audit

Constructively promote economical,

effective and efficient governance.

It also contributes to accountability and

transparency.

It promotes accountability by examining

whether decisions by the legislature or

the executive are efficiently and effectively

prepared and implemented, and whether

taxpayers or citizens have received value

for money.

Objectives of Performance Audit

It does not question the intentions and decisions of the legislature,

but examines whether any shortcomings in the laws and

regulations or their way of implementation have prevented the

specified objectives from being achieved.

Performance auditing focuses on areas in which it can add value for

stakeholders and which have the greatest potential for

improvement.

Performance auditing promotes transparency by affording parliament,

taxpayers, those targeted by government policies, media and other

stakeholders an insight into the management and outcomes of

different government activities.

It thereby contributes in a direct way to providing useful information

to the citizen, while also serving as a basis for learning and

improvements.

Elements of Performance Audit

(i) Three parties in the audit i.e.

◦ the auditor: clear-cut demarcation of roles

and responsibilities of officers and staff

◦ the responsible party- range of individuals or

entities (subject, problems, recommendations,

information)

◦ intended user - the legislature, government

agencies, the public

(ii) the subject matter information and

(iii) criteria to assess the subject matter.

Elements of Performance Audit Subject matter

The subject matter of a performance audit need not be

limited to specific programmes, entities or funds but

can include activities (with their outputs, outcomes and

impacts) or existing situations (including causes and

consequences). The subject matter is determined by the

objective and formulated in the audit questions.

Audit Criteria

Audit criteria within the context of performance audit are

audit specific, reasonable standards of performance

against which the economy, efficiency and

effectiveness of operations can be evaluated and

assessed. The auditor may sometimes be involved in

developing or selecting the criteria that are relevant to the

audit

Assurance and confidence

Users of performance audit reports seek

confidence about the reliability of

information in the reports

The performance auditors should,

therefore, provide findings based on

sufficient and appropriate evidence

and actively manage the risk of

inappropriate reports.

Assurance and confidence

Audit assurance should be communicated in a transparent

way

Overall view on aspects of 3E, when the audit objectives,

the subject matter, the evidence obtained and the findings

reached allow for such conclusions

Providing specific information in the report on different

points, including the audit objectives, the questions asked,

the evidence obtained, the criteria used, the findings

reached and the specific conclusions

Performance auditor is not normally expected to

provide an overall opinion

Assurance and confidence

The decisions made in drawing up a balanced

report, reaching conclusions and formulating

recommendations frequently need to be

elaborated in order to build user

confidence.

Performance auditors should specifically describe

how their findings have led to a set of

conclusions and, if applicable, an overall

conclusion.

This means explaining the criteria developed and

used and why, and stating that all relevant

viewpoints have been taken into account.

Outcomes of Performance Audit Provide entities and stakeholders with information and

assurance about the quality of management of public

resources and also assist public sector managers by

identifying and promoting better management

practices

One or many of the following objectives should be fulfilled:

◦ better accountability,

◦ improved economy and efficiency in the acquisition of

resources,

◦ improved effectiveness in achieving public sector

programme objectives,

◦ a higher quality in public sector service delivery and

◦ improved management planning and control.

Outcomes of Performance Audit

Means to an end and not an end by itself

Adding value to the Management by way of reliable, objective

and independent information, highlighting the shortcomings in

programme planning, implementation, information systems affecting

the outputs and outcome specifically and quality of expenditure or

management generally

Extent of fulfilment of the policy objectives to the stakeholders

including the Parliament, the State Legislatures and the general public

Good quality performance audit contributes to good governance

All performance audits should, thus, be planned and performed

keeping in view the expected outcome. It is a good practice to

evaluate the real impact of performance audit on entity

policies and programmes.

Mandate and General Principles

In many countries, the mandate of performance auditing will stop short

of review of the policy bases of government programs.

In these cases, performance auditing does not question the merits of

policy objectives but rather involves examinations of actions taken

to design, implement, or evaluate the results of these policies,

and may imply an examination of the adequacy of information leading to

policy decisions.

According to the Auditing Standards (AS 2.2.10-19), a SAI must be free

to determine the areas covered by its performance audits

The earliest point at which a SAI can examine efficiency and

effectiveness is after the government has made a decision on the

policy concerned

Mandate

General aims of legislature should be taken for granted

Political decisions and goals established by the legislature

are in general the frame of reference, which form the

basis of the audit criteria used in performance auditing. It is

not the role of a SAI to question these decisions

and goals.

However, a SAI may, as a result of its findings, make critical

comments on the goals, for example if they are

inconsistent or if it proves impossible to follow up

the extent to which they have been achieved.

Mandate

SAIs may seek to achieve one or more of the following general goals:

(1) To provide the legislature with independent examination of the

economical, efficient, or effective implementation practices of

government policies or programs.

(2) To provide the legislature with independent, ad hoc analyses of

the validity and reliability of performance measurement

systems, or statements or self-evaluations about performance

that are published by executive entities.

(3) To provide the legislature with independent analyses of

problems of economy, efficiency, and effectiveness in

government activities and thus contribute to improvements.

(4) To provide the legislature with independent assessments of the

intended and unintended direct or indirect impact of

government and agency programs and whether, and to what

extent, stated aims or objectives have been met or why they have not

been met.

Mandate: SAI India

The mandate of SAI India for performance audits is

described in Sections 13, 14, 15, 16, 17, 19 and 20 read with

Section 23 of this Act.

Chapter 7 of the Regulations on Audit and Accounts, 2007

provides the specific guidance on this subject.

Regulation 68 defines performance audit as an independent

assessment or examination of the extent to which an

organisation, programme or scheme operates economically,

efficiently and effectively.

Policy’ is usually defined as an effort to achieve certain aims

with certain resources and perhaps within a certain time.

A ‘program’ can be described as a set of interrelated means

– legal, financial, etc.– to implement a given government or

agency policy.

Mandate : SAI India

Are the right things being done?

Standards and Guidelines

In conducting a performance audit the auditors should follow the

INTOSAI Code of Ethics and Auditing Standards as well as relevant SAI

standards and guidelines applicable to performance auditing.

ISSAIs

◦ Level 3:

100: Fundamental Principles of Public Sector Auditing

300: Fundamental Principles of Performance Auditing

◦ Level 4:

3000- Implementation Guidelines for Performance Auditing

3100 – Performance Audit Guidelines: Key principles - Appendix

SAI India’s Performance Audit Guidelines 2014 based on ISSAIs

Emphasis on process than on explaining the underlying philosophy

Updated with policy decisions taken and lessons learnt

General Principles

Ethics and Independence: Auditor should comply

with the relevant ethical requirements and be

independent.

Audit Objectives: Auditors should set a clearly defined

audit objective that relates to the principles of economy,

efficiency and effectiveness.

◦ Descriptive (How are the things?),

◦ Normative (are things as they ought to be?) and

◦ Analytical (why are things not as they ought to be?).

Well defined audit objectives relate to a single entity or an

identifiable group of government undertakings, systems,

operations, programmes, activities or organisations.

Thematically related, complementary, not overlapping and

collectively exhaustive sub-objectives

General Principles : Audit Approach

A system-oriented approach, which examines the

proper functioning of management systems, e.g. financial

management systems;

A result-oriented approach, which assesses whether

outcome or output objectives have been achieved as

intended or programmes and services are operating as

intended;

A problem-oriented approach, which examines, verifies

and analyses the causes of particular problems or

deviations from criteria.

General Principles : Audit Approach

Audits can be pursued from a top-down or

bottom-up perspective.

Top-down audits concentrate mainly on

the requirements, intentions, objectives and

expectations of the legislature and central

government.

A bottom-up perspective focuses on

problems of significance to people and the

community.

General Principles : Audit Criteria Auditors should establish suitable criteria which correspond to

the audit questions and are related to the principles of

economy, efficiency and effectiveness

Criteria are the benchmarks used to evaluate the subject

matter.

Reasonable and audit specific standards of performance

against which the 3Es of operations can be evaluated and

assessed.

The criteria can be qualitative or quantitative and should

define what the audited entity will be assessed against.

The criteria may be general or specific, focusing on

◦ what should be according to laws, regulations or objectives;

◦ what is expected, according to sound principles, scientific knowledge and best

practice; or

◦ what could be (given better conditions).

Diverse but transparent / discussed

General Principles : Audit Criteria

In a problem-oriented performance audit, the

starting point is a known or suspected deviation from

what should or could be.

The main objective is, therefore, not just to verify the

problem (the deviation from the criterion and its

consequences) but to identify causes.

This makes it important to decide how to examine and

verify causes during the design phase.

Conclusions and recommendations are primarily based on

the process of analysing and confirming causes

General Principles : Audit Risk

Auditors should actively manage audit risk, which is

the risk of obtaining incorrect or incomplete conclusions,

providing unbalanced information or failing to add value

for users

◦ Not being able to provide new information or perspectives to

the risk of neglecting important factors

◦ Complex and politically sensitive

◦ Not possessing the competence

◦ Lacking access to complete and quality information

◦ Failing to collect or address the most relevant arguments

Auditors should, therefore, actively manage risk. Audit

planning documents should state the possible or known

risks of the work envisaged and show how these

risks will be handled.

General Principles : Communication

Auditors should maintain effective and proper communication with the

audited entities and other parties sharing the responsibility of the subject

matter throughout the audit process and define the content, process and

recipients of communication for each audit.

Non-recurring process, stakeholders other than regular ones

Often there are no predefined criteria (such as a financial reporting

framework), and therefore an intensive exchange of views with the audited

entity is necessary.

An active effort is required to obtain insight into the points of view of the

various stakeholders to bring out a balanced report

Likelihood that audit recommendations will be implemented

Written engagement letter and constructive interaction

Care should be taken to ensure that communication with stakeholders

does not compromise the independence and impartiality of the

Department

Audited entities should be given an opportunity to comment on the

audit findings, conclusions and recommendations before the Department

issues its audit report.

Any disagreements should be analysed and factual errors corrected.

General Principles : Skills

Collectively, the audit team should have the necessary

professional competence to perform the audit. This would include

in addition to the knowledge of the domain of the audited entity,

sound knowledge of auditing, research design, social

science methods and investigation or evaluation

techniques, as well as personal strengths such as analytical,

writing and communication skills.

Auditors should have a sound knowledge of government

organisations, programmes and functions.

Performance audits often involve a learning process and the

development of methodology as part of the audit itself.

On the job learning and training should therefore be available to

auditors, who should maintain their professional skills through

ongoing professional development.

External experts can be used to complement the knowledge of

the audit team

General Principles : Professional Judgement and

Scepticism

Auditors should exercise professional scepticism, but also be receptive

and willing to innovate

Exercise professional scepticism (An attitude that includes a questioning

mind, being alert to conditions which may indicate possible misstatement

due to error or fraud, and a critical assessment of evidence) and adopt a

critical approach, maintaining an objective distance from the

information provided.

This is necessary in order to avoid errors of judgement or cognitive

bias. If they are not receptive, they may miss important arguments

or key evidence.

As auditors work to develop new knowledge, they also need to be

curious, reflective and resourceful in their efforts to collect,

interpret and analyse data.

High standard of professional behaviour - work systematically, with

due care and objectivity and under appropriate supervision

General Principles : Quality Control

Auditors should apply procedures to safeguard quality, ensuring that the

applicable requirements are met and placing emphasis on

appropriate, balanced and fair reports that add value and answer

the audit questions

Performance audit is a process in which the audit team gathers a large

amount of audit specific information and exercises a high degree of

professional judgement and discretion concerning the relevant issues

If there is a difference of opinion between supervisors and the audit

team, appropriate steps should be taken to ensure that the audit

team's perspective is given sufficient consideration and that the

Department's policy is consistent.

Even if the report is evidence based, well documented and accurate, it

might still be inappropriate or insufficient if it fails to give a balanced

and unbiased view

Define clearly what constitutes a high quality report in the specific

context of an audit engagement

Control mechanisms should therefore be complemented by support,

such as on the job training and guidance for the audit team

General Principles : Materiality

Auditors should consider materiality at all stages of the audit

process. Thought should be given not only to financial but also to

social and political aspects of the subject matter, with the aim

of delivering as much added value as possible.

Relative importance of a matter within the context in which it is

being considered. The materiality of an audit topic should have

regard to the magnitude of its impact.

In performance audit, materiality by monetary value may, but need

not, be a primary concern. In defining materiality, the auditor should

consider also what is socially or politically significant and bear

in mind that this varies over time and depends on the perspective

of the relevant users and responsible parties.

Materiality concerns all aspects of performance audits, such as the

selection of topics, definition of criteria, evaluation of evidence

and documentation and management of the risks of

producing inappropriate or low impact audit findings or reports

General Principles : Documentation

Auditors should document the audit in accordance with the particular

circumstances thereof. Information should be sufficiently complete and

detailed to enable an experienced auditor having no previous

connection with the audit to subsequently determine what

work was done in order to arrive at the audit findings, conclusions

and recommendations.

The purpose and context of documentation are somewhat specific in

performance auditing

The audit methodology and criteria may have been developed

specifically for a single engagement, the auditor carries a special

responsibility to make his reasoning transparent.

The report describes the framework, perspective and analytical

structure that were adopted and the process that was followed to

arrive at the conclusions in addition to findings and recommendations

(in other audits it is done by standards)

Documentation should not only confirm the accuracy of facts, but also

ensure that the report presents a balanced, fair and complete

examination of the audited question or subject matter.

THANKS