Annual Results and Impact Evaluation Workshop for RBF - Day Three - Measurement of Results - Conditional Transfer Models - The Role of External Concurrent Audit

Audit | Tax | Advisory

MEASUREMENT OF RESULTS (CONDITIONAL TRANSFER MODELS) The Role of the External Concurrent Audit

March 2014


Agenda

• Audits, External Audits, Internal Audits, Concurrency

• Costs and Benefits expected from the ECA

• Experiences

• Lessons learned

• Conclusion


The 4 Qs

Sooner or later, during the design stage of any given program (the sooner, the better), the team in charge of strategic definitions will be faced with the 4 questions related to the Audit function:


The 4 Qs

Should we include the

Audit function in our

Project?

Should we define an

Internal Audit or an

External Audit?

Should our project

consider a Concurrent

Audit?

What should we do

with the results of

the Audit?


Building (possible) answers

• The Audit function is not equivalent to the Audit structure. Some organizations may create an Internal Audit structure, while others may outsource such task. Some may design a Monitoring and Evaluation Area. Smaller customer-driven organizations may develop this function in the Customer Care Area.

• However, every organization must evaluate the need for the Audit function, which is in charge of assessing what was done, what should have been done and of suggesting corrections in the event of discrepancies. Without such feedback, organizations operate blind-folded throuought their lifecycle, without being aware of the detours until the end of their operation.


External Audits vs. Internal Audits

• In the case of standard External Audits, the achievement of the audited organization’s goals is not seen as their mission. The goal is to ensure the veracity of the reported financial information.

• Internal Audits focus on the process and the organization’s compliance with the defined procedures. The result is relevant in terms of the measurable result of the process.

• The is no such dilemma: they are two models, with different goals and methods, which may be independent from each other in the worst case scenario, and complementary in the best.


The ECA Model

• The External Concurrent Audit (ECA) appears between these two models with the obvious tensions and overlaps in terms of conceptual boundaries.

• It maintains its mandate of external independence and opinion source , but it incorporates most of the tools of the Internal Audit for the evaluation of processes and measurement of results, as well as the improvement opportunity approach.


What kind of Projects are eligible for an ECA? • Long-term operation

• Distributed operation model

• Multiple parties involved

• High transactional component among players

• Introduction of New Management Models

• Results- Based Management Models

• Changing Operation Frameworks


What kind of Projects are eligible for an ECA?

The ideal projects are those in which the complexity of the operation in terms of change over time, the number of participants and geographic distribution requires the implementation of a frequent and close evaluation and report scheme to ensure that the information is provided in a timely manner to enable the adjustment or verification of the operation.


The Problem of Audit Results

• Regardless of the answer to the previous questions, we must always define what to do with the Audit results, taking into account the following:

– There is always the possibility of detours (even in the design).

– Detected variations can be taken as flaws or as opportunities for improvement.

– Hiding (or regulating) detours does not improve a project’s performance.

– The more detailed the expected scope, the more specific the Audit results will be.


Should we include

the Audit function

in our Project?

Should we define an

Internal Audit or an

External Audit?

Should our project

consider a Concurrent

Audit?

What should we do

with the results of

the Audit?

The Audit function is inevitable. Therefore, it is better to plan it appropriately.

Both standard models complement each other, which means they can co-exist within a Project.

Not every project is eligible to benefit from an ECA model. It is essential to identify those that are.

Audits create results which are commonly expressed as detours; we must take this into account from the design stage to plan any actions that should be taken.


Benefits of the ECA Model

• Early detection of detours and improvement opportunities

• Incorporation of the perception of the different players involved

• Fast adjustment of work programs to regulatory changes

• Cooperation in terms of the compliance of goals by means of uniform approval criteria

• Creation of multidisciplinary teams, which enables the fluent interaction of players


Costs of the ECA Model

• Intensive time application

• Steep learning curves

• Counterpart availability

• Careful strategic definition of scopes

• Strategic decision about what to do with the results

• Effort to document administrative decisions


Our Experience with ECA

PROAPS REMEDIAR

BID

2003

PISMIP PLAN NACER APL1

BIRF

2005

PISMIP PLAN NACER APL2

BIRF

2009

FESP

BIRF

2011


What did we learn?

• What to do

• When to do it

• How to do it

• What to do it with

• Who to do it with


ECA Roadmap Identification of Central

Process

Development of Work Plan

Implementation of Work Plan

Planning of work cycle

Fieldwork scheduling

Field Survey

Consolidation and Analysis

Report Completion


Risks associated with Project Management

• Scope Risks – The Work Plan might not cover the expected scope. – The Work Plan might cover the expected scope but it

might not meet the strategic needs.

• Control Risks – Excessive decentralization – Too inflexible

• Detection Risks – The Work Plan might not evolve fast enough to detect any

emerging variations. – Variations across independent evaluators

• Relevancy Risks – Observations in one site may not be relevant in another.


Challenges

• To include local realities into Program guidelines

• To coordinate the agendas of resources and counterparts in order to meet defined deadlines

• To match the consultant’s expectations with the auditor’s responsibilities

• To adjust work programs to regulatory changes without affecting deadlines


ECA Guidelines

• Multidisciplinary local team

• Ongoing revision of Work Programs

• Ongoing training and updating

• Important supervising and monitoring component

• Constant contact with counterparts


ECA Guidelines

• Multidisciplinary local professional team






AUDITORES DE CAMPO

PROJECT LEADER

IT COORDINATOR AUDIT COORDINATOR TECHNICAL AUDIT

COORDINATOR

AUDIT SUPERVISOR TECHNICAL AUDIT

SUPERVISOR

IT AUDITOR FIELD AUDITOR TECHNICAL FIELD

AUDITOR

AUDIT COORDINATION

TEAM

TECHNICAL AUDIT COORDINATION

TEAM

EXPERT IN STATISTICS CENTRAL BASE

LOCAL BASE

MIXED BASE


ECA Guidelines








ECA Guidelines








ECA Guidelines







Work Cycle

External Concurrent Audit

Month 1 Month 2 Month 3 Month 4

Two-month period

under analysis

Report

submission

Field auditor survey Supervising visits Report review

Field auditor survey

Set up of sampling

Report completion


ECA Guidelines







Key Success Factors

• Highly-trained work team

• Efficient communication

• Adaptability to changes in the operation model

• Constant counterpart feedback

• Careful planning of times and resources


Lessons Learned

• Steep learning curves call for the careful planning of changes in the ECA throughout the project’s lifecycle.

• The strengthening of the auditor’s role through the adoption of their observations as binding for the program requires a negotiation effort on the part of those running the Program in the short term, but it leads to a boost of results in the long term.


Conclusion

• The ECA scheme provides the independence of an External Audit with a methodological framework based on process evaluation

• Not every Program will benefit from an ECA model.

• Those Programs that benefit from an ECA model will clearly obtain greater benefits from this model than with a standard Audit model.

Healthcare

Annual Results and Impact Evaluation Workshop for RBF - Day Three - Measurement of Results - Conditional Transfer Models - The Role of External Concurrent Audit