Performance Audit Manual for Public Internal Auditors Case ... · The head of the internal audit unit assigned internal auditors Aylin Kaya and Mehmet Akın for this engagement, in

EDITION 1.0

INTERNAL AUDIT COORDINATION BOARD

Performance Audit Manual for Public Internal Auditors

Case Studies

ANKARA | April 2016

INTERNAL AUDIT COORDINATION BOARD

Performance Audit Manual for Public Internal Auditors

Case Studies

ANKARA | April 2016

TABLE OF CONTENTS

1. CASE 1

PERFORMANCE AUDIT OF THE LIBRARY PROCESSES OF UNIVERSITY A ........................................ 7

A. LAUNCHING THE AUDIT ENGAGEMENT ................................................................................... 8

A1. ASSIGNMENT ................................................................................................................. 8

A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY DOCUMENT .................... 8

A3. NOTIFICATION TO THE AUDITEE ......................................................................................... 8

B. PRELIMINARY WORK ........................................................................................................... 11

B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE ......................................... 10

B2. ASSESSMENT OF THE PERFORMANCE MANAGEMENT SYSTEM ................................. 13

B3. UNDERSTANDING THE LOGICAL FRAMEWORK ........................................................... 16

B4. IDENTIFICATION OF AUDIT COMPONENTS AND METHODS ....................................... 18

B5. KICK OFF MEETING ...................................................................................................... 18

B6. DEVELOPING THE AUDIT MATRIX ............................................................................... 21

B7. PREPARATION AND APPROVAL OF THE WORK PLAN .................................................... 23

C.FIELD WORK......................................................................................................................... 25

C1. DATA COLLECTION ...................................................................................................... 25

C2. DATA ANALYSIS ........................................................................................................... 26

C3. IDENTIFICATION OF FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS ......... 35

D.REPORTING ......................................................................................................................... 43

D1. OFFICIAL SHARING OF FINDINGS ........................................................................................ 43

D2. CLOSING MEETING ....................................................................................................... 48

2. CASE 2

PERFORMANCE AUDIT OF THE BUS OPERATIONS OF MUNICIPALITY A ..................................... 51

A. LAUNCHING THE AUDIT ENGAGEMENT ................................................................................. 52

A1. ASSIGNMENT ............................................................................................................... 52

A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY DOCUMENT .................. 52

A3. NOTIFICATION TO THE AUDITEE ................................................................................... 52


B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE.......................................... 55





B6. DEVELOPING THE AUDIT MATRIX ............................................................................... 64

B7. PREPARATION AND APPROVAL OF THE WORK PLAN .................................................... 68

C. FIELD WORK ........................................................................................................................ 70

C1. DATA COLLECTION....................................................................................................... 70

C2. DATA ANALYSIS ........................................................................................................... 72

D.R EPORTING.......................................................................................................................... 81

D1. OFFICIAL SHARING OF FINDINGS ........................................................................................ 81

D2. CLOSING MEETING ....................................................................................................... 85

3

4

3. CASE 3

PERFORMANCE AUDIT OF THE EMERGENCY CALL CENTRE OF PRESIDENCY A ........................... 89

A. DENETİM GÖREVİNİN BAŞLATILMASI ..................................................................................... 90

A1. GÖREVLENDİRME ........................................................................................................ 90

A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY DOCUMENT .................. 90

A3. NOTIFICATION TO THE AUDITEE ................................................................................... 90


B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE ......................................... 93





B6. DEVELOPING THE AUDIT MATRIX ............................................................................. 102

B7. PREPARATION AND APPROVAL OF THE WORK PLAN .................................................. 103

C. FIELD WORK ...................................................................................................................... 105

C1. DATA COLLECTION..................................................................................................... 105

C2. DATA ANALYSIS ......................................................................................................... 106

C3. IDENTIFICATION OF FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS ....... 116

D.REPORTING ....................................................................................................................... 118

D1. OFFICIAL SHARING OF FINDINGS ...................................................................................... 118

D2. CLOSING MEETING ..................................................................................................... 123

4. VAKA

PERFORMANCE AUDIT OF THE TRANSITION TO ELECTRONIC SYSTEM PROGRAM ................... 129

A. LAUNCHING THE AUDIT ENGAGEMENT .............................................................................. 130

A1. ASSIGNMENT ............................................................................................................. 130

A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY DOCUMENT ................ 130

A3. NOTIFICATION TO THE AUDITEE ................................................................................. 130

B. PRELIMINARY WORK ......................................................................................................... 133

B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE........................................ 133

B2. ASSESSMENT OF THE PERFORMANCE MANAGEMENT SYSTEM ............................... 134

B3. UNDERSTANDING THE LOGICAL FRAMEWORK ......................................................... 138

B4. IDENTIFICATION OF AUDIT COMPONENTS AND METHODS ..................................... 140

B5. KICK OFF MEETING .................................................................................................... 140

B6. DEVELOPING THE AUDIT MATRIX ............................................................................. 142

B7. PREPARATION AND APPROVAL OF THE WORK PLAN .................................................. 147

C. FIELD WORK ...................................................................................................................... 149

C1. DATA COLLECTION ..................................................................................................... 149

C2. DATA ANALYSIS ......................................................................................................... 149

C3. IDENTIFICATION OF FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS ....... 154

D.REPORTING ....................................................................................................................... 156

D1. OFFICIAL SHARING OF FINDINGS ...................................................................................... 156

D2. CLOSING MEETING ..................................................................................................... 159

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

Performance Audit of the Library Processes of University A

The content of the case study is not related to any organisation, practice or person. The content developed is completely fictional.

7

PERFORMANCE AUDIT OF THE LIBRARY PROCESSES OF UNIVERSITY A

University A is a well-established university with faculties of social sciences, engineering

and fine arts, where the language of instruction is English. According to leading indices, the

university ranks among the ten most successful universities nationwide.

University management declared 2017 as the library year. In this regard, a new objective

was set to enrich the university library, which has been in service since the establishment of the

university and is considered to be among the best of its kind in the country, and to increase the

number of users which was in a decline from 2011 to 2015.

Within the scope of the macro level risk assessment exercise conducted by the internal audit

unit, the library processes, which form part of the audit universe, were assessed against the risk factors

and in consideration of the goals and objectives of the strategic plan and the opinions of the top

manager and other senior managers. At the end of the assessment, it was decided to include the library

process in the audit program for 2016.

The university management is planning to use the audit results to take necessary actions for

the restructuring of the library in 2017.

The main expectations of the top management from the audit to be carried out are:

Assessment of the extent to which the existing library processes are sufficient to reach

the objectives set forth in the strategic plan of the university,

Assessment of the existing activities in comparison with the other city universities similar

to University A and with other good practices,

Identification of areas of development that need to be taken into account while

developing activities within the scope of 2017 library year.

In line with the objectives and expectations of the management, the internal audit unit

of the university decided to carry out a performance audit on the library processes. The audit has

been included in the audit program for 2016.

The audit is performed in line with the framework provided in the Performance Audit

Manual for Public Internal Auditors and Public Internal Audit Manual.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

8

A. LAUNCHING THE AUDIT ENGAGEMENT

A1. ASSIGNMENT

The head of the internal audit unit assigned internal auditors Aylin Kaya and Mehmet Akın

for this engagement, in accordance with the annual audit program. Senior auditor Ömer Başkale was

assigned as the Audit Supervisor. In the assignment of internal auditors, the head of internal audit

took into account the following elements:

- Complexity and size of the audit field

- Strategic importance of the audit field

- Technical knowledge requirement of the audit field

A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY DOCUMENT

Internal auditors assigned by the head of internal audit unit signed the impartiality and

confidentiality documents.

A3. NOTIFICATION TO THE AUDITEE

The head of internal audit sent a notification letter to the unit to be audited to give them

basic information about the audit.

A1.Engagement

Nr. : 76995536-662.02- 14/02/2016

Subject : Assignment

Ms. Aylin Kaya (6666/A3)

Internal Auditor

You have been assigned to audit the Library Processes of the Library and Documentation

Department, in particular the processes of user services, technical services, IT services,

administrative and financial affairs. You are kindly requested to perform the audit in line with the

Public Internal Audit Standards, the Public Internal Audit Manual and the Performance Audit

Manual and to submit the audit report to the Internal Audit Unit.

Head of the Internal Audit Unit

Type of Audit Performance Audit of the Library Processes of the Library and Documentation Department

Special Instructions The performance audit shall be carried out by applying the methods and techniques referred to in the Performance Audit Manual

Planned Audit Period 26.02.2016-15.05.2016

Other Internal Auditors Assigned

Mehmet Akın (4444/A3)

Audit Supervisor Ömer Başkale (5555/A3)

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

9

A2.Preparation of the Impartiality and Confidentiality Document

IMPARTIALITY AND CONFIDENTIALITY DOCUMENT

Audited unit: Library and Documentation Department

Audit subject: Library processes

Performance Audit of Library Processes 19.1. User Services Process 19.2. Technical Services Process 19.3. IT Services Process 19.4. Administrative and Financial Affairs Process

All activities of the Library and Documentation Department taking place from 01.01.2015 to 31.12.2015

I hereby declare that

- I have not undertaken any administrative assignments within the last year related to the activities that are included in the audit scope,

- None of my first, second and third degree relatives by blood and by law are employed by the audited unit,

- I have not been assigned to audit the same unit for three years in a row, - I have not been assigned with any consultancy work related to the topics covered by the audit scope

within the last year, - I bear no prejudices against the audited unit, its employees or its managers.

In the event that I encounter with a situation during the performance of the audit which would distort my impartiality or which would lead to the impression that my impartiality is distorted, I promise that I will inform the head of the internal audit unit as soon as possible and I will protect the confidentiality of the information that I acquire during the performance of the audit.

26.02.2016 Aylin Kaya

Internal Auditor

DECLARATION

NAME OF THE AUDIT

AUDIT SCOPE

WARRANTY

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

11

A3. Notification to the auditee

Nr. : 76995536-679- ..../02/2016

Subject : Audit Notification

TO THE DEPARTMENT OF LIBRARY AND DOCUMENTATION

Within the scope of the 2016 Internal Audit Program carried out in accordance with the

approval of the University Presidency signed on 20.02.2016, a performance audit will take place in

your unit from 26.02.2016 to 15.05.2016, to assess the library processes of your department

related to user services, technical services, IT services and administrative and financial services.

The audit in question will be launched on 26.02.2016 and is planned to be completed on

15.05.2016.

The audit will be performed under the supervision of Internal Auditor Ömer Başkale

(5555/A3), by internal auditors Aylin Kaya (6666/ A3) and Mehmet Akın (4444/A3). You will receive

the findings and the report to be drafted at the end of the audit.

The audit scope in general consists of Library Processes (user services process, technical

services process, IT services process and administrative and financial affairs process); the exact scope

and audit objectives will be determined on the basis of the discussions between our audit team and

your unit.

The performance audit exercise will be primarily based on data analysis. Therefore it is of

utmost importance that the data requested by the internal auditors be provided on a timely manner

and accurately. Contribution and participation of process owners will help the auditors to develop

feasible recommendations for improvement.

Successful completion of the audit engagement depends strongly on your co-operation

and open attitude for sharing information.

Kindly submitted for your information and due action.

Head of the Internal Audit Department

CIRCULATION

To the attention of

Department of Library and Documentation

Information

Aylin Kaya (6666/A3)

Ömer Başkale (5555/A3)


Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

10

Identification of Audit Components and Methods

Kick off meeting

Understanding

the Logical

Framework

Audit Matrix Preparation and

approval of the Work

Plan

B. PRELIMINARY WORK

At the preliminary work stage of the audit, the following steps set forth in the Public

Performance Audit Manual were followed.

1 2 3 4 5 6 7 Assessment of

the Performance

Management

System

B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE

The audit supervisor, in consultancy with the internal auditors within the audit team,

prepared the audit engagement time schedule form. In addition to the form, he prepared a

detailed audit plan which also shows the interim stages of the audit.

Task Output Start end date date

1 2 3 4 5 6 7 8 9 10 11 12

Wee

k

Wee

k

wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

1 Launching of the audit engagement 26.02.16 26.02.16 Assignment Assignment Letter Preparation of the Impartiality and Confidentiality Document Impartiality and

Confidentiality Document

Notification to the Auditee Audit Notification Letter 2 Preliminary work 02.03.16 27.03.16

Audit Engagement Time Schedule Form Audit Engagement Time

Schedule Form

Performance Management System Maturity Analysis Maturity Analysis Form Understanding the Logical Framework Requesting Preliminary Data Preliminary Data Request

Form

Preliminary Analysis and Assessments Documenting the Logical Framework Logical Framework Form

Identification of Audit Components and Methods Audit Matrix Kick-off meeting Kick-off meeting minutes Drafting the Audit Matrix Audit Matrix Preparation of the Work Plan Work Plan

3 Field work 30.03.16 24.04.16 Data collection Requesting for data Data Request List Establishing the Audit Data Base Audit Data Base Analysis Identifying the findings and developing recommendations Consolidated list of findings

4 Reporting 27.04.16 15.05.16 Official sharing of findings Findings form Closing meeting Closing meeting minutes

Taking action plans Action Plan Audit report Audit Report

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

Developing the

Audit

Engagement

Time Schedule

12

B1. AUDIT TIME PLAN FORM

Performance Audit of the Library Process

Planned Realised

Start date

End

date

Start

date

End date

PRELIMINARY WORK 02.03.2016 27.03.2016

Performance Management System Maturity Analysis

03.03.2016

Understanding the Logical Framework

09.03.2016

Kick-off meeting 17.03.2016

Approval of the work plan (Engagement Work Program and Audit Matrix)

27.03.2016

FIELD WORK 30.03.2016 24.04.2016

Collecting data and launching the analyses (tests)

30.03.2016

Completing the analyses (tests) 24.04.2016

REPORTING 27.04.2016 15.05.2016

Preparing the findings and developing the recommendation

27.04.2016

Communicating the findings 01.05.2016

Closing meeting 08.05.2016

Presenting the audit report 15.05.2016

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

13

B2. ASSESSMENT OF THE PERFORMANCE MANAGEMENT SYSTEM

The internal auditor assessed the maturity of the performance management system of the

audit field in order to ensure that the results of the performance audit exercise to be conducted

are successful. In this regard, the maturity of the audit field was analysed on the basis of the

following topics:

▪ Assessment of the maturity of the performance management system,

▪ Examination of whether the area to be audited is suitable for performance audit,

▪ Evaluation of whether the performance audit components are applicable on the area to

be audited.

In the assessment of the maturity of the performance management system, the maturity

analysis table provided in the Performance Audit Manual was used. The internal auditor asked the

following questions in this assessment.

Have the goals and objectives been defined at the level of the administration, its units and activities?

Have performance indicators been developed related to these goals and objectives?

Are the goals and objectives at the administration, unit, process, program and activity

level and related performance indicators followed through a system?

Is data related to performance indicators monitored and reported?

The preliminary study revealed that goals and objectives were defined at the level of units and activities of the University. Also, the goals and objectives of the library unit were described in the performance program. These goals and objectives were related with performance indicators. To monitor the performance indicators in a systematic manner, a software called “performance management system” is used. Some of the performance indicators are monitored through this program, which, however, lacks the necessary capacity to report on all performance indicators. The management attaches importance to this program and plans to make necessary investments for its further improvement.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

14

As a result of the interviews, the internal auditors came up with the following scores in the

maturity analysis assessment.1

Planning and implementation

Weight coefficient

%30

1 The administration does not have a strategic plan.

2 Only a few goals and objectives are defined in the strategic plan of the administration.

3 Goals and objectives are set out in the strategic plan of the administration.

Distribution of tasks within relevant units is at basic level.

4

The goals and objectives are set out in the strategic plan of the administration at a reasonable level and they are related with each other and with performance indicators.

Distribution of tasks and responsibilities related to performance indicators are made in detail within related units.

Resource allocation is planned in line with goals and objectives

5

The goals and objectives are set out in detail in the strategic plan of the administration and they are related with each other and with comprehensive performance indicators.

Tasks are distribution to related units in a detailed manner. Resource allocation is planned in line with goals and objectives. A Monitoring and follow up mechanism exists for the strategic plan.

Performance indicators

Weight coefficient %40

1 No indicators exist for monitoring performance.

2 Some indicators have been set out for monitoring performance however

they are not comprehensive nor systematic.

3

Indicators have been systematically set out for monitoring performance.

The performance programs do not show the connection of indicators with the objectives of relevant units

4


The performance programs show the connection of indicators with the objectives of relevant units.

5



A system to monitor indicators has been established and a system for reporting to make revisions where necessary is in place.

1 Indicators are in line with SMART criteria. See: 3.5.2.3. Understanding the logical framework

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

15

Monitoring and Reporting

Ağırlık Katsayısı

%20

1 Performance results of the organisation are not monitored

2 Performance is measured for some indicators, but not on a regular basis.

3 Performance results related to all performance indicators set out in the

strategic plan are monitored

4 Performance is measured for all activities and processes on a regular basis

and the results are shared with related people.

5

Performance is measured for all processes through a regular monitoring mechanism and the results are shared with related people. Based on the results, improvements are recommended and the strategic plan is revised where necessary.

Management awareness and support

Weight

coefficient %10

1 Monitoring results are not reported to the management

2 Management takes into account the results of monitoring for only some of

the units and activities.

3 Management takes into account the results of performance monitoring for

all units within the scope of the performance program.

4

Management takes into account the results of performance monitoring for all units within the scope of the performance program.

Feedback on results is given to related units.

5



Improvements are made based on the results. Management supports the realization of recommendations on improvement.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

16

Weight

coefficient

Score

Weighted score

Total score

Design and implementation of the strategic plan and activity program

30%

4

1,2

3,6 Performance indicators 40% 4 1,6

Monitoring and reporting 20% 2 0,4


10% 4 0,4

The maturity level of the performance management system in the administration is

calculated as 3,6. This score has established that the audit field is available to conduct a

performance audit at a scale described in the Performance Audit Manual for Public Internal

Auditors.

The analyses have also led to the conclusion that the existing situation of the performance

management system does not involve any restrictions as far as the audit components and audit

fields concerned.

After this assessment the internal auditors continued with the stage that concerns the

understanding of the logical framework.

B3. UNDERSTANDING THE LOGICAL FRAMEWORK

At the preliminary work stage, the internal auditors prepared the list of preliminary data

needed to carry out the necessary analyses and assessments for understanding the library

processes. The list that contains the data which the internal auditors are planning to examine is as

follows:

# Document or information

1 Strategic Plan of University A (2015-2019)

2 Strategic Plan of University A (2010-2014)

3 Performance Program of University A (2011, 2012, 2013, 2014, 2015)

4 Library Business Flow Charts

5 Library sub processes

6 Library Accountability Reports

Internal auditors also used publicly available documents and reports during the

preliminary work stage. The reports and documents examined are as follows:

Independent Annual Library Report, UK

Library Accountability Reports (from national and international universities)

World Libraries Association– Contemporary Library Processes and Good Practices

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

17

The logical framework to be applied during the audit is provided below:

Goal Objective

Milestone (Program objectives)

Activities Performance Indicators

Source of the

indicator

2015-2019 Strategic Plan, Goal 3 To improve the national and international recognition of the university

Objective 3.1. To improve the national and international recognition of the university library

- To ensure that the university library comes at the top of the academic search results - To improve the library’s resource diversity and capacity and to develop the library as a means of promoting the university

Main activities under the program:

- Improvement of the library inventory in terms of quality and quantity. - Increasing the variety of the library databases. - Development of the library management system - Development of the HR serving the library

Ratio of the library users to the total number of students and academics

Strategic Plan

Ratio of the number of printed materials to the number of students and academics

Performance Program

Change in the rate of borrowing printed materials from 2011 to 2015

Program Indicators

Ratio of the number of borrowings to the number of library users

Indicators proposed by the internal auditor

Objective 3.4. To establish the necessary infrastructure to increase the number of academic publications generated as a result of academic studies performed at the university

Enhancing the physical and spatial capacity of the library

- Development of the variety and actuality of the resources based on student expectations

Ratio of the budget allocated to the purchase of new materials to the overall library budget

Performance Program

Total number of training hours

Number of staff participating in the training

Program Indicators

Number of students and academics taking the orientation training


Ratio of the number of databases to the total number of materials

Rate of periodicals per user in 2011 – 2015

Breakdown of the library budget of 2011 – 2015 based on materials.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

18

B4. IDENTIFICATION OF AUDIT COMPONENTS AND METHODS

After getting the views of the internal auditors, the audit supervisor decided that the

audit exercise, which is based on the expectations of the management and the audit scope, be

conducted focusing on the components of productivity, effectiveness, consistency and

conformity.

During the audit, the aim is to assess the activities and practices that form part of the library

processes against the following audit components and to identify areas of further improvement:

Productivity; assessment of whether the output generated from the available resources is

at an adequate level.

Effectiveness; capacity to reach objectives and realise intended results

Consistency; coherence between the defined goals, objectives and strategies of public

administrations and the planning and design of their programs and projects

Conformity; compliance of the activities and practices with the regulations and

procedures in force and adequacy of the institutional capacity for implementation

The methods that are planned to be applied in auditing the abovementioned components are

benchmarking and measuring and reporting on performance.

The benchmarking analysis will focus on:

the performance trends of the library from 2011 to 2015

performance of the library as compared to the libraries of national and international university libraries.

B5. KICK OFF MEETING

During the kick-off meeting, the internal auditor informed the auditee on the following topics:

audit team and its organisation

audit scope

stages of the audit

time schedule

identification of the comparison group

next steps

During the kick-off meeting, the auditee expressed its expectations as follows:

international university libraries are also used as a benchmark as part of the analysis on

the library activities

universities without a faculty of medicine are included in the comparison group

human resources and physical/spatial conditions of the library are taken into account

during the assessment.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

19

since the student gateway of the library became operational towards the end of 2015 is

still under development, conceptual design of the gateway is assessed within the report.

B5-Kick-off meeting minutes

AUDIT SUBJECT Performance Audit on the Library of University A

AUDITEE Library of University A

DATE OF MEETING 17.03.2016

PLACE OF MEETING Meeting room of University A Presidency

KATILIMCILAR

Name Title Signature

1 Burcu Demir Head of Internal Audit

2 Aylin Kaya Internal Auditor

3 Ömer Başkale Internal Auditor

4 Mehmet Akın Internal Auditor

5 Hasan Işık Expert – Library and Documentation Department

6 Mustafa Polat Expert – Computer Research and Implementation Center

7 Ahmet Aktaş Head of the Library and Documentation Department of University A

ISSUES RAISED DURING THE MEETING

During the kick off meeting the internal auditor provided the auditee with information on the following subjects

▪ Audit team and its organisation, audit scope, stages of audit

▪ Time schedule

▪ Identification of the comparison group

▪ Next steps

▪ List of requested data

During the kick-off meeting, the auditee expressed its expectations.

international university libraries are also used as a benchmark as part of the analysis on the library activities

universities without a faculty of medicine are included in the comparison group

human resources and physical/spatial conditions of the library are taken into account during the assessment.

since the student gateway of the library became operational towards the end of 2015 is still under development, conceptual design of the gateway is assessed within the report

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

21

ANNEX – 1: Kick-off meeting – Meeting notes

Topic Notes

Audit team and its organisation

It was stated that the performance audit will be carried out by the Internal Audit Unit of University A and the Library and Documentation Department of University A. Support of the IT expert of the Computer Research and Implementation Center (CRIC) will be taken in areas which require expertise on IT.

Audit Scope and Stages of Audit

Head of Internal Audit, Burcu Demir stated that the management takes into account the results of performance monitoring for all units within the scope of the performance program and that the units are being informed about the performance results.

Head of the Library and Documentation Department, Ahmet Aktaş stated that the human resources and physical/spatial conditions of the university must be taken into consideration while assessing the performance of the library.

IT expert of CRIC, Mustafa Polat stated that conceptual design of the student gateway which became operational in 2015 must be assessed as well.

Time schedule It was decided not to make any changes on the time schedule.

Identification of the comparison group

Expert librarian Hasan Işık underlined the importance of including international universities to the pool of benchmarks in the comparison group.

Internal auditor Ömer Başkale emphasized the necessity of including in the comparison group the universities without a medical faculty since University A does did not have one and the libraries of such universities had a much different collection of library resources.

Internal auditor Aylin Kaya stated that the comparison group must consist of universities whose language of instruction is English which is an important factor with direct impact on the library collection.

It was also mentioned that the universities to be selected following the filtering exercise must be among the top 10 in terms of academic publications and their libraries must be recognized by the world libraries union.

Finally, it was stated that University A had a student population of 10000-15000 and the universities in the comparison group must have similar populations.

Expert Hasan Işık stated that it was necessary to check whether the library of the university in the comparison group would be able to provide sufficient data.

Next steps

Information was provided related to the data requests, completion of the conceptual design and weekly progress meetings.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

20

B6. DEVELOPING THE AUDIT MATRIX As a result of the analyses and interviews conducted within the scope of the preliminary

work stage, internal auditors prepared the audit matrix which would set the general framework

of the field work.

RISK CONTROL TEST/ANALYSIS

Audit component

Audit question

Criteria Performance indicators

Data source Analysis method

Analysis strategy

PR

OD

UC

TIV

ITY

Does the library offer sufficient resource variety to enable the improvement of training and instruction?

University’s printed and electronic resource inventory is sufficient.

- Ratio of the

printed and electronic resource inventory of the university to the number of students

- Number of subscriptions to periodicals

Number of publications in the printed and electronic resource inventory

Benchmarking method

The number of research documents downloaded from the electronic data base from 2010 to 2014 will be provided. This data will be compared with: - the trends of 2011 to 2015 - Comparison group data.

E

FFEC

TIV

ENES

S

Is the level of borrowed resources from the university library sufficient to support academic research?

Library resources are sufficient to allow borrowing in a way to support academic research.

- Ratio of the number of borrowed printed materials to the number of library users.

- Ratio of the number of printed materials to the number of students and academic staff.

Number of borrowed resources according to years

Benchmarking method

The number of entries to the library’s online gateway from 2011 to 2015 will be provided. This data will be compared with: - the trends of 2011 to 2015 - comparison group data.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

22


Audit component

Audit question



Analysis strategy

PR

OD

UC

TIV

ITY

Does the library offer sufficient resource variety to enable the improvement of training and instruction?

University’s printed and electronic resource inventory is sufficient.

- Ratio of the

printed and electronic resource inventory of the university to the number of students

- Number of subscriptions to periodicals

Number of publications in the printed and electronic resource inventory

Benchmarking method

The number of research documents downloaded from the electronic data base from 2010 to 2014 will be provided. This data will be compared with: - the trends of 2011 to 2015 - Comparison group data.

E

FFEC

TIV

ENES

S

Is the level of borrowed resources from the university library sufficient to support academic research?

Library resources are sufficient to allow borrowing in a way to support academic research.

- Ratio of the number of borrowed printed materials to the number of library users.

- Ratio of the number of printed materials to the number of students and academic staff.

Number of borrowed resources according to years

Benchmarking method

The number of entries to the library’s online gateway from 2011 to 2015 will be provided. This data will be compared with: - the trends of 2011 to 2015 - comparison group data.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

23

B7. PREPARATION AND APPROVAL OF THE WORK PLAN

The work plan which was drafted following the preparation of the logical framework and

the audit matrix was approved by the audit supervisor.

B7- Work plan

SUBJECT OF THE AUDIT Auditing the Performance of the Library Process

AUDIT NUMBER

PLANNED DURATION OF AUDIT

Field work 30.3.2016 – 24.4.2016

Reporting 27.4.2016 – 15.5.2016

PURPOSE(S) OF AUDIT

In the audit to be carried out, it is aimed to evaluate the activities and practices related to the library process within the scope of the following audit components and to determine the development areas.

- Productivity: assessment of whether the level of outputs generated with the available resources is sufficient

- Effectiveness: capacity to realise the objectives and intended results - Consistency: the coherence between the goals, objectives and strategies of public

administrations and the planning and implementation of programs and projects, - Conformity: adequacy of the existing organisational capacity of public administrations

to conduct projects, programs, processes and activities

AUDIT SCOPE

The audit will cover the following sub-processes as well as practices and activities thereof, related to the library processes of University A in 2015.

- User services o User training and information o Reception and reservation o Shelf services o Borrowing section o Visual services o Audio services o Microfilm services

- Technical services o Provision o Cataloging and Classification o Technical affairs o Periodicals

▪ Services related to periodicals - current periodicals ▪ Services related to periodicals – hardback periodicals

- IT services - Administrative services - Financial services

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

24

AUDIT METHOD

The audit work will involve the auditing of the performance of library processes. The following

audit methods will be applied:

Me

tho

ds o

f A

na

lysis

Components

Econom

y

Pro

ductivity

E

ffectivene

ss/E

ffi

cie

ncy

C

on

sis

ten

cy

Susta

ina

bili

ty

Confo

rmity

Benchmarking

Measuring and Reporting on Performance

Assessment of Program and Implementation Results

Input-Output Analysis

Timeliness Analysis

Quality Analysis

INFORMATION ON THE PREVIOUS AUDIT

The findings of the system audit conducted in 2014 on the library process were examined during the preliminary work stage of the performance audit to be conducted.

PREPARATORY WORK

At this stage, preliminary data collection and analysis work was conducted. The following documents were prepared:

- Logical Framework of the Audit - Audit Matrix (Draft)

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

25

C. FIELD WORK

C1. DATA COLLECTION

During the data collection stage, the internal auditors looked into the data requested in

the opening meeting as well as the libraries of the universities which were planned to be included

in the benchmarking exercise. With the aim to find examples of good practices from Turkey and

abroad, the communication network of university libraries, publications and indices of the World

Libraries Union and the publications of the Higher Education Administration from Turkey were

investigated.

Following filters were used to select libraries to the comparison group. The resulting

comparison group was evaluated together with the officials in charge of the library process.

Filter 1

Filter 2

Filter 3

25 national

and

international

libraries

whose

accountability

reports can be

reached

15 Libraries 10 Libraries 7 Libraries

Filter 1

Filter 2

Filter 3

The comparison group involves 7 university libraries. The accountability reports of these

libraries were examined to understand whether they provide suitable data for the indicators that

are planned to be included in the audit scope.

Universities without a medical faculty. Universities whose language of instruction is English.

Top 10 Turkish universities in terms of academic publications. Universities whose libraries are recognized by the World Libraries Union

Universities with similar student populations as University A

Co

mp

ari

so

n

gro

up

lis

t

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

26

Indicators

Data availability

Univ.

B

Univ.

C

Univ.

D

Univ.

E

Univ.

F

Univ.

G

Univ.

H Ratio of library users to the number of students and

academic staff

Ratio of the number of printed materials to the number of students and academic staff

Change in the rates of borrowing printed materials from 2011 to 2015

Ratio of the number of borrowed printed materials to the number of users

Ratio of the budget allocated to purchasing new materials to the overall budget of the library

A A A A A A A

A A A A A A A

A A A A A A A

N/A N/A A A N/A N/A A

N/A A A A N/A N/A A

In light of this examination, universities D and H from Turkey, and university E from

abroad were included in the comparison group. It was also decided to use data from the last 5

years to compare the trends related to the key indicators of the library of University A. The

performance audit would focus on the components of productivity, effectiveness/efficiency,

consistency and conformity and the audit matrix was updated accordingly.

C2. DATA ANALYSIS

During the field work stage, secondary data was collected and University A was compared

with its own trends and with other university libraries.

# C2A1

Analysis

-7,6%

10,57 10,30 10,25 10,04

9,76 Ø 10

2011 2012 2013 2014 2015

Ratio of the library users (students and academic staff) to the number of annual visits to the library

Analysis method

Trend analysis

Existing situation

The examinations showed that the annual number of visits to the library by its users (students and academic staff) was in a downward trend from 2011 to 2015.

By 2015, the number of visits per user per year fell by 7.6% as compared to 2011 and reached 9.76.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

27

# C2A2

Analysis -15%

0,20 0,20 0,19

0,18 Ø 0,19

0,17

2011 2012 2013 2014 2015

Ratio of number of printed material borrowing to the number of library users

Analysis method Trend analysis

Existing situation

It was understood that the rate of borrowing printed materials from the library was in a downward trend between 2011 and 2015.

The rate of borrowing per student fell by 15% In 2015 and reached 0,17.

# C2A3

Analysis 19%

0,19 0,18 0,18

0,17 0,16

0,14 0,14

0,12 0,12 0,11

0,06 0,05 0,05 0,05

0,04

2011 2012 2013 2014 2015

Budget allocated to purchasing printed materials

Budget allocated to e-materials and e-databases

Budget allocated to purchasing new materials

Ratio of the budget allocated to purchasing new materials to the overall library budget – University A


Existing situation

It was seen that the share of the library’s budget allocated to purchasing new materials increased from 2011 to 2015.

The budget allocated to purchasing new materials is mainly spent on printed materials.

The budget allocated to printed materials have increased over the years.

The budget allocated to e-sources and e-databases has not changed significantly.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

28

# C2A4

Analysis

0,00%

7,00 7,00 7

6,00 6,00 6,00 6

5 6%

3,78 3,83 3,89 3,93 4,02 4

3 -11%

2 1,20 1,15 1,13 1,10 1,07

1

0

2011 2012 2013 2014 2015

Number of printed materials per student/academic staff

Number of e-sources per student/academic staff

Number of electronic databases to which the library has subscription

Ratio of the number of printed materials to the number of students and academic staff


Existing situation

The analysis revealed that;

The number of printed materials per student or academic staff member increased over the years.

The number of subscribed electronic databases did not increase.

The ratio of the number of electronic sources to the number of students and academic staff decreased by 11% between 2011-2015.

# C2A5

Analysis

8%

0,25 0,25 0,25 0,26

0,24 Ø 0,25

2011 2012 2013 2014 2015

Ratio of the number of subscriptions to periodicals to the number of students and academic staff.

Method Trend analysis

Existing situation

The analysis showed that the subscription to periodicals was in an upward trend between 2011 – 2015.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

29

# C2A6

Analysis

14,89

13,16

12,06 Ø 12

10,45

9,76

Uni A Uni D Uni E Uni H Average

Ratio of library users to the number of students and academic staff in 2015 (comparison group)

Method Benchmarking

Existing situation

The benchmarking analysis revealed that;

the ratio of library users to the number of students and academic staff in 2015 was lower compared to the university libraries within the comparison group.

# C2A7

Analysis 4,02

3,83

3,56 Ø 4

3,37

3,02


Ratio of the number of printed materials to the number of students and academic staff.

Method Benchmarking

Existing situation

The benchmarking analysis showed that;

The printed material inventory of University A was higher than those of the universities in the comparison group.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

37

# C2A8

Analysis 0

-0,13 -0,12

-0,15

-0,20

-0,26


Change in the rates of borrowing printed materials from 2011 to

2015 (Comparison group)

Method Benchmarking

Existing situation


From 2011 to 2015, the rate of borrowing printed materials was in a downward trend in all university libraries.

Library of the University A is the one with the sharpest decrease in this rate.

# C2A9

Analysis 0,19 0,19

0,16 0,16

0,14

Ø 0,12

0,09 0,08

0,07 0,07

0,05

Uni A Uni D Uni E Uni H Av.

Ratio of printed materials to the #of students and academic

staff Ratio of the budget allocated to e-sources

and databases

Ratio of the budget allocated to purchasing new materials to the

overall budget of the library (comparison group)

Method Benchmarking

Existing situation


In 2015, the budget allocated to purchasing new materials was higher in University A than the average of the comparison group.

The budget allocated to purchasing e-sources and e-databases on the other hand was lower than the group average in 2015.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

30

The analysis of the secondary data was followed by an analysis of the primary data related

to the below-mentioned stakeholders. The method of collecting primary data and the stakeholder

groups are stated below.

# C2B1

Stakeholder Academic staff

Analysis method Focus group meeting

Analysis strategy

Participants were chosen from all faculties of the university, for a focus group meeting on library processes.

It was requested that the participants would be identified by the faculty management.

The number of academic members representing each faculty was proportional to the overall number of academic members in that faculty.

The subject of the focus group meeting, its content, general topics to be discussed and the assessment method to be applied were communicated to the participants beforehand.

Existing situation

As a result of the focus group meeting, following conclusions were made:

Majority of the academic staff have subscriptions to e-databases outside the university library.

The main reasons for this are stated below:

The inventory of the e-database in the university library is limited.

The library may purchase printed materials upon demand from the academic staff members. The budget for this type of purchases is sufficient. Such demands from academic staff members are generally accepted. However, due to the procedures in the library regulation and the time taking procedure of acquiring printed materials, the process may take a long time.

For this reason, the academic staff members prefer to have memberships in the e-databases so that they can have quick access to the resources they need in their academic work.

Individual purchase of these memberships by many academic staff members in sum reaches a much higher amount compared to the general membership of the university library.

Academic members therefore conduct their work by using the resources and databases outside the library.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

32

# C2B2

Stakeholder group

Library staff

Analysis method In-depth interview

Analysis strategy

Face to face interviews were conducted with the library staff members in consideration of their experiences and job titles.

In this context, 20 out of the 75 staff members were interviewed with.

The responses given by staff members to pre-determined questions were analysed.

Existing situation

Following conclusions were reached:

The areas which may have an impact on the satisfaction of the students using the library and their rate of using the library were analysed during the interviews. These areas are summarized as follows:

Performance indicators of the library staff have not been sufficiently identified. This results in a situation where good performance is not rewarded and the actual performance may vary depending on the sense of responsibility of the staff members.

Due to lack of resources related to the IT services in the library user demands in this area cannot be met at a satisfactory level.

Since the qualities that a library professional must possess, such as experience, command of foreign languages, education level, etc. are not stated in the library regulation, the existing human resources of the library may lack such qualities.

# C2B3

Stakeholder group

Students

Analysis method Questionnaire

Analysis strategy

A questionnaire consisting of 5 questions was prepared for all students.

The questionnaire was published on the web site of the university. 410 students responded via this medium.

Also, face to face interviews were conducted within the library and 140 students answered the questionnaire.

Existing situation

The results of the questionnaire are as follows:

The data sources used by students while conducting a research are listed as follows:

Internet (97%)

Publicly available electronic books and academic studies (90%)

Online databases and resources (78%)

Printed publications available in the library (25%)

The library should focus more on the following resources in its inventory. (The questionnaire allowed selecting more than one option.)

Online memberships (80%)

E-databases (75%)

Printed periodicals (90%)

Printed books (40%)

Only 40% of the students are aware that they can ask the library to acquire e-resources or printed materials. The number of students who know how to place such requests constitutes 10% of the students who have responded to the questionnaire.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

33

# C2B4

Stakeholder group Students

Analysis method

Questionnaire /

Coding

Analysis strategy

One of the questions in the questionnaire asked the students to list the most frequent problems they encounter related to library services.

Coding method was used to analyse the responses. In this exercise, the answers provided were assessed through the below-mentioned scores.

The problems stated by participants were scored according to the ranking provided by participants.

In this regard;

- The problem at the top of the list gets 3 points

- The second problem gets 2 points and

- The third problem gets 1 point.

Existing situation

Using the results of the questionnaire, 10 basic problems faced by library users were identified. These problems were assessed by using the coding method defined in the analysis strategy and the resulting first five problems are stated below.

# C2B5


Analysis method

Control group analysis

Analysis strategy

Whether the students participating in the questionnaire have taken the library orientation guidance or not was identified.

Regardless of whether they took the guidance, the students were asked whether they visited the library in the first year of their education.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

Definition of the problem

Number of users

indicating the problem

Value reached through coding

#

Problems related to remote access to the library system

240 540 2

Insufficient library databases 210 565 1

Unavailability of searched printed materials

50 75 5

Insufficient guidance by library staff

81 222 4

Poor physical conditions 95 280 3

34

Ory

an

tasy

on

Eğ

itim

ine

Ka

tıla

n Ö

ğre

nci

leri

n İl

k Y

ıl K

ütü

ph

an

e

Ku

lla

nım

Ora

nla

rı

# C2B6


Analysis method

Bi-variate analysis

Analysis strategy

The library usage rate and the ratio of students who have taken the orientation training to the total number of students were compared to analyse the relationship between these variables. The dependent variable was set as the library usage rate (which is found by dividing the total number of visits to the library to the number of students in the university). Number of students who have taken the orientation training was taken from the library data.

Existing situation

Relationship between the orientation training and library usage rate

74%

73%

72%

71%

70%

Library Usage rate

69% Projected library usage rate

68%

67%

66%

54,00% 56,00% 58,00% 60,00% 62,00% 64,00% 66,00%

Ratio of students who have taken the orientation training to the entire student population

The analysis shows that there is high correlation between the library usage rate and the participation to orientation training (the calculations result in a rate of 85%).

Existing situation

2014

Students who have taken the orientation guidance Students who have not taken the orientation guidance

The students who took and who did not take the library orientation guidance between 2010-2015 were asked if they visited te library in their first year.

The results revealed that 70% of the students who took the guidance visited the library at least once during their first year at school. For students who did not take the guidance, this rate is 39%.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

35

C3. IDENTIFICATION OF FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS

C3.1. Analysing the results of the benchmarking exercise and sharing the results with the management

Following conclusions were reached as a result of the benchmarking exercise. The results

were assessed together with the process owners.

# Analysis topic Analysis result Finding Working paper

C3.1.1

Library usage rate

The library usage rate was in a downward trend between 2011 and 2015 and in 2015 reached 9,76 per user after a 7,6% decrease compared to 2011.

Yes

C2A1

C3.1.2

Rate of borrowing publications

Rate of borrowing printed publications was in a downward trend from 2011 to 2015 among students and academic staff. The decline in 2015 was 15% as compared to 2011.

Yes

C2A2

C3.1.3

Purchase of new

resources

The share of the budget allocated to the purchase of new resources increased from 2011 to 2015.

This budget is mainly allocated to the purchase of printed materials.

Yes

C2A3

C3.1.4

Library inventory

Number of printed materials per student or academic staff member increased throughout the years.

The number of e-database memberships also increased.

The ratio of number of electronic resources to the number of students and academic staff fell by 11% from 2011 to 2015.

Yes

C2A4

C3.1.5 Library inventory It was understood that the number of memberships

to periodicals increased from 2011 to 2015.

C2A5

C3.1.6

Library usage rate

In 2015, library of the University A was visited by fewer users than the libraries of the universities within the benchmark group.

Yes

C2A6

C3.1.7

Library inventory

In 2015, the printed publications inventory of Library A was higher than the inventories of the benchmark libraries.

C2A7

C3.1.8

Rate of borrowing printed materials.

Between 2011 and 2015, there was a downward trend in borrowing printed materials in all university libraries.

In this period, Library A experienced the sharpest fall within the benchmark group.

Yes

C2A8

C3.1.9

Purchase of new

resources

In 2015, library of University A allocated a higher budget than the other libraries within the benchmark group for the purchase of new materials.

On the other hand, the budget allocated for e-resources and e-databases in 2015 was lower than the average of the benchmark group.

Yes

C2A9

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

36


C3.1.10

Focus group meeting (Academic members)

- Library usage rate

- Library inventory

- Purchase of new resources

It has been understood that a major part of the academic members have subscriptions to e-databases outside the university library.

According to the academic members, the main reasons for this are as follows:

The inventory of the e-database of the university is limited.

The library has sufficient financial means to purchase printed materials upon the demand of the academic members. Such demands are met positively most of the time. However, due to the procedures provided in the regulation of the library and the time taking nature of the acquisition of new books, this process may take a long time.

For this reason, the academic members prefer to subscribe to e-databases which provide the fastest access to the materials they need in conducting their researches.

When the money spent by individual academic members for such subscriptions are put together, the overall sum is much greater than the amount that would be spent by the library for collective memberships.

For the above reasons, the academic members are conducting their academic studies over the databases and resources outside the university library.

Yes

C2B1

C3.1.11

In depth interviews

(Library staff)

- User satisfaction

- Operations

As a result of the interviews, the areas that might affect the satisfaction and library usage rate of students have been identified as follows:

Performance indicators of the library staff have not been sufficiently identified. This results in a situation where good performance is not rewarded and the actual performance may vary depending on the sense of responsibility of the staff members.

Due to lack of resources related to the IT services in the library user demands in this area cannot be met at a satisfactory level.

Since the qualities that a library professional must possess, such as experience, command of foreign languages, education level, etc. are not stated in the library regulation, the existing human resources of the library may lack such qualities.

Yes

C2B2

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

37


C3.1.12

Questionnaire

(Students)

- User satisfaction

- Operations

The data sources used by students while conducting a research are listed as follows:

Internet (97%)

Publicly available electronic books and academic studies (90%)

Online databases and resources (78%)

Printed publications available in the library (25%)

The library should focus more on the following resources in its inventory. (The questionnaire allowed selecting more than one option.)

Online memberships (80%)

E-databases (75%)

Printed periodicals (90%)

Printed books (40%)

Only 40% of the students are aware that they can ask the library to acquire e-resources or printed materials. The number of students who know how to place such requests constitutes 10% of the students who have responded to the questionnaire.

For students, the most common shortcomings related to the library services are as follows:

Problems related to remote access to the library system (%88)

Insufficient library databases (%65)

Unavailability of searched printed materials (%20)

Insufficient guidance by library staff (%30)

Poor physical conditions (%30)

Yes

C2B3, C2B4

C3.2. Analysis of the findings, risks and root causes that have emerged as a result of the benchmarking exercise

Several findings have been identified as a result of the analysis. These findings are

grouped under different headings. A fishbone diagram has been used to analyse the causes of the

identified finding.

The analyses and interviews revealed that the proportion of the budget allocated by the

university for library activities within the overall budget increased from 2011 to 2015. In terms of its

inventory and the budget allocated for acquiring new resources, the library is among the top

university libraries both at national and international levels.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

38

The library usage rate however is in a downward trend. When the library of University A is

compared with the other university libraries in terms of the library usage rate and the rate of

borrowing resources, the performance of library A is lower.

The reasons of the decline in the library usage rate and the rate of borrowing were assessed

under four main headings.

Budget and fund use

Expectations of academic members

Management and organization

Expectations of students

BUDGET AND FUND USE

ACADEMIC M.

Academics

Insufficient number of e-resource and e-database memberships

Priority given to printed materials when

Failure to consider user demands when acquiring new resources

preferring to subscribe to e-databases that are not available

Long procedure for requesting the acquisition of

acquiring new resources.

in the inventory of the library.

new resources.

Performance indicators of the library staff not sufficiently identified.

Shortcomings related to remote access to library system

Lack of sufficient personnel in the IT department of the library

Students preferring e-databases and electronic resources over the printed ones

Lack of awareness among students about the new resource request process.

MANAGEMENT AND ORGANIZATION

STUDENTS

- Library

usage rate showing a downward trend over the years

- Decline in the demands to borrow books

- Library inventory failing to meet user expectations.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

39

Reasons Working paper

Areas of development

Budget and fund use

Insufficient number of e-resource and e-database memberships

Priority given to printed materials when acquiring new resources.

Failure to consider user demands when acquiring new resources

C2A4

C2A9

Budget and fund use

User demands and current needs must be taken into account when establishing the budget for the acquisition of new resources for the library. In this regard, the budget allocated to the acquisition of e-resources and databases must be increased.

Expectations of academics

Academics preferring to subscribe to e-databases that are not available in the inventory of the library.

Long procedure for requesting the acquisition of new resources.

C2B1


The library must plan to subscribe to the electronic databases that are most frequently used by the academics.

The system for acquiring new resources must be strengthened and the procedure for requesting new resources must be simplified.

Student expectations

Students preferring e-databases and electronic resources over the printed ones

Lack of awareness among students about the new resource request process.

C2B3

C2B4

Student expectations

The number of electronic databases and other electronic resources within the library’s inventory must be increased.

Students must be informed about the procedure for the acquisition of new resources to the library.


Performance indicators of the library staff not sufficiently identified.

Shortcomings related to remote access to library system

Lack of sufficient personnel in the IT department of the library

C2B2


Necessary performance indicators for the library staff must be identified and monitored on a regular basis.

Effective remote access to the library’s system must be ensured.

Sufficient number of competent staff must be recruited to the IT department of the library.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

43

Kıy

asla

ma

Gru

bu

Lis

tesi

D. REPORTING

D1. OFFICIAL SHARING OF FINDINGS

1 FINDING SHARING FORM

Subject of the finding

Downward trend of library use rate and requests for borrowing

publications by years

Level of importance

HIGH

Relevant unit Department of Library and Documentation

Current situation

Within the scope of the audit conducted, analyses on students’ and academics’ library use rate and borrowed source request rates were conducted in line with the following framework.

- 2011 – 2015 trends of the library of University A.

- Performance realisations of the library of University A were compared with the comparison group determined together with the process owners.

- A long list of 25 universities, either national or international, was determined for the comparison group. A short list of 7 libraries, out of the mentioned long list, meeting the following criteria was developed. Libraries D, E and H from this short list were taken into the comparison group within the framework data availability.

Filter 1

Filter 2

Filter 3

Faaliyet

Raporuna

Ulaşılan 25

Ulusal ve 15 Lİbraries 10 Libraries 7 Libraries

Uluslararası

Küt üphane

Filter 3

Filter 2 Filter 1

The issues detected as a result of the analyses conducted are as follows. .

1. Library use rate of University A had a downward trend in 2011-2015.

As indicated above, average number of library visits of a student or an academic within a year fell by 7,6% from 2011 to 2015 to reach 9,76.

Yearly average library use rate was 12.06 in 2015 in the libraries in the benchmark group.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

Lis

t o

f B

en

ch

ma

rk

Gro

up

25 National

and International libraries providing access to

accountability report

Universities with no Medical School.

Universities whose language of instruction is English.

University libraries ranking in top 10 in Turkey in terms of academic publications.

Universities internationally recognized by World Union of Libraries.

Universities close to

University A in terms of number of students

40

10,57

10,30

-7,6%

10,25

10,04

9,76

Ø 10

2011 2012 2013 2014 2015

Ratio of number of library users (students and academic personnel) to the number of yearly entries to the library

9,76

10,45

14,89

13,16

12,06

Ø 12

A Üni. D Üni. E Üni. H Üni. Ort.

Ratio of number of library users to the number of students and academic personnel in 2015 (Benchmark Group)

2. The assessment on the requests for borrowing revealed that a library user requested to borrow 0.22 printed sources on average in 2011. This rate had a downward trend from 2011 to 2015 and was 0.17 in 2015.

2 The Ratios indicated in the tables were calculated by dividing the number of printed materials borrowed within the year by the total number of users. Number of users is the number of students and academics who are subscribed to the library.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

42

-15%

0,20 0,20 0,19

0,18 Ø 0,19 0,17

2011 2012 2013 2014 2015

Ratio of the cases of borrowing printed materials to the number of library users.

0

-0,13 -0,12

-0,15

-0,20

-0,26

A Üni. D Üni. E Üni. H Üni. Ort.

Change in the rate of borrowing printed materials from 2011 to 2015 (Benchmark group)

In addition, assessment of the current library implementations and the other university libraries included in benchmark group revealed that requests from the libraries for borrowing books had a downward trend. When the change in the requests for borrowing publications in the benchmark group from 2011 to 2015 was analysed, it was observed that requests for borrowing sources (including periodicals) in the benchmark group decreased by 15% on average. The rate of decline was 26% in University A.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

43

3. It was found that the budget allocated for purchasing new resources increased year by year.

Allocated budget was utilised for purchasing electronic resources and printed resources. Priority was given to the purchase of printed resources.

As stated above, as a result of the analyses, in 2011 -2015;

- Number of printed materials per student increased by 6%.

- Number of electronic resources per student decreased by 11%.

- Number of memberships to e-database remained the same.

0,16

0,11

0,17

0,12

19%

0,18

0,12

0,18

0,14

0,19

0,14

0,05

0,05 0,06

0,04

0,05

2011

2012

2013

2014 2015Budget allocated to the purchasing of printed resources

Budget allocated to e-resources and e-databases

Budget allocated to the purchasing of new materials

Ratio of the budget allocated to the purchasing of new materials to the overall budget of Library A

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

44

0,00%

7,00 7,00 7

6,00 6,00 6,00 6

5 6%

3,78 3,83 3,89 3,93 4,02 4

3 -11%

2 1,20 1,15 1,13 1,10 1,07

1

0

2011 2012 2013 2014 2015

Number of printed materials per student/academic member

Number of e-sources per student/academic member

Number of memberships to e-databases

Ratio of the number of printed materials to the number of students and academic members

Consequently, although global budget of the university library and budgets for purchase of new resources increase every year, library use rates and requests for borrowing books are at a decline. Current performance results fall behind the performance results of the universities included in the benchmark group.

Cause As a result of trend analyses and best practices comparisons, root causes of downward trend of university library use frequency and decrease in requests of borrowed publications are assessed under 4 main headlines:

I. Student expectations

As a result of the questionnaires conducted with the university students, the factors causing decrease in library use rates and borrowing rates are listed as follows:

▪ The questionnaire revealed that, in an academic research 78% of the students preferred using e-databases and online resources, while the ratio of students preferring using printed resources via the library was 25%.

▪ 75% of the students believe that e-resources and e-databases should be focused on in library inventory.

▪ Only 40% of the students know that they can request the acquisition of any e-resource or printed publication from the library.

▪ Ratio of the students who know how to request acquisition of e-source or printed publication from the library is 10%.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

45

Cause II. Expectations of the academics

As a result of the focus group meetings held with the academics selected by the faculty management, in line with the academic personnel breakdown of the faculties, from among the academics conducting studies in the faculties, the factors causing a decrease in use of library and borrowing were listed as follows:

▪ It was understood that most of the academics had individual memberships to the e-databases other than the university library due to limited e-database inventory of the university and book provision process taking too long following their request for a printed source.

▪ This results in decrease of university library use by academics.

III. Management and organization

In the in-depth interviews made with library personnel, the fields which may affect the satisfaction of the students, who are users of the library, and library use rates were summarized as below.

▪ Performance indicators of the library personnel were not adequately defined. This results in changes in performance depending on the employees’ devotion and responsibility.

▪ Insufficient resources of the library informatics services result in an inability to adequately meet the requests of the users in terms of information systems.

▪ Lack of remote access to the library system makes it harder for the students to scan available source inventory of the library.

▪ It was found out that library human resources are inadequate since the library by-law did not set out the criteria sought in library personnel such as experience, foreign language knowledge and education level.

IV. Budget and fund use

▪ The examinations revealed that budget allocated to purchase of new sources for the library increased by 19 % from 2011 to 2015. Priority was given to the purchase of printed publications. Budget allocated to e-source and e-database did not change over years.

▪ Budget utilisation may not be consistent with the expectations of the users due to absence of systematic infrastructure and awareness required for receiving the expectations of the students and academics regarding purchase of new sources.

Risks and impacts

In its 2016 – 2020 strategic plan, University A targeted increasing its recognition level and being placed among top 100 universities which are followed, both nationally and internationally, for academic publications.

Failure to diversify the university inventory in line with the improving student requests and decrease in library use may result in a decrease in academic publications of the university and inability to increase the recognition of the university in national and international arena. This may prevent the university from achieving its relevant strategic objectives.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

46

Recommendations

Following actions are recommended with regard to the finding detected within the framework of the audit:

1. Necessary questionnaires, pre-request process and focus group meetings are recommended to be performed at least twice a year with the aim of efficiently taking the expectations of the students and academics while determining what type of sources will be purchased with the library budget allocated to this purpose.

2. The system for requesting new resources should be delivered online, through the library portal, to increase the demands of students and academics for new resources and to provide rapid response to such demands. This system is recommended to meet certain criteria including but not limited to the following. - Ability to make new resource requests with student IDs

- Regularly notifying the applicant about the progress made on resource request

- Sending the student informative e-mails when the decision to provide the resource is taken and when the resource is actually provided

3. New resource requesting procedure is recommended to be updated to increase and rapidly meet the new resource requests of the students and academics. New resource request procedure should include, but is not limited to, the system to be used in request receiving process (Library Management System and University Web Site) and tools (Questionnaire, focus group meetings, etc.).

4. With the aim of increasing the awareness on new source request process, necessary informative activities and training are recommended to be periodically held via university portal and other promotion channels at least once in a semester.

5. As for recruitment of library personnel, it is recommended that minimum criteria sought from the candidate to the vacant positon be stated in the library by-law.

6. It is recommended to plan human resources and technical support required for improving the library system and ensuring remote access to the system

Criterion Criteria set within the framework of the root causes can be listed as follows.

Budget and fund use

Requests of the users and current needs should be considered in the budget allocated from library budget to purchase of new sources. In this framework, e-resources and databases must be focused on while purchasing new sources.


Library must plan institutional subscriptions to the e-databases mostly used by the academics.

Systematic infrastructure required for supply of new sources should be strengthened and new resource request procedure should be simplified.

Expectations of students

Ratio of e-database and other electronic sources included in library inventory should be increased.

Students should be informed and trained about the process of requesting acquisition of new sources to the library.


Performance indicators for library personnel should be set and regularly followed.

Remote access to library system should be ensured efficiently.

Sufficient number of competent personnel should be employed in the branch of library communication services.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

47

1

Responsible person

Action to be taken Completion

date


Necessary questionnaires, pre-request process and focus group meetings will be planned to be performed twice a year starting in fall with the aim of efficiently taking the expectations of the students and academics while determining what type of resources will be purchased with the library budget allocated to this purpose.

Department of Library and Documentation / Department of Information Processing

System design required for new resource request system running online via library portal. This system will be designed in a way to meet the following requests.

- Ability to make new resource requests with student ID information

- Regularly notifying the applicant about the progress made on resource request

- Sending the student informative e-mails when the resource is decided to be acquired and when the resource is acquired


New resource request procedure will be updated. Updated procedure will be shared with the students during the registration periods. In addition, informative e-mails will be sent at the beginning of each semester.


Minimum criteria sought from the library personnel will be determined and shared with the Office of the University President.

Department of Library and Documentation /

Department of Information Processing

Human resources and technical support required for improving the library system and ensuring remote access to the system will be planned and relevant budget requirement will be determined.

Necessary action will be taken after the budget request is approved by the Office of the University President.

Opinion of the auditee

[X] We agree with the finding.

[ ] We do not agree with the finding.

[X] We agree with the finding.


[ ] We do not agree with the importance level of the finding.

A Ü

niv

ers

ites

i Kü

tüp

han

e Sü

reci

Per

form

ans

Den

etim

i

48

D2. CLOSING MEETING

The audit team provided the auditee with information on the following topics in the closing

meeting:

▪ Information on the analyses and studies performed

▪ Sharing the findings with the people responsible for relevant processes

▪ Analysis of the root causes of the findings

▪ Identification of areas of improvement in line with the findings and development of recommendations

D2-Minutes of the closing meeting

AUDIT SUBJECT Performance Audit on the Library of University A

AUDITEE Library of University A


PLACE OF MEETING Meeting room of the office of the university president

PARTICIPANTS






5 Hasan Işık Expert – Department of Library and Documentation

6 Mustafa Polat Expert - BAUM

7 Ahmet Aktaş Head of the Library and Documentation Department of University A

ISSUES RAISED

The audit team provided the auditee with information on the following topics in the closing meeting:

Information on the analyses and studies performed

Sharing the findings with the people responsible for relevant processes

Analysis of the root causes of the findings

Identification of areas of improvement in line with the findings and development of recommendations.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

49

Performance Audit of the Bus

Operations of Municipality A

The content of the case study is not related to any institution, practice or person. The content is purely fictitious.

50

51

PERFORMANCE AUDIT OF THE BUS OPERATIONS OF MUNICIPALITY A

Municipality A is a medium size municipality providing transport services to 500 thousand

citizens per annum.

The municipality runs satisfaction surveys related to this service with regular intervals.

However, they have not yet performed a self-assessment study to identify the measures to be

taken in line with the survey results.

In its macro level risk assessment exercise, the internal audit unit focused on the bus

operations process, which is included in the audit universe, from the perspective of risk factors

and also by taking into account the goals and objectives of the strategic plan and the opinions of

the top manager and other senior managers. As a result of this assessment, the internal audit unit

decided to include the bus operations process in its annual audit program for 2016.

In 2014, the top management of the municipality purchased buses equipped with

technological features providing accessibility to citizens with disabilities. However, the survey

results show that citizens with disabilities in particular are not satisfied at the desired level. The

top management asked the internal audit unit to analyse the root causes of this situation during

audit exercise.

The internal audit unit of the municipality decided to carry out a performance audit on

the bus operations process, in consideration of the goals and expectations of the top

management. The audit in question was included in the 2016 audit program.

The audit was performed in line with the framework provided within the Performance

Audit Manual and Public Internal Audit Manual.

52


A1. ASSIGNMENT




took into account the following elements:





Internal auditors assigned by the head of internal audit unit signed the impartiality and

confidentiality documents.

A3. NOTIFICATION TO THE AUDITEE



A1.Engagement

Nr. : 76995536-662.02- 14/02/2016


Ms. Aylin Kaya (6666/A3) Internal Auditor

You have been assigned to audit the bus operations process of the Department of

Transportation Services. You are kindly requested to perform the audit in line with the Public

Internal Audit Standards, the Public Internal Audit Manual and the Performance Audit Manual and

to submit the audit report to the Internal Audit Unit.

Head of Internal Audit

Type of Audit Performance Audit of the bus operations process of the Department of Transportation Services


Planned Audit Period 26.02.2016-15.05.2016

Other Internal Auditors Assigned


Audit Supervisor Ömer Başkale (5555/A3)

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

53

A2. Preparation of the Impartiality and Confidentiality Document


Audited Unit: Department of Transportation Services

Audit subject: Bus Operations

Performance Audit on the Bus Operations Process

19.1. Passenger Services Process

19.2. Maintenance and Breakdown Follow-Up Process

19.3. Informatics and Technical Services Process

19.4. Administrative and Financial Services Process

All activities of the Transportation Services Department taking place from 01.01.2015 to 31.12.2015



- none of my first, second and third degree relatives by blood and by law are employed by the audited unit,




26.02.2016

Aylin Kaya

Internal Auditor

DECLARATION

TITLE OF THE AUDIT

AUDIT SCOPE

COMMITTMENT

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

54

A3. Notification to the audited unit

Nr. : 76995536-679- ..../02/2016

Subject : Audit Notification

TO THE DEPARTMENT OF TRANSPORTATION SERVICES

Within the scope of the 2016 Internal Audit Program carried out in accordance with the

approval of the Office of the Mayor signed on 20.02.2016, a performance audit will take place in your

unit from 26.02.2016 to 15.05.2016, to assess the bus operations processes. The audit in question will

be launched on 26.02.2016 and is planned to be completed on 15.05.2016.


(5555/A3), by internal auditors Aylin Kaya (6666/ A3) and Mehmet Akın (4444/A3). You will receive the

findings and the report to be drafted at the end of the audit.

The audit scope in general consists of the bus operations; the exact scope and audit objectives

will be determined on the basis of the discussions between our audit team and your unit.







Kindly submitted for your information and necessary action.

Head of the Internal Audit Department

CIRCULATION

To the attention of

Department of Transportation Services

Information




Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

55

Identification of Audit Components and Methods

Kick off meeting

Understanding

the Logical

Framework

Audit Matrix Preparation and

approval of the Work

Plan

B. PRELIMINARY WORK At the preliminary work stage of the audit, the following steps set forth in the Public


1 2 3 4 5 6 7 Assessment of

the Performance

Management

System

B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE The audit supervisor, in consultancy with the internal auditors within the audit team, prepared the audit engagement time schedule form. In addition to the form, he prepared a detailed audit plan which also shows the interim stages of the audit

Task Output Start End date date

1 2 3 4 5 6 7 8 9 10 11 12

Wee

k

Wee

k

wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

1 Launching of the audit engagement 26.02.16 26.02.16

Assignment Assignment Letter Preparation of the Impartiality and Confidentiality Document Impartiality and

Confidentiality Document

Notification to the Auditee Audit Notification Letter 2

Preliminary work 02.03.16 27.03.16 Audit Engagement Time Schedule Form Audit Engagement Time

Schedule Form

Performance Management System Maturity Analysis Maturity Analysis Form Understanding the Logical Framework Requesting Preliminary Data Preliminary Data Request

Form

Preliminary Analysis and Assessments Documenting the Logical Framework Logical Framework Form

Identification of Audit Components and Methods Audit Matrix Kick-off meeting Kick-off meeting minutes Drafting the Audit Matrix Audit Matrix Preparation of the Work Plan Work Plan

3 Field work 30.03.16 24.04.16

Data collection Requesting for data Data Request List Establishing the Audit Data Base Audit Data Base Analysis Identifying the findings and developing recommendations Consolidated list of findings

4 Reporting 27.04.16 15.05.16

Official sharing of findings Findings form Closing meeting Closing meeting minutes

Taking action plans Action Plan Audit report Audit Report

Developing the

Audit

Engagement

Time Schedule

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

56

B1- AUDIT TIME PLAN FORM

Performance Audit of the Bus Operations Process

Planned Realised Start

date End

date

Start date

End

date PRELIMINARY WORK 02.03.2

016 27.03.2

016

Performance Management System Maturity Analysis 03.03.2

016


09.03.2016

Kick-off meeting 17.03.2016

Approval of the work plan (Engagement Work Program and Audit Matrix)

27.03.2

016

FIELD WORK 30.03.2016

24.04.2016

Collecting data and launching the analyses (tests)

30.03.2016

Completing the analyses (tests)

24.04.2

016

REPORTING 27.04.2016

15.05.2016

Preparing the findings and developing the recommendation

27.04.2016

Communicating the findings 01.05.2016

Closing meeting 08.05.2016

Presenting the audit report 15.05.2016


The internal auditor assessed the maturity of the performance management system

of the audit field in order to ensure that the results of the performance audit exercise to

be conducted are successful. In this regard, the maturity of the audit field was analysed

on the basis of the following topics:

▪ Assessment of the maturity of the performance management system,

▪ Examination of whether the area to be audited is suitable for performance audit,

▪ Evaluation of whether the performance audit components are applicable on the area to

be audited.

In the assessment of the maturity of the performance management system, the

maturity analysis table provided in the Performance Audit Manual was used. The

internal auditor asked the following questions in this assessment.

Are the goals and objectives defined at the level of the administration, its units and activities?

Are there performance indicators related with these goals and objectives?

Are the goals and objectives at the administration, unit, process, program and activity

level and related performance indicators being monitored through a system?

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

57

Is data related to performance indicators being monitored and reported?

The preliminary study revealed that certain goals and objectives were defined at the level

of units and activities of the Municipality. However, performance indicators related to these

goals and objectives were not sufficiently identified. Some of the performance indicators in the

Transportation Services Department, the auditee, were set forth by the management. No

mechanism is available to regularly report on and monitor the performance indicators. Reports

have been designed to monitor the results of the performance indicators that were tied to the

process in question. However the reporting mechanism only becomes functional to meet the

periodical demands of the manager.

To give an example; performance indicators such as the “accessibility of bus stops” or

“the sufficiency of buses in terms of getting on an off easily” are included in the satisfaction

surveys however, the municipality has not conducted any study related to the actions that

need to be taken in accordance with the survey results. In addition to the performance

indicators related to the goals and objectives of the performance program and the strategic

plan, other performance indicators have been developed by the management of the

department of transportation services. As a result of the interviews, the internal auditors

came up with the following scores in the maturity analysis assessment.

Planning and implementation

Weight coefficient

%30


2 Only a few goals and objectives are defined in the strategic plan of the administration.

3 Goals and objectives are set out in the strategic plan of the administration.

Distribution of tasks within relevant units is at basic level.

4

The goals and objectives are set out in the strategic plan of the administration at a reasonable level and they are related with each other and with performance indicators.


Resource allocation is planned in line with goals and objectives.

5

The goals and objectives are set out in detail in the strategic plan of the administration and they are related with each other and with comprehensive performance indicators.

Tasks are distribution to related units in a detailed manner. Resource allocation is planned in line with goals and objectives.

A Monitoring and follow up mechanism exists for the strategic plan.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

58

Performance indicators

Weight

coefficient

%40


2 Some indicators have been set out for monitoring performance however

they are not comprehensive nor systematic.3

3


The performance programs do not show the connection of indicators with the objectives of relevant units

4



5



A system to monitor indicators has been established and a system for reporting to make revisions where necessary is in place

Monitoring and reporting

Weight coefficient

%20

1 Performance results of the organisation are not monitored.

2 Performance is measured for some indicators, but not on a regular basis.

3 Performance results related to all performance indicators set out

in the strategic plan are monitored

4

Performance is measured for all activities and processes on a regular basis and the results are shared with related people.

5


3 Indicators are in line with SMART criteria. See: 3.5.2.3. Understanding the logical framework

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

59


Weight

coefficient

%10

1 Monitoring results are not reported to the management.

2 Management takes into account the results of monitoring for only some of

the units and activities.

3 Management takes into account the results of performance monitoring for

all units within the scope of the performance program.

4



5




Weight coefficient

Score Weighted score

Total score

Design and implementation of the strategic plan and activity program

30% 2 0,6

2,0

Performance indicators 40% 2 0,8


Management awareness and support 10% 2 0,2

The maturity level of the performance management system in the administration is

calculated as 2. This score shows that the level of available data and the performance

management mechanisms of the administration may not be sufficient enough to conduct an

effective performance audit.

The internal auditors believe that the audit may be conducted by identifying certain

performance indicators together with the management to be able to measure and report on

performance, and by using these indicators during the audit exercise.

After this assessment the internal auditors continued with the stage that concerns the

understanding of the logical framework.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

60


At the preliminary work stage, the internal auditors prepared the list of preliminary data needed

to carry out the necessary analyses and assessments for understanding the bus operations

process. They gave the list that contains the preliminary data they need from the department,

apart from those that are publicly available.

The list that contains the data which the internal auditors are planning to examine is as follows.

# Document or information

1 Strategic Plan of Municipality A (2015-2019)

2 Strategic Plan of Municipality A (2010-2014)

3 Performance Program of Municipality A (2010, 2011, 2012, 2013, 2014, 2015)

4 Bus Operations Process Flow Charts

5 Bus Operations Sub Processes

6 Bus Operations Satisfaction Surveys 2013-2015

Internal auditors also used publicly available documents and reports during the preliminary

work stage. The reports and documents examined are as follows.

▪ Results of the Mystery Shopper and User Satisfaction Surveys of Different Municipality Transport Services

▪ Union of Municipalities – Smart City Services Report

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

61

The logical framework to be applied during the audit is provided below:

Purpose

Goal

Objective

Activities

Performance Indicators

Source of the

indicator

Municipality Strategic Plan Purpose 1. To extend the use of information technologies in internal and external public relations.

Goal 1.1 To use the information technologies at the highest level possible

- Providing the opportunity to follow the municipality services via the website. - Tracking of municipality service vehicles on a digital platform

Public awareness about the services delivered online

Satisfaction surveys, Municipality Strategic Plan (2014-2016)

Number of municipality vehicles equipped with the tracking system

Municipality Strategic Plan (2014-2016)

Municipality Performance Program Purpose 2. To ensure accessibility of all citizens to the transportation services delivered by the municipality

Goal 2.1. To make sure that disadvantaged groups benefit from the transportation services at the same level as all other citizens.

-To increase

the service usage rate of disadvantaged groups - To increase the quality of services delivered to disadvantaged groups

- To increase

the use of buses with low floor access for people with disabilities - To improve the design of bus stops that would provide easy access to disadvantaged groups.

Sufficiency of reaching the stops

Satisfaction surveys

Sufficiency of buses in terms of getting on and off


Number of buses with low floor access

Program Indicators

Number of bus stops suitable for low floor access


(annual) Number of passengers injured during a journey


Sufficiency of online information related to bus stops and timetables


Goal 2.4. To reduce the negative impacts of the bus services on the environment

Rate of reduction in the emissions


Goal 6.3. To increase the number of public transport vehicles providing accessibility to people with disabilities

Number of buses converted for better accessibility


Goal 6.8. To strengthen the transportation infrastructure

Rate of updating the transportation information system


A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

62

Purpose

Goal

Objective

Activities


Source of the

indicator

Municipality Strategic Plan Purpose 13. To provide modern, comfortable and economic public transport services

Goal 13.1 To establish new systems through the procurement of new transport vehicles

- To procure new and modern vehicles of public transportation -To equip the bus stops with passenger information panels with maps -To create express and feeder lines through preference roads -To establish a public transport monitoring and control centre -To take measures for making public transport attractive for people with disabilities

Number of secure stops in the pilot area


Number of vehicles equipped with a tracking system


Number of campaigns promoting public transport Public awareness about the new practices


Number of low floor vehicles among the new additions to the transportation fleet.


Meetings with the senior management Purpose 2. To increase citizen satisfaction related to the transportation services

Goal 2.1. To increase the service quality in bus operations

- To increase the accessibility of disadvantaged groups to public transportation services

User satisfaction rates

Program Indicators


After getting the views of the internal auditors, the audit supervisor decided that the

audit exercise, which is based on the expectations of the management and the audit scope, be

conducted focusing on the components of productivity, effectiveness, conformity and

sustainability.

During the audit, the aim is to assess the activities and practices that form part of the bus

operations process against the following audit components and to identify areas of further

improvement.

▪ Productivity; assessment of whether the output generated from the available resources is at an adequate level.

▪ Effectiveness; capacity to reach objectives and realise intended results.

▪ Conformity; compliance of the activities and practices with the regulations and

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

63

procedures in force and adequacy of the institutional capacity for implementation.

64

▪ Sustainability; using the outputs on a continuous basis and rolling them out to be reused

by other administrations and stakeholders.

The methods that are planned to be applied in auditing the abovementioned components are quality analysis and measuring and reporting on performance.

Satisfaction surveys conducted by the Municipality will also be included in the analysis

work.

B5. KICK OFF MEETING

During the kick-off meeting, the internal auditor informed the auditee on the following

topics.

▪ audit team and its organisation

▪ audit scope

▪ stages of the audit

▪ time schedule

▪ identification of the comparison group

▪ next steps

During the kick-off meeting, the auditee expressed its expectations as follows.

▪ Municipality management became aware of the fact that satisfaction from bus operations is not at the desired level especially among citizens with disabilities. The management asked the internal audit team to examine the root causes of this situation.

B5-Kick off meeting minutes

AUDIT SUBJECT Performance Audit on the Bus Operations Process of Municipality A

AUDITEE Transportation Services Department of Municipality A


PLACE OF MEETING Meeting room of the office of the Mayor

PARTICIPANTS






5 Hasan Işık Expert – Department of Bus Operations

6 Mustafa Polat Expert – Informatics Department

7 Ahmet Aktaş Head of the Transportation Services Department

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

65

B6. DEVELOPING THE AUDIT MATRIX

As a result of the analyses and interviews conducted within the scope of the preliminary

work stage, internal auditors prepared the audit matrix which would set the general framework of

the field work.


Audit component

Audit question


Data source

Analysis method

Analysis strategy

EF

FEC

TIV

ENES

S

Are there enough opportunities to ensure the accessibility of bus services?

Bus stops are within a reasonable distance of access.

Number of stops in proportion to the distance of the line

User Satisfaction Surveys, Line distances and number of stops of Municipality A in 2015

Quality Analysis Measuring and Reporting on Performance

Information on line distances and number of stops will be acquired and analysed.

EF

FEC

TIV

ENES

S

The number of ticket sales points is sufficient.

Number of sales points in proportion to the number of passengers

User Satisfaction Surveys, Number of ticket sales points in 2015


The list of bus ticket sales points will be analysed.

EF

FEC

TIV

EN

ESS

Stops are accessible for people with disabilities.

User complaint rate

User Satisfaction Surveys

Quality Analysis

2015 User Satisfaction Surveys will be analysed.

EF

FEC

TIV

EN

ESS

Buses provide adequate accessibility in terms of getting on and off the bus.

User complaint rate


Quality Analysis


TOPICS RAISED DURING THE MEETING

During the kick off meeting the internal auditor informed the auditee on the following topics:

Audit team and its organisation, audit scope, stages of audit

Time schedule

Identification of the comparison group

Next steps

List of requested data

During the kick-off meeting, the auditee expressed its expectations as follows.

Municipality management became aware of the fact that satisfaction from bus operations is not at the desired level especially among citizens with disabilities. The management asked the internal audit team to examine the root causes of this situation

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

66


Audit component

Audit question



Analysis strategy

EF

FEC

TIV

ENES

S

Is user and citizen information on bus operation and services made at an adequate level?

The information at the bus stops (guidance, vehicle direction signs, time, ticket type information, etc.) is sufficient.

User complaint rate


Quality Analysis


EF

FEC

TIV

ENES

S

Guidance and voice information within the vehicles is sufficient.

User complaint rate


Quality Analysis


EF

FEC

TIV

EN

ESS

Information on lines and routes outside the vehicle is sufficient.

User complaint rate


Quality Analysis


EF

FEC

TIV

EN

ESS

Information provided on the website is clear.

User complaint rate


Quality Analysis


C

ON

FOR

MIT

Y

Is there a competent and sufficient source of personnel to serve at the desired quality level?

The attitude and behaviour of the drivers against the passengers are in line with the standards of the institution.

The attitudes and behaviors of the staff in the Saftch channels are suited to the institutional standards.

Environmental awareness of exhaust fumes is well suited to internationally recognized standards.

The environmental awareness of vehicles in terms of noise conforms to internationally recognized standards.

User complaint rate


Quality Analysis


C

ON

FOR

MIT

Y

Personnel attitudes and behaviours in sales channels are in line with institutional standards.

User complaint rate


Quality Analysis


EF

FEC

TIV

ENES

S

Are the environmental impacts of vehicles in line with minimum standards?

Environmental awareness of vehicles in terms of exhaust fumes is in line with internationally recognized standards.

Minimum level of exhaust fume

User Satisfaction Surveys, Exhaust gas measurement results


2015 User Satisfaction Surveys and exhaust fume measurement results will be analysed.

EF

FEC

TIV

ENE

SS

The environmental awareness of vehicles in terms of noise conforms to internationally recognized standards

Minimum level of bus noise

User Satisfaction Surveys, Noise measurement results


2015 User Satisfaction Surveys and noise measurement results will be analysed.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

67


Audit component

Audit question


Data source

Analysis method

Analysis strategy

EF

FEC

TIV

ENES

S

Are the vehicles' safety levels and technological features in line with the maximum standards?

Necessary technological features for passenger safety are available on vehicles.

Annual accident and injury rates

User Satisfaction Surveys, Annual accident and injury records


2015 User Satisfaction Surveys and Annual accident and injury records will be analysed. Also, on site examinations will be made to measure the sufficiency of technological safety features of vehicles.

EF

FEC

TIV

ENES

S

Do the physical conditions and comfort level of vehicles meet maximum standards?

Vehicles provide sufficient means for a comfortable journey (seats, handles and other equipment)

User complaint rate



2015 User Satisfaction Surveys will be analysed. Also, on site examinations will be made to measure the sufficiency of technological safety features of vehicles.

EFFE

CTI

VEN

ESS

Passenger density within the vehicle is at a reasonable level.

Number of passenger between 7.00h-9.30h and 16.30h -20.00h per bus

User Satisfaction Surveys, Number of passengers and rides within the specified intervals.


2015 User Satisfaction Surveys and number of passengers and rides within the specified intervals will be analysed.

EF

FEC

TIV

EN

ESS

AC and ventilation systems within the vehicles are operating sufficiently.

User complaint rate


Quality Analysis


EF

FEC

TIV

ENE

SS

Vehicles are cleaned properly (seats, handles and other equipment)

User complaint rate




EF

FEC

TIV

EN

ESS

Sound/noise level and mechanical shock level within the vehicles are below the maximum limit.

Maximum noise level within the bus


Quality Analysis


Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

68

RİSK KONTROL TEST/ANALİZ

Audit component

Audit question


Data source

Analysis method

Analysis strategy

E

FFEC

TIV

ENES

S

Do the vehicles arrive at the stop on a timely basis?

Buses arrive at the stop on time.

User complaint rate


Quality Analysis


Focus group meeting with a selected group of drivers will be conducted.

C

ON

FOR

MIT

Y

Is there an effective grievance mechanism related to the delivered services?

Complaints and demands are answered within a reasonable time.

Average response time to online complaints


Quality Analysis


C

ON

FOR

MIT

Y

The website provides sufficient guidance, information and content for users to lodge a complaint or request for information.

User complaint rate


Quality Analysis


P

RO

DU

CTI

VIT

Y

Do the vehicles go through timely, accurate and complete periodic maintenance?

Failure to perform timely, accurate, and complete maintenance of vehicles and response to breakdowns can cause disruptions in service safety and sustainability.

Periodic maintenance of vehicles is performed on a timely, accurate and complete manner.

Maintenance frequency based on distance and time travelled.

User Satisfaction Surveys, Vehicle maintenance charts


2015 User Satisfaction Surveys and vehicle maintenance charts will be analysed.


P

RO

DU

CTI

VIT

Y

Is the response to vehicle breakdowns timely, accurate and complete?

Response to vehicle breakdowns is quick.

Average response time to recorded cases of breakdown within the year.

User Satisfaction Surveys Vehicle maintenance charts


2015 User Satisfaction Surveys and vehicle maintenance charts will be analysed.


A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

69

RİSK KONTROL TEST/ANALİZ

Audit component

Audit question



Analysis strategy

SU

STA

INA

BIL

ITY

Did the low bus floor investment made to increase the access of buses to disadvantaged groups serve in the determined economic life?

Failure of the low bus floor feature to provide services at the planned time and quality may hinder the access of disadvantaged groups to services.

Accessibility equipment in buses remain in service throughout their economic lifespan

Actual period of service of accessibility mechanisms as compared to their economic lifespan.

Technical specifications related to the procurement of accessibility equipment, Accessibility equipment maintenance charts.


Technical specifications related to accessibility equipment and accessibility equipment maintenance charts will be analysed. Focus group meeting with a selected group of drivers will be conducted.




B7- Work Plan

SUBJECT OF THE AUDIT Performance Audit of the Bus Operations Process

AUDIT REGISTRY NUMBER


Field work 30.03.2016 – 24.04.2016

Reporting 27.04.2016 – 15.05.2016

PURPOSE OF AUDIT

During the audit, the aim is to assess the activities and practices that form part of the bus operations process against the following audit components and to identify areas of further improvement.

- Productivity; assessment of whether the output generated from the available resources is at an adequate level.

- Effectiveness; capacity to reach objectives and realise intended results.

- Conformity; compliance of the activities and practices with the regulations and procedures in force and adequacy of the institutional capacity for implementation,

- Sustainability; using the outputs on a continuous basis and rolling them out to be reused by other administrations and stakeholders

AUDIT SCOPE

The scope of the audit to be realized consists of the activities and practices related to the “Bus Operations Process” of Municipality A in 2015 and the sub processes thereto, including:

- Preparation of the vehicle for service - Periodic maintenance - Response to breakdowns, damages - Operational practices - Filtering and waste management - Informatics - Administrative Services - Financial Affairs

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

70

AUDIT METHOD

The audit work will involve the auditing of the performance of bus operations process. The following audit methods will be applied

Me

tho

ds o

f A

na

lysis

Components

Econom

y

Pro

ductivity

E

ffectivene

ss/E

ffi

cie

ncy

C

on

sis

ten

cy

Susta

ina

bili

ty

Confo

rmity

Benchmarking



Input-Output Analysis

Timeliness Analysis

Quality Analysis

INFORMATION REGARDING THE PREVIOUS AUDIT

The area in question has not been audited before.

PREPARATORY WORK

At this stage, preliminary data collection and analysis work was conducted. The following documents were prepared:

- Logical Framework of the Audit

- Audit Matrix (Draft)

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

71

A. SAHA ÇALIŞMALARI

C1. DATA COLLECTION

In addition to the data requested at the kick off meeting, the internal auditor also asked

for the provision of the below mentioned data.

The requested data was provided to the internal auditor in a file.

▪ User Satisfaction Surveys (2015)

▪ Length of bus routes and number of bus stops in 2015

▪ Number of ticket sales points in 2015

▪ Results of exhaust gas measurements and noise measurements

▪ Accident and injury records related to the year in question

▪ Total number of passengers and rides within the specified time intervals

▪ Vehicle maintenance charts, list of bus stops equipped with accessibility features

▪ Fleet information (2012-2015)

As part of the user satisfaction study, the municipality interviewed with 1000 people from 10 different regions of the municipal area. The number of people interviewed with in each region is proportional to the population of that region.

The survey questions and results are presented below.

▪ Do you have any disabilities?

▪ How satisfied are you with the distance from your domicile/work to the bus stop?

▪ How satisfied are you with the ticket sales services?

▪ How satisfied are you with the bus stops in terms of their accessibility for citizens with disabilities?

▪ How satisfied are you with the suitability of the bus stops for buses equipped with

accessibility equipment (low steps, etc.)?

▪ How satisfied are you with the buses in terms of easiness of getting on and off?

▪ How satisfied are you with the guidance provided at the bus stops (information, vehicle

direction signs, timetables, information on the type of tickets, etc.)

▪ How satisfied are you with the guidance provided within the vehicle through LDC screens and voice announcements?

▪ How satisfied are you with the guidance provided outside the vehicle about lines and routes?

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

72

▪ How satisfied are you with the information provided on the website?

▪ How satisfied are you with the attitude and behaviours of the drivers against the passengers?

▪ How satisfied are you with the attitude and behaviours of the ticket sales staff?

▪ How satisfied are you with the environmental awareness of vehicles in terms of exhaust

fumes?

▪ How satisfied are you with the environmental awareness of vehicles in terms of noise?

▪ How satisfied are you with the conditions related to the safety of passengers during a

journey? (Approach of the vehicle to stops, drivers being careless, allowing passengers to

get on and off the bus in between stops, etc.)

▪ How satisfied are you with the comfort of the vehicles? (Seats, handles and other

equipment and their suitability for people with disabilities.)

▪ How satisfied are you with the AC and ventilation systems of the vehicles?

▪ How satisfied are you with the hygiene of the vehicles (seats, handles and other

equipment?

▪ How satisfied are you with the sound/noise level within the vehicles?

▪ How satisfied are you with the timely arrival of the buses?

▪ How satisfied are you with the way your demands/complaints were handled?

The answers were given on a scale of one to five, as defined below.

▪ Not satisfied at all - 1

▪ Not satisfied - 2

▪ Neutral - 3

▪ Satisfied - 4

▪ Very satisfied - 5

72

C2. DATA ANALYSIS Some of the analyses conducted at the field work stage are given below.

# C2A1

Analysis 40

20

10

5

1 2

Disadvantaged individuals

All users

30

3

40

10

4

20

0

5

25

Analysis method

Existing situation

Results of the satisfaction survey performed by the municipality were analysed. The answers given to the question of “How satisfied are you with the accessibility of bus stops?” were examined on the basis of the above mentioned scale, in terms of people with disabilities as well as all users. Sufficiency of accessibility of stops 15% of the people surveyed found the accessibility of stops insufficient (options 1 and 2 taken together). This rate is 60% among the disadvantaged groups.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

73

# C2A2

Analysis

10

5 5 3

1 2


All users

50

3

52

25

4

25

10

5

15

Analysis method

Existing situation

Results of the satisfaction survey performed by the municipality were analysed. The answers given to the question of “How satisfied are you with the buses in terms of easiness of getting on and off?” were examined on the basis of the above mentioned scale, in terms of people with disabilities as well as all users.

Sufficiency of buses in terms of easiness of getting on and off

8% of the people surveyed found the buses insufficient in terms of easiness of getting on and off (options 1 and 2 taken together). This rate is 15% among the disadvantaged groups.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

74

# C2A3

Analysis

15 10

5 0

1 2


All users

70

3

85

10

4

5

0

5

0

Analysis method

Existing situation

Results of the satisfaction survey performed by the municipality were analysed. The answers given to the question of “How satisfied are you with the information provided on the website of the municipality?” were examined on the basis of the above mentioned scale, in terms of people with disabilities as well as all users.

Sufficiency of information provided on the website about the bus stops and timetables.

85% of the people surveyed picked option 3 as their answer to this question. The explanations show that people in this group are either not aware of the online services delivered by the municipality or do not use the website. Level of dissatisfaction among the disadvantaged group is 20%.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

75

# C2A4

Analysis

15

10

5

1


All users

2

10

40

3

40

15

4

20

20

5

25

Analysis method

Existing situation

Results of the satisfaction survey performed by the municipality were analysed. The answers given to the question of “How satisfied are you with the conditions related to the safety of passengers during a journey? (Approach of the vehicle to stops, drivers being careless, allowing passengers to get on and off the bus in between stops, etc.)?” were examined on the basis of the above mentioned scale, in terms of people with disabilities as well as all users.

Passenger safety during the journey

(Approach of the vehicle to the bus stop, careful driving, allowing passengers to get on and off the bus only at the designated stops) 45% of the people surveyed found the safety of the buses sufficient.

This rate is 35% among the disadvantaged groups.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

76

# C2A5

Analysis 60

20

15

A B


All users

10

15

C

8

25

D

15

Analysis method

Existing situation

Comparison of satisfaction of disadvantaged groups and all users from the service quality.

The results were analysed to measure the level of satisfaction from the following services;

A. Accessibility of bus stops

B. Sufficiency of the website

C. Accessibility of buses

D. Passenger safety during the journey

Example

A. As far as the accessibility of bus stops is concerned, the level of dissatisfaction is 15% among all users while it is 60% among the disadvantaged individuals.

Examinations show that the main problem for people with disadvantages is the poor quality of accessibility of bus stops.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

77

# C2A6

Analysis %30

%25

%20

%15

2012 2013 2014 2015

Ratio of the number of bus stops providing accessibility to buses to the total number of bus stops

60

36

8

4

2012 2013 2014 2015

Number of buses with low floor for accessibility

Analysis method

Trend analysis

Existing situation

Number of buses with low floor arrangements and the ratio of the accessible bus stops to the total number of bus stops.

The number of accessible buses in the municipality fleet reached 60 in 2015.

The ratio of the bus stops providing accessibility to the total number of bus stops is 30% as of 2015.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

78

# C2A7

Analysis

30

15


All users

Analysis method


Existing situation

Physical conditions of accessible bus stops

The 30 stops that were built to accommodate buses with low floor arrangements were identified as the sample group.

It was observed that 15 of the stops within this group did not have the necessary physical conditions. Since these stops did not meet the quality criteria, they became obsolete in time.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

79

# C2A8

Analysis -45%

60

12

15

33

Buses with Buses Buses out of Buses a low where service with a step for the step due to functioning accessibility does not poor low step

work conditions at the stops

Analysis method


Existing situation

Number of buses with a functional low step

The analyses showed that 60 buses equipped with a low step for accessibility of people with disabilities were registered to the municipality inventory.

Although the life span of this equipment is 2 years, the steps have become non-operational in 12 buses.

15 out of the remaining 48 buses also cannot provide services due to the lack of physical conditions in the bus stops along the route they operate.

Following the analysis of the secondary data, primary data on the below mentioned

stakeholders have also been analysed. The methods and stakeholder groups in this primary data

collection exercise are also mentioned below.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

80

# Participa

nts

Analysis method

Analysis strategy

Existing situation

C2B1

Bus drivers

Focus group meeting

▪ Focus group meetings were conducted with 15 drivers employed by the municipality.

▪ The topic and the scope of the meeting, the subject of the questions to be raised and the analysis methods were shared with the department of bus operations before the meeting and their comments were received.

The results of the focus group meetings are as follows:

▪ One of the main factors that affect the user satisfaction is the lack of sufficient physical conditions at the bus stops.

▪ The failure of the bus stops to meet the standards makes it difficult for the buses to approach the stop.

▪ The physical conditions of certain stops are not sufficient to accommodate accessible buses.

▪ Maintenance and repair works are performed rather fast.

▪ Maintenance and repair works do not negatively affect the bus operations.

▪ Number of drivers is sufficient.

C2B2

Municipality’s maintenance and repair staff

In-depth interview

▪ Face to face interviews with the municipality staff members selected according to their seniority and positions.

▪ 5 staff members were interviewed with.

▪ The answers given by the interviewees to the pre-determined questions were analysed.

The conclusions drawn from the in-depth interviews are as follows:

▪ Following issues were highlighted related to the maintenance and repair works.

o Maintenance and repair requests can only be communicated to the related unit verbally. No notification or reporting can be done over the web site.

o This leads to the failure to effectively plan and monitor the maintenance and repair works.

o The life span of accessible buses was specified as 2 years.

o The breakdowns of such buses are not communicated in time, causing problems in the repair management process.

o Bus stops are not maintained/repaired at a satisfactory level.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

81

D.R EPORTING


1 FINDING SHARING FORM

Subject of finding

Physical conditions of the bus stops along the city bus lines are not sufficient to provide accessibility for disadvantaged groups.

Level of importance

HIGH

Related unit


Existing situation

The conclusions drawn from the analyses and assessments on the bus operations of the municipality are provided below.

1. The number of accessible buses in the municipality fleet reached 60 in 2015.

60

36

8

4

2012 2013 2014 2015

Number of buses with low floor for accessibility

%30

%25

%20

%15

2012 2013 2014 2015

Ratio of the number of bus stops providing accessibility to buses to the total number of bus stops

2. In the period between 2012 and 2015, the municipality started designing bus stops that are compatible with the accessible buses. By 2015, 30% of the bus stops had been rearranged to be suitable for the accessibility of people with disabilities.

3. An analysis of the user satisfaction levels related to the accessibility of buses show that individuals among the disadvantaged group (elderly, children and people with disabilities) are not satisfied with the accessibility of the buses.

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

82

40 40

30

25

20 20

10 10

5

0

1 2 3 4 5


All users

Accessibility of the buses from the bus stop (2015 Satisfaction Survey conducted by the municipality)

15% of the people surveyed in 2015 found the accessibility of stops insufficient. This rate is 60% among the disadvantaged groups.

Reason According to the analyses, the reasons of the situation in question are as follows:

I. The analyses show that there is a sufficient number of bus stops that accommodate buses equipped with a low step for accessibility.

The focus group meetings and interviews with the drivers and maintenance staff on the other hand revealed that a significant number of these bus stops were not built in line with the quality measures set out by the municipality.

The internal auditors selected a sample of 30 stops that are deemed to provide accessibility to people with disadvantages. The auditors examined whether these stops are suitable for the approach of the buses with a low step. It was understood that these buses became obsolete due to the failure to meet minimum quality standards.

II. The analyses show that the municipality fleet includes 60 buses equipped with a low step to provide accessibility for people with disabilities. Although the life span of these vehicles were specified as two years, the accessibility step of 12 of those vehicles have ran out of service. 15 out of the remaining 48 buses also cannot provide services due to the lack of physical conditions in the bus stops along the route they operate. Therefore, only 33 of the 60 accessible buses serve their purpose.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

83

-45%

60

12

15

33

Buses with a

low step

Buses with a non-

performing low step

Buses out of service

due to poor

physical conditions

of stops

Buses with functional low steps

Number of buses with functioning low steps for accessibility

III. It was understood from the meetings conducted with the maintenance/repair staff that the requests for maintenance and repair are communicated to the related unit only verbally. The lack of a documented system for placing such requests result in a failure to keep records of the repair work.

IV. It was also understood that the web site does not provide any means for citizens to lodge their complaints or find information on accessible bus stops. Public awareness about the available information and services on the website is also limited.

85% of the people surveyed did not express any views about the content of the website.

85

70

15 10 10

5 5 0 0 0

1 2 3 4 5


All users

Sufficiency of information available on the website about the bus stops and time tables

A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

84

Risks and impacts

Although the municipality has a sufficient number of accessible buses, the design of bus stops does not meet the specified quality criteria, which makes it impossible for disadvantaged groups to use buses with means of accessibility. The loss that occurs due to inadequate design quality at approximately 50% of the sample bus stops result in the fact that approximately 32% of the buses with means of accessibility for people with disabilities cannot actually be used by such groups. Considering that 15% of the total number of passengers is made up of disadvantaged groups, 75,000 citizens are affected by the inadequate design quality at the stops. This situation causes the municipal services to not function effectively and causes the citizens' dissatisfaction.

Criterion Criteria that are identified on the basis of the above mentioned root causes are:

▪ It is necessary that the designs of the bus stops are suitable for disabled access.

▪ Maintenance and repair requests must be made through the website, both by the personnel and by the citizens. Mechanisms for effective follow-up of maintenance and repair activities should be established.

▪ Website content must be in line with citizen needs. Promotional activities should be carried out to increase citizens' awareness of the services provided through the website.

Recommendation

It is recommended that the following actions be taken regarding the findings found within the scope of the audit activity performed

1. It must be determined whether the bus stops that are compatible with the buses with means for people with disabilities comply with the quality standards set by the municipality.

2. It is recommended to plan the necessary systematic design which will enable the maintenance / repair requests for the bus stops to be made via a system which is accessible to the citizens as well.

3. Through this system it is recommended that the results of the following performance indicators, including others, be followed regularly:

- Number of bus stops maintained within the year

- Number of complaint applications made by citizens.

- Number of bus stops that require maintenance for more than twice a year

- Response time to requests

- Completion time of maintenance works

4. It is recommended that the content of the website of the municipality has content appropriate to the needs of the citizen. In this context, the website may have the following specific content, among others:

- Bus rides and stops that provide accessibility to people with disabilities.

- Application system for citizen complaints

Also, it is recommended that the promotional activities should be organized to increase the use of the web site by citizens.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

85

1

Responsible person

Required action Date of completion


Whether or not the accessible bus stops in the municipality inventory are in line with the quality standards set by the municipality will be assessed by a commission to be established by the Department of Transportation Services.

Department of Transportation Services /

IT Department

The budget requirement will be determined for the design of a system which will enable the maintenance/repair requests for the bus stops to be made through this system which is also accessible to the citizens.

Following the approval of the budget by the top management, necessary arrangements should be made regarding the system design.

Through this system, it is aimed to regularly monitor the results of the following performance indicators.

- Number of bus stops maintained within the year

- Number of complaint applications made by citizens.

- Number of bus stops that require maintenance for more than twice a year

- Response time to requests

- Completion time of maintenance works

-

Department of Transportation Services /

IT Department

In order for the municipality website to have a content suitable for the needs of the citizen, the necessary design project will be carried out with the IT Department.

Announcements for promoting the website will be distributed twice a year on the municipality buses to start from 2017.

D2. CLOSING MEETING

During the closing meeting, the audit team provided information on the following topics to

the auditee.





Opinion of the auditee

[X] We agree with the finding

[ ] We do not agree with the finding

[X] We agree with the recommendation.

[ ] We do not agree with the recommendation.


A B

eled

iyes

i Oto

bü

s İş

letm

eler

i Per

form

ans

Den

etim

i

86

D2- Minutes of the closing meeting

AUDIT SUBJECT Performance Audit of the Bus Operations Process of Municipality A

AUDITEE Municipality A


PLACE OF MEETING Meeting room of the office of the mayor

PARTICIPANTS






5 Hasan Işık Expert-Department of Transportation Services

6 Mustafa Polat Expert – IT Department

7 Ahmet Aktaş Bus Operations Department

TOPICS RAISED

During the closing meeting, the audit team provided information on the following topics to the auditee:




▪ Identification of areas of improvement in line with the findings and development of recommendations.

Kam

u İç

Den

etçi

leri

İçin

Per

form

ans

Den

etim

i Reh

ber

i (V

aka

Çal

ışm

alar

ı)

87

Performance Audit of the Emergency Call Centre Management of Presidency A

The content used in case study is not associated with any organisation, practice or person.

The content developed is completely fictional.

88

89

PERFORMANCE AUDIT OF THE EMERGENCY CALL CENTRE OF PRESIDENCY A

Presidency A has a call centre whose task is to provide information to citizens during

natural disasters and emergencies. The call centre has 250 employees who work in shifts.

The administration did not conduct any performance assessment on the call centre, which

became operational 2013, for three years following its establishment. The administration intends

to set, in its 2016 performance program, the objectives and relevant performance indicators to

follow such objectives with regard to the call centre.

As part of the macro risk assessment conducted by the internal audit unit, activity of the

call centre included in audit universe was evaluated considering the goals and objectives set in the

strategic plan as well as the opinions of the head of public administration and high level directors.

As a result of such evaluation, call centre activity was planned to be included in 2016 audit

program.

Primary expectations of the top management from the audit can be summarised as follows:

▪ Assessment of current activities in terms of achievement of the objectives included in the

administration’s strategic plan;

▪ Assessment of current activities in comparison with the similar call centres and other best

practices;

▪ Determining the improvement areas that should be attached importance while developing

activities in 2017.

In line with the objectives and expectations of the management, the internal audit unit of

the university decided to carry out a performance audit on the library processes. The audit has

been included in the audit program for 2016.

The audit is performed in line with the framework provided in the Performance Audit

Manual for Public Internal Auditors and Public Internal Audit Manual.

90


A1. ASSIGNMENT




took into account the following elements.




A2. PREPARATION OF THE IMPARTIALITY AND CONFIDENTIALITY PAPER

The internal auditors appointed by the head of IAU signed impartiality and confidentiality

documents.

A3. NOTIFICATION OF THE AUDITEE



A1. Assignment

No : 76995536-662.02- 14/02/2016


Mrs. Aylin Kaya (6666/A3)

Internal Auditor

You have been assigned to audit the call centre activities. You are kindly requested to

perform the audit in line with the Public Internal Audit Standards, the Public Internal Audit

Manual and the Performance Audit Manual and to submit the audit report to the Internal Audit

Unit

Head of IAU

Type of Audit Conducting Performance Audit on Call Centre Activities


Planned audit duration 26.02.2016-15.05.2016

Other internal auditors assigned


Audit supervisor Ömer Başkale (5555/A3)

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

91



Auditee: Department of Data Processing

Subject of the Audit: Activities of the Call Centre

Performance audit of call centre activities

User Services Process

Technical Services Process

Information Services Process

Administrative and Financial Services Process

AUDIT SCOPE

It includes all activities of the Department of Data Processing between 01.01.2015 – 31.12.2015.

DECLARATION





within the last year,

- I bear no prejudices against the audited unit, its employees or its managers.


26.02.2016

Aylin Kaya

Internal Auditor

NAME OF THE AUDIT

WARRANTY P

resi

de

ncy

A M

anag

eme

nt

of

Emer

gen

cy C

all C

ente

r Pe

rfo

rman

ce A

ud

it

92

A3. Notification to the Auditee

No : 76995536-679- ..../02/2016

Subject : Notification of Audit

TO THE DEPARTMENT OF DATA PROCESSING

Within the scope of 2015 Internal Audit Program executed in line with the approval of

the Undersecretary of 20.02.2016, performance audit will be conducted in your unit regarding call

centre activities between 26.02.2016-15.05.2016. Mentioned audit will start on 26.02.2016, and it

is expected to completed on 15.05.2016.


(5555/A3), by internal auditors Aylin Kaya (6666/ A3) and Mehmet Akın (4444/A3). You will receive

the findings and the report to be drafted at the end of the audit.

Scope of the audit is mainly the call centre activities, and the exact scope and audit goals

will be determined as a result of the interviews to be made between our audit team and your

unit.

Performance audit will mostly be conducted on data analyses. Therefore, it is important

that the data to be requested by the internal auditors be fully and accurately provided on time.

Support and participation by the process owners during the audit process will contribute to

provision of feasible improvement suggestions by internal auditors.

We attach great importance to your cooperation and information sharing for a successful

audit engagement.

Kindly submitted to your information and for due action.

Head of Internal Audit Unit

CIRCULATION

For due action

Department of Data Processing

For information




Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

93

Determining

Audit

Components

and Methods

Kick-off Meeting

Understanding

Logical

Framework

Forming

Audit Matix

Drafting and

Approving

Work Plan

B. PRELIMINARY SURVEY

Following stages included in Performance Audit Manual were followed during the

preliminary work stage of the audit.

1 2 3 4 5 6 7 Evaluating

Performance

Management

System

B1. DEVELOPING AUDIT ENGAGEMENT TIME SCHEDULE

The audit supervisor, in consultancy with the internal auditors within the audit team,

prepared the audit engagement time schedule form. In addition to the form, he prepared a

detailed audit plan which also shows the interim stages of the audit.

.

Work Output Start Completion Date Date

1 2 3 4 5 6 7 8 9 10 11 12

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

1 Launching Audit Engagements 26.02.16 26.02.16 Assignment Letter of Assignment Drafting Impartiality and Confidentiality Paper Impartiality and

Confidentiality Paper

Notification to the Auditee Audit Notification Letter 2 Preliminary Survey 02.03.16 27.03.16


Schedule Form

Performance Management System Maturity Analysis Maturity Analysis Form Understanding Logical Framework Requesting Preliminary Data List of Preliminary Data

Request Preliminary Analysis and Assessments Documenting Logical Framework Logical Framework Form

Determining Audit Components and Methods Audit Matrix Kick-off Meeting Kick-off Meeting Minutes Drafting Audit Matrix Audit Matrix Preparing Work Plan Work Plan

3 Field Work 30.03.16 24.04.16 Data Collection Requesting Data List of Data Request Setting up Audit Database Audit Database Analysis Preparing Findings and Developing Recommendations Consolidated List of Findings

4 Reporting 27.04.16 15.05.16 Sharing Findings Officially Findings Form Closing Meeting Closing Meeting Minutes

Receiving Action Plans Action Plan Audit Report Audit Report

Developing

Audit

Engagement

Time Schedule

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

94

B1-Audit Time Schedule Form

Performance audit of the call centre activity

Planned Realised

Start Date

End Date

Start Date

End Date

PRELIMINARY WORK 02.03.2016 27.03.2016

Maturity Analysis of Performance Management System

03.03.2016

Understanding Logical Framework

09.03.2016

Kick-off Meeting 17.03.2016

Approval of the Work Plan(Engagement work program and audit matrix)

27.03.2016

FIELD WORK 30.03.2016 24.04.2016

Data Collection and Launching Analyses (Tests)

30.03.2016

Completing the Analyses (Tests)

24.04.2016

REPORTING 27.04.2016 15.05.2016

Preparing Findings and Developing Recommendations

27.04.2016

Sharing Findings 01.05.2016

Closing Meeting 08.05.2016

Presenting the Audit Report 15.05.2016


Aiming to ensure the success of the performance audit results, the internal auditor

evaluated the maturity of the performance management system of the audit field. In this

framework, maturity of the audit field was analysed based on the following:

an assessment of the maturity of the performance management system,

an examination of the suitability of the area for performance audit,

evaluation of the applicability of performance audit components in the area to be audited

evaluation of the competency of the internal auditor who will conduct the performance

audit.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

95

The maturity of the performance management system was assessed by using the maturity analysis table provided in the manual. During this analysis, the internal auditor raised the following questions.

Have the goals and objectives been defined at the level of the organisation, its units and

their activities?

Have performance indicators been developed related to these goals and objectives?

Are the goals and objectives at the organisation, unit, process, program and activity level and related performance indicators followed through a system?

Is data related to performance indicators monitored and reported?

The preliminary analysis made it clear that goals and objectives were set out for the institution

and activities. In this scope, the goals and objectives of the Department of Data Processing were

set out in the performance program. Such goals and objectives were linked with performance

indicators. Call centre is a recently established part of the organisation. It is not included in the

main activity areas of the administration. Therefore, although some performance indicators were

determined for call centre activities, no mechanism exists for following up and reporting such

indicators. In addition, a “performance management system” program is being designed in the

institution. It is understood that the management plans to follow call centre activities via this

program, as is the case in other processes. Below is the scoring made by the internal auditors in

maturity analysis evaluation as a result of the interviews made.

Planning and Implementation

Weight

Coefficient

30%


2 Only a few goals and objectives are defined in the strategic plan of the organisation.

3 Goals and objectives are set out in the strategic plan of the organization.

Distribution of roles within relevant units is at basic level.

.

4

The goals and objectives are set out in the strategic plan of the organisation at a reasonable level and they are related with each other and with performance indicators.



5


Distribution of tasks is made in detail within relevant units. Resource allocation is planned in line with goals and objectives. Monitoring and follow-up mechanism is determined for strategic plan.

96


Weight

Coefficient

40%

1 No indicators exist for monitoring performance

ır. 2

Some indicators have been set out for monitoring performance however they are not comprehensive nor systematic.4

3

Indicators have been systematically set out for monitoring performance

The performance programs do not show the connection of indicators with the objectives of relevant units.

.

4



.

5





Weight

Coefficient

20%


2

Performance is measured for some indicators, but not on a regular basis.

3 Performance results related to all performance indicators set

out in the strategic plan are monitored.

. 4


yönelik ölçümler düzenli olarak tüm faaliyet ve süreçler için yapılmaktadır ve ilgililer ile paylaşılmaktadır.

5


I Indicators complying with SMART criteria. See: 3.5.2.3. Understanding Logical Framework

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

97


Weight

Coefficient

10%


. 2

Management may take into account the results of monitoring for only some of the units and activities.

3 Management takes into account the results of performance monitoring

for all units within the scope of the performance program.

.

4



.

5




Weight Coefficient

Score Weight

ed Score

Total Score

Design and implementation of the strategic plan and activity program 30% 3 0,9




The analyses show that the maturity score of the organisation’s performance

management system is 2.3. On the basis of this score, the audit area is considered to be available

for performance auditing within the scale provided in the Performance Audit Manual for Public

Internal Auditors as long as some audit risks of performance audit are taken into account.

In this scope, the internal auditors estimate that they can determine some performance

indicators together with the process owners and report on performance results.

The analyses results have also proven that the existing situation of the performance

management system does not pose any restrictions in terms of the audit components and audit

areas to be followed during the auditing exercise. This assessment is followed by understanding

the logical framework by the internal auditors.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

98


During the preliminary work stage, the internal auditors prepared a list of preliminary data

needed to perform necessary analyses and assessments to understand the call centre activity. In

this scope, they shared with the relevant unit lists of preliminary data requests, apart from the

reports and other data which they could obtain with their own means.

The internal auditors plan to examine the following data at the preliminary work stage.

# Information or Document

1 Strategic Plan of the Organisation (2015-2019)

2 Strategic Plan of the Organisation (2010-2014)

3 Performance Program of the Organisation (2010, 2011, 2012, 2013, 2014, 2015) 4 Call Centre Work Flow Charts

5 Call Centre Sub Processes

6 Call Centre Accountability Reports

During preliminary work stage, the internal auditors worked on other publicly available

relevant documents and reports. In this framework, following report and documents are

examined.

▪ Call Centre Accountability Reports (National and International Organisations)

Below is the logical framework to be used within the scope of the audit.

Goal Objective Milestone (Program

objectives)

Activities Performance Indicators Source of the indicators

Goal 1. To perform call centre activities in compliance with international standards

Objective 1.1. To provide human resources and systematic infrastructure required for effective and efficient call centre activities

- Providing reasonable quality service with the norm personnel of call centre - Systematic infrastructure of call centre having the capacity to ensure adequate service in emergencies

Main activities of the program:

- Providing adequate call centre services in emergencies -Creating adequate quantitative and qualitative human resources for call centre operations

Average talk time

Strategic Plan

Ratio of time allocated to emergency calls to total service duration

Performance Program

Ratio of emergency calls to all incoming calls

Program Indicators

Average duration of post-call actions

Indicators recommended by internal auditors

Objective 1.2. To increase satisfaction of the citizens who use the services of the call centre

- Ensuring call centre services at the level of international standards

Occupancy Performance Program

Service level Program Indicators

Ratio of missed calls Indicators recommended by internal auditors

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

99


Taking the opinions of the internal auditors, AS decided to perform an audit aiming at

effectiveness, efficiency and consistency, within the framework of expectations of the

management.

The audit to be conducted aims at assessing the work and transactions of the call centre

within the framework of the audit components below and determining improvement areas.

▪ Effectiveness; Capacity to reach the objectives and intended outcomes

▪ Productivity; assessment of whether the output generated from the available resources is

at an adequate level

▪ Consistency; coherence between the defined goals, objectives and strategies of public

administrations and the planning and design of their programs and projects.

In the audit activities aiming at audit components set in this scope, they mostly plan measuring and reporting performance, and using the methods of efficiency analysis and timeliness analysis.

Within the scope of the analysis, the following are aimed:

▪ Performance results on efficiency and timeliness of call centre activities between 2013 –

2015

▪ Comparing international best practices and call centre performance indicators.

B5. KICK-OFF MEETING

The internal auditor provided the auditee with information on the following issues during the

kick-off meeting.

▪ Audit Team and its organisation

▪ Audit Scope

▪ Audit Stages

▪ Time Schedule

▪ Determining Comparison Group

▪ Following Steps

In the kick-off meeting, the auditee stated its expectations as follows.

▪ Level of the current activities in terms of achieving the objectives included in the strategic

plan of the organisation

▪ Evaluating current activities by comparing the call centre and similar call centres and

other best practices

▪ Determining the improvement areas that should be attached importance in the activities

to be performed in 2016

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

101

ANNEX-1 Kick-off Meeting Notes

Subject Notes

General

Information

Ahmet Aktaş, Head of Data Processing Department, gave information about how sufficient current activities are in terms of achieving the objectives included in the strategic plan of the organisation.

The auditee highlighted the improvement areas which need to be attached importance in the activities to be performed in 2016.

It is decided to use the performance indicators for evaluating current activities of call centre in comparison with similar call centres and other best practices.

Selected performance indicators are presented in the table below.

Performance Indicators Performance Indicators’ Targets

Average talk time Average talk time per staff is reasonable.

Ratio of time allocated to emergency calls to total service time

When responding incoming calls, priority is given to emergency calls. Ratio of emergency calls to all calls received

Average after-call work time Average time spent for post-call actions should not exceed 120 seconds.

Occupancy rate * Call centre occupancy rate should be 70%.

Service level 80% of the call centre’s incoming calls should be answered in the first 20 seconds.

Ratio of missed calls Ratio of call centre’s missed incoming calls should not be more than 5%.

Average Speed of Answering Call centre’s incoming calls should be answered in 30 seconds on average.

Average Rate of Quitting Maximum waiting time for the calls cancelled by the citizens should be 130.

Longest waiting call On a daily basis, maximum waiting time for a call should be 180 seconds.

(*) Occupancy rate is ratio of time when the personnel actively serves for a call or answers a call to total working time, except for time spent for training, post-call actions, shift changes, etc.

.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

100

B5-Kick-off Meeting Minutes

AUDIT SUBJECT Call Centre Activity Performance Audit

AUDITEE Organisation A Department of Data Processing

MEETING DATE 10.04.2016

MEETING VENUE Organisation A Meeting Room of the Department

PARTICIPANTS

Order No


1 Burcu Demir Head of IAU




5 Hasan Işık Expert- Strategy Development Unit

6 Mustafa Polat Expert- Department of Data Processing

6 Ahmet Aktaş Head of Strategy Development Unit

ISSUES RAISED

The internal auditor provided the auditee with information on the subjects below in the kick-off meeting.


▪ Audit Scope

▪ Audit Stages, Time Schedule

▪ Determining Comparison Group

▪ Following Steps

▪ List of Data Request

In the kick-off meeting, the auditee expressed its expectations as follows.

▪ Level of the current activities in terms of achieving the objectives included

in the strategic plan of the organisation

▪ Evaluating current activities by comparing the call centre and similar call centres and other best practices

▪ Determining the improvements areas that should be attached importance

in the activities to be performed in 2017

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

102

B6. DEVELOPING THE AUDIT MATRIX As a result of the analyses and interviews performed in the preliminary work stage, the

internal auditors prepared an audit matrix which will set the overall framework of audit field

work.


Audit Component

Audit Question

Criterion & Control

Performance Indicator

Data Source

Analysis Method

Analysis Strategy

P

RO

DU

CTI

VIT

Y

Is the number of call centre staff sufficient to respond to incoming calls within the specified quality standards?

Average talk time per staff is reasonable.

Average talk time

Call Centre System records

▪ Measuring

and reporting on performance

▪ Efficiency analysis

The analyses will find out average talk time in 2013 -2015. Measurement results will be compared with 2013 - 2015 trends.

P

RO

DU

CTI

VIT

Y

Are the necessary actions after the call made on time?

Average time spent for post-call actions is reasonable.



The analyses will find out average duration of calls in 2013 -2015. Measurement results will be compared with 2013 - 2015 trends.

P

RO

DU

CTI

VIT

Y

Is the call centre staff able to deliver services actively during the working hours?

Call centre occupancy is at a sufficient level.

Occupancy rate


The analyses will find out occupancy rates in 2013 -2015.

Measurement results will be compared with 2013 - 2015 trends and objectives of the organisation.

EF

FEC

TIV

ENES

S How long does it take the call centre to answer incoming calls?

80% of the incoming calls must be answered within the first 20 seconds.

Service Level


▪ Measuring

and reporting on performance

▪ Timelines

s analysis

The analyses will find out service level durations in 2013 -2015. Measurement results will be compared with 2013 - 2015 trends and objectives of the organisation.

EF

FEC

TIV

ENES

S How many calls cannot be answered by call centre?

Maximum 5% of the incoming calls cannot be answered by call centre.

Ratio of missed calls


The analyses will find out ratio of missed calls in 2013 -2015. Measurement results will be compared with 2013 - 2015 trends and objectives of the organisation.

EF

FEC

TIV

ENES

S

How long does it take to answer the incoming calls of call centre on average?

Call centre’s incoming calls are answered in 30 seconds on average.

Average speed of answering


The analyses will find out average speed of answering in 2013-2015. Measurement results will be compared with 2013 - 2015 trends and objectives of the organisation.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

103

EF

FEC

TIV

ENES

S How long after do the citizens cancel their call on average when on hold?

Maximum waiting time before a call is cancelled by the citizens is 130 seconds

Average cancellation time


The analyses will find out average cancellation times in 2013 -2015.

Measuring results will be compared with 2013 - 2015 trends and objectives of the organisation.

EF

FEC

TIV

ENES

S

What is the longest waiting time for incoming calls?

On a daily basis, maximum waiting time for a call is 180 seconds.

Call with longest waiting time


The analyses will find out longest waiting call time in 2013 -2015. Measurement results will be compared with 2013 - 2015 trends and objectives of the organisation.

104


Audit Component

Audit Question

Criterion & Control


Data Source

Analysis Method

Analysis Strategy

C

ON

SIST

ENC

Y

Can the emergency calls be taken as a priority?

When taking incoming calls, priority is given to emergency calls.

▪ Ratio of

the time allocated to emergency calls to the total service duration

▪ Ratio of emergency calls to all incoming calls.


Measuring and reporting on performance

The analyses revealed that no classification was made on whether the calls received were urgent or not.

In this respect, an analysis was conducted by using the method below.

Calls received and answered during 30 days selected from among the days when call centre was operational and emergency cases occurred were classified.

The level of answering emergency calls was analysed.


Work plan drafted by the internal auditor after the preparation of the logical framework

and audit matrix was approved by AS.

B7- Work Plan

AUDIT SUBJECT Performance Audit of the Call Centre Activity

AUDIT NUMBER

PLANNED AUDIT DURATION

Field Work 30.03.2016 – 24.04.2016

Reporting 27.04.2016 – 15.05.2016

PURPOSE(S) OF THE AUDIT

The audit aims at;

Evaluating the works and transactions on call centre activities within the framework of the following audit components and determining improvement areas.

- Effectiveness; capacity to realise the objectives and intended results - Productivity; assessment of whether the level of outputs generated with the available

resources is sufficient - Consistency; the coherence between the goals, objectives and strategies of public

administrations and the planning and implementation of programs and projects,

AUDIT SCOPE

The audit encompasses the activities and practices on the sub-processes below regarding the call centre activities in 2015.

- User Services

- Technical services

- Informatics

- Administrative Services

- Financial Services

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

105

AUDIT METHOD

Within the scope of the audit, performance of the call centre will be assessed. Below are the audit

methods to be used in this respect.


The findings detected in 2014 system audit report were examined during the preliminary work stage of the performance audit exercise.

PREPARATORY WORK

Preliminary data were collected and analyses were made within the scope of the audit. In this scope, following documents were prepared.



Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

Me

tho

ds o

f A

na

lysis

Components

Econom

y

Pro

ductivity

E

ffectivene

ss/E

ffi

cie

ncy

C

on

sis

ten

cy

Susta

ina

bili

ty

Confo

rmity


Efficiency Analysis

Timeliness Analysis

106

C.FIELD WORK

C1. DATA COLLECTION

The internal auditor conducted researches together with the process owners with the aim

of determining performance indicators on call centre activities, apart from the list of data

requested in kick-off meeting during data collection stage.

In this scope, it was decided to analyse the below mentioned performance indicators

which are followed in similar organisations. Objectives related with the performance indicators

were set out within the framework of best practices and objectives of the organisation.5

Performance Indicators Performance Indicators’ Objective Source of Objective

Average Talk Time Average talk time per personnel is reasonable.

Strategic Plan

Ratio of time allocated to emergency calls to total service time

When responding incoming calls, priority is given to emergency calls.

Performance Program

Ratio of emergency calls to all calls received

Program Indicators

Average after-call work time Average time spent for post-call actions should not exceed 120 seconds.

Performance Indicators’ targets set out by top management Occupancy rate * Call centre occupancy rate should be 70%.

Service level 80% of the call centre’s incoming calls should be answered in the first 20 seconds.

Call Centre Best Practices and Standards5

Ratio of missed calls Ratio of call centre’s missed incoming calls should not be more than 5%.

Average Time of Answering Call centre’s incoming calls should be answered in 30 seconds on average.

Average Rate of Cancelling the Call Maximum waiting time for the calls cancelled by the citizens should be 130 seconds.

Longest waiting call On a daily basis, maximum waiting time for a call should be 180 seconds.

5 For the performance indicators’ targets whose source is Call Centre Best Practices and Standards. Union of Call Centres 2015 evaluation results were examined. Mentioned best practices’ outcomes were taken under the scope of the evaluation.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

107

C2. DATA ANALYSIS

During the field work, performance outcomes related to the identified performance

indicators were measured and reported.

# C2A1

Analysis 120

100

95

85 -15,00%

2012 2013 2014 Criterion

Average Talk Time

Analysis Method

Efficiency Analysis

Analysis Method

Input: Number of calls answered

Output: Total talk time on calls

In the analysis, the total talk time of the call centre staff is divided to the number of answered calls. Performance indicator shows how many seconds it takes to provide a solution related to the incoming call.

Current Situation

Average Talk Time (seconds)

The analysis shows that average talk time has decreased over the years and is currently performing under management’s target.

This indicates that the calls last shorter than targeted.

The most important reason behind the decrease in the average talk time is

that some of the procedures that used to be handled during the call are now handled after the call is ended. This helps to complete the provision of a solution to the citizen after the call, without keeping the citizen waiting on hold.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

108

# C2A2

Analysis 145

120 120 +20,83%

80

2012 2013 2014 Criterion

Average duration of post-call actions (Seconds)

Analysis Method

Efficiency Analysis

Analysis Method

Input: Number of incoming calls

Output: Total duration of post-call actions

How much time the personnel taking the call spend on average on post call actions was analysed.

Performance indicators show how many seconds the post-call action takes on average.

Current Situation

Average duration of post-call actions (Seconds)

The examinations show that the post-call action takes more than the targeted 120 seconds.

This results in the personnel answering the subsequent call later.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

109

# C2A3

Analysis

85 80 80 70 +14,29%

2012 2013

2014

Criterion

Occupancy rate

Analysis Method

Efficiency analysis

Analysis Method

Input: Number of personnel

Output: Total time used efficiently (except for training, post-call actions, shift changes)

The ratio of time a staff member spends for answering a call to the total working time of that staff member is analysed.

Performance indicator is the ratio of the available time to answer a call to total working time, except for time allocated to training, post-call actions, shift changes.

Current situation

Occupancy Rate

The examinations show that the occupancy rate exceeds the targeted rate of 70%.

This reveals that the time allocated by the personnel to fundamental services is at an adequate level for an efficient work.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

110

# C2A4

Analysis

45

40

35

+75,00%

20

2012 2013 2014 Criterion

Service Level Time

Analysis Method

Timeliness Analysis

Analysis Method

Input: Number of incoming calls

Output: Total waiting time for the calls answered

The analysis showed that the target is to answer 80% of the calls within 20 seconds, considering best practices.

Current Situation

Service Level Time

The examinations showed that;

In the scope of service level time, 80% of the incoming calls are answered within 35 seconds on average. This is more than 20 seconds taken as basis in best practices.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

110

# C2A5

Analysis

18

15

11

+120,00%

5

2012 2013

2014

Criterion

Ratio of Missed Calls

Analysis Method

Timeliness Analysis

Analysis Method

Input: Number of Total Calls

Output: Number of Missed Calls

The percentage of the incoming calls cancelled without being answered was analysed.

The good practice is 5%.

Current Situation


Ratio of missed calls is 11% in 2015, which is over the targeted rate of 5%.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

117

# C2A6

Analysis

60

50

40

30 +33,33%

2012 2013

2014

Criterion

Average Speed of Answering

Analysis Method

Timeliness Analysis

Analysis Method

Input: Number of Calls Answered

Output: Total Waiting Time for the Answered Calls

How many seconds it takes to answer an incoming call was analysed.

Management’s target was 30 seconds.

Current Situation


Average Speed of Answering is 40 seconds in 2015. P

resi

de

ncy

A M

anag

eme

nt

of

Emer

gen

cy C

all C

ente

r Pe

rfo

rman

ce A

ud

it

112

# C2A7

Analysis

150

140 135 130 +3,85%

2012 2013 2014 Criterion

Average Rate of Quitting

Analysis Method

Timeliness Analysis

Analysis Method

- Input: Number of Abandoned Calls

- Output: Total Waiting Time for the Abandoned Calls

How long a caller waits before abandoning the call was analysed.

Management’s target was 130 seconds.

Current Situation

Average Rate of Abandoning Calls

Average rate of abandoning is the average time of calls abandoned by the caller before being answered.

Average rate of abandoning was 135 seconds in 2015. This shows that the caller waits for 135 seconds before abandoning the call.

Management’s target is to answer the calls in maximum 130 seconds.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

113

# C2A8

Analysis

210

195 190 180 +5,56%

2012 2013 2014 Criterion

Longest waiting call time

Analysis Method

Timeliness Analysis

Analysis Method

Longest waiting incoming call time was analysed. The rate set by the management for

this indicator was 180 seconds.

Current Situation

Longest waiting call time

Longest waiting call time was measured as 190 seconds in 2015.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

114

# C2A9

Analysis

40%

25% 25%

20%

15%

10%

A B C

Ratio of answering priority calls

15%

D

50%

Calls answered

Returned calls

Analysis Method

Measuring and Reporting on Performance, Multi-criteria analysis

Analysis Method

The analyses revealed that no classification was made on whether the incoming calls were urgent or not.

In this respect, an analysis was conducted by using the method below.

Calls received and answered on a period of 30 days selected from among the days when call centre was operational in 2015 were classified by using multi-criteria analysis. The level of answering the emergency calls was analysed.*

Current Situation

Ratio of answering priority calls

Calls included in the sample group were analysed by using multi-criteria analysis method, based on the records of services provided. In this scope, the calls were divided into four groups in terms of their priority.

A-Very High Priority,

B-High Priority,

C-Medium Priority,

D-No Priority

60% of the calls answered during the selected days are under group A and B, which are priority calls.

(*) Analysis conducted by using multi-criteria analysis was performed in stages as follows.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

115

All the calls received were classified as per the services provided for the call (urgency of call is evaluated by analysing the service provided).

I. The calls received per service were classified according to the following criteria. The

criteria are as follows.

DECISION CRITERION

Incoming Call

A B C D Priority

Order

Period Location Emergency Type of Service

A Period of

Emergency

Call Received

from Emergency

Location

Emergency Call

Police Force and Judicial Case A

Single Rescue Case (Fire, Accident, etc.)

A

Multi-rescue Case (Disaster, Security, etc.)

A

Other C

Non-emergen

cy Call

Police Force and Judicial Case B


B


C

Other D

Call Received from Other Locations

Emergency Call



A


A

Other C

Non-emergen

cy Call



B


B

Other D

Normal

Period

Emergency Call



A


A

Other D

Non-emergen

cy Call



B


B

Other D

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

116

Evaluating the

recommendations

for improvement

areas

C3. IDENTIFICATION OF FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS

1 2 3 Analysing the

results of the

benchmarking

exercise and

sharing them with

the management

Analysing the

findings, risks and

root causes

C3.1. Analysing the results of the benchmarking exercise and sharing them with the management

Following results are obtained as a result of the benchmarking analysis. Analysis results

were examined together with the relevant process owners.

# Subject of Analysis

Analysis Result

Finding Work Paper

C3C1

Average Talk Time

The analyses show that average talk time has decreased over the years and is currently performing under management’s target.

This indicates that the calls last shorter than targeted.

A1

C3C2


The examinations show that the post-call action takes more than the targeted 120 seconds.

This results in the personnel answer the subsequent call later.

A2

C3C3

Occupancy rate

The examinations show that the occupancy rate exceeds the targeted rate of 70%.

This reveals that the time allocated by the personnel to fundamental services is at adequate level for an efficient work.

A3

C3C4

Service level

In the scope of the service level time, 80% of the incoming calls are answered within 35 seconds on average. This is more than 20 seconds taken as basis in best practices.

Evet

A4

C3C5 Ratio of missed calls

60% of the calls answered during the selected days are under group A and B calls which are priority calls.

Evet A5

C3C6 Average Speed of Answering

Average Speed of Answering was 40 seconds in 2015. Evet A6

C3C7


Average rate of abandoning was 135 seconds in 2015. This shows that the caller waits for 135 seconds before abandoning the call. Management’s target is to answer the calls in maximum 130 seconds.

A7

C3C8

Longest waiting call

Longest waiting call time was measured as 190 seconds in 2015.

A8

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

117

C3.2. Analysing the findings, risks and root causes

The findings identified as a result of the analyses are consolidated under several headlines.

▪ The audit revealed that 80% of the incoming calls are answered within 35 seconds on

average. Yet, it is understood in the interviews that the resources allocated when

planning the call centre system including its human resources are enough to answer the

incoming calls within 20 seconds on average.

▪ Average speed of answering calls was 40 seconds in 2015.

▪ The late reaction to incoming calls results in 11% of the calls being abandoned without being answered.

Main reason is that some services related with a call are carried out after the call is ended.

This way, the management aims at reducing average time spent on a call. However, as there is no

monitoring and control system on timeliness and efficiency of the post-call actions, there is

unnecessary time loss in the mentioned stage. Post-call action time for a call was 145 seconds in

2015.

6 2014 data.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

118

D. REPORTING


1 FINDINGS

Subject of finding

Answering time to an incoming call in call centre is higher than the planned.

Importance Level of the Finding

HIGH

Relevant Unit


Current Situation

Within the scope of the audit, 2013 – 2015 performance results of the call centre were measured and reported.

Performance results were measured on the basis of the performance indicators below. As no performance indicator had been set out by the organisation regarding the call centre activities, the indicators below were set during the audit. When determining the indicators, opinions of the process owners, best practices and standards were taken as basis.

Following findings were obtained as a result the analysis.

1. The audit revealed that 80% of the incoming calls are answered within 35 seconds on average. Yet, it is understood in the interviews that the resources allocated when planning call centre system including the human resources were enough to answer the incoming calls within 20 seconds on average.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies


Performance Indicators’ Objective Source of Objective

Average Talk Time Average talk time per staff is reasonable.

Objectives Set by Top Management


Average time spent for post-call actions is reasonable.

Occupancy

Call centre occupancy is at a sufficient level.

Service level

80% of the incoming calls must be answered within the first 20 seconds.

Call Centre Best Practices and Standards


Maximum 5% of the incoming calls cannot be answered by call centre.


Incoming calls are answered in 30 seconds on average.

Average Rate of Quitting

Maximum waiting time before a call is abandoned by a citizen is 130 seconds.

Longest Waiting Call

On a daily basis, maximum waiting time for a call is 180 seconds.

119

45

40

35

+75,00%

20

2012 2013 2014 Criterion

Service level

2. Average speed of answering in the call centre was 40 seconds in 2015.

60

50

40

30 +33,33%

2012

2013

2014

Criterion


3. The late reaction to incoming calls results in 11% of the calls being abandoned without being answered.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

125

18

15

11

+120,00%

5

2012 2013

2014

Criterion


Cause As a result of the trend analyses and comparison with best practices, it is understood that the main reason is that some services related with a call are carried out after the call is ended.

This way, the management aims at reducing average time spent on a call. However, as there is no monitoring and control system on timeliness and efficiency of the post-call actions, there is unnecessary time loss in the mentioned stage. Post-call action time for a call was 145 seconds in 2015.

145

120 120 +20,83%

80

2012 2013 2014 Criterion

Duration of post-call actions (seconds)

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

120

Risks and Effects

Taking a call later than the targeted time may result in the caller abandoning the call before a conversation occurs. This may lead to dissatisfaction.

Due to the lack of a prioritisation of calls and the length of taking the calls, the call centre is unable to take emergency calls in a timely manner. This is expressed as follows.

50%

Calls answered

40% Returned calls

25% 25%

20%

15% 15%

10%

A B C D

Rate of Taking Priority Calls

As stated above, priority of the calls taken within a period of randomly selected 30 days were analysed. The priority of calls were listed beginning from very high (A) to low (D). The analysis revealed that 60% of the taken calls were in group A and B, i.e. priority calls. During the same period, 25% of the calls under this group could not be answered due to occupancy.

This may cause that the call centre does not operate in compliance with one of its main objectives, which is delivering services related to emergency calls.

Criterion 80% of the incoming calls must be answered within the first 20 seconds. In this respect, time spent for post-call actions should not exceed 130 seconds.

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

122

Recommendation

Following actions are recommended within the scope of the audit regarding this finding.

7. It is recommended that the content and targeted completion time of the post-call actions be determined. These stages should be expressed in detail, in the work flow chart of the call centre.

8. Designing a new organisation structure to ensure that post-call actions are followed up by a different and specialised team and an automation system to support this team is recommended.

9. Necessary actions must be taken for the systematic follow-up of the performance indicators listed below (the list is not comprehensive):

Average talk time

Average Time Spent for Post-Call

Actions Occupancy Rate

Service Level




Longest Waiting Call Time

10. It is recommended that a process and technological infrastructure be set up to systematically categorize the incoming calls. In this respect, the hierarchy provided in Annex-1 may be used among others.

11. Objectives should be set out for the performance indicators. It is recommended that the results obtained for each objective be published in the accountability reports of the call centre.

Opinion of

the Auditee




[ ] We do not agree with the recommendation.


Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

123

1

Responsible Person

Action to Be Taken Date of

Completion


Content and targeted completion time of the post-call actions will be set in the 2017 performance program.


Budget required for designing a new organisation structure to ensure that post-call actions are followed up by a different and specialised team and an automation system to support this team will be submitted to the office of the President as included in the 2017 budget proposal.


Actions necessary for systematic follow-up of the below mentioned performance indicators will be taken as of 2017 (the list is not comprehensive). The results obtained for each objective will be published annually within the accountability reports of the call centre.

Average Talk Time


Occupancy

Service level




Longest Waiting Call Time


Resource required for setting up the process and information technologies infrastructure for the systematic categorization of the incoming calls in terms of their priority will be determined and shared with the Presidency.

D2. CLOSING MEETING

The audit team provided the auditee with information on the subjects below in the closing

meeting.

▪ Giving information about the analyses and work carried out

▪ Sharing the findings with the process owners

▪ Analysing the root causes of the findings

▪ Determining improvement areas on the findings and developing recommendations

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

124

D2-Closing Meeting Minutes

AUDIT SUBJECT Presidency A Emergency Call Centre Performance Audit

AUDITEE Presidency A


PLACE OF MEETING Presidency A Meeting Room

PARTICIPANTS

NO Name Title Signature








ISSUES RAISED

The internal auditor provided the auditee with information on the subjects below in the closing meeting.

▪ Information on the analyses and work carried out

▪ Sharing the findings with the process owners

▪ Analysing the root causes of the findings

▪ Determining improvement areas on the findings and developing recommendations

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

ANNEX-1 Call Priority Order

DECISION CRITERION

Incoming Call

A B C D Priority

Order

Period Location Emergency

Type of Service

A Period

of Emerge

ncy

Call

Received from

Emergency Location

Emergency Call


Single Rescue Case (Fire, Accident, etc.) A


A

Other C

Non-emergen

cy Call


Single Rescue Case (Fire, Accident, etc.) B


C

Other D

Call Received from Other Locations

Emergency Call




A

Other C

Non-emergen

cy Call




B

Other D

Normal Period

Emergency Call




A

Other D

Non-emergen

cy Call




B

Other D

Pre

sid

en

cy A

Man

agem

en

t o

f Em

erge

ncy

Cal

l Cen

ter

Perf

orm

ance

Au

dit

127

Performance Audit of the Transition to Electronic System Program within Ministry A

The content used in case study is not associated with any organisation, practice or person.

The content developed is completely fictional.

129

PERFORMANCE AUDIT OF THE TRANSITION TO ELECTRONIC SYSTEM

A central government administration provides services for the businesses operating in its

field of service through its decentralised organisation. The operations and transactions related to

these services were made on paper with written notifications but recently the administration has

shifted to an electronic and internet based system, with the "Electronic Transformation Program".

The transformation program was carried out for 3 years from 2012 to 2014, and following

activities were executed within the scope of the program:

▪ Purchase of inventory stock and furnishing in central and decentralised units for setting up the infrastructure of the system,

▪ Purchase of software and integration service,

▪ Improving the legislation,

▪ Training for administration’s personnel,

▪ Dissemination activities aiming at users.

Transformation program has four primary objectives:

▪ To make the transactions simple and easy

▪ To standardise practical implementation

▪ To increase efficiency for administration and users

▪ To facilitate monitoring and control of the transactions and operations

Management of the organisation requested the internal audit unit to perform a performance audit on the program in order to understand its effects and benefits. Primary expectations of the management from the audit are as follows:

▪ Understanding the extent to which the program achieves its objectives and makes an impact

on the internal and external user activities.

▪ Understanding the differences between the current situation where the program is

operational and an alternative scenario without the program.

▪ Revealing whether there is any improvement area in the operation and use of new electronic

system.

In line with the objectives and expectations of the top management, Internal Audit Unit launched the performance audit on the Electronic Transformation Program.

Auditing activities are carried out within the framework of Public Internal Audit Manual and

Performance Audit Manual.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

130


A1. ASSIGNMENT




took into account the following elements.



- Need for technical information on audit field


The internal auditors assigned by the head of IAU signed impartiality and confidentiality

documents.

A3. NOTIFICATION OF THE AUDITEE

The head of internal audit sent a notification letter to the unit to be audited to give them basic information about the audit.

1. Assignment

No : 76995536-662.02- 14/02/2016


Mrs. Aylin Kaya (6666/A3)

Internal Auditor

You have been assigned to audit the transition to electronic system. You are kindly

requested to perform the audit in line with the Public Internal Audit Standards, the Public Internal

Audit Manual and the Performance Audit Manual and to submit the audit report to the Internal

Audit Unit.


Type of Audit Performance Audit of the transition to Electronic System Program

Special Instructions The performance audit shall be carried out by applying the methods and techniques referred to in the Performance Audit Manual.

Planned audit duration 26.02.2016-18.05.2016

Other internal auditors assigned


Audit supervisor Ömer Başkale (5555/A3)

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

131



Auditee: Strategy Development Unit

Subject of the Audit: Transition to Electronic System Program

Performance Audit of the Transition to Electronic System Program

It includes all activities of Strategic Development Unit taking place between 01.01.2015 – 31.12.2015.

DECLARATION







26.02.2016 Aylin Kaya

Internal Auditor

NAME OF THE AUDIT

AUDIT SCOPE

WARRANTY

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

132

A3. Notification to the Auditee

No :76995536-679- …./02/2016

Subject : Notification of the Audit

TO THE STRATEGY DEVELOPMENT UNIT

Within the scope of 2016 Internal Audit Program executed in line with the approval of

the Undersecretary of 20.02.2016, a performance audit of the transition to electronic system

program will be conducted in your unit between 26.02.2016-18.05.2016. The audit in question

will start on 26.02.2016, and is planned to be completed on 18.05.2016.

The audit will be conducted by the internal auditors Aylin Kaya (6666/ A3) and Mehmet

Akın (4444/A3) under the supervision of Internal Auditor Ömer Başkale (5555/A3), and you will

receive the findings and report to be prepared at the end of the audit.

Scope of the audit is mainly shift to electronic system program, and the exact scope and

audit goals will be determined as a result of the interviews to be made between our audit team

and your unit.







Kindly submitted for your information and due action.


DISTRIBUTION

For Due Action


For Information




Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

133

Determining

Audit

Components

and Methods

Kick-off Meeting

Understanding

Logical

Framework

Developing

Audit Matix

Drafting and

Approving

Work Plan

A. PRELIMINARY WORK

At the preliminary work stage of the audit, the following steps set forth in the Public


1 2 3 4 5 6 7 Evaluating

Performance

Management

System

B1. DEVELOPING THE AUDIT ENGAGEMENT TIME SCHEDULE

AS developed audit time schedule form considering the opinions of the internal auditors

taking part in the audit team. In addition, a detailed audit plan indicating interim stages of the

audit was also prepared.

Work Output Start Completion Date Date

1 2 3 4 5 6 7 8 9 10 11 12

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

Wee

k

1 Launching Audit Engagements 26.02.16 26.02.16 Assignment Letter of Assignment Drafting Impartiality and Confidentiality Paper Impartiality and

Confidentiality Paper

Notification to the Auditee Audit Notification Letter 2 Preliminary Survey 02.03.16 27.03.16


Schedule Form

Performance Management System Maturity Analysis Maturity Analysis Form Understanding Logical Framework Requesting Preliminary Data List of Preliminary Data

Request Preliminary Analysis and Assessments Documenting Logical Framework Logical Framework Form

Determining Audit Components and Methods Audit Matrix Kick-off Meeting Kick-off Meeting Minutes Drafting Audit Matrix Audit Matrix Preparing Work Plan Work Plan

3 Field Work 30.03.16 24.04.16 Data Collection Requesting Data List of Data Request Setting up Audit Database Audit Database Analysis Preparing Findings and Developing Recommendations Consolidated List of Findings

4 Reporting 27.04.16 15.05.16 Sharing Findings Officially Findings Form Closing Meeting Closing Meeting Minutes

Receiving Action Plans Action Plan Audit Report Audit Report

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

Developing

Audit

Engagement

Time Schedule

134

B1- Audit Time Schedule Form Performance Audit of the Transition to Electronic System Program

Planned Realised

Start Date

End Date

Start Date

End Date

PRELIMINARY SURVEY 02.03.2016 27.03.2016

Evaluating Auditability 03.03.2016


09.03.2016

Kick-off Meeting 17.03.2016

Approving the Work Plan (Engagement work program and audit matrix)

27.03.2016

FIELD WORK 30.03.2016 24.04.2016

Data Collection and Preliminary Analyses (Tests)

30.03.2016

Completion of the Analyses (Tests) 24.04.2016

REPORTING 27.04.2016 15.05.2016

Preparing Findings and Developing Recommendations

27.04.2016

Sharing Findings 01.05.2016

Closing Meeting 08.05.2016

Presenting the Audit Report 15.05.2016


The internal auditor assessed the maturity of the performance management system of the

audit field in order to ensure that the results of the performance audit exercise to be conducted

are successful. In this regard, the maturity of the audit field was analysed on the basis of the

following topics.

Evaluating maturity of the performance management system,

▪ an examination of the suitability of the area for performance audit

▪ evaluation of the applicability of performance audit components in the area to be audited

▪ evaluation of the competency of the internal auditor who will conduct the performance audit

The maturity of the performance management system was assessed by using the maturity analysis table provided in the Performance Audit Manual for Public Internal Auditors. During this analysis, the internal auditor raised the following questions.

▪ Have the goals and objectives been defined at the level of the organisation, its units and their activities?

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

135

▪ Have performance indicators been developed related to these goals and objectives?

▪ Are the goals and objectives at the organization, unit, process, program and activity level and

related performance indicators followed through a system?

▪ Is data related to performance indicators monitored and reported?

The preliminary study made it clear that the goals and objectives as well as the indicators to measure them were set for the institution and activities. Such goals and objectives were linked with performance indicators by means of the performance program. However, no mechanism exists for monitoring. Although some of the performance indicators are irregularly monitored, no mechanism exists for monitoring and reporting on all indicators.

Below is the scoring made by the internal auditors through the maturity analysis as a result of the interviews made.

Planning and Implementation

Weight

Coefficient

%30

1 The organisation has no strategic plan.

2 Only a few goals and objectives are defined in the strategic plan of the organisation.

3 Goals and objectives are set out in the strategic plan of the organization.

Distribution of roles within relevant units is at basic level.

4




5

The goals and objectives are set out in the strategic plan of the organisation at a reasonable level and they are related with each other and with comprehensive performance indicators.

Distribution of tasks is made in detail within relevant units.


Monitoring and follow-up mechanism is determined for strategic plan.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

136


Weight

Coefficient

%40


2 Some indicators have been set out for monitoring performance

however they are not comprehensive nor systematic.6

3


The performance programs do not show the connection of indicators with the objectives of relevant units.

4



5





Weight

Coefficient

%20


2 Performance is measured for some indicators, but not on a regular basis

3 Performance results related to all performance indicators set

out in the strategic plan are monitored.

4


5


7 Indicators complying to SMART criteria. See: 3.5.2.3. Understanding Logical Framework

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

137


Weight

Coefficient

10%


. 2 Management may take into account the results of monitoring for only

some of the units and activities.

3 Management takes into account the results of performance monitoring

for all units within the scope of the performance program.

.

4



.

5




Weight Coefficient

Score Weighted

Score Total

Score

Design and implementation of the strategic plan and activity program 30% 4 1,2




The analyses show that the maturity score of the organisation’s performance

management system is 3.5. On the basis of this score, the area is considered to be suitable for

performance auditing within the scale provided in the Performance Audit Manual for Public

Internal Auditors.

In addition, whether the Electronic Transformation Program meets prerequisites regarding

the “method of evaluating the results of program and implementation” was analysed.

Accordingly;

▪ It was understood that as the program is completed and some time elapsed after its

completion, the results could be measured.

It was estimated that the results and realisations regarding the program are measurable,

some secondary data is available and primary data could be collected through questionnaires and

interviews. The analysis results have also proven that the existing situation of the performance

management system does not pose any restrictions in terms of the audit components and audit

areas to be followed during the auditing exercise.

This assessment is followed by understanding the logical framework by the internal

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

138

auditors.

139


The internal auditors prepared a list of preliminary data needed to perform necessary

analyses and assessments to understand the program. In this scope, they obtained the reports and

other available data and requested process owners to provide the others.

At the preliminary work stage, the internal auditors plan to examine the following data.

# Information or Document

1 Strategic Plan (2015-2019)

2 Strategic Plan (2010-2014)

3 Performance Program (2012, 2013, 2014, 2015)

5 2012-2015 Accountability Reports

6 Electronic Transformation Program planning documents

The documents listed above are used in the analyses aiming at understanding the content

of the program and its logical framework.

The analysis aiming at understanding the content and logical framework is provided below:

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

140

Below is the logical framework to be used within the scope of the audit:

Goal Objective Milestone (Program

objectives)

Activities Performance Indicators Performance Indicator

Source

2010-2014 Strategic Plan, Goal 5. To ensure efficient, fast and good quality paper work in the sectors operating in the duty field of the institution.

Objective 5.1. To take measures for improving efficiency, speed and quality of the transactions performed by the institution

- Making the transactions simple and easy - Increasing

efficiency for administration and users

Main activities of the program:

- -Improving legislation

- -Purchase of inventory stock and furnishing

- -Purchase of software and integration service

- -Training for administration’s personnel

- -Dissemination activities aiming at users

- Ratio of transactions completed in compliance with service standards (%)

Strategic Plan

- Service receivers’ satisfaction rate (%)

Performance Program

- Change in average transaction closing time (%) - Change in the transactions’

error rate (%)

Program Indicators

- Change in average transaction duration for the administration’s personnel (%) - Change in average

transaction duration for the users (%) - Change in average cost

of users per transaction (%) - Total time saving for

the administration - Total time saving for

the users - Total cost saving for

the users


Objective 5.4. To increase the efficiency of service units of provincial organisation

- Standardising the implementations

- -Number of trainings offered - -Number of personnel

participating in the trainings

Performance Program

- -Total training time offered - -Number of personnel

participating in the trainings

Program Indicators

- Total average training time per personnel regarding program


2010-2014 Strategic Plan, Goal 7. Strengthening institutional capacity.

Objective 7.1. To facilitate monitoring and control of the transactions and operations carried out in the duty field of the institution

- To facilitate monitoring and control of the transactions and operations

- Change in number of investigation initiated for irregularity (%) - Change in number of

punishments imposed due to irregularity (%)

Program Indicators

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

140


Taking the opinions of the internal auditors, AS decided to assess the economy,

sustainability, effectiveness, efficiency, consistency and conformity of the program during the

audit to be conducted within the framework of expectations of the management.

The audit to be conducted aims at assessing the activities under the program within the

framework of the audit components below and determining improvement areas.

▪ Economy; Obtaining appropriate quality and amount of resources at the most reasonable cost.

▪ Productivity; assessment of whether the output generated from the available resources is at an adequate level.

▪ Effectiveness; capacity to reach objectives and realise intended results

▪ Consistency; coherence between the defined goals, objectives and strategies of public

administrations and the planning and design of their programs and projects

▪ Sustainability; continuing to use activity outputs and making them widespread, as the case

may be, and reuse of them by other organisations and stakeholders

▪ Conformity; examining whether the projects, programs, processes and activities executed by

public administrations comply with relevant legislation, procedures and generally accepted

principles, and the maturity of the process regarding the institutional capacity of the

organisations executing such activities

Within the scope of the audit aiming at assessing the above mentioned components, the internal auditors mostly plan to evaluate the program and implementation results and to use the method of measuring and reporting on performance.

Performance audit to be executed using these methods aims at;

▪ measuring and reporting on the results in the areas affected by the program

▪ understanding the difference between the current situation with the program and the alternative scenario without the program and understanding the real effect of the program, i.e. causality of the intervention on the results via the program.

B5. KICK-OFF MEETING

The internal auditor provided the auditee with information on the subjects below in the kick-

off meeting.


▪ Audit Scope

▪ Audit Stages

▪ Time Schedule

▪ Following Steps

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

141


▪ It is requested that the difference between the current situation with the program and the alternative scenario without the program be analysed, and the real effect of the program be understood, i.e. causality of the intervention on the results via the program.

▪ This way, it is intended to create a roadmap for further improvement of the program.

B5-Kick-off Meeting Minutes

AUDIT SUBJECT Shift to Electronic System Program Performance Audit

AUDITEE Presidency A Strategy Development Unit


PLACE OF MEETING Meeting Room of the Ministry

PARTICIPANTS









ISSUES RAISED

The internal auditor provided the auditee with information on the subjects below in the kick-off meeting.


▪ Audit Scope

▪ Audit Stages

▪ Time Schedule

▪ Following Steps

▪ List of Data Request


▪ It is requested that the difference between the current situation with the program and the alternative scenario without the program be analysed, and the real effect of the program be understood, i.e. causality of the intervention on the results via the program.

▪ This way, it is intended to create a roadmap for further improvement of the program.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

142

B6. DEVELOPING THE AUDIT MATRIX

As a result of the analyses and interviews conducted within the scope of the preliminary

work stage, internal auditors prepared the audit matrix which would set the general framework of

the field work. Below is a part of the audit matrix:


Audit Component

Audit Question

Criterion & Control Performance Indicator

Data Source Analysis Method

Analysis Strategy

EC

ON

OM

Y

Are there options similar in quality but more cost-effective for the purchase of goods and services?

- The tenderer who offered the lowest cost has been awarded with the tender to purchase the goods and services under the program. - Appropriate competitive conditions have been established during the tendering process. -There is no significant difference between the actual cost and the estimated cost of the works.

- Number of tenderers who participated in the tender - Difference between the offered cost of the purchase and the estimated cost (%)

- Program planning documents - Tendering

minutes - Estimated cost information - Tender specifications - Contracts

- Analytical review - Qualitative evaluation

- Reviewing the tendering documents, making an analysis by comparing the bids and unit costs in the bids separately - Making an analysis regarding the tenders by comparing the estimated cost and the bids received - Analysing the number of bids received.

EC

ON

OM

Y

Were a technology and platform fitting the needs and existing infrastructure selected for purchase of goods and services?

Needs analysis was conducted to understand the needs of the organisation and users when selecting technology and platform for purchase of goods and services. Technology and platform selected for purchase of goods and services fit the existing infrastructure of the organisation and the needs of both the organisation and users.

- Appropriateness of technology selection of purchase of goods and platform of software purchase to the existing infrastructure of the organisation - Assessments of the internal and external stakeholders on the subject

- Program planning documents - Interviews with Department of Data Processing - Interviews with process owners - Interviews with contractors

- -Qualitative assessment

- Analysing the information obtained in the interviews - Confirming together with the process owners the subjects and opinions highlighted during the interviews

EC

ON

OM

Y

Did the administration incur any additional cost other than the estimated cost for the program?

- No additional cost other than the estimated cost.

- Ratio of the unanticipated cost items to total budget under program budget (%) -Difference between planned and realised budget items (%)

- -Program planning documents

- - -Budget and

spending information

- -Interviews with process owners

- - Analytical review

- - -Qualitative

assessment

- -Analysing budget and spending

- - -Detecting and

analysing unanticipated spending, if any

- - -Obtaining

additional information from process owners on unanticipated spending

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

143


Audit Component

Audit Question

Criterion & Control



Analysis Strategy

PR

OD

UC

TIV

ITY

Were the activities and outputs envisaged under the program realised?

- Activities envisaged under the program and the expected outputs were realised.

- Realisation rate of the activities envisaged under the program (%) - Realisation rate of the outputs envisaged under the program (%)

- Program planning documents - Program interim and final report - Program monitoring records


- -Qualitative evaluation

- Defining the activities envisaged under the program and their outputs - Analysing the realisation percentage of activities and outputs

PR

OD

UC

TIV

ITY

Is there any problem preventing the electronic system from functioning as planned?

- No problem inherent in the design of the electronic system. - No systematic problem in software and hardware.

- Errors and problems arising from the platform used by the software - Errors and problems arising from software interface - Errors and problems arising from integration

- Interviews with the officials of the program department - Interviews with the officials of the Department of Data Processing - Interviews with the internal and external users - Tests via electronic system for each transaction type



- Receiving feedback from implementers and users on software - Tests via electronic system for each transaction type –Conducting quantitative analysis

EFFE

CTI

VEN

ESS

Did the program contribute to making the transactions easier and simpler?

- Interventions under the program reduced the work steps in transactions for the organisation’s personnel and the users.

- Number of work steps reduced for the organisation’s personnel - Ratio of the number of work steps reduced for the organisation’s personnel to total work steps (%) - Number of work steps reduced for the users - Ratio of the number of work steps reduced for the users to total work steps (%) - Percentage of the transactions completed in compliance with the service standards (%) - Satisfaction rate of the service takers (%)

- Process and work flow charts on manual transactions

- Process and work flow charts on transactions performed via electronic system - Interviews with process owners - Focus group meetings with the organisation’s personnel - Focus group meetings with the users



- Developing process and work flow charts used when working via manual system, comparing them with process and work flow charts used along with the electronic system, and analysing the number and qualities of the work steps reduced - Analysing through weighting the share of the number of work steps reduced to total work steps - Revising, and repeating if necessary, the analysis according to the feedbacks of the organisation’s personnel and users - Confirming the results together with the process owners

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

144


Audit Component

Audit Question

Criterion & Control



Analysis Strategy

EFFE

CTI

VEN

ESS

Did the program contribute to standardise the implementations?

- In parallel with the program, different implementations in the organisations decreased.

- Differentiation rate in the implementations of manual transactions (%) - Differentiation rate in the implementations of electronic transactions(%) - Feedback of the organisation’s personnel and users

- -Transaction records archive of the organisations

- -Interviews with process owners

- -Questionnaire with the organisation’s personnel

- - Questionnaire with the users



- Comparing the samples selected from the documents on the manually executed transactions in the organisations’ archives; analysing different practices - Taking records on electronic transactions from the archives of the same organisations, analysing them along with the documents taken from electronic data base, calculating the standardisation percentage through comparing them with manual transaction practices - Obtaining information on pre- and post-electronic system practices through a questionnaire with the organisation’s personnel and the users, and analysing it

EFFE

CTI

VEN

ESS

Did the program contribute to increasing efficiency for the organisation and users?

- Time spent on each transaction by the organisation’s personnel decreased. - Time spent on each transaction by the users decreased. - Cost per transaction incurred by the users decreased. - Time elapsed between application for and closing of the transaction decreased.

- Change in average transaction time for the organisation’s personnel (%) Change in average transaction time for the users (%) - Change in average cost per transaction incurred by the users (%) - Change in average transaction closing time - Change in transaction error rates (%) - Total time saved for the organisation - Total time saved for the users - Total cost saved for the users

- Transaction time measuring through on-the-spot observations - Questionnaire with the personnel of provincial organisation - Questionnaire on the sample to be selected from the users - Transaction statistics to be obtained from the software

- -Evaluating observation results

- -Analytical review

- Calculating transaction times through measuring with different types of transactions and questionnaires - Analysing total time and cost savings

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

145


Audit Component

Audit Question

Criterion & Control



Analysis Strategy

EFFE

CTI

VEN

ESS

Did the program contribute to making it easier to monitor and control the transactions and operations?

- Incorrect transactions are now monitored more easily. - Transactions with risk of irregularity are now monitored more easily. - Effectively detecting and punishing the irregularities

- Change in the transactions’ error rate (%) - Change in number of investigations launched due to irregularity (%) - Change in number of penalties imposed due to irregularity (%)

- Transaction statistics to be obtained from the software - Investigation and penalty statistics to be obtained from monitoring and control department - Interviews with the officials of monitoring and control department

- Analytical review - Qualitative evaluation

- Comparing error, investigation and penalty statistics - Analysing the trends

CO

NSI

STEN

CY

Are program objectives in the same direction with goals and objectives prioritised in strategic plan of the administration?

- Program objectives, priorities and activities are directly related with goals, objectives and activities prioritised in strategic plan of the administration.

- Objectives tree - Strategic plan - Program planning documents - Program closing reports

- Analytical review - Qualitative assessment

- Comparing and associating goals and objectives included in strategic plan of the administration with program objectives, priorities and activities - Performing objectives tree analysis; detecting directly and indirectly related objectives and activities, if any

SU

STA

INA

BIL

ITY

Are software, furnishing and inventory stock purchased under the program used actively?

- Furnishing and inventory stock purchased under the program are integrated with infrastructure of electronic system. - Furnishing and inventory stock are actively used. - Software and its sub-modules purchased under the program are actively used.

- Number of transactions performed each month as per electronic system sub-modules. - Assessments by process owners. - Assessments by personnel of the administration.

- Interviews with process owners - Focus group meetings with personnel of the administration - Electronic system database

- Sample

- Analytical review

- Qualitative assessment

- It will be assessed through the interviews how active the furnishing and inventory stock purchased are used in electronic system, if there is any equipment either inactive or unused. - It will be assessed whether there is any furnishing and inventory stock purchase which is not integrated with electronic system infrastructure, and its reasons will be questioned in interviews. - Detailed transaction records on software modules will be scrutinised, and it will be researched if there is any module not in active use.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

146

RİSK RISK CONTROL

Audit Component

Audit Question

Criterion & Control



Analysis Strategy

C

OM

PLI

AN

CE

Is the capacity and maturity level of the executive structure of the program adequate?

- Capacity and maturity level of the department responsible for executing the program is appropriate for performing program activities timely and at the desired quality. - Know-how and experiences of the officials charged with managing the program, implementing the activities and controlling the outputs are appropriate for executing relevant activities. - In-house or external experts are assigned during implementation of the program, when necessary.

- Ratio of number of technical personnel charged with management of the program to number of personnel assigned - Whether the personnel responsible for controlling the program outputs are experienced on software platform of electronic system - Number of experts included in implementation and total assignment time

- Interviews with process owners - Records on use of human resources

- -Analytical review

- -Qualitative assessment

- Analysing number of technical personnel charged with management of the program and their duration of assignment through proportioning it to total number of personnel assigned in the program and total duration of assignment - Analysing the number of experts included in implementation and their duration of assignment; comparing the processes which may require know-how with estimated durations - Qualitative assessments on know-how and experience of other human resources

C

OM

PLI

AN

CE

Are the competences of administration personnel included in program target group adequate for using electronic system?

- Adequate training, information meetings and promotion activities were carried out with the aim of taking the competences of administration personnel, target group of the program, up to a level capable of using electronic system.

Average training hours offered per personnel under the program

Assessments on trainings

Number and qualities of the information and promotion documents prepared under the program

Program monitoring records and program final report

Questionnaire with the personnel of the administration

Focus group meetings with personnel of the administration

Statistics on material on promotion and information and material distribution

Sample

Analytical review

Qualitative assessment

Regarding training offered under program, training types, total hours, and number of participants will be analysed and average training hour per participant will be calculated.

Feedback of the administration personnel obtained through questionnaire and focus group will be assessed.

Types and quality of material on promotion and information will be assessed.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

147




B7- Work Plan

AUDIT SUBJECT Performance Audit of the Transition to Electronic System Program

AUDIT NUMBER


Field Work 15.4.2016 – 8.5.2016

Reporting 11.5.2016 – 29.5.2016

PURPOSE(S) OF THE AUDIT

The audit aims at;

Evaluating the work and transactions related to the Transition to Electronic System Program within the framework of the following audit components and determining improvement areas.

- Economy; Obtaining appropriate quality and amount of resources at the most reasonable cost.

- Productivity; assessment of whether the level of outputs generated with the available resources is sufficient

- Effectiveness; capacity to realise the objectives and intended results - Productivity; The amount of input used per unit output or the amount of output per unit

input - Consistency; the coherence between the goals, objectives and strategies of public

administrations and the planning and implementation of programs and projects, - Sustainability; continuing to use activity outputs and making them widespread, as the case

may be, and reuse of them by other organisations and stakeholders - Conformity; examining whether the projects, programs, processes and activities executed

by public administrations comply with relevant legislation, procedures and generally accepted principles, and the maturity of the process regarding the institutional capacity of the organisations executing such activities

-

AUDIT SCOPE

The audit encompasses the activities and practices on the sub-processes below regarding the shift to electronic system program.

- Planning

- Program development

- Conceptual design

- Implementation and its results

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

148

AUDIT METHOD

Within the scope of the audit, the performance of the transition to electronic system program will be assessed.

Below are the audit methods to be used in this respect.

- Assessment of program and implementation results

- Measuring and reporting on performance.


The findings specified in the 2014 systems audit report were examined during the preliminary work stage of the performance auditing exercise.

PREPARATORY WORK

Preliminary data were collected and analyses were made within the scope of the audit. In this scope, following documents were prepared.



Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

Me

tho

ds o

f A

na

lysis

Components

Econom

y

Pro

ductivity

E

ffectivene

ss/E

ffi

cie

ncy

C

on

sis

ten

cy

Susta

ina

bili

ty

Confo

rmity



149

C. FIELD WORK

C1. DATA COLLECTION

During the field work, secondary data and primary data were collected respectively, and

the trends of A, B and C type transactions executed on electronic system were analysed.

C2. DATA ANALYSIS

During the field work, performance results on the already available performance

indicators were measured and reported. Secondary data and primary data were collected

respectively, and the trends of A, B and C type transactions executed on electronic system were

analysed.

One of the most important issues regarding the method of evaluating the program and

implementation results is to understand the difference between the situation where the program

is realised and the presumptive situation with no program, i.e. real effect of the program. The

analyses were made by taking this into account.

In the questionnaires conducted for calculating the change in transaction completion

times, the users and the organisation’s personnel were asked to state the average completion

time of the transactions performed on paper and those performed electronically:

# C2A1

Analysis

Analysis Method

Analytical review / Trend Analysis

Preliminary Survey

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

Transactions 2011 2012 2013 2014 2015

Type A transaction

Electronically - - 45.407 539.762 694.031

On paper 586.344 570.405 612.097 127.276 -

Total number of transactions (A) 586.344 570.405 657.504 667.038 694.031

Type B transaction

Electronically - - 12.860 69.208 79.123

On paper 67.987 59.431 52.158 8.474 -

Total number of transactions (B) 67.987 59.431 65.018 77.682 79.123

Type C transaction

Electronically - - 11.983 22.809 36.879

On paper 33.278 30.146 23.006 12.862 -

Total number of transactions (C) 33.278 30.146 34.989 35.671 36.879

150

Current Situation

Yearly transaction numbers are taken from the system and compared.

This comparison reveals the pre-electronic system, the transitional period and post-transition system trends and develops some inputs for other analyses.

150

# C2A2

Analysis

Analysis Method


Current Situation

Time saved (man hour) for the organisation’s personnel in Type A transactions as a result of the electronic system was analysed.

While it took the staff 12-15 minutes on average to manually (on paper) perform a type A transaction, after the electronic system, the same procedure takes 5-6 minutes, indicating a 60% time saving.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

Transaction Time

Organization’s Personnel

User

Type A Transaction

Electronically 0,1

0,75

On paper 0,25 2,5

Total Transaction Times for

the Organization’s Personnel

2011 2012 2013 2014 2015

Type A transaction

Electronically - -

4.541

53.976

69.403

On paper 146.586

142.601

153.024

31.819

-

Total number of transactions (A)

146.586

142.601

157.565

85.795

69.403

Presumptive scenario (On paper )

146.586

142.601

164.376

166.760

173.508

Saving - -

6.811

80.964

104.105

Current Situation 146.586

142.601

157.565

85.795

69.403

Total Transaction Times for

User

2011

2012

2013

2014

2015

Type A transaction

Via electronic system

-

-

34.055

404.822

520.523

On paper 1.465.860 1.426.013 1.530.243 318.190 -

Total number of transactions (A)

1.465.860

1.426.013

1.564.298

723.012

520.523

Presumptive scenario (On paper )

1.465.860

1.426.013

1.643.760

1.667.595

1.735.078

Saving - - 79.462 944.584 1.214.554

Current Situation - - 1.564.298 723.012 520.523

155

# C2A3

Analysis Time saved by the user in type A transactions thanks to electronic

system (man hour)

2013 2014 2015

1.643.760 79.462

1.564.298 -5%1.667.595 1.735.078

944.584 -57% 1.214.554 -70%

723.012

520.523

No program Saving Current No program Saving Current No program Saving

Current situation situation situation

Analysis Method


Current Situation

- While it took a user 2-2.5 hours on average to manually perform a type A transaction, the same takes 40-45 minutes with the electronic system, meaning a 70% time saving.

- The analysis revealed that saving is much higher for the users.

# C2A4

Analysis

65

No

program

5

Off-program

saving

35

Saving

due to

program

25

Current

situation

-58%

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

Type A transaction

On paper Electronically Amount of

saving

Price of valuable paper 22,00 TL 0,00 TL 22,00 TL

Transaction fee 18,00 TL 15,00 TL 3,00 TL

Fee 15,00 TL 10,00 TL 5,00 TL

Commission of the intermediary

10,00 TL 0,00 TL 10,00 TL

Total 65,00 TL 25,00 TL 40,00 TL

152

Analysis Method


Current Situation

Direct cost saving of the businesses using the electronic system was also analysed.

It was understood that, as for type A transactions, 40 TL was saved per transaction in the current situation.

However, during the interviews, process owners stated that such saving cannot be wholly attributed to the program; the fee was reduced by another legal amendment. Therefore, 5 TL saved from fees were associated with off-program factors and removed from the impact analysis.

After external factors were separated, amount of saving attributed to the program was calculated as 35 TL per transaction.

Another indicator examined is the rate of irregularity. Relevant analysis results are provided below.

# C2A5

Analysis

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

Numbers 2011 2012 2013 2014 2015

Total number of transactions performed on paper

687.609

659.982

687.261

148.612

-

Total number of transactions performed on electronic system

- - 70.250 631.779 810.033

Number of transaction errors on paper

NA

Number of electronic transaction errors

- - 11.834 79.239 93.045

Number of Transactions Investigated

2011

2012

2013

2014

2015

Number of transactions performed on paper

24.273

26.069

23.779

4.666

-

Number of transactions performed electronically

-

-

6.913

56.292

58.727

Number of Transactions Punished

2011 2012 2013 2014 2015

Transactions performed on paper

3.851

3.894

3.986

921

-

Transactions performed electronically

-

-

857

6.507

10.125

Ratio of Transactions Performed on Paper

2011

2012

2013

2014

2015

Ratio of number of transactions investigated to total number of transactions

3,53%

3,95%

3,46%

3,14%

-

Ratio of number of sanctioned transactions to total number of transactions

0,56%

0,59%

0,58%

0,62%

-

153

Number of transactions investigated and punished for all transaction types

Analysis Method Analytical review / Trend Analysis

Current Situation For all types of transactions, number of transactions investigated and punished was examined.

A significant difference occurs when the number of manually performed transactions investigated and punished are compared with those performed electronically.

As for the electronic system, the transactions investigated and punished are much higher when compared with the other method. However, share of the transactions punished out of those investigated is lower when compared with the transactions performed on paper.

Secondary data analysis was followed by the analysis of primary data on the below

mentioned stakeholders. The method and stakeholder groups used in primary data collection

studies are presented below.

# C2B1

Stakeholder Group

Organisation’s Personnel Using the Program

Analysis Method

Focus Group Meeting

Analysis Strategy

A group of 20 was selected from among the organisation’s personnel using the program considering the regions where the service is provided. Participants of the focus group were determined considering the number of organisation’s personnel in the region and by taking the opinions of the auditee.

Subject and content of the focus group as well as the topics to be asked and assessment methods were shared with the participants before the meeting.

Current Situation

Following results were obtained at the end of the focus group meeting.

It was stated that the personnel of the decentralised organisation frequently made mistakes while using the electronic system. Such errors mostly include:

Entering incorrect information

Not entering the information into assigned areas

Missing interim approvals required for some transactions

Completing the transaction without filling the cells assigned to some important information

The officials stated that last two errors automatically leads the system to flag the transaction as irregular and to call for an investigation.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

Ratio of Transactions Performed Electronically

2010

2011

2012

2013

2014

Ratio of number of transactions investigated to total number of transactions

-

-

9,84%

8,91%

7,25%

Ratio of number of sanctioned transactions to total number of transactions

-

-

1,22%

1,03%

1,25%

154

# C2B2

Stakeholder Group

Organisation’s Personnel Using the Program

Analysis Method

Questionnaire

Analysis Strategy

A sample group of 90 was selected from among the organisation’s personnel using the program considering their regional distribution. An online questionnaire was conducted with this group.

A sample of 120 firms was selected from among the businesses using the system considering the distribution of number of transactions. The personnel executing the transactions in the mentioned firms was reached by phone and the questionnaire was conducted on the phone.

Current Situation

Following results were obtained regarding the transaction times:

In the questionnaire conducted with the organisation’s personnel, it was stated that;

Type A transactions performed manually (on paper) takes 12-15 minutes on average,

This reduced to 5-6 minutes after the electronic system.

Following answers were received in the questionnaires conducted with the personnel of the businesses using the system.

Type A transactions performed manually (on paper) takes 2-2.5 hours on average,

This reduced to 40-45 minutes after the electronic system.

C3. IDENTIFICATION OF THE FINDINGS AND DEVELOPMENT OF RECOMMENDATIONS

C3.1. Analysing the results and sharing them with the management

Following results are obtained as a result of the benchmarking analysis. Analysis results

were examined together with the relevant process owners.


Analysis Result Finding Work Paper

C3.1C1

Change in number of transactions over years

Yearly transaction numbers are taken from the system and compared.

Number of all types of transactions per year increased.

A1

C3.1C2

Time saved by the organisation’s personnel due to electronic system in type A transactions (man hour)

Time saved by the organisation’s personnel with the electronic system in type A transactions (man hour) was analysed.

▪ While type A transactions performed manually (on paper) take 12-15 minutes on average for the organisation’s personnel, it decreased to 5-6 minutes with the electronic system, indicating a 60% saving.

▪ The analysis revealed that saving is much higher for the user’s personnel.

▪ While type A transactions performed on paper take 2-2.5 hours on average for the users, it decreased to 40-45 minutes with the electronic system, which means a 70% time saving.

A2

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

155


Analysis Result

Finding Working Paper

C3.1C3

Change in direct cost incurred by the users per type A transaction

▪ Direct cost saving of the businesses using electronic system was also analysed.

▪ It was understood that 40 TL was saved per transaction in the current situation.

▪ However, during the interviews, process owners stated that such saving cannot be wholly attributed to the program; the fee was reduced by another legal amendment. Therefore, 5 TL saved from fees were associated with off-program factors and removed from the effect analysis.

▪ After external factors were separated, amount of saving attributed to the program was calculated as 35 TL per transaction.

A3

C3.1C4

Number of transactions investigated and punished for all transaction types

▪ Number of transactions investigated and sanctioned was examined for all transaction types.

▪ A significant difference occurs when the number of manually performed transactions investigated and sanctioned are compared with those performed electronically. As for the electronic system, the transactions investigated and sanctioned are much higher when compared with the other method. However, share of the transactions sanctioned out of those investigated is lower when compared with the transactions performed on paper.

Evet

A4

C3.2. Analysing the findings, risks and root causes

The analyses revealed that the number of transactions investigated remarkably increased,

yet the number of penalties imposed did not increase by the same ratio. This makes the detection

of irregularities harder, which causes extra work load in the relevant department.

Main reason is that investigation is automatically launched for any transaction related to

which an error is made on the electronic flow. In this respect, two improvement areas stand out:

Conducting a specific study for separating the errors and irregularities on the system

Reducing the errors made during the transaction.

As for the root causes of the errors made during the transaction, improvement areas

concluded from the interviews are listed below.

Lack of adequate input provided by training activities for implementation,

Seminars not focusing on the benefits and modules of the system,

During the trainings, sample transactions are performed by the trainer but not by the

participants

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

156

Sample transactions shown by the trainers are simple and smooth, the sample pool used

in the training does not represent the entire typology of transactions, nor contains

complicated transactions,

Although the informative materials are rich in variety (help menu, Frequently Asked

Questions, expressive videos, leaflets and user manuals), they do not meet the needs

regarding the content,

“Help Menu” and “Frequently Asked Questions” part have not been updated since the

system was first launched.

D.REPORTING


1 FINDING

Subject of the Finding

High number of incorrect transactions and extra work burden for the organisation due to investigation of these transactions

Importance Level of the Finding

MEDIUM

Relevant Unit

Strategy Development Unit

Current Situation

In the scope of the audit, it was researched whether the electronic system contributes to making it easy to monitor and control the transactions and operations. The analyses revealed that the number of transactions investigated remarkably increased, yet the number of penalties imposed did not increase by the same ratio. This makes it harder to detect the irregularities and causes extra work load in the relevant department. Number of transactions investigated and punished for all transaction types

Transactions performed on paper Transactions performed electronically

3,53% 2011

3,95% 2012

3,46 9,84% 2013

1,22%

3,1 8,91%

2014 1,03%

7,25%

2015 1,25%

Ratio of transactions investigated to total number of transactions

Ratio of transactions punished to total number of transactions

A significant difference occurs when the number of manually performed transactions investigated and sanctioned are compared with those performed electronically. As for the electronic system, the transactions investigated and sanctioned are much higher when compared with the other method. However, share of the transactions sanctioned out of those investigated is lower when compared with the transactions performed on paper.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

0,56%

0,59%

% 0,58%

4%

0,62%

157

As a result of the interviews with the Department of Monitoring and Control, it was stated that the reason is the increasing number of errors in transactions. We intended to analyse whether the number of errors increased or not, but failed to make any comparison, because no record of error had been kept in the paper-based system. We researched this in the interviews and questionnaires conducted during primary research, and the results revealed that the decentralised organisation frequently make mistakes while using the electronic system. Such errors mostly include:

Entering incorrect information

Not entering the information into assigned areas

Missing interim approvals required for some transactions

Completing the transaction without filling the cells assigned to some

important information

The officials stated that last two errors automatically leads the system to flag the transaction as irregular and to call for an investigation.

Cause As a result of the interviews with the Department of Monitoring and Control, it was stated that some errors made by the organisation’s personnel were defined as irregularity by the system and this resulted in an investigation launched on the transaction concerned. It was stated that relevant officials of the Department of Monitoring Control determined, as a result of a preliminary review, that such transactions were not irregular, but incorrect, and they decided that no investigation was necessary. As a result of the interviews and questionnaires conducted with the personnel of the decentralised organisation, we received a feedback that the training programs on the use of the electronic system were not sufficient, either qualitatively or quantitatively. The results of the analyses conducted with the personnel of decentralised organisation revealed that the most frequent feedback on dissemination activities is about training. Accordingly, most significant comments are as follows:

Lack of adequate input provided by training activities for implementation,

Seminars not focusing on the benefits and modules of the system,

During the trainings, sample transactions are performed by the trainer but not by the participants

Sample transactions shown by the trainers are simple and smooth, the sample pool used in the training does not represent the entire typology of transactions, nor contains complicated transactions,

Although the informative materials are rich in variety (help menu, Frequently Asked Questions, expressive videos, leaflets and user manuals), they do not meet the needs regarding the content,

“Help Menu” and “Frequently Asked Questions” part have not been updated since the system was first launched.

Risks and Effects

High number of errors and transactions against which an investigation is launched result in extra work burden in the Department of Monitoring and Control. Such extra work burden could not be managed with existing resources, and the transactions against which investigation is launched may be closed without being examined in detail. This does not make monitoring and control easy, and it also increases the risk of the irregular transactions be ignored.

High number of incorrect transactions constitutes quality cost for the organisation. Correcting the incorrect transactions may take time either equal to or more than the ordinary transaction time, depending on the feature of the error. This reduces efficiency in the provincial organisation.

Criterion As for the transactions performed electronically, a reasonable level is aimed for transactions being investigated or sanctioned. The aim is to have a maximum of 5% of transactions being investigated. In cases with a higher ratio, rationality controls must be made on the system for the whole investigation.

Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies

158

Recommendation

Following actions are recommended to be taken regarding the findings detected within the scope of the audit performed.

12. Conducting a specific study for separating on the system the transactions which seem to be an irregularity due to the information entered incorrectly.

In this respect:

Categorising existing errors and analysing them in detail

Making a prioritisation of more critical areas to detect the irregularities

Updating the interface in the electronic system open to the organisation in line with such prioritisation

13. As for the training activities on the transition to electronic system, training content is recommended to be updated in such a way to include the following:

▪ Program conceptual design

▪ Key controls and control objectives

▪ Application examples

An important part of the training is recommended to be allocated to the studies on implementing the relevant transactions on test environment.

14. It is recommended that the content of the user support features of the program be enriched with practical implementations and transactions.

User support feature is recommended to encompass the following points among others.

▪ Help menu,

▪ FAQs,

▪ Expressive videos,

▪ Leaflets and user manuals

Opinion of

the Auditee




[ ] We do not agree with the

recommendation.


Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies)

159

1

Responsible Person

Action to Be Taken Date of Completion

Strategy Development Unit /


A study group will be formed for conducting studies with the aim of separating, on the system, the transactions which seem to be an irregularity due to the information entered incorrectly. Such study group will start serving as of 2017. All implementation in 2015 – 2016 will be included in the study.

Categorising existing mistakes and analysing them in detail

Making a prioritisation of the more critical areas to detect the irregularities

Updating the interface in the electronic system open to the organisation in line with such prioritisation



Training programs will be organised on the transition to electronic system in a way to include the topics below. It will be ensured that all users participate in these training programs.

Program conceptual design

Key controls and control objectives

Application examples



Design development program required for the program user support feature to include the parts below will be completed as of 2017.

Help menu,

FAQs,

Expressive videos,

Leaflets and user manuals

▪

D2. CLOSING MEETING

The internal auditor provided the auditee with information on the subjects below in the

closing meeting.





Min

istr

y A

Sh

ift

to E

lect

ron

ic S

yste

m P

rogr

am P

erfo

rman

ce A

ud

it

160

D2-Closing Meeting Minutes

AUDIT SUBJECT Shift to Electronic System Program Performance Audit

AUDITEE Presidency A Strategy Development Unit


PLACE OF MEETING Meeting Room of the Ministry

PARTICIPANTS








ISSUES RAISED

The internal auditor provided the auditee with information on the subjects below in the closing meeting.




▪ Identification of areas of improvement in line with the findings and development of recommendations.

Per

form

ance

Au

dit

Man

ual

for

Pu

blic

Inte

rnal

Au

dit

ors

(Cas

e St

ud

ies)

Documents

Performance Audit Manual for Public Internal Auditors Case ... · The head of the internal audit unit assigned internal auditors Aylin Kaya and Mehmet Akın for this engagement, in