In the first session we discussed Performance Auditing – Introduction – Planning & Resourcing – Performance Auditing involving IT System Development

In the first session we In the first session we discusseddiscussed

Performance Auditing–Introduction–Planning & Resourcing–Performance Auditing involving IT System Development–Performance Auditing involving operational IT Systems–Performance Aspect of Auditing in IT Environment

Session II Coverage-Performance Auditing in IT Environment

Evidence gathering techniques Evidence Analysis techniques

EVIDENCE GATHERING EVIDENCE GATHERING TECHNIQUESTECHNIQUES

The module contains a broad framework of evidence gathering techniques:

1. Survey2. Statistical sampling3. Benchmarking4. Focus groups5. Interviews6. Case studies

Appropriate evidence gathering techniques enhances the authenticity of audit findings

This is not an exhaustive list

Appropriate other techniques can also be used

1.SURVEY

Gathering specific information from a group of people or an organization by questionnaire

Used in planning and execution phases Planning – identifying issues or key concerns in

an issue Execution phase – necessary audit evidence Used to collect quantitative information to

estimate output or evaluate process of a project

STEPS TO USING SURVEY Decide the objective and

target population Decide the sample size and

method of drawing sample Decide the nature of

survey-direct interview, through post, phone, email

Frame the questionnaire Test the validity of the

questionnaire-use of counter questions, language, words etc.

Administer the survey and collect information

Checking the completed questionnaire

Analyze the results of the survey-qualitative and quantitative techniques, avoid bias, errors

Need for expert in large surveys

Strengths and concerns







Need for Sampling

It saves timeit involves less costIn some cases it may not be possible to

check 100%It may yield better results than 100%

checking in some cases

It means----

Audit sampling is the technique of evaluating the population

by testing a part of the population Two types of sampling 1. Statistical 2. Judgmental Statistical sampling is important as it

gives almost accurate results.

Audit sampling Types

Statistical sampling can be defined as any sampling procedure that uses the Laws of probability for selecting and evaluating a sample from a population for the purpose of reaching a conclusion of population.

Implication of sampling

If the auditor states that he has 95% audit assurance it means he is confident of a statistical sampling technique for the audit assurance.

In non- statistical sampling he can not state the accuracy in terms of certain amount.

Statistical Sampling Sampling means testing less than 100% of the cases in the population for

some characteristic and then drawing a conclusion about that characteristic for the entire population.

Traditionally, auditors use ‘test check’ (or judgmental sampling, non-statistical sampling) approach. This means checking a pre-determined proportion of the cases on the basis of the auditor’s judgment. This sampling technique can be effective if properly designed.

Test check does not have the ability to measure sampling risk and thus audit conclusions reached becomes rather difficult to defend.

For statistical sampling techniques, there is a measurable relationship between the size of the sample and the degree of risk.

Statistical sampling procedure uses the laws of probability and provides a measurable degree of sampling risk. Accepting this level of risk, (or conversely at a definite assurance level) the auditor can state his conclusions for the entire population.

In sum, statistical sampling provides greater objectivity in the sample In sum, statistical sampling provides greater objectivity in the sample selection and in the audit conclusion. selection and in the audit conclusion.

Attributes and Variable sampling Statistical sampling may be used in different auditing situations. The

auditor may wish to estimate how many departures have occurred from the prescribed procedures; or estimate a parameter in the population. Based on whether the audit objective is to determine a qualitative characteristic or a quantitative estimate of the population, the sampling is called an attribute or variable sampling.

Attributes sampling estimates the proportion of items in a population having a certain attribute or characteristic. In an audit situation, attribute sampling could estimate the existence or otherwise of a error. Attribute sampling could be used when drawing assurance that prescribed procedures are being followed properly. For example, attribute sampling may be used to derive assurance that procedures for classification of vouchers have been followed properly. Here, the auditor estimates through attribute sampling the percentage of error (vouchers that have been mis-classified) and sets an upper limit of error that he is willing to accept and still be assured that the systems are in place. Variables sampling would estimate a quantity, e.g., the underassessment in a tax circle.

Attribute sampling for test of controls.

Attribute sampling is defined as a sampling plan in which the sampling unit is defined as an account balance, a purchase invoice or any other constituent of the accounting population.

Attribute sampling is required to test whether the internal control is working or not.

Stages involved in Attribute sampling

Determining the sample size Selecting the sample Performing test of control procedures Evaluating the test results How to calculate the sample size? Define population Determine the controls Define exception/error

Stages Attribute Sampling Contd...

Determine the tolerable error– This is the maximum deviation the auditor is willing

to tolerate and still conclude that the audit assurance is there

Determine the expected occurrence rate Select the statistical table Locate the TE Read the table in the that column to the line that

contains error to obtain the sample size

Sampling Methods

Random Selection - each item in the population has a equal chance of selection.

Random selection assumes that the population is homogeneous.

Stratified selection - Population is non-homogeneous. Population is sub-divided into homogeneous groups and then a random sampling is done on the groups, ensuring a better representative sample.

Auditor to use his judgment in determining which kind of sampling is best suited to his audit job

Random selection

In this each and every item has an equal chance of selection

For.Eg. If 5 items out of 1000 are to be selected, it

can be 1,4,6,8,9 or 14,25,36,998.999. Random number tables can be used.

Random Sampling Methods

Simple random sampling :- each member of the population has an equal chance of selection.Useful when the population is uniform.

Stratified random sampling :-Population is divided into strata and random sample is drawn from each strata. This is useful when there exists stratification in the data and the method will ensure that members from each strata are represented

Systematic sampling :-Population members at equal intervals get selected. Often it might be easier to draw systematic sample than random sample. This would be particularly useful when cases are ordered by size, type or region. Then by selecting systematically one can ensure that cases having different attributes have been adequately represented.

This method should be used for samples to evenly cover a population range.

Multistage sampling :- In this sampling one can extract random sample of a sample. Ex. – Sample states, Sample districts within select states, sample blocks within selected district, sample villages within selected blocks, sample beneficiaries within selected villages. At each stage a suitable method of sample could be used

Random Sampling Methods

Methodology in selecting sample

Once the method is finalized design the actual sample.

For simple random sampling the following method could be used. For other types of sampling it is advisable to consult expert.

Simple Random Sampling (Attribute Sampling)

Used when audit desires to estimate an attribute in a population.

Useful for testing internal control If errors above certain level- auditor may

conclude – Internal Controls are weak The attribute, which the auditor is

interested here are errors/ abbreviations from process

Basic stages involved are :- a) Determining the sample size,

b) Selecting the sample and perform substantive audit tests on the sample,

c) project the results


(a) Determining sample size After defining the target population and the

attribute that audit wishes to test, the size of the sample required to be tested need to be determined. This can be done with through understanding of the following parameters :-

• Precision

• Confidence level

• Occurrence rate (p) or population proportion


Determining sample sizeParameters

Precision (E) Audit test on the sample will throw up an

estimate of the attribute for the population. The true population value of the attribute could be more/less than this estimate. The gap between the sample estimate and the actual population is the precision. The auditor has to decide the precision he desires to provide in his estimates.


Confidence Level The confidence level or the level of assurance

that audit needs to provide is to be defined. Confidence level states how certain the auditor is, that the actual population measure is within the sample estimate and its associated precision level. In case of performance audit, this level can be taken at 95 percent.


Occurrence Rate

The rate (p) or population proportion which is the proportion of items in the population having the attribute that audit wishes to test.

This is based on the judgment of the auditor.

Basic stages involved are :- a) Determining the sample size,

b) Selecting the sample and perform substantive audit tests on the sample,

c) project the results


(b) Simple Random Sampling ( Attribute sampling)

The sample could be selected using random number tables or through computers. Auditing software, e.g. IDEA is an efficient tool for sample selection. Once the sample selected, identified audit tests are to be applied on the sample. The proportion of the sample having the attribute that is under test is determined through audit.

( c ) Projecting the results The test results are to be projected to the population. Precision

can be calculated at the desired confidence level and sample size. Loading the precision on the sample value the upper estimate for the population can be made.

In example of testing internal controls, this estimate is the maximum error/aberration that is expected the the given confidence level. In case this estimate is less than the threshold of error/ aberration that the auditor can tolerate, the auditor can place assurance on the controls. When estimate is higher than the tolerable error/ aberration the auditor can not derive assurance from the controls. The auditor may, in such situations reduce the assurance he derives from the controls and increase the assurance required from substantive tests.







3. BENCHMARKING

Benchmarking is a process for measuring an organization’s performance or process against such organizations that consistently distinguish themselves in the same categories of performance

External or internal benchmarking can be done to identify opportunities of achieving better economy, efficiency and effectiveness

Can be used in planning and execution phases Planning-setting the audit criteria Execution-cause effect analysis by comparison

Steps for using benchmarking Deciding the aspects of

performance or process that will be benchmarked

Deciding the type of comparison and benchmarking partners

Collect data Determine the

performance gap Framing conclusions and

recommendations for betterment

Strengths: Objective review of processes,

policies and systems Objective data on methods of

operation Better ways of operating Supports recommendations for

change Target for improvement Concerns: High degree of skill Acceptability of the findings







4. Focus groups A qualitative research technique Selection of some individuals to discuss specific issues in

an informal setting Reactions of the group are used to explore attitudes,

beliefs, perceptions, and problems Causes and effects of problems to achievement of

economy, efficiency and effectiveness 6 to 12 participants, auditee staff or beneficiaries of the

programme assisted by facilitator, 90 to 180 minutes Used both during the planning and execution phases Execution phase for validation of findings

Steps for using this technique

Selecting a facilitator Determining the number of

focus groups Deciding the participants of

the focus groups A topic guide Conducting the focus group Recording the results of a

focus group Analyzing the results of a

focus group

Strengths: Different perspectives,

opinions and ideas Express opinions freely in

a group Less expensiveConcerns: Not selected statistically Cannot be projected to the

population at large Needs to be backed up







5. Interviews

An interview is a question-answer session to elicit specific information

Structured or individual (unstructured) Structured-specific wording and are asked in set

order Open ended or closed ended questions Elicit explanations, impressions and opinions Used both in the planning and execution phases Planning stage-identifying potential key issues Execution phase-to corroborate evidence

Steps for using interviews Preparing the questions-

determine the minimum data to be obtained

Determining the interviewees-structured interview selection by statistical sampling

Conducting and recording the results-video audio recording, notes of interview; expert interview and auditor interview in complex/controversial issues

Analyzing the results

Individual interview-strength: Flexible Used for probing New areas broadens the

perspectiveConcerns Weak evidence needs

collaboration Problem in keeping focus Can guess answers, not well

formulated







6. Case studies Case study is the examination of a selection of

incidents, events, transactions or item in order to understand or examine a programme or activity. It is an in depth study of individual cases to explore the audit issues

Used in the planning and execution stages Case study can help to develop key questions to be

focused on later in the main study Through examination of specific cases and can identify

reasons for bad performance Causes of bad performance and impact of a specific act

Steps for using case studies Deciding the issue to be

audited: specific question to be studied using the criteria need to be identified

Selecting the areas to be studied, judgmental selection, rationale clear and defensible

Conducting the case study: PA techniques used, document review, interviews, focus groups

Analyzing the results: qualitative and

Strengths: Cheaper than studying a larger

sample More accurate as greater in

depth investigation is possible Problems, cause and effect

analysis and pragmatic recommendations possible

Concerns: Judgment in selection Open to bias Difficult to determine generic

or an aberration

Session Coverage-Performance Auditing in IT Environment

Evidence gathering techniques Evidence Analysis techniques

EVIDENCE ANALYSIS TECHNIQUES

The module introduces some of the evidence analysis techniques :

1Quantitative data analysis

2 Data Spread 3 Regression analysis 4 Ratio analysis 5 Qualitative data analysis 6 Qualitative data matrices 7 Programme logic model

Appropriate analysis techniques enhances authenticity of audit findings


Other appropriate techniques can also be used

1. Quantitative data analysis

Measures of central tendency-compresses information about a distribution into a single number. The common measures of central tendency are mean, median and mode

Quantitative information regarding a variable is collected over a period of time.

The data can be presented in tables or can be presented diagrammatically through bar charts, line curves, histograms, etc

A single number can be determined which will summarize the variable. This is called descriptive statistics

The entire distribution can be presented

Mean, median and mode

Mean: Arithmetic average-influenced by extreme values and hence not a good choice in asymmetric data distribution

Median: Middle value of the observation-not affected by extreme value greatly

Mode: Most common value of a variable

Used when typical value of a variable is required

In performance audit variable could be a performance indicator

Depends on correctness and completeness of data

Mean used in case of symmetric/uniform data and mode of skewed data








2. Data spread

Refers to the extent of variation among cases

Range: Difference between the largest and the smallest observations-solely based on extreme values

Inter quartile range:Difference between two points in a distribution that bracket the middle 50 % of the cases

Standard deviation:Square root of the average of squares of deviations of each case from the mean

Normal curve Symmetric curve Irregular shape curve








3. Regression analysis

Assesses the degree to which two variables are co-related

Scatter plot is prepared of the variables

Linear relationship-line of best fit and use the equation

Multi-variate regression analysis assesses the influence of a number of variables on a dependent variable

Causes of audit findings or identify reasons for problems

Use of Regression analysis in PA

Test a relationship Identify unusual values Identify causal relationship

between variables Help in framing proper

recommendations E.g.:Audit observes very

few underprivileged students go to higher education. Causes could be:

Lower awareness

Lower attainment Perception Economic, social

conditions Used in identifying

the dominant reason Concerns-higher level

skills and prone to misinterpretation








4. Ratio analysis for comparison

Compares one quantity with the other

Easy to understand and apply

Actual with the expected values

To place an audit finding in context

To observe a change in variable over time

Concerns While calculating ratios

the base should be clearly chosen

Cases being compared legitimately comparable

Same units are used for both numerator and denominator








5. Qualitative data analysis

Coding the data Identifying themes

within the data Generating coding

categories on the basis of identifying themes

Analyzing and drawing conclusions

Slot data in terms of conditions, actions, intervening factors or consequences

Used in analysis of qualitative data

Planning stage to identify programme objectives and activities

Examination phase to summarize audit findings

Concerns: time consuming and tedious, needs expertise








6. Qualitative data matrices

Data matrices structure qualitative data in a matrix making it possible to highlight and derive conclusions

Used to identify causal links between data

Steps: Identify the dimensions of

a subject to be assessed and place them in horizontal axis

Identify what is to be used to assess the dimensions and list them along the vertical axis

Search the evidence gathered to find contributions to each box within the matrix

Useful in drawing inferences from a large amount of qualitative data








7. Programme logic model

Schematic representation of life cycle of a programme

Logical flow of programme design from mandate to results

Outputs in relation to objectives

Stages: Objectives

Inputs Processes Outputs OutcomeFlow chartsTo understand the

processes involved in an activity

System based auditing

In this session we discussed following In this session we discussed following Evidence Gathering TechniquesEvidence Gathering Techniques






We also discussed following Evidence Analysis Techniques







Documents

In the first session we discussed Performance Auditing – Introduction – Planning & Resourcing – Performance Auditing involving IT System Development