Chapter [4] Created By Manish Mathur. Testing : Definition Testing is the process of assessing Correctness, Completeness and Quality of system. Testing

Created By Manish Mathur

Chapter [4]

~ Testing ~General &

Automated Controls


Testing : Definition• Testing is the process of assessing Correctness,

Completeness and Quality of system.• Testing is a process to determining whether the

controls are adequately protect the system.• Types –– SUBSTANTIVE : To prove the integrity of the actual

processing and To ensure that processes work to produce reliable results.

– COMPLIANCE : To ensure that system controls adhere to management directives.


• Phases –

– PLANNING : Here auditor determines the way to collect the evidence to achieve objectives IS audit.

– Testing : Here auditor tests the effectiveness of IS controls.

– Reporting : Here auditor concludes and reports the result of audit to the management.


Audit Planning

• Planning occurs throughout the audit and includes the following activities – Obtain an understanding of the entity and

its operations. Obtain an understanding of internal

controls. Assess the risk. Design the nature, extent and timing of

audit procedure.


• Auditor uses the concept of Materiality and Significance.

• According to these concepts –Auditor is not required to spend resources on item that are not material and significance i.e. those that would not affect the judgment of users of audit report.


AUDIT TESTING

• Testing Methodology-– Auditor must find testing methods to determine

that controls are effective. This may include reviewing documentary evidence, conducting personnel interview and personal observation.

• File interrogation-– Auditor must browse directories of PC to

investigate use developed application files.

Some important decisions before testing begins ~


• Test pack-– Auditor uses valid and invalid data to test the

ability to prevent, detect, and correct errors.– The intensity and extent of testing depends upon

importance of the application.

• Automated tools –– Audit team can use GAS (Generalised Audit

Software) to do sampling, data extraction, summarizing and reporting.


Tasks – Understanding the entity and key business process. Understanding entity’s network structure. Identify key area of audit interest. Assessing IT risks. Identify critical control points Understanding of IS controls. Performing other audit procedures.


Types of Control Audit Test

Financial Audit –• If IS control audit is performed as a part of

financial audit, the auditor understand the controls over financial reporting to assess the risk of misrepresentation.

Performance Audit –• If IS control audit is performed as a part of

performance audit, the auditor should evaluate the design and operating effectiveness of all the controls.


The following factors assist the auditor to determine which audit procedure to use to collect audit evidences –

The extent to which internal controls are to be tested.

The availability of evidences outside the system. The relationship of system controls and data

reliability. Audit Objectives.


Key areas of Audit interest

• Key areas are those applications and files that are critical in achieving audit objectives.– For financial audit : key financial applications– For performance audit : all key system

applications• For each key area, auditor should document

–– Operational location– Significant components (h/w, s/w)– Other support systems/resources– Prior audit reports


Understanding IS controls

• Identify entity wide(component level) controls and determine that they are effectively designed and implemented.

• Identify business process level controls for key application and check for effectiveness.

• Any internal audit or third party reviews performed during last year.

• Management’s plan for corrective action for the IS weakness and IS control weakness.

• Status of the prior year’s findings.


• Review any significant security incident for the last year.

• Review entity’s security plan.• Review risk assessment for the system.• Certification and accreditation document for

system.• Review BCP and DRP.• Review description of outsourced activities.• Relevant laws and regulation and their relation

with audit.• Procedure to consider risk of fraud that could

affect audit objectives.


• Plan audit resources.• Review current multi year testing plan.• Communication with entity’s management.• Audit procedure for service organization audit.• Decision to use the work of others.• Develop audit plan : objectives, scope,

methodology etc.• Decision to reduce testing of IS controls.

Auditor should document all these information as their Preliminary Investigation Documentation.


IS CONTROL AUDIT TEST

• Auditor uses information obtained in the investigation phase to test the effectiveness of IS controls.

• While performing audit, auditor should assess evidence to identify any revision needed in audit plan. For example –– If significant weakness is found the auditor may

decide to perform less testing in remaining areas.


Auditor determine effectiveness of controls at following levels –

• Processes designed to achieve control activities.

• E.g. Configuration mgmt., database updation, authorisation process etc.

Entity/Component

• Processes designed to control resources related to general support system

• E.g. Network, Operating system, Infrastructure application (e-mail, browser, utilities i.e. not directly related to business process.

System• Policies and procedures for controlling

specific business processes.• E.g. General controls i.e. security

guard, CCTV, door access locks etc.

Business process

application


Testing Critical Control Point

• Critical control point is that component of system which is of significant importance.

• Auditor tests controls related to the component, its operating system and its applications.

• For e.g. – Router. Auditor tests the control related to the router itself, its operating system and applications.


Effectiveness of IS controls

• Auditor should conduct test of those control technique that are effective in operation.

• To do so the best way is to test control in tired basis. Starting with –– Entity wide controls– System level controls– Business process application level controls– Data management controls

• Ineffective IS controls at each tier generally prevent effective control at the subsequent tier.


General controls• General controls at entity wide and system level

can be tested using techniques such as Inquiry, Observation, Inspection and re-performance thru test software.

• After reaching favorable conclusion on general controls at these level auditor test general control at business process application level.

• If general control are not effectively operating then auditor should-– Determine nature and extent of risk resulting from

ineffectiveness.– Identify and test any manual controls as compensating

control.


Application controls• Auditor tests those application controls that

achieve control objectives when other general controls are ineffective.

• If application controls are not likely to be effective auditor –– Understand the risk in terms of impact on audit

objectives.– Identify any manual controls that achieve the control

objectives.• If in the previous year controls were ineffective

and management have not significantly improved it, the auditor need not to test them.


Appropriateness of Control test

To keep appropriateness of control tests the auditor should perform appropriate mix of audit procedure that includes the following –

Inquiries of IT and management personnel Questionnaires Review documentation of control procedures Inspection of approvals(authorisation) Analysis of system information(configuration) Analysis of output (accuracy of processing) Review of data file Re-performance of the control (use of test data)


Multi year testing plan

• Where auditor regularly performs control audit of the entity, the auditor may develop a multi year plan for control audit.

• These plan should cover not more then 3 years and include schedule and scope of assessment.

• Under multi year plan each control is tested at least once during the multi year period.


• This concept allow auditor to test controls on risk basis rather then testing every control every year.

• For example a multi year plan for an entity with 7 applications might include comprehensive test of 2-3 application annually.

• Multi year plans are not appropriate in all situations. For example –– They are appropriate for first time audit.– They are not appropriate where audit has not been

tested within a recent period.– For entity that do not have strong entity wide

controls.


Documentation of control testing phase

Information gathered during testing phase should be documented. This include –

Understanding of IS. IS control objectives and activities. Description of control techniques used by entity. Specific test performed. Description of nature, extent and timing of test. Evidence of effectiveness of controls. If ineffective then compensating controls. Auditor’s conclusion about effectiveness of controls. For each weakness; material weakness, significant

deficiency or just deficiency.


Audit Reporting After completing testing auditor summarizes the

audit result, draw conclusion on the control weakness.

Auditor prepares this report on entitywide, system and BPA level collectively.

Such documentation may be developed as the audit progresses, allowing auditor to demo. that the weakness exist and can be exploited.

Auditor should also document the potential impact of weakness on completeness, accuracy, validity, confidentiality of system.


Some audit terms

Substantive testing Substantive testing is used to determine the

accuracy of information generated by a process. Auditor generate and process test data to verify

the processing steps. Where controls are evaluated as ineffective,

substantive testing may be required. Auditor uses CAAT to generate test pack and

conduct the test.


Some audit terms..

Analysis Interviews and tests provide the raw facts for

drafting a audit report but does not guarantee to produce a quality audit report.

Analysis is important to convert this raw material into finished product.

Timely analysis gives the auditor time to conduct further test and allow more time for corrective actions.

Thorough analysis includes the following 4 steps –


Steps

Re-examination

Cause of deviation

Materiality Exposure

conclusion


Step 1 : Re-examination– The two factors to be re-examined are : Standard

and Facts.– Standard are the rules, procedures and

practices that defines how an operation under audit should function.

– The standards must be clearly understood by the auditors, because wrong understanding leads to incorrect findings.

– Facts are evaluated after standards are reviewed. For accuracy the sample should be • Large enough to reflect behavior of population.• Representative of current control activity


Step 2 : Cause of Deviation– After understanding standards and facts, auditor

identify the causes of the deviation.– Determining the cause is like answering the

following questions –• Who (responsible)• What (initiating event)• Where (system component)• Why (contributing factor)• When (timing)

– Cause determination helps to identify exposure and formulating recommendations.


Step 3 : Exposure and Materiality• These are consequences of deviation. • Exposure is the potential loss, harm, damage,

theft or inefficient use and • Materiality is a qualitative judgment about

whether a deviation’s frequency of occurrence and degree of exposure are significant enough for the deviation to be corrected.

• Degree of exposure is related to Proximity and Severity of risk.

• Proximity refers to the extent of asset availability to the users or environment. Limited access – less proximity.


• Severity refers to the amount of loss. The greater the value of asset and higher the proximity, higher will be severity.

• Frequency refers to how often the deviation will occur.

• With understanding of Materiality and Exposure auditor can identify why corrections should take place.


Step 4 : Conclusion• Conclusions are auditor’s opinion on, whether

the audit subject area meets the audit objectives.• Conclusions must be supported by factual

evidences.


Concurrent audit techniques

• There are two categories of CAT : Embedded modules and Special audit records.

• Some of the CAT are –Snapshot

s

ITF

SCARF

CIS


Snapshot• Special audit module built into the system

where transaction processing occurs.• It takes images of the transactions of audit

significance and stores them in auditor’s file.• Main issues to decide are –– Location of snapshot– Condition to capture the image– Reporting system


Integrated Test Facility

• It involves creation of dummy entity in the client system and processing special audit data against that.

• Methods of creation of test pack –– An embed audit module, recognize transaction

having certain characteristic. These tagged tr. can be used as test pack.

– Auditor may use test data specially prepared for audit.


• Method of removing effect of ITF Tr. –– Application system may be programmed to

recognize ITF Tr. and ignore them in reporting.– Auditor may submitting additional inputs that

reverses the effect of ITF Tr.


System Control Audit Review File

• It involves use of special audit module within system under audit.

• It provide continuous monitoring of system’s transactions.

• Collected information is stored in special audit file : SCARF.– Application system errors– Policy and procedure variance– System exceptions– Statistical sample– Snapshot and extended records– Profiling data– Performance measurement


Advantages of CAT1 • Reduction in the cost of audit.

2• Reduction in the time of

audit.

3• Improvement in the quality of

audit.

4• Comprehensive and detailed

audit.

5 • Surprise test capability

6 • Information to system staff on meeting objectives

7 • Training to new users


Disadvantages of CAT1

• Auditor should be able to obtain resources.

2• Useful where audit is

involved in development.

3• Audit need to have computer

background.

4• Useful where audit trail is less

visible.

5• Useful where cost of error is

very high.

6• Effective when system is

stable.


Hardware Testing• H/w testing is done against FRS and SRS.• Types –

Function testingUser interface testing

Usability testingCompatibility testingModel based testing

Error exist testingUser help testing

Security testingCapacity testing

Performance testingReliability testing

Installation testingMaintenance testingAccessibility testing


Auditor’s review of hardware

• Review of capacity management and performance evaluation procedure to determine –– Ensure continuous review of performance and

capacity.– Whether historical data obtained from : system

trouble log, processing schedule, system report, preventive maintenance are used on performance monitoring.

– Decision of buy and sell h/w is based on capacity planning and workload forecast.


• Review of Hardware Acq. plan to determine –– Mgmt issued a written policy regarding h/w acq.– Criteria for acquisition is laid out.– Procedure is established for acq. approval process.– There is awareness of budget constraint.– Request for acq. Is supported by C/B analysis.– All h/w are purchased thru IS purchase deptt.– Envi. is conducive & space is adequate for new h/w– Acq. Plan considers technology obsolescence.– plan considers lease expiration.– Document for h/w i.e manual, warranty card etc is

properly maintained.


• Review of change mgmt control to determine –– Changes in h/w are planned and scheduled.– Time for adequate installation and testing.– Operator’s manual is properly updated.– Cross reference between changes and cause.– Programmers and IS staff has been informed of all

h/w changes.


• Review of preventive maintenance practice–– Understand frequency of PM and compare it with

contract.– Vendor compliance with agreement.– Ascertain PM does not have adverse effect in

production scheduling.– Check that PM log is maintained.– Ensure PM contract commences when warranty

expires. – Verify PM contract has call response time defined.


OPERATING SYSTEM REVIEW

• Interview IS manager, system programming manager and others regarding –– Process of option selection.– Test procedure for system software– Review and approval procedure fro test results.– Implementation procedure– Documentation requirement

• Review the feasibility study– Same selection criteria are applied to all proposals.


• Review cost benefit analysis –– Direct financial cost of the product.– Cost of product maintenance.– Hardware capacity requirement.– Training and support requirement.– Impact of the product on the processing.– Impact on data security.– Financial stability of the vendor.

• Review control over installation of changed System software –– All updates are implemented.– Installation of changes SS is scheduled when they

least impact processing


– There is a written plan for testing.– Problems encountered during testing were

resolved and changes were re-tested.– Test procedures ensure that changes do not create

new problems.– Restoration procedure are in place.– Software must be properly authorised prior moving

from test to production environment.– Access to libraries is limited to individual’s need.

• Review system software’s maintenance activities –– Changes made to the SS are documented.– Vendor support current version of software.


• Review SS documentation –– Installation control statement.– Parameter tables.– Exit definition.– Activity log.

• Review SS for adequacy of controls, such as –– Change procedure controls– Authorisation controls – Access privileges controls– Documentation controls– Testing controls– Audit trails


• Review authorization document to determine –– Addition, deletion or change to access

authorisation is documented.– Attempted violation reporting and response is

documented.• Review SS security, to determine –– Procedure have been established to prevent bypass

of access control.– Procedure have been established to limit access to

system interrupt capability.– Physical and logical access controls are adequate.– Vendor supplied passwords are changed.


• Review database supported controls –– Access to shared data is appropriate.– Data organization is appropriate.– Change procedures are established to ensure

integrity of DBMS.– Integrity of data dictionary is maintained.– Data redundancy is minimised.


Network Review• Review the LAN, to understand –– LAN Architecture– Cost benefit analysis– LAN topology– LAN components– Internetworking– LAN uses– LAN administrator– LAN users


• Review LAN to make an assessment of –Threat Impact Controls

• Review physical access controls –

– Ensure that LAN h/w, file server and documentation are located in secured area.

– Verify that LAN wiring is physically secured.– Observe LAN file server and verify that it is secure.– Keys to file server facility is controlled.– Obtain copy of key log for the file server room and

determine that keys are assigned to appropriate persons.

– Select keys held by people and determine that these keys do not permit to access LAN facilities.


• Review Environment controls to –– Ensure that LAN file server is protected from

electric surges.– Ensure that AC and humidity control system are

adequate to maintain temperature.– Ensure that LAN server is equipped with UPS.– LAN file server is free of dust, smoke and

pollutants.– Backup disks are protected from environmental

damage.– Fire extinguishers are nearby.– Food and beverages are prohibited.


• Review Logical access controls to ensure –– Users have unique password, password are change

periodically and does not appears on screen while entry.– LAN access should be based on written authorization.– Remote access to the system supervisor should be

prohibited.– All log-on attempts should be logged.– LAN supervisor should maintain up-to-date information

of all outside communication.– Evaluate LAN server access profile.– Attempt to gain access using unauthorised ID/PWD.– If LAN is connected to an outside source through a

modem attempt to gain access to the LAN thru correct and incorrect means.

Documents

Chapter [4] Created By Manish Mathur. Testing : Definition Testing is the process of assessing Correctness, Completeness and Quality of system. Testing