Ian M. Harper, CISA, CISM, CISSPChief Technology OfficerIan.Harper@twminc.com

Dealing with Your IT Process Failures

Improving Security with Root Cause Analysis

Lisa Johnson, CPA, CISA, CISM, CITP

Lisa.johnnson@twminc.com

Agenda

• Control Environment Survey• Risk Environment Survey• Control Weaknesses as Critical Process Failure• Root Cause Analysis Tools & Techniques• General Application of Root Cause Analysis to

Control Weaknesses• Common Pitfalls• War Stories

The Controls Equation

Increased complexity of control drivers

Increased complexity of systems

Dramatically increased complexity of IA controls environment

+

Explosion of IA Drivers

• General Business• US Sarbanes-Oxley (SOX)• EU 8th Company Law Directive• Japanese Financial Instruments and Exchange Law• Australia’s Corporate Law Economic Reform Program

• Specific Market Regulations• US Gramm-Leach-Bliley (GLB)• EC Markets in Financial Services Directive (MiFID)• US Health Insurance Portability and Accountability Act (HIPAA)

• Market Forces (the real driver)• Protection of business, shareholders and other market players’

interests per best practices

OrganizationalControl Frameworks

• May be home-grown or based off of internationally recognized frameworks• COSO, COBIT, etc.

• From the perspective of RCA, all implemented control frameworks are generally equally valid• Unless the framework is the cause for the

weakness!

The Role of Risk

• A necessary evil• "If the primary aim of a captain were to preserve his

ship, he would keep it in port forever." –Thomas Aquinas

• "Risk is what separates the good part of life from the tedium." –John “Johnny Zero” Foley

• But one that must be controlled• “From others' slips some profit from one's self to

gain.” –Publius Terentius Afer

Risk LifecycleU

NK

NO

WN

RIS

KA

CC

EP

TE

D R

ISK

IDENTIFICATION OF RISK

Concept Plans Components

InitialSystem

OperationalSystem

Initial RiskAssessment

RiskAssessment

Developmental ST&E

Inspection & Acceptance

Certification

Mo

nito

ring

&P

eriod

ic As

se

ssm

en

t

SYSTEM DEVELOPMENT

What Risks?

• Financial – risks that directly cost us money• Strategic – risks that affect our ability to meet our

various goals and objectives• Operational – risks that introduce inefficiencies in

achieving our goals• Compliance – risks that

may put us in jail or may cost us in fines

• Reputation – risks that undercut the trust in us held by others

Control WeaknessIdentification

• Risk Management Process• Allow us to fix before the weakness is

exploited

• Incident• Pressure-cooker environment• Personal and business credibility is at

stake

ControlWeaknesses

• Identification of presence of unacceptable risk

• Critical Process Failure (Key Point)• In manufacturing: a defect• Ultimately, a quality issue in the

manufacture of controls

“Weakness”in Context

Control Framework

Controls

Business desires to produce quality controls

Continued commitment to manage risks

Weakness Control framework or controls have failed (execution or quality)

Object Denotes

Control WeaknessObservations

• Usually occur in batches• Typically affect various areas of the

organization• Pareto Principle

• 80% of cost of control issues can be attributed to 20% of the total population of defective controls

Key Point: These observations typically indicate systemic causes

Root Cause Analysis

Enter Root Cause Analysis

Root Cause Analysis

• Definition:• A structured approach to identify factors causing

an undesirable event• Goal is to prevent recurrence

• A toolbox of tools and techniques tailored to a case, not a rote process• Each team/practitioner will use different

techniques and tools• Same team/practitioner may use different

approaches on different cases

RCA Roots &Applicability

• Manufacturing• Focus on quality production processes• Causes related to workforce, materials,

methods, and machinery• IT

• Need to view control framework as an assembly line

• Have to shift focus to those aspects that deal with IT and with virtual objects

• Machinery is never the cause

RCA is Not New to IT

• Auditors, remember the Cause statement of standard findings?

• So why don’t we do this in practice?• Blame game and rejection of findings by

management• Lack of resources and return on investment• Environment that does not promote creativity

RCA Stages

• Information Gathering & Problem Identification

• Process Analysis and Problem Understanding

• Brainstorming• Data Collection• Root Cause Determination• Root Cause Elimination

Information Gathering/Problem Identification

• Goals:• Gather information about the observed

weakness• Identify the actual problem

• Tools & Techniques:• Timeline

• Used throughout the RCA effort• Needs to identify pertinent data as a reference

point

Information Gathering/Problem Identification

• Tools & Techniques (continued):• WWWWH

• What? equipment involved, what was wrong.

• Who? who took what action, may be unknown

• When? day, date, time of occurrences, SDLC

stage, etc.• Where? physical, logical locations involved • How? rate of occurrence, scope of effect

Information Gathering/Problem Identification

• Concerns:• Integrity starts here• Data needs to be attributable• Data preservation is key here• Identified problem must be accepted by the

team and management• Problem definition should not jump to

conclusions (avoids “no” or “lack of” statements)

Process Analysis/Problem Understanding

• Goals:• Understand the basic function of applicable

business processes and controls• Understand the relationship of the problem

to processes and controls• Tools & Techniques:

• Document review• Interview of personnel

• Need to focus on developing rapport and furthering openness

Process Analysis/Problem Understanding

• Concerns:• Continued need to preserve data• Need to be resourceful in finding sources

of information• Trouble tickets, security plans, development

artifacts, audit logs, helpdesk statistics, etc.• Need to focus on developing rapport and

furthering openness of the process• Remember people develop personal worth from

their work

Brainstorming

• Goals:• Identify the full set of possible causes for the

defined problem based on information obtained• Prioritize possible causes for further review

• Tools & Techniques:• Unstructured Brainstorm Session

• Effective in symbiotic teams• All personnel provide possible causes while one member

records ideas• Trades risk of domination for more freedom

Brainstorming

• Tools & Techniques (continued):• Structured Brainstorm Session

• Effective in stratified teams (personality or position)

• Round Robin – Each individual gets a chance to provide ideas in turn

• Secret Ballot – Ideas are turned in on chits secretly

• Trades uninhibited expression for fairness of process

Brainstorming

• Tools & Techniques (continued):• Comparison Tables

• Used to resolve conflicts on prioritization• Priority of each possible cause is voted on by

each team member• The cause with the most votes tallied is

prioritized over the least number of votes• Can only be used for small sets of data

Comparison Table Example

– 3 causes, 10 team members– Resulting Prioritization: A, C, B

A

A/B 7

A/C 6

B

3

C

4

B/C 2 8

Total 13 5 12

Brainstorming

• Concerns:• At this point do not attempt to limit factors

based on feasibility or data collected to this point

• Prioritization is key to making sure resources are spent wisely

• Prioritization should focus on the best fit of the causes to the observed effect

• The team must be able to deal with strong personalities or perceived positions of power

Data Collection

• Goal:• Obtain any additional information needed to document

applicability of possible causes

• Tools & Techniques:• Interview and Document Review• Physical Inspection

• Concerns:• This step may not be needed if all necessary data was

collected in Information Gathering• Corroborative information may be necessary to assess

potential causes identified during Brainstorming

Determination of Root Causes

• Goal:• Organize and analyze potential causes to

identify the systemic root causes of an effect

Determination of Root Causes

• Tools & Techniques• 5-Whys

• A technique used by Toyota Motor Corporation• Based on the concept that by answering why 5

times, you get to the cause (may take less)• Prone to rote memorization of rationalized

causes, so need to ensure proper mindset• Also limited to the set of knowledge of the

questioner

Determination of Root Causes

• Tools & Techniques (continued)• Pareto Charting

• Based on 80/20 rule• Provides a visual way to distinguish the “vital

few” from the “important many”• Need to collect the occurrence of potential

causes related to an effect• Usually more effective with larger sets of data

Pareto Chart(Example)

• Potential Causes A and B are the “vital few”

Example Pareto Chart

0

0.1

0.2

0.3

0.4

0.5

0.6

A B C D E

Potential Cause

Oc

cu

ren

ce

(%

)

0

0.2

0.4

0.6

0.8

1

Cu

mu

lati

ve

P

erc

en

tag

ePercentage Cumulative

Determination of Root Causes

• Tools & Techniques (continued)• Fishbone Diagramming

• Based on Kaoru Ishikawa’s approach• Provides a graphical representation of causes in a

causal factor chain• Uses cause groupings for default causes:

• Policy – Unclear or non-existent policies• Procedure – Lack of sufficient, clear procedures• Organization – Ineffective business organization• People/Culture – Performance of personnel or groups

• Machines are not causes in IA control failures• Acceptance is an issue within other process groupings

Fishbone Diagram (Example)

Weak Passwords

Procedure

Password procedures not implemented

Procedures draftedbut not approved Password management

system being revamped

Organization People/Culture

Policy

Determination of Root Causes

• Tools & Techniques (continued)• Event Causal Factor Chart

• Based on Max Ammerman’s approach• Provides identification of potential and root

causes based on a workflow methodology• Standard charting methodology

• Circles – Start and end conditions• Rectangles – correctly-functioning events• Diamonds – Incorrectly functioning events• Light Ovals – Secondary causes• Dark Ovals – Root causes

Event Causal FactorChart (Example)

Baseline Established

XYZ Application Down 1/1/05

Helpdesk Creates

Trouble Ticket

System Moved to Production

Helpdesk Assigns TroubleTicket to Admin

Change NotDocumented

Admin FindsProblem/

Applies Fix

Helpdesk Contacts Admin

To Close

HelpdeskCloses Trouble

Ticket

Baseline Differs FromProduction

InsufficientResources

Helpdesk Not Trained

ElevatedHelpdesk Privileges

Admin Reassigned

Helpdesk Avoiding

Notification

Determination of Root Causes

• Concerns:• Go beyond the “pilot error” knee-jerk

defense• Make sure you don’t stop until fundamental

causes are identified• Be creative in the techniques used, use the

techniques that are most effective• “To thine own self be true”

Root Cause Elimination

• Goals:• Suggest controls or changes to existing

controls to eliminate the root cause of undesired events

• Identify metrics to ensure that proposed recommendations resolve root causes within the field

Root Cause Elimination

• Tools & Techniques:• Prioritized potential fixes are identified with effort

represented with all costs• For each root cause, metrics are defined to allow

measurement of the effectiveness of an implementation

• The findings of analysis and recommended fixes are provided to management for decision

• A champion is identified to pilot the implementation of recommended actions

Root CauseElimination

• Concerns:• Effectiveness should be provided as a

function of return cost of future events against up-front and ongoing costs for each recommendation

• Credibility of the team in front of management is key

• Typically the best champion is a member of the RCA team with sufficient organizational power to effect change

Common Pitfalls

• Human nature• Team dynamics

• Lack of organizational knowledge• Lack of necessary skills (technical and professional)• Lack of team credibility

• Organizational dynamics• “Gotcha” mentality• Blame game/personnel-rating• Insufficient resources• “Drive-by” RCA

War Stories

Conclusions

• IT organizations should consider effectiveness of IA controls as a key measure of quality for the processes used to develop or acquire systems

• Root cause analysis is directly applicable to and can provide great benefits to IT organizations, and specifically within the IA controls environment