A Solution for Identifying the Root Cause of Problems in IT Change Management

Malia Project – HP Labs & Institute of Informatics – UFRGS © 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

A Solution for Identifying the Root Cause of Problems in IT Change Management Ricardo L. dos Santos, Juliano A. Wickboldt, Roben C. Lunardi, Bruno L. Dalmazo, Lisandro Z. Granville and Luciano P. Gaspary

Federal University of Rio Grande do Sul, Brazil

Claudio Bartolini

Hewlett Packard Laboratories, Palo Alto, USA

Marianne Hickey

Hewlett Packard Laboratories, Bristol, UK

Malia Project – HP Labs & Institute of Informatics – UFRGS

• Introduction

• Related Work

• Proposed Solution

• Diagnosis Process

• Information Model

• Conceptual Architecture

• Root Cause Analyzer

• Conclusion and Future Work

Outline

Wednesday, March 06, 2013 2


Introduction

• Context

• The complexity of IT infrastructure makes the IT processes a critical mission

• ITIL has become important for IT processes

• IT Change Management

• Defines how the IT infrastructure must evolve in a consistent and safe fashion

• Defines how changes should be conducted



Introduction

• IT Problem Management

• Defines how lifecycle of IT problems is managed

• Its main goals are

• Preventing the occurrence of IT related problems

• Eliminating recurring problems

• Minimizing the impact of problems that cannot be prevented

• Allows to simplify procedures for detecting the root cause of IT problems and reducing the associated costs



Introduction

• Current Scenario

• Organizations use tools for to help in analysis and identification of the root cause of failures

• When they do not point directly into the root cause, is able to provide important information to the responsible technical staff

• Enables the reuse of knowledge, which results in a reduction of associated costs

• However, these tools have serious limitations, such as

• The description of cases in diagnostic tools is performed in ad hoc fashion

• Poor reusability of cases

• Low maintainability



Introduction

• Problems

• Often diagnoses are not completed correctly due to outdated cases

• The complexity of IT infrastructure where the failure occurs is ignored

• Data of previous diagnoses are not considered



Introduction

• Objectives

• Propose an interactive method for identifying causes of failure in change processes

• Based on information of the diagnosis of previous failures in change activities

• Considering the affected IT infrastructure

• Optimize the root cause identification process of failures in IT changes

• Reuse the knowledge acquired in previous diagnoses



Iterative Diagnosis

Proposed Solution Diagnosis Process

Incident Report Answered

Question

Root Cause Question

Selection


IR RC

Help Desk Root Cause

Analyzer

Operator


Config. Mgmt.

Database

Diagnosis System

Change Management System

Change

Planner

Change

Designer

Proposed Solution Conceptual Architecture

Operator


Deployment

System CI CI CI

Diagnosis Log

Recorder

RC

Root Cause

Analyzer

Root Cause Analyzer

Input

Processor

Weight

Calculator

Question

Selector

Question

Verifier IR RC

CI CI CI CI CI L

RC RC RC


Proposed Solution Information Model


▲ determinesProblem

possiblesAnswers

determinesOthersQuestions

CategoryParentChild

1 1..*

0..1 1

0..1

1..*

*

ServiceProblem

SolutionCategory

* 1..*

ManagedSystem

Element

ManagedElement

Logical Element

EnabledLogical

Element

MessageLog

ExchangeElement

SolutionElement

*

QuestionCategory

RecordLog

RecordedAnswers

RecordedQuestions

Category

1

0..1

Question

RootCause

1..* *

1 1 1 1

1

0..*

PRS_ServiceInciden

t

PRS_Problem

Answer

0..1

1..*

RecordedProblem


Log Log Log CI CI CI

Proposed Solution Root Cause Analyzer

14

Reports the

incident

CI

Priority

…

Operator


Root Cause Analyzer

Weight Calculator Question Verifier



Input Processor Question Selector

CI CI L

RC RC RC Identif ication

based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer




16

IDENTIFIES:

•Logs of previous diagnoses

•RCs, Questions and Answers (categories)

•Affected CIs

Operator



• An example of the affected IT infrastructure



18


CI Category

CI Category

Internet Third Party Services

WEB Server (215) System Computer System Server Web Server

DNS Server (423) System Computer System Server DNS Server

Apache 2.2 Software Web Server

PHP 5.0.2 Software Programming Language

Hard Disk Device Storage Devices

Network Card Device Network Cards


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



20


Select questions and

answers based on the

categories of CIs

identified

RC RC RC RC RC

Configuration

Mgmt. System

Set of RCs, Answers, and Questions



21

Sy

ste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n

Co

mp

ute

r Sy

ste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

20

RC

RC

RC RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC

RC RC

RC

System Computer System Server Web Server

Application Web Server

Device Network Cards

Third Party Services



22

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Solução Proposta Root Cause Analyzer

24 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Example Selection of Questions Current Version

26 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



27 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight Obvious?

Threshold

80% with the

same answer



29 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Questions RC 1,3 (50)


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the


Threshold

80% with the

same answer



31 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Questions RC 1,3 (50)


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the


Threshold

80% with the

same answer


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer


Results


Question Path of Apache

cfg is Ok?

Question Network adapter

is configured?

Question Switch ports are

configured?

Question Exim is installed?

Question POP3 is running?

Question Apache is

installed?

Executions: 25 Answers: 88% YES

Operator’s Answer: YES

Executions: 22 Answers: 63,6% NO


Executions: 16 Answers: 93,7% YES







Operator’s Answer: NO

Question Switch is

reachable by

ping? Executions: 36 Answers: 77,8% NO


Question Server is

reachable by

ping? Executions: 41 Answers: 73,2% NO


Question Switch is up?

Executions: 48 Answers: 75% NO


Question Path of Apache

cfg is Ok?

Question Exim is running?

Question Path of Exim cfg is

Ok?

Question Exim is installed?

Question POP3 is running?

Question Apache is

installed?














Conclusion and Future Work


• The proposed solution proved to be feasible with the following features

• Interactive

• Considering the affected IT infrastructure

• Based on information of the previous diagnoses

• Reuse the knowledge acquired

• Optimize the root cause identification process of failures in IT changes


Conclusion and Future Work


• Melhorar o bootstrpping do sistema

• Improve the selection algorithms

• Consider other metrics (age)

• Consider other selection methods (popularity)

Thanks for your attention!

Questions?


Root Cause Analyzer





CI CI L


based on

categories

Identif ication

based on IR

Identif ication

based on RCs

Calculates the

weight of RCs

Calculates the

weight of

Questions and

Categories

Calculates the

weight of

possible

Answers

Question with

higher level

Question that

has the

greatest weight

Category that

has the

greatest weight

Obvious?

Threshold

80% with the

same answer



39 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



40 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

Ap

plic

atio

n Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



41 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20



42 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Qestions RC 1, 3 (50)



43 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

We

b S

erv

er

De

vic

e

Ne

two

rk C

ard

74

30

5

95

70

35

30

26

20

We

b S

erv

er

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions



44 6 March, 2013

How many

bits is the

application?

16 bits 32 bits

64 bits


We

b S

erv

er

70


45 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

So

ftwa

re

We

b S

erv

er


We

b S

erv

er

70


46 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

So

ftwa

re

We

b S

erv

er


We

b S

erv

er

70


47 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

95 35

30

26

20

95 – 70 = 25

Questions RC 1 (10)

Questions RC 2 (20)

Questions RC 3 (40)

Questions RC 4 (25)

Common Questions

We

b S

erv

er


We

b S

erv

er

70


48 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

70


49 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

70


50 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

We

b S

erv

er


We

b S

erv

er

We

b S

erv

er

70


51 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)

Questions RC 11 (25)


We

b S

erv

er

We

b S

erv

er

70


52 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




53 6 March, 2013

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

No Yes


We

b S

erv

er

We

b S

erv

er

70


54 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)



We

b S

erv

er

We

b S

erv

er

70


55 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




56 6 March, 2013

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

Network

card

detected?

No Yes

No Yes


We

b S

erv

er

We

b S

erv

er

70


57 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)



We

b S

erv

er

We

b S

erv

er

70


58 6 March, 2013

Syste

m

Th

ird P

arty

Se

rvic

es

So

ftwa

re

Co

mp

ute

r Syste

m

Se

rve

r

De

vic

e

Ne

two

rk C

ard

74

30

5

25 35

30

26

20

Questions RC 9 (2)

Questions RC 6 (5)

Questions RC 7 (12)

Questions RC 8 (12)

Common Questions

Questions RC 5 (15)

Questions RC 10 (3)




59 6 March, 2013

How many

bits is the

application?

16 bits 32 bits

64 bits

The devices

are well

plugged?

Network

card

detected?

Network

card is

configured

correctly?

Network card

not

configured

No Yes

No Yes

Yes

No

Documents

A Solution for Identifying the Root Cause of Problems in IT Change Management