Evaluating the Impact of UML Modeling on Software QualityAn Industrial Case Study

Introduction

Research Design

Case Study

Results

Conclusions

Ariadi Nugroho Michel R.V. Chaudron

Introduction

Research Design

Case Study

Results

Conclusions

Modeling, is it worth it?

Well-thought solutions

Better comprehension

Improved Communication

Reduced maint. effort

Reduced reworks

Reduced defects

Improved productivity

Cost-saving

Introduction

Research Design

Case Study

Results

Conclusions

UML is used diversely: style, completeness, level of detail, formality...

But, this is often how people model...

Introduction

Research Design

Case Study

Results

Conclusions

Exploring the effects of quality in modeling

• Defects in UML models often remain undetected and cause misinterpretation (Lange & Chaudron, ICSE 2006)

• Higher level of detail in UML models increases comprehension performance (IST Journal, 2009)

• Higher level of detail in sequence diagrams corresponds to lower defect density in software (MODELS 2008)

This study focuses on the use of UML and its effect on the quality of software

Does model quality matter?

In modeling a system, some parts might have been left unmodeled for various reasons...Introduction

Research Design

Case Study

Results

Conclusions

Implementation::Package AUML Model::Package A

do_A()aClass

do_B()bClass

do_C()cClass

do_something()letterClass

do_A()aClass

do_B()bClass

do_Z()zClass

do_something()letterClass

do_BB1()bb1Class

do_BB2()bb2Class

• Implementation may not be well-thought of or well-designed• Implementation may deviate from requirements

What may be the results of not modeling?

“Do the modeled and not modeled classes have significantly different quality in the implementation?”

0

25

50

75

100

ModeledNot Modeled

Defect densityIntroduction

Research Design

Case Study

Results

Conclusions

Research Question

Introduction

Research Design

Case Study

Results

Conclusions

Hypothesis:Modeled classes have significantly lower defect density than those that are not modeled

Design of Study

No Modeling

UML Modeling

Main factor

Confounding factors

CodeComplexity

CodeCoupling

THE USE OF UML RESULTING IMPLEMENTATION

Defect

DensityAffects?

Project characteristics

• A web-based healthcare system

• System modeling was done in Holland

• Sixty percent of coding and testing were done in India

• UML model was created using Rational XDE

• Textual specifications exist

Project Size # of staffs Off-shoring Model Size Code size

20 man-years 25 people India104 UCs

266 classes341 SDs

152, 017

Introduction

Research Design

Case Study

Results

Conclusions

System under Study

Introduction

Research Design

Case Study

Results

Conclusions

Approach

Payment-amount

-cashTenderedCash

+authorized()

-name-bankID

Check

+authorized()

-number-type-expDate

Credit

+calcTax()+calcTotal()+calcTotalWeight()

-date-status

Order

+calcSubTotal()+calcWeight()

-quantity-taxStatus

OrderDetail

1

1

1* Preprocessor Parser Analyser Inheritance Relator Statistic Filter Statistic CalculatorDB Creator DB Filler DB Checker

SAAT.BAT

component C2 requires I2 requires I3uses I2.guses I3.h

component C2 requires I2 requires I3uses I2.guses I3.h

defectdensity

Modeled vs Unmodeled

0

13.75

27.50

41.25

55.00

Not Modeled Modeled

Collect implementation classes, UML model, and

defect data

Determine defect density of every

implementation class

Check whether an implementation class has been modeled in UML

Project Repository

Statistical tests

• The core part of the system contains 812 Java classes

Classes that were not modeled tend to have higher defect density than the modeled ones

Introduction

Research Design

Case Study

Results

Conclusions21%

79%

Modeled

Not Modeled0

0.005

0.010

0.015

0.020

Not Modeled Modeled

Mean Defect Density

Descriptive Statistics

Class coupling and complexity might confound the effect of UML modeling on defect density

Introduction

Research Design

Case Study

Results

Conclusions

0

7.5

15.0

22.5

30.0

Not Modeled Modeled

Coupling ComplexityMean

Descriptive Statistics

Modeled classes have significantly higher coupling and complexity

Eliminating the effects of class complexity and couplingIntroduction

Research Design

Case Study

Results

Conclusions

CBO

Total Variance in Defect Density

Variance explained by the use of UML

Unexplained Variance

MCC

Controlling for Confounding Factors

Introduction

Research Design

Case Study

Results

Conclusions

Mean

0

11.25

22.50

33.75

45.00

Not Modeled (59) Modeled (37)

Coupling Complexity

Accounting for CBO

Accounting for CBO & MCC

2

1Implementation Classes (812 classes)

CBO 8 - 10 (113 classes)

Random sampling

(80% = 96 classes)

Mann-Whitney Test

ANCOVA Test

Pairwise sampling

Controlling for Confounding Factors

Result of Mann-Whitney Test

• After controlling for the effect of class coupling

Not modeled classes have significantly higher defect density (p=0.003; 1-tailed)

Introduction

Research Design

Case Study

Results

Conclusions

0

13.75

27.50

41.25

55.00

Not Modeled Modeled

Defect Density (Mean Rank)

Testing the Hypothesis

Result of ANCOVA Test

• The use of UML remains a significant factor that explains the variability of defect density in the implementation

Variables Sum of Squares

df F Sig.

The use of UML 0.002 1 6.825 0.010

Coupling 7.562E-8 1 0.000 0.987

Complexity 0.001 1 5.224 0.025

Introduction

Research Design

Case Study

Results

Conclusions

Testing the Hypothesis

So,...Introduction

Research Design

Case Study

Results

Conclusions

The hypothesis is confirmed:

Modeled classes have significantly lower defect density than those that are not modeled..

after controlling for the effects of class complexity and coupling

Defect Density (Mean Rank)

0

0.005

0.010

0.015

0.020

Not Modeled Modeled

• The use of UML has a positive effect on the quality of the system (i.e., reduced defect density)

• Increasing model coverage might help reduce incorrect implementation of systems

• Deciding which classes to model is an open issue

Introduction

Research Design

Case Study

Results

Conclusions

Conclusions

Introduction

Research Design

Case Study

Results

Conclusions

• Replication using more industrial cases

• Accounting for the effects of other confounding factors

• Performing similar studies in experimental settings

What’s Next?

Introduction

Research Design

Case Study

Results

Conclusions

Questions?