Change Impact Analysis for Natural Language Requirements

.lusoftware verification & validationVVS

Change Impact Analysis for Natural Language Requirements:

Chetan Arora, Mehrdad Sabetzadeh, Arda Goknil, Lionel Briand

Frank Zimmer

University of Luxembourg, Luxembourg

SES TechCom, Luxembourg

An NLP Approach

Problem Definition

• Requirements change frequently

• Large number of requirements and interdependencies

• Consistency must be maintained

• Handling change is expensive

• Support is required for impact analysis among requirements

2

3

Scope

Requirements

Architecture & Design Source Code

TestCases

SoftwareDocumentation

Majority of Research

Challenges

Example

• R1: The mission operation controller shall transmit satellite status reports to the user help desk.

• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.

• R3: The mission operation controller shall transmit any detected anomalies to the user help desk.

5

What Changed?

• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.



6

user help desk? operator help desk?

station maintenance crew help desk?

What Changed?




7

user help desk

Why Was the Change Made?




8

Possible Reason : Replace (user help desk) with (user document repository)

Why Was the Change Made?




9

Another Possible Reason : No communication between (user help desk) and

(mission operation controller)

Approach

Approach

11

Apply Change

Identify Differences

Specify Propagation

Condition Propagation Condition

Process Requirements

2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

PhrasesSimilarity

Matrix

Requirements Document

SortRequirements

SortedRequirements

Syntactic Similarity: satellite transmission licence

license of satellite transmission

Semantic Similarity: transmittransfer

Propagation Condition as a Boolean Query:

(user help desk) AND

(mission operation controller) AND

(transmit)

Approach

Apply Change


Specify Propagation



2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

PhrasesSimilarity

Matrix


SortRequirements

SortedRequirements

Processing Requirements Statements




13

Processing Requirements Statements




14

Phrase Detection Similarity Calculation

1.0transmit

transfer

Noun Phrase (NP)Verb Phrase (VP)

Approach

Apply Change


Specify Propagation



2

6666664

s11 · · · s1n... . . . ...sn1 · · · snn

3

7777775

PhrasesSimilarity

Matrix


SortRequirements

SortedRequirements

16

NAtural language Requirements Change Impact Analyzer

https://sites.google.com/site/svvnarcia/

Demo

monitoring desk

user help desk

Requirement Phrases

Impact Likelihood Computation

18

user priority list of his help desk

PropagationCondition

19

Sorted Requirements

Using the Sorted List

Inspect till the point after which the quantitative measure loses the capability to sufficiently differentiate the impacted requirements

20

When to Stop Inspecting?Graph Builder

MembershipDifference

Membership & Difference vs. %Inspected

%Inspected0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 0,40 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00

Mem

bers

hip

0,000,200,400,600,80

Diff

eren

ce

0,00

0,10

0,20

0,30

Where(40160 rows excluded)

h

Del

ta

% of requirements traversed in the sorted list

h

0 20 40 60 80 1000.0

0.1

0.2

0.30.0

1.0

0.8

0.6

0.4

0.2

Mat

chin

g sc

ore

max

h /3 max last

Graph Builder



%Inspected0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 0,40 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00

Mem

bers

hip

0,000,200,400,600,80

Diff

eren

ce

0,00

0,10

0,20

0,30


h

Del

ta


h

0 20 40 60 80 1000.0

0.1

0.2

0.30.0

1.0

0.8

0.6

0.4

0.2

Mat

chin

g sc

ore

max

h /3 max last

Impa

ct Li

kelih

ood

Inspect till the point after which the quantitative measure loses the capability to sufficiently differentiate the impacted requirements

Impa

ct Li

kelih

ood

Delta

When to Stop Inspecting?Graph Builder



%Inspected0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 0,40 0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00

Mem

bers

hip

0,000,200,400,600,80

Diff

eren

ce

0,00

0,10

0,20

0,30


h

Del

ta


h

0 20 40 60 80 1000.0

0.1

0.2

0.30.0

1.0

0.8

0.6

0.4

0.2

Mat

chin

g sc

ore

max

h /3 max last

Recommended point in sorted requirements to stop inspecting

Empirical Evaluation

Research Questions

• Which similarity measures perform best?

• How effective is our approach, using the guidelines?

23

Case Studies

24

Case-A

• 160 Requirements• 9 Change Scenarios

Case-B

• 72 Requirements• 5 Change Scenarios

Change Scenarios

25

ID Propagation Condition Pattern Size of Impact SetA.1 ⟨NP⟩ AND ⟨NP⟩ 4

A.2 ⟨NP⟩ OR ⟨NP⟩ 8A.3 ⟨NP⟩ 39A.4 (⟨NP⟩ OR ⟨NP⟩) AND ⟨NP⟩ 5A.5 ⟨NP⟩ OR ⟨NP⟩ 10A.6 ⟨NP⟩ AND ⟨NP⟩ 3A.7 ⟨NP⟩ AND ⟨NP⟩ 7A.8 ⟨NP⟩ OR ⟨NP⟩ 5A.9 ⟨verbatim-text⟩ AND ⟨NP⟩ 3B.1 ⟨NP⟩ AND ⟨NP⟩ 2B.2 ⟨NP⟩ 9B.3 ⟨NP⟩ AND ⟨NP⟩AND ⟨NP⟩ 1B.4 ⟨NP⟩ AND ⟨NP⟩ 1B.5 (⟨NP⟩ OR ⟨NP⟩) AND (⟨NP⟩ OR ⟨NP⟩) 9

26

Syntactic MeasuresBlock Distance

Cosine SimilarityDice’s coefficient

EuclideanJaccard

Jaro Jaro WinklerLevenstein

Monge Elkan SOFTTFIDF

Semantic MeasuresHSO JCNLCHLESK

LESK_TANIMLIN

PATHRESWUP

Which Similarity Measures Perform Best?

RecommendedBest inCase-A Best in

Case-B

“touristic attraction” is a

“point of interest”

Reason: Lack of a Domain Model

1 impacted requirement missed out of a total of 106 impacted

requirements.

Effectiveness of Our Approach

27

Futil

e Ins

pect

ion Ef

fort

1% - 7% 6% - 8%

45%

Key Points from Evaluation

28

Choice of Similarity Measures Effectiveness Execution Time

Related Work

• A. Goknil et. al. 2014 - Change Impact Analysis in requirements using dependency model with formal semantics

• J. Cleland-Huang et. al. (2005) - Soft goal dependencies for analysing the impact of changes in functional requirements to non-functional requirements

• Yang et. al. (2011) - Use of NLP (text chunking) for resolving ambiguities in requirements

• J. Cleland-Huang, “Traceability in agile projects,” in Software and Systems Traceability, J. Cleland-Huang, O. Gotel, and A. Zisman, Eds. Springer, 2012.

29

Just-In-Time Traceability

Future Work

• Address the limitation concerning tacit dependencies between the requirements

• More empirical studies - especially user studies - relations between other NL artefacts, such as test cases

30

31

Additional Artefacts

Guidelines

https://sites.google.com/site/svvnarcia/