Documented Requirements are not Useless After All!

Documented Requirements !are not Useless After All! !

!The Case for Supporting Change, !

Configuration, and Testing

Lionel Briand XP 2016, May 27th, 2016 Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg

Fundamentals of Agile Methods

2

•  Individuals and Interactions over Process and Tools

•  Customer Collaboration over Contracts •  Working Software over Documentation •  Responding to Change over Planning

Requirements in Agile Methods

3

•  “Customer” on site •  Whole development team involved in eliciting

requirements •  Use of a “common” language, e.g., user stories •  Test cases as “executable requirements” (TDD) •  Minimize traceability •  Prioritize requirements •  Accommodate requirements changes

•  Like all principles, these are meant to be used thoughtfully and adapted in context

These principles are often used !in a simplistic and dogmatic

fashion

4

Agile development, in practice, tend to underplay the role of documented requirements

5

Objectives of the Talk

6

•  In reality, whether and how you write requirements is a trade-off

•  When are documented requirements important?

•  What are their main applications? •  What form do they take in practice? •  What are the challenges? •  Project examples and lessons learned

7

Challenges

Natural Language

8

•  Most requirements in practice are expressed using natural language

•  Ambiguity and incompleteness •  Trade-off between precision and cost •  Current support is insufficient, especially for

large sets of requirements

Domain Knowledge

9

•  All requirements depend, more or less explicitly, on domain knowledge

•  Common concepts and terminology •  Not always consistent among all stakeholders •  Software engineers often have a superficial

understanding of the application domain they target

Change

10

•  Requirements change frequently •  Changes have side-effects on other

requirements, design decisions, test cases … •  How do we support such changes in ways

that scale to hundreds of requirements?

Configuring Requirements

11

•  Many software systems are part of product families targeting varying needs among multiple customers

•  Requirements typically need to be tailored or configured for each customer

•  Because of interdependencies among such decisions, this is often error-prone and complex.

Challenges

12

•  Dealing effectively with natural language •  Capturing domain knowledge •  Dealing with frequent changes •  Configuring requirements with customers in

product families •  These challenges also apply to agile

development

13

Addressing the Challenges

14

Analyzing and Handling Natural Language Requirements

Context

15

Challenges

16

•  Large projects in satellite domain (e.g., ESA) •  Hundreds of natural language requirements •  Three tiers of requirements •  Many stakeholders •  Requirements capture a contract •  Requirements change

Research Objectives

Checking compliance

to boilerplates

Domain Model

extraction

Change Impact

analysis in requirements

17

Research Objectives

Checking compliance

to boilerplates

Domain Model

extraction

Change Impact


18

Compliance with Boilerplates

19

•  Rupp’s boilerplate:

•  As soon as an unplanned outage is detected

the Surveillance and Tracking Component shall notify the SOT operator at the local station.

•  Automation required for compliance checking •  No glossary, scalable parsing

[<When?><under what

conditions?>]

THE SYSTEM<system name>

SHALL

SHOULD

WILL

PROVIDE <whom?> WITH THE ABILITY TO

<process>

<process>

BE ABLE TO<process>

<object> [<additional details about the object>]

Approach and Results

20

•  Natural Language Processing (GATE) •  Text chunking •  Boilerplates: RUPP and EARS •  Boilerplates are expressed as BNF grammars

and then pattern matching rules •  Four industrial case studies •  Low number of false positives and negatives •  Hundreds of requirements in a few minutes

Tool: RETA

21

GATE NLPWorkbench

ConformanceDiagnostics

(within GATE)Requirements

Lists of modals,conditional words,

ambiguous terms, etc.

LSTLSTLSTLSTRules for checking

templateconformance

JAPEJAPEJAPEJAPERules for checking

best practices

Glossary(optional)

JAPEJAPEJAPEJAPE

Requirements Analyst

Requirements Authoring &Management

http://sites.google.com/site/retanlp/

Research Objectives

Checking compliance

to boilerplates

Domain Model

extraction

Change Impact


22

Change Impact Analysis

23

•  A change in requirements may lead to changes in other requirements

•  Hundreds of requirements •  No traceability •  We propose an approach based on: (1) Natural

Language Processing, (2) Phrase syntactic and semantic similarity measures

•  Results: We can accurately pinpoint which requirements should be inspected for potential changes

Example

• R1: The mission operation controller shall transmit satellite status reports to the user help desk.

• R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.

• R3: The mission operation controller shall transmit any detected anomalies with the user help desk.

24

Change

• R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.



25

Challenge #1 Capture Changes Precisely




26

Challenge #2 !Capture Change Rationale




27

•  R1: The mission operation controller shall transmit satellite status reports to the user help desk document repository.

•  R2: The satellite management system shall provide users with the ability to transfer maintenance and service plans to the user help desk.

•  R3: The mission operation controller shall transmit any detected anomalies with the user help desk.

Challenge #2 !Change Rationale

Rationale:

R1: We want to globally rename “user help desk” R2: Avoid communication between “mission operation controller” and “user help desk” R3: We no longer want to “transmit satellite status reports” to “user help desk” but instead to “user document repository”

28

Solution Characteristics

• Account for the phrasal structure of requirements The mission operation controller shall transmit satellite

status reports to the user help desk document repository.

user help desk, Deleted user document repository, Added

• Consider semantically-related phrases that are not exact matches and close syntactic variations

29

Tool: Narcia

30 https://sites.google.com/site/svvnarcia/

Research Objectives

Checking compliance

to boilerplates

Domain Model

extraction

Change Impact


31

Domain Models A domain model is a representation of conceptual entities or

real-world objects in a domain of interest.

32

Motivation •  In practice, domain models are not preceding the elicitation and writing of

requirements

• Representation of important domain concepts and their relations

•  Facilitate communication between stakeholders from different backgrounds

• Help identify inconsistencies in terminology, etc.

• Support is required for building domain models

• A lot of information required to build a domain model is automatically extractable

33

Approach

The simulator shall continuously monitor its connection to the SNMP manager and any linked devices.

dobj

advmoddet

NP NP NP

NP

possdet

amod

conj_and

nsubj

prep_to

prep_to

1 2 3 4 5 6 7 8 9 10 11 12

1 1 2 3

4

13 14 15

aux

nndet

prep_toprep_to

wor

dde

pend

enci

es

depe

nden

cies

de

rived

by

Alg

. of F

ig. 8

(Sec

tion

4.2)

14

amod

153 4

advmodaux

VB

R4:

ref_to

6

dobj

nsubj

(identified directly by coreference resolution)

Figure 5: Results of dependency parsing for requirement R4 of Fig. 4(a).

(ROOT (S (NP (DT The) (NN simulator)) (VP (MD shall) (ADVP (RB continuously)) (VP (VB monitor) (NP (PRP$ its) (NN connection)) (PP (TO to) (NP (NP (DT the) (NNP SNMP) (NN manager)) (CC and) (NP (DT any) (VBN linked) (NNS devices)))))) (. .)))

R4: The simulator shall continuously monitor its connection to the SNMP manager and any linked devices.

(b)

(a)

Figure 4: (a) A requirement and (b) its parse tree.

Phrase structure parsing [18] is aimed at inferring thestructural units of sentences. In our work, the units of in-terest are noun phrases and verbs. A noun phrase (NP) isa unit that can be the subject or the object of a verb. Averb (VB) appears in a verb phrase (VP) alongside any di-rect or indirect objects, but not the subject. Verbs can haveauxiliaries and modifiers (typically adverbs) associated withthem. To illustrate, consider requirements statement R4 inFig. 4(a). The structure of R4 is depicted in Fig. 4(b) us-ing what is known as a parse tree. To save space, we donot visualize the tree, and instead show it in a nested-listrepresentation commonly used for parse trees.

Dependency parsing [29] is aimed at finding grammaticaldependencies between the individual words in a sentence.In contrast to phrase structure parsing, which identifies thestructural constituents of a sentence, dependency parsingidentifies the functional constituents, e.g., the subject andthe object. The output of dependency parsing is representedas a directed acyclic graph, with labeled (typed) depen-dency relations between words. The top part of the graphof Fig. 5 shows the output of dependency parsing over re-quirements statement R4. An example typed dependencyhere is nsubj(monitor{5},simulator{2}), stating that “simula-tor” is the subject of the verb “monitor”.

Syntactic parsing is commonly done using the pipeline ofNLP modules shown in Fig. 6. There are alternative imple-mentations available for each of the modules in this pipeline.The pipeline can therefore be instantiated in di↵erent waysusing di↵erent combinations of module implementations. Inour work, we instantiate the pipeline using the module im-plementations provided by the Stanford Parser Toolkit [24].

The first module in the pipeline is the Tokenizer, whichsplits the input text, in our context a requirements docu-ment, into tokens. A token can be a word, a number, or asymbol. The second module, the Sentence Splitter, breaksthe text into sentences. The third module, the POS Tagger,attaches a part-of-speech (POS) tag to each token. POStags represent the syntactic categories of tokens, e.g., nouns,

Tokenizer

SentenceSplitter

POS Tagger

Parser(Phrase Structure

+ Dependency)

NLRequirements

1

2

3

5

Named Entity Recognizer 4

StructureParse Tree NP NPVP

TypedDependencies

nsubjnn dobj

Annotations

CoreferenceResolver 6

Figure 6: Parsing pipeline.

adjectives and verbs.The fourth module,the Named-Entity Rec-ognizer, identifies en-tities belonging topre-defined categories,e.g., proper nouns,dates, and locations.The fifth and mainmodule is the Parser,encompassing bothphrase structure pars-ing and dependencyparsing. The finalmodule is the Coref-erence Resolver. This is an optional module whose functionis to find expressions that refer to the same entity. We con-cern ourselves with pronominal coreference resolution only,which is the task of identifying, for a given pronoun suchas “its” and “their”, the NP that the pronoun refers to.Fig. 5 shows an example of pronominal coreference resolu-tion, where the pronoun “its” is linked to the referenced NP,namely “the simulator”, via a ref to dependency.

4. APPROACH

NPs, VBs Dependencies

Process Requirements Statements

NLRequirements

Lift Dependencies to Semantic Units

nsubjnn dobj

DomainModel

Construct DomainModel

0..1 1..*

relation

ExtractionRules

1

23

Figure 7: Approach Overview.

Fig. 7 presentsan overview of ourdomain model ex-traction approach.The input to theapproach is an NLrequirements doc-ument and the out-put is a UML classdiagram. Below,we elaborate thethree main steps of our approach, marked 1-3 in Fig. 7.

4.1 Processing the Requirements StatementsThe requirements processing step includes the following

activities: (1) detecting the phrasal structure of the require-ments, (2) identifying the dependencies between the words inthe requirements, (3) resolving pronominal coreferences, and(4) performing stopword removal and lemmatization; theseare common NLP tasks, respectively for pruning words thatare unlikely to contribute to text analysis, and for trans-forming words into their base morphological form.Activities (1), (2), and (3) are carried out by the pipeline

of Fig. 6. From the parse tree generated by this pipeline foreach requirements statement, we extract the atomic NPs

4

34

Requirements to Design IA Requirements

Design

Objective: Identifying impact of requirements changes on SysML design

35

Motivating Scenario!

Example Change Requests:

•  Change to R11: Change over temperature detection level to 147 C from 110 C

•  Change to R12: Temperature range should be extended to -40/150 C from -20/120 C

text = "The CP controller shall provide temperature diagnostics."id = "R1"

«requirement»Temperature Diagnostics

text = "The CP controller shall detect temperatures exceeding 110 ºC."id = "R11"

«requirement»Over-Temperature Detection

text = "The CP controller shall be able to measure temperatures between -20 ºC and 120 ºC."id= "R12"

«requirement»Operational Temperature Range

36

:Digital to Analog Converter

:DC Motor Controller

«satisfy»

:Over-TemperatureMonitor :Diagnostics

Manager

Over-Temperature

Motordrive mode

Error: Diagnostics and Status

Signal Generation

…

Motor position

B1

B2B3

B4

«requirement»Over-Temperature

Detection(R11)

……

«requirement»Operational

Temperature Range(R12)

…

:Temperature Processor

«satisfy»

B5

B6

Temperature

Voltage(digitized)

TemperatureMotor speed

…

Motortorque

…

Block Diagram

37

Actually Impacted Elements

• Both changes are related to ``Temperature Diagnostics’’, but they impact two drastically different parts of the system

• Change to R11 impacts a decision statement in software

• Change to R12 impacts electronic voltage divider circuit (hardware) and a lookup table (software)

38

:Digital to Analog Converter

:DC Motor Controller

«satisfy»

:Over-TemperatureMonitor :Diagnostics

Manager

Over-Temperature

Motordrive mode

Error: Diagnostics and Status

Signal Generation

…

Motor position

B1

B2B3

B4

«requirement»Over-Temperature

Detection(R11)

……

«requirement»Operational

Temperature Range(R12)

…

:Temperature Processor

«satisfy»

B5

B6

Temperature

Voltage(digitized)

TemperatureMotor speed

…

Motortorque

…Impacted by R12

Impacted by R12

Impacted by R12

Impacted by R11

Impacted Blocks …!

39

Approach

• Algorithm to compute estimated impacted elements in SysML designs for a requirements change

• Combination of structural analysis, behavioral analysis, and natural language processing (model labels)

• Number of design elements to inspect (average): 4.8% of design elements that can possibly be impacted

• Only one missed impacted element across all changes

• Frequent changes: Significant savings and better quality assurance

40

41

Supporting Product Lines and Requirements Configuration

Context

International Electronics !& Engineering (IEE)

IEE develops real-time embedded systems: •  Automotive safety sensing systems •  Automotive comfort & convenience systems,

e.g., Smart Trunk Opener

42

Smart Trunk Opener (STO)

STO Provides automatic and hands-free access to a vehicle’s trunk (based on a keyless entry system)

43

IEE Requirements Engineering Use Case Driven

Development

Use Case !Diagram

Use Case! Specifications

Domain !Model

44

Dealing with Multiple Customers

45

STO Requirements from Customer A

(Use Case Diagram and Specifications, and Domain Model)

Customer Afor STO

modify modify

modify modify

STO Test Cases for Customer A

evolves to

(clone-and-own)

STO Requirements from Customer B


Customer Bfor STO

evolves to

(clone-and-own)

STO Test Cases for Customer B

evolves to

(clone-and-own)

STO Requirements from Customer C


Customer Cfor STO

evolves to

(clone-and-own)

STO Test Cases for Customer C

Product Line Approach

46

•  A Product Line approach was clearly needed •  Restricted and analyzable use case specifications

(NLP) •  Variability modeling in use case diagrams and

specifications •  Automated configuration guidance for configuring

requirements with each customer •  Automated generation of product-specific use case

models based on decisions

Use Cases And Domain Model

Customer A for Product X

Product-Line Use Cases And Domain Model

Identify Commonalities and

Variabilities Configurator

Customer B for Product X


Customer C for Product X


configure evolves

reconfigure

evolves

reconfigure

reconfigure

reconfigure

47

Product Line Use Case Diagram for STO (Partial)

48

• RUCM is based on a (1) template, (2) restriction rules, and (3) specific keywords constraining the use of natural language in use case specifications

• RUCM reduces ambiguity and facilitates automated analysis of use cases

Restricted Use Case Modeling: !RUCM

49

• Flow of events is described in restricted natural language

RUCM

Basic Flow

1. INCLUDE USE CASE Identify System Operating Status.2. The system VALIDATES THAT the operating status is OK.3. The system REQUESTS the move capacitance FROM the UpperSensor.4. The system REQUESTS the move capacitance FROM the LowerSensor.5. The system VALIDATES THAT the movement is a valid kick.6. The system VALIDATES THAT the overuse protection feature is enabled.7. The system VALIDATES THAT the Overuse protection status is inactive.8. The system SENDS the valid kick status TO the STOController.Post condition: The gesture has been recognised and the STO Controller hasbeen informed.

50

• Keyword: INCLUDE VARIATION POINT: ... • Inclusion of variation points in basic or alternative flows of

use cases:

Use Case: Identify System Operating StatusBasic Flow1. The system VALIDATES THAT the watchdog reset is valid.2. The system VALIDATES THAT the RAM is valid.3. The system VALIDATES THAT the sensors are valid.4. The system VALIDATES THAT there is no error detected.Specific Alternative FlowRFS 41. INCLUDE VARIATION POINT: Storing Error Status.2. ABORT.

!Example Variability Extension

51

• Tool Support (PUMConf): https://sites.google.com/site/pumconf/

• Positive feedback from engineers, both about the modeling approach and configuration tool

• They confirmed they benefited from:

• Understanding the commonalities and differences across product requirements

• Automated guidance in a configuration that is often complex, i.e., many (interdependent) decisions

Results

52

53

Legal Compliance

Regulations - Requirements •  Many (government) systems need to be

compliant with the law and remain so over time

•  E.g., Tax systems •  How can we help with making sure IT

engineers implement systems that comply with the law?

•  How can we verify they do?

54

Solution Overview

Test cases

Actual software system

Traces to

Traces to

Analyzable interpretation

of the law Generates

Results match?

Impact of legal changes

Simulates

55

Domain Model

56

Art. 105bis […]The commuting expenses deduction (FD) is defined as a function over the distance between the principal town of the municipality on whose territory the taxpayer's home is located and the place of taxpayer’s work. The distance is measured in units of distance expressing the kilometric distance between [principal] towns. A ministerial regulation provides these distances.

Interpretation + Traces

Procedures as Activity Diagrams

57

The amount of the deduction is calculated as follows: If the distance exceeds 4 units but is less than 30 units, the deduction is € 99 per unit of distance. The first 4 units does not trigger any deduction and the deduction for a distance exceeding 30 units is limited to € 2,574.

Interpretation + Traces

Discussion •  Models understandable by both legal experts and IT

specialists

•  We addressed the gap between legal experts and IT specialists

•  Modeling effort was considered reasonable given the life span of such eGovernment systems

•  Traceability to the law was considered a significant asset given frequent and complex changes in the law

58

59

Requirements-Driven Testing

Context • Context: Automotive, sensor systems

• Traceability between system requirements and test cases

• Mandatory when software must comply with ISO 26262

• Customers also require such compliance

• Use-case-centric development TC4

TC3

TC2

Requirements!!

Test cases

TC1

60

Objectives • Automatically generate test cases from requirements

• Capture and create traceability information between test cases and requirements

• Requirements are captured through use cases

• Use cases are used to communicate with customers and the system test team

• Complete and precise behavioral models are not an option: too expensive (Model-based testing)

61

Strategy

• Analyzable use case specifications

• Automatically extract test model from the use case specifications

• Minimize modeling, domain modeling only

• No behavioral modeling

62

THE ACTOR SEND THE SYSTEM VALI THE SYSTEM DIS THE ACTOR SEND



ERRORS ARE ABSENT

TEMPERATURE IS LOW

STATUS IS VALID

Identify Constraints 4

Constraint descriptions

Errors.size() == 0 Status != null

t > 0 && t < 50

Generate Scenarios and

Inputs

6

Elicit Use Cases 1

Missing Entities

Specify Constraints 5

OCL constraints

Model the Domain 2

Evaluate Consistency

3 Domain Model RUCM Use Cases

Generate Test Cases

7

Test Cases Object

Diagrams Test

Scenarios Mapping Table

64

https://sites.google.com/site/umtgTestGen/

Toolset integrated with IBM DOORS and Rhapsody

65

Discussion

Applications

66

•  Requirements to support a shared understanding among many stakeholders in large projects

•  Requirements as contract with customers •  Requirements to support communication

between software engineers and domain experts

•  Requirements to support compliance with standards, e.g., traceability to tests

•  Requirements to support change

Forms of Requirements

67

•  Natural language statements, complying or not with boilerplates

•  Use case specifications, possibly structured and restricted

•  Models, e.g., class and activity diagrams •  Agile practice: User stories are a basis for

communicating requirements but are not analyzable or complete – This is not always acceptable

The form of requirements depends on how one intends to apply them and many

contextual factors

68

Contextual Factors

69

•  Regulatory compliance, e.g., standards •  Project size, team distribution, and number of

stakeholders •  Background of stakeholders and communication

challenges •  Domain complexity •  Presence of product lines with multiple customers •  Importance of early agreement on and full compliance

with requirements (e.g., Contract, safety certification) •  Frequency and consequences of changes in

requirements

Conclusions

70

•  Documented requirements are useful after all! •  The extent to which requirements and their traceability are

documented depends on multiple factors •  They also come in different forms, mostly depending on

analysis requirements •  Many challenges to be addressed by research:

(1) Ambiguity in Natural Language requirements (2) Automation of time consuming tasks: Scalability (3) Change impact

•  NLP technology is now mature and provides many opportunities for automation and lowering the documentation overhead

The challenge is to find the right trade-off!in a given context

71

Acknowledgements

72

•  Mehrdad Sabetzadeh •  Arda Goknil •  Shiva Nejati •  Chetan Aurora •  Ines Hajri •  Ghanem Soltana •  Chunhui Wang

Documented Requirements !are not Useless After All! !

!The Case for Supporting Change, !

Configuration, and Testing

Lionel Briand XP 2016, May 27th, 2016 Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg

Natural Language Requirements •  [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP

Approach

•  [TSE 2015] C. Arora et al., Automated Checking of Conformance to Requirements Templates using Natural Language Processing

•  [ESEM 2014] C. Arora et al., Improving Requirements Glossary Construction via Clustering

Requirements-Driven Testing •  [ISSTA 2015] C. Wang et al., Automatic Generation of System Test Cases from Use Case

Specifications

74

SysML Traceability and Safety Analysis •  [TOSEM 2014] L. Briand et al., Traceability and SysML Design Slices to Support Safety

Inspections: A Controlled Experiment

•  [IST 2012] S. Nejati et al., A SysML-Based Approach to Traceability Management and Design Slicing in Support of Safety Certification: Framework, Tool Support, and Case Studies

Legal Modeling and Analysis •  [MODELS 2014] G. Soltana et al., UML for Modeling Procedural Legal Rule

•  [SoSYM 2016] G. Soltana et al., Model-Based Simulation of Legal Policies: Framework, Tool Support, and Validation

75

Product Families and Configuration •  [MODELS 2015] I. Hajri et al., Applying Product Line Use Case Modeling in an Industrial

Automotive Embedded System: Lessons Learned and a Refined Approach

•  [SoSYM 2016] I. Hajri et al., A Requirements Configuration Approach and Tool for Use Case-Driven Development

Impact Analysis •  [FSE 2016] S. Nejati et al., Automated Change Impact Analysis between SysML Models of

Requirements and Design

•  [RE 2015] C. Arora et al., Change Impact Analysis for Natural Language Requirements: An NLP Approach

76