Quantitative Non-Functional Requirements Publication

8/14/2019 Quantitative Non-Functional Requirements Publication

http://slidepdf.com/reader/full/quantitative-non-functional-requirements-publication 1/8

An Approach To Qunantitative Non-Functional RequirementsIn Software Development

Andrew J. Ryan

School of Information Technology and EngineeringGeorge Mason UniversityMS 5D3 Fairfax, VA 22030

{[email protected]}

Abstract. Non-functional requirements are commonly calledthe qualitative aspects of a system -- testability, mobility, andscalability, to name a few. However, when taking a holisticview of a system, non-functional requirements take on aquantitative nature. This paper will describe the RequirementsHierarchy Approach (RHA), a quantifiable method to measureand manipulate the effect non-functional requirements have on

a system by capturing the utility of functional requirements.Through the use of agent-oriented programming, multi-attribute utility analysis (MAUA), and decision science theory,non-functional attributes (of a system) can be used ascontainers in an attempt to capture the broadest cross-sectionof functional requirements, based on stakeholder input. Thefinal result will be an optimal set of requirements that satisfythe stakeholder’s needs and are not in opposition to oneanother.

INTRODUCTION

Requirements drive any systems engineering effort. Most of the focus has been placed in the area of functionalrequirements. According to Alford, “In nearly every softwareproject which fails to meet performance and cost goals,requirements inadequacies play a major and expensive role inproject failure”(Alford 1979). Although written “requirementsinadequacies” it should read “functional requirementsinadequacies.” There has been a tremendous amount of moneyand effort earmarked to solve this issue. With the emphasis onfunctional requirements, it is hard to understand why systemsengineers still struggle to create a complete set of requirementsfor a stakeholder. One reason for this has been the inattentionto non-functional requirements (NFRs) and the affect theyhave on fulfilling functional requirements. This is most

notable in the omission of non-functional requirements by thetwo documents which are considered the guidelines for theEngineering of Systems in the United States: ElectronicsIndustry Association (EIA) 632 and the International StandardsOrganization (ISO) 15228. The choice of this topic is a smallstep in an attempt to give non-functional requirements a moreprominent role in the field of systems engineering.

BACKGROUND

There has been some prior work on non-functionalrequirements (NFRs) in systems/software engineeringKenneth Chung presented one of the earlier attempts to capture

knowledge in this domain (Chung 1993). His work was latefollowed up by Myopoulos et al. (Myopoulos 1999)Myopoulos suggested that all requirements be viewed as goalsEach goal would be an umbrella for related functional and non-functional requirements. As these requirements are satisfied orsatisficed1 the system goals are effectively satisfied. Missingfrom this formula is the impact the stakeholder can have on therequirements elicitation process and the gap that is oftenpresent between stakeholder vision and requirementsrepresentation.

One of the strong points of the RequirementsHierarchy Approach (RHA) is that it creates a visual toolwhich illustrates to stakeholders how their requirements relateto each other and the effect this may have on total systemperformance. Oftentimes, stakeholders have overlappingintentions (not to mention differences in opinion with thedevelopers) and the need for tradeoff becomes necessaryBarber et al. espouses this belief, noting “System stakeholdershave varying notions on the purpose of requirements gatheringand these differences may lead to communication problemsFurther complicating the communication process is the‘requirements gap’ that separates the developer from theclient” (Barber et al. 1997). The requirements gap is one of themain targets of this paper. Moreover, the RequirementHierarchy Approach (RHA) can lead to positive results inrequirements elaboration, requirements tradeoff, requirementsvalidation, and verification.

NON-FUNCTIONAL REQUIREMENTS EXPLORED

Non-functional requirements are best described as qualitybased requirements which can only be realized at the end of thesystem development process. The overall measurement of a

1 To get a result that is good enough. Initially coined by Herbert Simon in his1957 book Models of Man.



non-functional requirement is based on the functionality of thesystem as a whole, and cannot be viewed as independententities. The ISO has identified a set of quality characteristics(or non-functional requirements) which provides a solidtaxonomy for NFRs (see figure 1).

Figure 1. ISO/IEC 9126 Taxonomy of Quality

Requirements (ISO/IEC 1991)

Taking this a step further, non-functional requirements can bequantified by the percentage of functional requirements they

satisfy. That is, by placing non-functional requirements at thetop of the “requirements hierarchy” and allowing theirpresence to filter down into the functional requirements, ameasurable unit can be established. The rest of this paper willbe dedicated to explaining this process in greater detail.

ROLE OF NON-FUNCTIONAL REQUIREMENTS IN A

SYSTEMThere is much debate about the role non-functionalrequirements should play in the requirements elicitation andsystem development process. Systems (and software)engineers fall into two camps when it comes to this question,with the line of demarcation being the “qualitative” nature of

non-functional requirements. A qualitative requirement isanalogous to an immeasurable requirement, which in anydiscipline is something to be avoided at all costs. In recentyears, there has been a push to make non-functionalrequirements more quantitative in nature (Chung 1993, Franch1998, Jacobs 1999). One of the more promising ideas was putforth by a group at the University of Texas (Tran et al. 1999),which includes a CASE tool to view the decision process of their tool.

Stakeholders often describe the system they want builin layman’s terms. Usually citing specifications qualitativelywith accompanying verbiage describing how it should happenFor example, one requirement may be: “The system shall allowanonymous login for unrestricted functions.” A secondrequirement states: “The system shall have protective firewallto enforce varying levels of access.” Implicitly, thestakeholder is contracting two non-functional requirementsusability and security. For the most part, when viewedindependently, non-functional requirements can be addressedroutinely. However, there are times (such as the one abovewhen they present a clash which initiates a ripple affect felt byother requirements – in attempting to satisfy one requirementthe developer is negating another requirement.

Dealing with Requirements Clash. When two or more nonfunctional requirements directly contradict each other, in termof fulfillment, we call this a requirements clash. This type osituation often goes unnoticed unless a visual representationsuch as the Requirements Hierarchy Approach, is utilized as atool to model requirements. There are two approaches that canbe employed when this situation occurs:

1. Decrease the measure of satisficability of one or more ofthe requirements.

2. Use the requirements hierarchy as a basis for requirementstrade-off.

The first approach is the preferred method. Thepower of satisficing goals, comes in the ability to alter thesequence which satisfies a goal. Therefore, if a goal has fivesub-goals, of which three are satisfied, one can then reduce themeasurement for satisficability to three with customer

approval. By decreasing the impetus placed on satisficing agoal, it also decreases the dependency the requirements haveon one other. Doing this can eliminate the clash or at leasdepreciate it to a point that it is manageable.

The second approach involves meeting with thestakeholder and, based on the requirements and the soft goatree, to create a list of trade-off requirements. Theserequirements can then be re-prioritized to alleviate the clashamong the requirements.

The hidden danger is that the conflicting nature of thedesired functionality (impacted by NFRs) of a system is norealized until late in the development life cycle. By managingthese potential conflicts early, the developer protects the

success of the project from being compromised.When bringing a system into being, great measures

are taken to appease the stakeholder – to build a trustingrelationship and confidence that the completion of the projectis in fact a feasible scenario. This action usually appears inconcrete form by a developer insisting they can meet alrequirements. Taken at face value, there is nothingintrinsically wrong with this, until non-functional requirementare brought into the picture.



NON-FUNCTIONAL REQUIREMENTS IGNORED

Late in 1999, NASA’s Mars Climate Orbiter andPolar Lander were lost in Mars’ atmosphere due to inadequateattention paid to a pair of non-functional requirements:interoperability and testability. In terms of interoperability, theOrbiter was developed by separate companies: Lockheed

Martin Aeronautics (main contractor) and the Jet PropulsionLaboratory (JPL). In calculating thrust for the unit, LockheedMartin scientists used English measurements, while JPLscientists used the metric system. This oversight was onlyrealized after the probe was deployed, a definite cause of thefailure

Secondly, the modules that were developed for thePolar Lander were not as succinct as initially thought. Duringsystem testing, one module was excluded (there was a knownproblem with that module) and tests were run. At a later time,the excluded module was tested independently and finaldelivery was made. It was not until the system failed in spacethat that scientists recognized that the ommission of themodule during system testing (they only ran one full test),affected the functionality of the landing gear. These twomistakes cost the United States taxpayers $319M.

RHA METHODOLOGY

A non-believer in the importance on NFRs might question theirimportance by stating that systems have been built withoutmuch attention being paid to NFRs. It has been proven thatomission of non-functional requirements can be detrimental tothe success of a system. Alan Davis states, “Knowing how tospecify behavioral requirements for a software system is onlyhalf the battle. All applications, from the most trivial to themost complex, have additional requirements that define theoverall qualities or attributes to be exhibited by the resultingsoftware system” (Davis 1993). Even if these non-functionalrequirements are not explicitly defined, it is still up to thedeveloper to create a design which takes these into account.Through the creation of a requirements hierarchy, this step isingrained into the system development process.

Linking functional requirements to non-functionalrequirements is not a trivial undertaking. It is worth reiteratingthat this method is not intended to be an elicitation tool, but atool which aids in trade-off and understanding of requirementselaboration, tradeoff, validation, and verification. Moreover,many non-functional requirements will have to be impliedfrom functional requirements or reworded to betterdepartmentalize the requirement within the ISO/IEC 9126

taxonomy. It is the opinion of this author that all requirementscan be reasonably positioned in the hierarchy represented inFigure 1. Although duplication within a category is allowed, itis encouraged to explore why the requirement cannot be placedin one category versus the other. The following section willfurther build on this idea and the process which goes intocreating a quantitative aspect to NFRs. Figure 2 illustrates therecommended cycle when employing this methodology with astakeholder.

Meet with

Stakeholders

Domain Expert

Requirements

Engineer

Document

Results

Run

Simulator

Knowledge

Engineer

Get requirementswith weights

Parse weights amongst top

level requirements; Add

additional sub-characteristics

Create Taxonomical hierarchy

from requirements

Add heuristics;

Look for clashes

Show possible clashes; Maximize strategy;

Risk analysis

Present

results

START

Figure 2. NFR Analysis Cycle

RHA AND MULTI-ATTRIBUTE UTILITY ANALYSIS

Multi-Attribute Utility Assessment (MAUA), firsused by Adelman and Donnel (Adelman et al. 1981), wasinitially used to evaluate knowledge-based systems. MAUAuses scoring and weighting procedures to evaluate the overalutility of a system. Employing the hierarchy presented inFigure 3, it is possible to assign weights to each requiremen(on a scale from .01 [wish list] to 1.0 [mandatory]) and traverseupward until a final number is reached. The weight given toeach requirement can come either directly from the stakeholder

(preferably) or a domain (subject matter) expert.The final number to be evaluated is for the top-level

NFRs. Functionality, reliability, usability, efficiencymaintainability, and portability need to be assessed in relationto each other. That is, the entire system is deemed to be (at100% when all requirements are satisfied. This number is thesummation of the lower six NFRs. It is up to the domainexpert to allocate priority across the six categories, based onthe project requirements. Once this has been done, the processnow switches to the requirements engineer to fill in theskeleton.

The process of linking functional requirements to quality

based attributes is a very objective task. The requirementsengineer should work closely with the domain (subject matter)expert in doing this. It will be shown later in this paper (viaexperiment) that agreement (in a hierarchy) among multipleparties is possible even without communication. The ability tospeak with experts while creating the hierarchy will onlystrengthen the reliability of the hierarchy



Suitability

wt=0.40

R1 (.4)

R2 (.3)

R5 (.1)R20 (.2)

Validity wt=0.3

Accuracy wt=0.7

Accuracy

wt=0.60

Functionality

wt=0.15

Reliability

wt=0.25

Usability

wt=0.05

Efficiency

wt=0.20

Maintainability

wt=0.10

Portability

wt=0.25

SYSTEMD o m a i n e x p e r t s e ts w e ig h t s

D o m a i n e x p e r t c h o o s e c a t e g o r ie s

a n d s e t s w e i g h t s

R e q u i re m e n t s E n g i n e e r c h o o s e c a te g o r ie s

a n d s e t s w e i g h t s

S t a k e h o l d e r (s ) s e t w e i g h t s

Figure 3. Partial Requirements Hierarchy

QUANTITATIVE ANALYSIS OF RESULTS

It would be impractical to believe a staticrepresentation of non-functional requirements can solve anyproblems a requirements engineer may encounter. In using theRHA, subject matter experts and experiencedrequirements/knowledge engineers are invaluable. Without arigid taxonomy of NFRs to act as a baseline, familiarity withthis approach becomes essential.

While only experience can provide the requirementsengineer with the intuition to analyze the results, the followingis a prescribed routine, once the hierarchy has been created:

1. Comparison of each of the top level nodes(functionality, reliability, usability, efficiency,maintainability, portability)

a. A low score does not necessarily meanproduct failure. Portability may not be of relative importance to a stakeholder if thesystem is an electrical heating system for a

skyscraper; however, if it were a solarpowered generator, portability wouldbecome of prime importance to capture asmuch of the sun as possible.

2. Check for uniqueness among requirements:

a. It is the responsibility of the requirementsengineer to parse out requirements in anunbiased manner, regardless of duplication.

An inexperienced requirements engineer maywant to limit their hierarchy to the 28 sub-NFRs in the ISO/IEC model, while aseasoned requirements engineer can be moredetailed and have more categories.

b. While there is nothing intrinsically wrong

with requirements appearing in more thanone category, on a case-by-case instance itmay be necessary to reevaluate therequirement in terms of scope andfulfillment.

i. To avoid bias, if possible, havemore than one knowledge engineerdo this (include something aboutsensitivity analysis).

ii. Another approach can be splittingrequirements into sub-parts,therefore capturing individualfunctionality in each respective sub-

requirement.3. Take a heuristic view of the requirements breakdownand make suggestions to the stakeholder wheredeemed appropriate.

Having the entire team sit in on this evaluation process isalways a good idea so all disciplines and viewpoints arepresent to provide perspective.



PRESENTATION OF RESULTS TO STAKEHOLDERS

The Requirements Hierarchy Approach (RHA) lends itself more to prototype-driven development efforts (spiral,incremental, evolutionary) than iterative approaches (i.e.waterfall). But, it can be looped and used in a waterfall

process too. Regardless of the method used, there will be apoint in time when the stakeholder will reenter the picture.This meeting will give birth to opportunities for tradeoff talks,issues of satisficing versus satisfying, and can serve as abaseline for initial user acceptance. Stakeholders will beimpressed by the simplicity of the requirements hierarchy andwill be able to match their project goals against the aspirationlevels listed for each requirement. Figure 4 demonstrates theactions a stakeholder can take after the first iteration of thecycle.

THE REQUIREMENTS HIERARCHY APPROACH INPRACTICE

To demonstrate the effectiveness of the RHA, a two-part experiment was performed via a web-based questionnaire.The first part consisted of twenty-five requirements whichwere taken from a System Requirements Specification (SRS)used in a graduate level course at George Mason University.The participants were given the system’s Statement of Work (SOW) and then asked to rate the relative importance (on an 11point scale) of each requirement with respect to the SOW. In

the second part, selected respondents were then asked tocategorize each requirement using the first tier (the top six) ofthe ISO 9126 framework. The users were given official ISOdefinitions to fully understand the intended meaning of eachcategory. The results of the survey were then queried byprofession, gender, and education to determine trends (if any).

In the first part of the experiment, respondentscollectively identified the same requirements as moreimportant than others. Among those receiving the highesscores were: “The system shall operate on a universallyaccepted platform” and “Users shall receive 24 hour technicasupport.” Low importance were given to requirements dealingwith the inclusion of a voice recognition interface and theusage of Commercial off the Shelf (COTS) solutions wherepossible. In addition to the average of each relativeimportance, the variance was also calculated as a measure of

uncertainty among thetotal. Requirementswith high/low relativeimportance but high

variance are furtherexamined. Oneinteresting discoverywas that the entirescale (0-10) was noused by many

respondentsespecially those with

non-technicabackgrounds. Thisidea is not a new oneas math/sciencedisciplines use the

total scale much morethan their liberaarts/social science

counterpartsUnderstanding thestakeholder is an

important aspect of using theRHA.The second part of theexperiment tested the

respondents’ ability to place each requirement into their non-functional requirement categories. This exercise is importantas it forces the stakeholder to anticipate how their requirements

will affect the system. Stakeholders will often agree to arequirement in principle without realizing that it has a broaddefinition and may affect various functions of the system. Thisis especially common when the stakeholders represent variousdepartments within the organization. Respondents wereallowed to choose two categories: a first and a second choiceThe first choice was given ten (10) points and the second five(5) points. The results were then measured against a silvestandard that was established prior to conducting the surveyRespondents successfully matched the silver standard in 16 o

M e e t w i t h

S ta ke ho lde rs

D o m a i nE x p e r t

R e q u i r e m e n t s

E n g i n e e r

D o c u m e n t

R e s u l t s

R u nS i m u l a t o r

K n o w l e d g e

E n g i n e e r


T r a d e o f f

Sat i s fy


Sat is f ice


- S t a k e h o l d e r u n d e r s t a n d s

l im i ta t ions on pro pose d se t o f

r e qu i re m e n t s , i s w i l li ng to

sacr i f ice func t ional i ty

- O n c e c o m p l e te a n o t h e r i te r a t io n

o f the c y c le i s r e qu i re d

- S tak e ho lde r w i l ling to m od i f y

p r ior i ti za t ion o f r e q u i re m e n t s in

a t t e mp t inc re ase poss ib le

func t ional i ty

- Se c on d i te ra t ion no t ne c e ssary ,

bu t c ou ld a id in r i sk ana ly s i s

- C u s t o m e r a g r e e s to c u r r e n t

c on f igura t ion

- C h a n g e s ( i f a n y ) s h o u l d b e n o t e d

in de s ign an d in i tia l p ro to ty pe c an

b e g i n

Figure 4. Stakeholder Options after first iteration



21 (76%) requirements. This illustrates that many of therequirements are broadly understood and only those otherrequirements need to be reviewed. Moreover, if face-to-facecommunication were possible, this percentage would be evengreater.

With the experiment complete, the final part was tomap the results. For the sake of exhibiting the full potential of the RHA, each category has been broken down into its secondtier. Furthermore, under usability, the sub-category markedunderstandability has been branched to show an alternateconfiguration (note: only one can be chosen). The overallsystem score shows that trade-off is necessary if fullfunctionality is to be achieved. Finally, the bold on R15 andR24 indicate requirements clash (see Appendix A for fullfigure).

CONCLUSION

Benefits of the Requirements Hierarchy Approach. In using

the Requirements Hierarchy Approach (RHA), clear gains canbe found in the areas of requirements elaboration, tradeoff, andrequirements validation and verification. Involving thestakeholder in the initial concept phase helps clarify theirvision of the system. Furthermore, categorizing requirementsand integrating the level of importance stakholders identifiedcreates a synergistic effect and sets the table for requirementstradeoff and preliminary acceptance metrics. A complimentarybenefit of establishing acceptance levels early in the life cycleis decreasing the risk involved in building a system thestakeholders asked for, but not what they wanted. Moreeloquently put, oftentimes developers appropriate resources tofunctionality that a user may care little about. In using theRHA, developers can be more in tune with the stakeholder’s

aspiration level as it pertains to specific functionality.

Another benefit in using the RHA is requirementsmanagement/project monitoring. With requirements listedcategorically, a project manager is able to track progress back to the hierarchy. Although developers would not be writingcode against the hierarchy, it creates a measurable correlationbetween the functionality they are working on and theunderlying hierarchy in which these requirements are a part.

Areas for Future Study. The methodology explained in thispaper is ripe for further study. The main area of particulainterest is automation. Knowledge-based systems are nowentering their fourth generation – reflecting the usage oautonomous agents to capture data. Moreover, the field ofknowledge engineering is just beginning to appreciate thepower of agents. Organized correctly, there is great promisefor a system that uses agents as the cognitive engine which“assists” in mapping the requirements and then performsanalysis on the hierarchy that was created. In time, throughrefinement, the RHA could ultimately be viewed as anobjective measure (which would be helpful for requirementsverification and validation) as opposed to a subjective one.

Finally, adopting an agreed upon taxonomy for NFRsis of prime importance. It is unrealistic to expect designers anddevelopers to incorporate an entity they cannot readily identifyWhile taxonomies aim to be inclusive of the entire set ofentities in question, a one or two level taxonomy would sufficeinitially as according to Chung et al. (Chung et al. 2000), thereare over 161 identifiable NFRs. Once an endorsed taxonomyis in place, researchers and practitioners would have the abilityto operate around this standard, as opposed to operating from ade facto standard.

REFERENCES

Adelman, Leonard; Donnell, M.L., Evaluating decision

support systems: A General framework and case

study. In S.J. Andriole (Ed.), MicrocomputerDecision Support Systems: Design, Implementationand Evaluation (pp285-310). Wellesley, MA: QEDInformation Science 1986.

Alford, M and Lawson, J., “Software RequirementEngineering Methodology (Development)”, U.S.A.FRome Air Development Center RADC-TR-79-1681979.

Barber, K. Suzanne, et al., “The Systems Engineering ProcessActivities (SEPA – Supporting Early RequirementsAnalysis and Intgration Prior to ImplementationDesign” The Laboratory for Intelligent Processes

and Systems The University of Texas at Austin 1997.Chung, K. L., “Representing and Using Non-functiona

Requirements for Information System DevelopmentA Process Oriented Approach”, Ph.D. Thesis, alsoTech. Rpt. DKBS-TR-93-1, Department of ComputerScience, University of Toronto, June 1993.

Chung, Lawrence et al., Non-Functional Requirements iSoftware Engineering Boston: Kluwer AcademicPress 2000.

Davis, Alan. Software Requirements: Objects, Functions, &

States, New Jersey: Prentice Hall, 1993.Franch, Xavier, “Systematic Formulation of Non-Functiona

Characteristics of Software” Proceedings of the Third

IEEE International Conference on Requiremen

Engineering 1998.



ISO/IEC Information Technology – “Software Productevaluation – Quality Characteristics and guidelinesfor their Use” ISO/IEC 9126 1991 (E).

Jacobs, Stephan, “Introducing Measurable QualityRequirements: A Case Study” Fourth IEEE

International Symposium on Requirements

Engineering 1999.Myopoulos, John, et al., “From Object-Oriented to Goal

Requirements” Transactions of the ACM January1999.

Tran, Quan and Chung, Lawrence, “NFR-Assistant: ToolSupport for Achieving Quality” IEEE Symposium on

Application-Specific Systems and Software

Engineering & Technology 1999.

BIOGRAPHY

Andrew J. Ryan was raised in the Bronx, New York. Heholds a Bachelors degree in Computer Science fromBinghamton University, a Master’s degree in Systems

Engineering and is currently working on a PhD in InformationTechnology – both at George Mason University. His interestsinclude agent programming, knowledge based systems, anddecision science theory. He is currently employed at MetronInc. as a software analyst and also teaches at George MasonUniversity. He is a member of IEEE, ACM, NSBE, INCOSE,and Who’s Who of International Professionals.



Appendix A – Requirements Hierarchy for Proposed System

.400

SYSTEM.883

Functionality

0.4

Portability

.15Maintainability

Efficiency

.1

Usability

.35Reliability

Understandability[.25]

R4 [.230]

R7 [.192]

R8 [.230]

R10 [.115]

R14 [.115]

R15 [.038]

R22 [.076]

Learnability [.1]

R14 [.750]

*R15 [.250]

Attractiveness

[.6]R7 [.192]

R8 [.230]

R9 [.076]

R10 [.115]

R11 [.153]

R12 [.230]

Compliance [.05]

R19 [1]

Time Behavior [1]

R13 [.428]

R14 [.428]

R15 [.142]

Availability [.1]

R21 [1]

Adaptability

[.6]

R17 [1]

Co-existence

[.4]

R24 [1]

Accuracy [.4]

R1 [.257]

R2 [.257]

R3 [.257]

R5b [.228]

Interoperability

[.2]

R6 [.210]

R18 [.368]

R24 [.421]

Suitability [.1]

R5a [1]

Compliance [.2]

R20 [.5]

R23 [.5]

Understandability[.25]

R4 [.272]

R7 [.227]

R9 [.090]

R10 [.136]

R14 [.136]

R15 [.045]

R22 [.090]

.115

.100

.200

.100

.250

.050

.600

.075 .085

0

.090

.0900.085.3410.366

.250

Documents

Quantitative Non-Functional Requirements Publication