Risk-based testing:Risk analysis fundamentals and metrics for software testing including
a financial application case study
Stale Amland *
TietoEnator Consulting AS, Koppholen 4, N-4313 Sandnes, Norway
Received 1 December 1999; accepted 1 December 1999
The idea of risk-based testing is to focus testing and spend more time on critical functions. By combining the focused process with
metrics it is possible to manage the test process by intelligent assessment and to communicate the expected consequences of decisions
taken. This paper discusses an approach to risk-based testing and how risk-based testing was carried out in a large project in a
financial institution. The paper concludes with how practical risk-based testing experience should inform theory and provide advice
on organizational requirements that are necessary to achieve success. 2000 Elsevier Science Inc. All rights reserved.
Keywords: Risk; Risk analysis; Risk-based testing; Risk management
1. Why risk-based testing?
IT projects are very rarely on time, schedule orbudget. Very often early project phases are delayed.Consequently, when it eventually comes down to testing,the time to delivery is extremely short and there is nobudget left due to the development overrun! Also, dur-ing development everybody is as a rule extremely busyresulting in test preparation activities not receiving theappropriate attention.
Risk-based testing helps to address these problems.A methodology was needed to assist in prioritizing whatto test and in determining how thorough the testingshould be, in order to deliver perceived good qualitywithin the deadline. Also a technique was needed tocommunicate the prioritization and the level of testingto the testers and to convince the management that thetesting would be good enough. The methodologydefined was inspired by vstedal and Stalhanes A goaloriented approach to software testing (vstedal andStalhane, 1992). They presented a theory this projectput it into practice.
A technique was needed to optimize the eort of thetesting given a particular set of constraints (budget,
hours and deadline). It was also a requirement to be ableto measure and thereby communicate the conse-quences of this particular set of constraints in an ob-jective manner.
Traditional testers (working with hardware or soft-ware) have always used risk-based testing, but in an adhoc fashion based on their personal judgement. Whatthe project tried to achieve was an agreed upon priori-tization with an objective foundation.
2. The case
This paper is based on the system test stage of aproject developing a retail banking application. Theproject included an upgrade of a customer informationsystem being used by clients as a central customer, ac-count and product database, and a complete reengi-neering of a retail banking system. The project scopeincluded reengineering of the data model, technologychange from IMS/DL1 to CICS/DB2, rewrite from JSPCOBOL to COBOL-2 and a completely new physicaldesign.
During this rewrite a large investment was made inproductivity tools, design, quality assurance and testing.Test coverage tools were used during unit test requiring100% code coverage. It is reasonable to believe that the
The Journal of Systems and Software 53 (2000) 287295www.elsevier.com/locate/jss
* Tel.: +47-51-96-30-19; fax: +47-51-96-30-01.
E-mail address: email@example.com (S. Amland).
0164-1212/00/$ - see front matter 2000 Elsevier Science Inc. All rights reserved.PII: S 0 1 6 4 - 1 2 1 2 ( 0 0 ) 0 0 0 1 9 - 4
project had substantial benefits from the extensive unittesting that took place before system testing, especiallysince the programmers in general were lacking businessknowledge. The programmers were mostly universitygraduates with no industry experience, but during theinduction programme and through senior personnelcoaching, they became a very proficient programmingteam.
The programmers did all unit testing. A team wasestablished to undertake system testing. Some pro-grammers were transferred to the test team on comple-tion of their programming and unit test work.
The applications consist of approximately 300 onlinetransactions and 300 batch programs, a total of 730,000SLOC 1 and 187 dB2 tables. This is the server part only,no GUI-client was tested in this project. A PC-based testbed was used to test the server transactions.
The project started in June 1994 and was delivered inOctober 1995. The project took approximately 40 man-years eort over 17 months. This paper documents ex-periences from the system test stage, which consumedapproximately 22% of the total project resources.
For online functional testing there was a test team ofseven testers with one team leader. The online fix teamvaried from 3 to 6 people. The online test team leadermanaged both teams. Batch testing started approxi-mately two months after online testing, the total numberof people in test and fixed increased dramatically at thispoint. Including a small test automation team, a non-functional test team and test management, the totalnumber of people involved in testing peaked at 40people.
The system test stage included:1. Technical system test, i.e., what is usually referred to
as environment test and integration test. Due to dif-ferences between the development environment andthe production environment, the system test stagehad to test all programs in the production environ-ment. During system test, the test team had to dothe integration test of the online system by testingand documenting all online interfaces (called mod-ules). The team also had to perform the integrationtest of the batch system(s) by testing and document-ing that all modules had been called and also testingthe complete batch flow.
2. Functional system test, i.e., black box testing of allprograms and modules to detect any discrepanciesbetween the behavior of the system and its specifica-tions. The integration test verified that all moduleshad been called, and therefore the functional systemtest was designed based on application functionality.
3. Non-functional system test. The system test also testedthe non-functional requirements, i.e., security, perfor-
mance (volume and stress test), configuration (appli-cation consistency), backup and recovery proceduresand documentation (system, operation and installa-tion documentation).As for all projects, time and resources were limited.
At the beginning of construction (programming), thesystem test strategy was still not agreed upon. Since thedevelopment project was a very large project forthe vendor and therefore consumed nearly all availableresources, the number of people with experience avail-able for test planning was limited.
The final strategy for the system test was agreed ap-proximately one month before the end of construction,and the time for planning was extremely short. A tra-ditional approach to system test planning based on testpreparation done in parallel with design and construc-tion, could therefore not be used.
3. What are the concepts risk and risk management?
Risk is something that might happen, whereas aproblem is something we know will happen (or has al-ready happened). If an event is associated with risk thenthere is a potential loss or impact associated with thatevent.
A simple risk model was used in this project, utilizingonly two elements of risk exposure: 2
1. the probability of a fault (in the online transaction orbatch program);
2. the cost (consequence or impact) of a fault in the cor-responding function if it occurs in production.Of main interest here are business risks. That is also
why only the above mentioned two elements of riskexposure have been used in this project. Several otherelements of risk exposure have been used in risk analysiswhen identifying risk impact, such as impact on systemperformance, schedule delays, etc. The simple calcula-tion used here is based on the assumption that aschedule delay without economic (i.e., cost) conse-quences is not a business risk. The same thing applies tobad system performance, if poor performance will notaect your business (for example, giving rise to customercomplaints or end-user dissatisfaction), there is no needto fix it.
Mathematically risk exposure can be expressed as
Ref P f Cf ;where Ref is the risk exposure of function f; Pf theprobability of a fault occurring in function f and Cf isthe cost if a fault occurs (in production) in function f.
1 SLOC Source Line of Code, excluding comments.
2 An overview of risk management and how it is performed is given
by Pfleeger (this issue) who also discusses dierent problems with risk
management and provides a number of good references.
288 S. Amland / The Journal of Systems and Software 53 (2000) 287295
The risk managing process was similar to the onedefined by Karolak (1996) as shown in Fig. 1. Therectangular boxes are Karolaks original process forsoftware engineering risk management, while the ovalboxes are a mapping of the risk process to the testprocess. The diagram does not depict a complete testprocess, only those elements relevant to show the rela-tion between testing and risk management.
The start of risk identification is the test item tree,that is the hierarchical breakdown of functions andfeatures in the system to be tested. The system to betested is input to the test plan, which should also containthe risk strategy. The latter is defined by the project andwill depend on the type of system to be developed andtested. It will also depend on the developer and the userenvironment (i.e., the risk profile for an internet productmight be dierent from the risk profile for a financialapplication) and other quality and commercial require-ments.
The risk strategy should define the optimizing crite-ria, for example: no function should have a risk exposure higher then
X million USD; risk exposure for function 1 should be less than risk
exposure for function 2 (Ref1 < Ref2).The risk strategy will then define the metrics to be
collected and the tracking to be done.Risk assessment is based on the test item tree (func-
tions and features) and this is the point when theprobability and cost of a fault in each function andfeature should be defined.
Since the system to be tested is now defined, the riskstrategy identified in the test plan and the identificationof cost and probability have all been calculated, a pri-oritized list of functions and features to be tested can becreated and the actual risk mitigation (the testing) canbegin.
Metrics are an important part of risk-based testingbecause they will be used for monitoring progress andquality to detect process discrepancies. As well as forrisk management, prediction is important for risk-based
testing. It is always complicated to estimate the resourcerequirements for testing, it is therefore important to useinformation during the first part of a test to predict theresources required to complete the test and to identifyareas where most resources will be required.
4. Risk-based testing approach
The definition of the risk-based process above, wasthe easy part. To implement the process was morecomplicated. The approach used can be split into thefollowing steps:
Step 1: planning (risk identification/risk strategy). Thisstep is part of every test project. The test plan will beestablished including what to test, description of testenvironments, standards and procedures for test docu-mentation, execution and logging. An overall riskstrategy should be defined.
Step 2: identify risk indicators (risk assessment). Toidentify the probability of a fault in a function can bevery dicult. In this project, measures for each functionwere identified which were expected to have impact onthe quality (the number of faults was used). Thesemeasurements included size of each function, number ofchanges since previous release, design quality based onnumber of change requests raised on design documen-tation and complexity as in functional complexity 3 (notthe structural complexity of the code).
It is important to use indicators that are meaningfulto those participating in the process of quantifying theindicators (and the weights). Cf. Pfleeger (this issue),who states that We must personalize the quantified riskto make it meaningful.
One meeting was used to have everybody agree on thedierent indicators. Some preparation had been donebeforehand, like identifying the dierent functions andfunctional areas. An initial list of probability indicatorswas proposed in the meeting. This list was then dis-cussed and agreed on the people present were of theopinion that these indicators would be a true reflectionof the number of expected faults. The focus was on thebusiness risks related to faults in the application. Thetest manager facilitated the meeting setting the numericvalue for each indicator. Dierent roles were represent-ed: developers, designers, product specialists, qualitymanager, development manager, project management,sales manager and corporate management. The idea wasto obtain a consensus so that a risk exposure for eachfunction could be agreed on.
Fig. 1. Risk management process mapped to some of the corre-
sponding elements in a test process. Diagram according to Karolak
3 A subjective measure on complexity was used by using functional
complexity, that is how complicated is it for the programmer to
understand what the function he is about to create, will do in
S. Amland / The Journal of Systems and Software 53 (2000) 287295 289
Other indicators (not used in this case study) might befrequency of use and function points (if available).
For all indicators (ij) only three values were used: low(1), medium (2) and high (3). However, each indicatorwas weighted (wj) with a weight ranging from 1 to 5 with5 being most important. The total probability of func-tion f will therefore be
ij wj; where i 1 3 and w 1 5:
The project used the following values for weight wj:
For example, the function Close Account had thefollowing indicators:
The probability for Close Account is therefore
Pf 2 5 2 5 2 1 3 3 31:Step 3: identify cost of a fault (risk assessment). This is
a similar process to step 2. Again the project used low(1), medium (2) and high (3) for the cost. Together withthe customer, a cost was identified for the customer andfor the supplier if a fault occurred. The assumption wasthat the two were equally important.
The same meeting that was used for agreeing onprobability indicators was used to have everybody agreeon the cost of a failure. The important thing was to havedierent roles present so the cost for the business ofcustomer and supplier could be assessed. It was essentialthat, for example, the cost for the suppliers technicaldepartment should not be too focused on.
Mathematically the costs can be expressed as (dividedby 2 to find average)
Cf scf ccf 2
where Cf is the total cost if a failure occurs in functionf; scf the suppliers cost if a failure occurs in function fand ccf...