Topic 1 Introduction to Software Testing

7/31/2019 Topic 1 Introduction to Software Testing

1/9

INTRODUCTIONNowadays, we are increasingly dependent on software to assist as well asfacilitate our daily chores. In fact, whenever possible, most hardwareimplementations are currently being replaced by software. From the washingmachine controllers, mobile phone applications to the sophisticated airplane

control systems, our growing dependency on software can be attributed to anumber of factors. Unlike hardware, software does not wear out. Thus, the use ofsoftware can also help to control maintenance costs. Additionally, software isalso malleable and can be easily customised as the need arises.

The fact that software is malleable can be an issue as far as dependability andreliability are concerned. Here, software testing becomes immensely importantespecially if software is employed in harsh, critical, or life-threateningapplications. Covering as much as 40 to 50 percent of the total softwaredevelopment costs (Beizer, 1990), testing can be considered one of the most

ToTopic11IntroductiontoSoftwareTestingpic

5. Explain the roadmap as shown in section 1.4.4. Describe the terminologies, terms and assumptions used in reference

to software testing; and

3. State how software testing fits into the software engineering lifecycle;2. Explain what is software testing;1. Recognise the importance of software testing;By the end of this topic, you should be able to:

LEARNING OUTCOMES

7/31/2019 Topic 1 Introduction to Software Testing

2/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING2

important activities for software validation and verification. Lack of testing canlead to disastrous consequences including loss of data, fortunes and even lives.For instance, consider the accident that occurred during the European Space

Agencys launching of Ariane 5 in 1996. Investigation by independentresearchers from Massachusetts Institute of Technology reveals that the disasteris caused by the mismatch of the hardware and software component faults (Lion,2005). The component erroneously puts a 64 bit floating point number in to a 16

bit space, causing overflow error. This overflow error affected the rocketsalignment function, and hence, causing the rocket to veer off course andeventually exploded a mere 37 seconds after lift off. This incident wasted billionsof Euros as far as satellite equipments and rockets equipment on board.

Figure 1.1: Application Error Dialog BoxIf the abovementioned example is beyond our comprehension, consider thefollowing common disaster that may have happened to some of us. Let say thatwe are in the middle of preparing our report to be submitted to a class in 5minutes time. Suddenly, the following screen appears (see Figure 1.1) and wehave not saved our work. What happen is that we have lost our valuable dataand therefore unable to submit our report on time.

As noted earlier, lack of testing could also be fatal if software is being used in lifethreatening or safety critical applications. Consider the software that controls theairbag system of a car. Suppose that the airbag system needs to be activatedwithin 5 milliseconds upon impact. What if, due to a software failure, the airbagsystem is only activated after 5 seconds instead? People could die as a result of

this delay.

These aforementioned examples have demonstrated some of the incidents due tosoftware failures. The question is how can testing helps to minimise if not avoidsuch incidents? In order to answer this question, there is a need to understandand define as well position software testing within the software engineeringlifecycle. These issues will be the subject of the next section.

7/31/2019 Topic 1 Introduction to Software Testing

3/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING 3

OVERVIEW OF SOFTWARE TESTING1.1

Software testing is the process of executing a program with the intent to find

errors (Myers, 1976). The objectives for software testing include:(a) Find defects and anomalies As will be seen in later topics, software testing

communities have provided a number of useful techniques that can be usedto identify possible defects and anomalies.

(b) Reduce risks for software failures If most of the defects and anomalies aredetected, the risks can indeed be minimised.

(c) Prove that the program is good or otherwise In some applications, theremay be a need to demonstrate and prove whether the program is workingwell or not.

(d) Detect variation from specification Software testing can help developerdetects variation from the specification, that is, to establish confidence thata program does what it is supposed to do.

In order to fulfil the aforementioned objectives, software testing activities can bedivided into three main stages. Termed test cycles, these stages are Test Planningstage, Test Execution stage, and Test Monitoring stage (see Figure 1.2).

Figure 1.2: Software Testing Cycles(Source: (Zamli et al., 2007a))

7/31/2019 Topic 1 Introduction to Software Testing

4/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING4

As the name suggests, the Test Planning stage involves the planning on how andwhat kinds of test techniques to be employed (as will be covered by later topics)as well as the resources involved (e.g. in terms of manpower, costing, timing, and

tools). Test planning stage also includes consideration on failure empirical data,which is, based on known problems with similar systems. Test Execution stageinvolves the activities to define and execute the planned test cases. Finally, TestMonitoring stage involves analysing coverage as well as monitoring whether ornot the test results conform to the specifications.

Having described the activities within software testing, there is also a need tounderstand how software testing fits into the whole software engineeringproduct development lifecycle. Referring to Figure 1.3, software developmentstarts with the requirement elicitation phase. Here, the customers and

stakeholders interact with the requirement engineers to produce the softwarespecifications. Based on the specifications, software engineers and programmerscollaborate to produce software design and implementations. This activity occursin the implementation phase. Software testing falls under the validation phasewhich may occur in parallel with the requirement elicitation phase andimplementation phase. The independent verification and validation (V&V) teamneeds to consult the requirement engineers for software specification. Based onthe software specification, the V&V team produces the test cases to be executedagainst the software implementation. If the execution results satisfy therequirement specification, then the software is ready to be released. Otherwise,some additional work need to be done to the design and implementation untilconformance is achieved.

Figure 1.3: Software Engineering Product Lifecycle(Source: (Zamli et al, 2007b))

7/31/2019 Topic 1 Introduction to Software Testing

5/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING 5

As seen above, the purpose of testing is not to prove anything, rather to reducethe perceived risk of not working to an acceptable value. The key challenges insoftware testing are not only dependent on the actual execution of the test cases

but also the production of quality test cases. Here, quality test cases are test casesthat are likely to detect faults.

SELF-CHECK 1.1

(a) List and discuss why software testing is important.

(b) Identify some of the common incidents due to softwarefailures.

(c) How does software engineering product lifecycle relate to

software engineering lifecycle?(d) Why do software testing activities need to be in parallel with

development?

RELATED SOFTWARE FAULTTERMINOLOGIES

1.2

In order to facilitate understanding of the material in this text, there is a need to

understand the meaning of some of the important software fault terminologies.(a) Specification defines the product being created. Often, specification

includes a functional requirement that expresses the features that theproduct will support. Sometimes, specification also outlines a non-functional requirement that represents the constraint on the product.

(b) Verification is an attempt to find errors by executing a program in a test orsimulated environment.

(c) Validation is an attempt to find errors by executing a program in a realenvironment.

(d) Error/bug/defect is a mistake in design, coding, requirements,specification and even testing.

(e) Fault is the representation of the error.(f) Failure is the result of faults (i.e. faults generate failure).(g) Incident is the visible manifestation of the failure.(h) Test case has inputs, steps and execution conditions as well as expected

results (i.e. passed and failed criterion).

7/31/2019 Topic 1 Introduction to Software Testing

6/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING6

(i) Test suite is a collection of test cases.(j) Conformance implies the outputs match with expected results.(k) Software artifact refers to any piece of software (i.e. models/descriptions)developed and used during the software development and maintenance

process. Examples are requirements specifications, architecture and designmodels, source and executable code (programs), configuration directives,test data, test scripts, process models, project plans, variousdocumentations and etc.

SELF-CHECK 1.2

Can you recap and differentiate between the above mentioned fault

terminologies?

SOFTWARE TESTING ASSUMPTION1.3

To ensure objective understanding on the test results obtained, software testingactivities are often undertaken with the following assumptions:

(a) Software testing can be used to show the presence of bugs but never theirabsence.

(b) Only exhaustive testing can show that a program is free from defects.However, exhaustive testing is impossible due to timing and resourceconstraints.

(c) As the number of detected defects in a piece of software increases, theprobability of the existence of more undetected defects also increases.

(d) There is no rule of thumb on when to stop testing apart from coveragemetric criteria (which will be discussed in later topics). Hence, highcoverage metric does not guarantee high fault detection.

(e) It is unethical to test ones own program.

(f) A small change in ones program does not necessarily improve theprogram, but can potentially lead to newer bugs.

7/31/2019 Topic 1 Introduction to Software Testing

7/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING 7

ACTIVITY 1.1

(a)

Work in pairs and identify three software that are commonlyused in your workplace. List out at least 5 possibleconsequences as a result of their failures.

(b) In terms of ethics, discuss why V&V teams must beindependent from the development teams to underttesting?

ake

TOPIC ROAD MAP1.4

This topic has discussed important background on software testing. In particular,this topic highlights the importance of software testing, as well as illustrates howsoftware testing fits into the software engineering lifecycle. Additionally, thistopic also highlights the common terminologies used in reference to softwaretesting.

Referring to Figure 1.4, the rest of the topics will be as follows. Topic 2 discusseson static analysis activities for software testing. Topic 3 focuses on dynamicanalysis activities. Unlike static analysis activities which concentrate on error

prevention (i.e. through specification review as well as code walkthrough),dynamic analysis activities focus on error detection (i.e. through programexecution). As such, Topic 3 highlights black box testing methods (also termed

behavioural testing) as well as white box testing methods (also termed structuraltesting). Also, technique for test case design based on mutation testing is alsosummarised. Topic 4 discusses the planning, documents, and portfoliosnecessary for software testing. Here, the focus of discussion is based on the IEEEStandard for Software Test Documentation (IEEE 829 1998). Topic 5 highlightsthe main issues concerning test case execution. In particular, this topic willhighlight the pass or fail criteria, the cascading dependencies of test execution,

the scaffolding technique as well as the need for test automation. Additionally,Topic 5 highlights various testing level. In doing so, Topic 5 also demonstratesthe scaffolding technique with drivers and stubs enabling functional testing evenwhen the whole code has not yet been completed. Topic 6 describes techniques tominimise the number of test cases based on coverage analysis as well as the t-way combinatorial design. Topic 7 complements Topic 6 by introducing variousstopping criteria for software testing. Specifically, Topic 7 explores coveragemetrics analysis, fault seeding technique, as well as statistical technique on thefault detection rate over specified time. Additionally, Topic 7 will also discuss thecriteria for choosing the optimum interaction strength of coverage, t, for t-way

7/31/2019 Topic 1 Introduction to Software Testing

8/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING8

test case minimisation strategy. Topic 8 explores the discussion on regressiontesting as well as highlights the change impact analysis techniques in an effort toreduce the need for re-testing when changes are introduced in the existing

software. Finally, Topic 9 discusses some of the automated tools that can assistand facilitate the testing process.

Figure 1.4: Text Roadmap

Software testing is an integral part of software engineering.

Lack of testing can lead to disastrous consequences including loss of data,fortune and even lives.

Software testing activities are often conducted in parallel with development.

Software testing stages encompass planning, execution and monitoringphases.

7/31/2019 Topic 1 Introduction to Software Testing

9/9

TOPIC 1 INTRODUCTION TO SOFTWARE TESTING 9

Error Dialogue boxTest Execution Stage

Test Monitoring StageTest Planning Stage

Beizer, B. (1990). Software testing techniques testing. Van Nostrand Reinhold.

Lion, J.L (2005). Ariane 5 failure report. Retrieved January 3rd, 2008, from

http://sunnyday.mit.edu/accidents

Myers, G.J. (1976). The art of software testing. Boston: John Wiley and Sons.

Zamli, K.Z. et al. (2007a). A tool for automated test data generation (andExecution) based on combinatorial approach. International Journal ofSoftware Engineering and Its Applications, July 2007, pp. 19-34.

Zamli, K.Z. et al. (2007b). SFITGenerator: Development of combinatorialalgorithms for automatic test case generation. Proceedings of the

International Robotics, Vision, Information and Signal Processing Conference2007 (ROVISP2007), Penang, pp. 31-35.