Embed Size (px)
Citation preview
1/42
2014 REU Program at ECUSoftware Testing - Foundations, Tools,
and Applications
Lecture 1 May 27, 2014
Introduction to Software Testing
Dr. Sergiy Vilkomir
2/42
Dr. Sergiy Vilkomir
Instructor’s office: S & T Building, C-111
Email: [email protected]
http://www.cs.ecu.edu/reu/index.html
http://core.ecu.edu/vilkomirshttp://core.ecu.edu/STRG/
3/42
Software life cycle
• concept • requirements • design • implementation (coding)• testing • installation• operation and maintenance
testing
Is testing important ?
4/42
Software Errors
Software errors can be found and eliminated during software testing
“Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce's National Institute of Standards and Technology (NIST)”.
(NIST News, June 42, 2002 )
5/42
Software Errors
Buggy Software May Have Crashed Mars Polar Lander (March 2000)
• The software problem crashed the Mars Polar Lander into the Red Planet’s frozen ground
• The most likely cause of the lander’s failure, investigators decided, was that a spurious sensor signal associated with the craft’s legs falsely indicated that the craft had touched down when in fact it was some 130-feet (40 meters) above the surface.
• This caused the descent engines to shut down prematurely and the lander to free fall out of the martian sky.
• The error was traced to a single bad line of software code.
Causes of software defects
Software Bugs with Fatal Consequences
• 1985 - 1986 few patients were killed in a hospital in USA because the medicine dose was calculated wrongly (software bugs in Therac-25 treatment machine lead to radiation overdoses, approximately 100 times the intended dose).
6/42
7/42
Causes of software defects
• In1996 the Ariane 5 rocket of the European space agency was destroyed because they used the software from Ariane 4 rocket (was self-destroyed 40 seconds after takeoff; more than $370 million was lost ).
8/42
Causes of software defects
Software Bugs with Fatal Consequences
• In 1962 NASA lost their 80 Million Dollar "Mariner 1" because of a missing "hyphen" in the program code.
9/42
Causes of software defects
Software Bugs with Fatal Consequences
During the Gulf War in February, 1991, a Patriot missile system operating in Dhahran, Saudi Arabia, failed to track and intercept an incoming Scud. The Iraqi missile impacted into an army barracks, killing 42 U.S. soldiers and injuring another 98.
10/42
The cause of the missile system failing to defend against the incoming Scud was traced back to a bug in Patriot’s radar and tracking software.The bug occurs in the calculation of the next location of the incoming target. That modified software reached Dhahran the day after the attack.
Computers and Software Errors
11/42
• On Tuesday, June 3, 1980, at 1:26 a.m., the display system at the command post of the Strategic Air Command (SAC) near Omaha, Nebraska, indicated that two submarine-launched ballistic missiles (SLBMs) were headed toward the United States.
• The SAC duty controller directed all alert crews to move to their B-52 bombers and to start their engines
• Land-based missile crews were put on a higher state of alert, and battle-control aircraft prepared for flight
• Fortunately, there were a number of factors which made those involved in the assessment doubt that an actual attack was underway
• The cause of these incidents was eventually traced to the failure of a single integrated circuit chip in a computer which was part of a communication system
Causes of software defects
Software Bugs with Fatal Consequences
• Software bugs in a Soviet early-warning monitoring system nearly brought on nuclear war in 1983.
• The software was supposed to filter out false missile detections caused by Soviet satellites picking up sunlight reflections off cloud-tops, but failed to do so.
• Disaster was averted when a Soviet commander, based on what he said was a '...funny feeling in my gut', decided the apparent missile attack was a false alarm. The filtering software code was rewritten.
12/42
Software Errors
13/42
Investigation found that the satellite in question had picked up the sun's reflection off the cloud tops and somehow interpreted that as a missile launch.
CNN Money - August 9, 2012
In less than an hour, Knight Capital's computers executed a series of automatic orders that were supposed to be spread out over a period of days. Millions of shares changed hands. The resulting loss, which was nearly four times the company's 2011 profit, crippled the firm
and brought it to the edge of bankruptcy.
14/42
Computer glitch that set fire to $440 million of Knight Capital Group's funds
Top Software Failures Of 2011(from http://www.businesscomputingworld.co.uk)
1. Financial services giant fined $25 million for hiding software glitch that cost investors $217 million
2. Computer system bugs cause Asian banking facilities’ downtime
3. Cash machine bug benefits customers by giving them extra money
4. 22 people wrongly arrested in Australia due to failures in new NZ $54.5 million courts computer system
5. 42,420 cars recalled after airbag-related software glitch
15/42
Top Software Failures Of 2011(from http://www.businesscomputingworld.co.uk)
Conclusions
• Software failures are costing companies and consumers large amounts of money
• The worst software failures have damaged reputations, impacted negatively on financials and caused stress to users.
• Each of the top 2011 software failure examples could easily have been avoided through an effective quality management and proper software testing 16/42
17/54
• Systems where failure means: death, injury, destruction, chaos, severe financial loss, etc.
• Such systems often are safety-critical systems or mission-critical systems
Student project
18/54
19/54
• Nuclear power plant systems
• Military systems
• Space systems
• Avionic software
• Medical software
• Large telecommunication systems
• . . .
20/42
Software testing
• Software errors can be found during software testing
• Testing can help but it is not for free
• The cost of software testing could be around 40% of the overall costs of software development
21/42
Terminology
Software testingError
FaultFailure
Input domain
Oracle
Regression testingUnit testing
Functional testingVerification Validation
V&V
22/42
Terminology
• Unfortunately, SE terminology in textbooks is often contradictory
• Two different textbooks could provide two different definitions for the same term
• What to do in this situation?
Solution: use terminology from standards
23/42
IEEE (Institute of Electrical and Electronics Engineers) standard
IEEE Std 610.12 1990
IEEE Standard Glossary
of Software Engineering
Terminology
24/42
New standard
• IEEE Std 610.12 1990 has not been updated for many years• Some terms are obsolete
• Instead: ISO/IEC/IEEE 24765:2010 “Systems and software engineering – Vocabulary”
New International Software Testing Standard
“Software and systems engineering — Software testing”
• ISO/IEC/IEEE 29119-1: Concepts & Definitions (published September 2013)
• ISO/IEC/IEEE 29119-2: Test Processes (published September 2013)
• ISO/IEC/IEEE 29119-3: Test Documentation (published September 2013)
• ISO/IEC/IEEE 29119-4: Test Techniques (draft)
• ISO/IEC/IEEE 29119-5: Keyword Driven Testing (draft)
25/42
26/42
From IEEE Std 610.12 1990
testing - the process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component
Contrast with: Static Analysis, Inspection
27/42
From IEEE Std 610.12 1990
• Static Analysis - the process of evaluating a system or component based on its form, structure, content, or documentation.
• Inspection - a static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems.
28/42
From IEEE Std 610.12 1990
• Fault - an incorrect step, process, or data definition in a computer program. Note: In common usage, the terms "error" and "bug" are used to express this meaning.
• Failure - the inability of a system or component to perform its required functions within specified performance requirements. The result of the fault.
29/42
From IEEE Std 610.12 1990
• test case - a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
30/42
• Input domain – the set of all possible inputs to a program
• test oracle - a source to determine expected results to compare with the actual result of the software under test. An oracle may be the existing system (for a benchmark), a user manual, or an individual’s specialized knowledge, but should not be the code
31/42
Y= a X + b – it is very easy to determine expected results
expected results ?
32
Goal of software testing
Test case that did not find an error Test case that discovered a new error
- successful
- successful
Increasing the quality and reliability of the program after removing the error
Increasing our confidence on the quality and reliability of the program
33
Goal of software testing
Dual Goal
• To find undiscovered errors AND
• to get confidence on the quality and reliability of the program
34
Lifecycle phase in which testing takes place
© Aditya P. Mathur 2005
35
From IEEE Std 610.12 1990• black-box testing -Testing that ignores the
internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions.
Syn:. functional testingContrast with: structural testing.• white-box testing -Testing that takes into
account the internal mechanism of a system or component. Types include branch testing, path testing, statement testing.
Syn: structural testing, glass-box testing Contrast with: functional testing
Is testing always done in the same way?
Types of software Consequences of defects
Program A
Program B
Program C
Program D Safety-critical software
Personal software
Company’s internal software
Commercial software
internal problems
nothing serious
financial losses, damaged reputation
loss of human life
36/42
• Testing is done differently in different contexts.– Type of software– Recourses (constraints): human recourses,
financial recourses, time– Regulatory requirements
• Conclusion: testing is context dependent
37/42
Causes of software defects
Human mistake (error)
Defect (bug, fault)in software
Failureduring software execution
38/42
How much testing is enough?Exhaustive testing is impossible
Testing everything (all combinations of inputs and preconditions) is not feasible
Example:• The exam has 30 questions and four possible answers for each question.• What is a number of all possible combinations of answers?• If we can test 1000 combinations per second, how long will we test all combinations? (An hour? A day? A month?)
Total number of combinations: 444 …. 4= 430 1.000.000.000.000.000.000
30 times
1 year 42.000.000 sec. We need 42.000.000 years for testing 39/42
How much testing is enough?
Not possible to say in generalDepends on each specific situation
• Pressures on a project include time and budget• Risk assessment to decide how much testing to do.
Important:• prioritization (do the most important tests first)• setting criteria when to stop testing• Take into account technical and business risk• Take into account project constraints.
40/42
Testing principles
Testing shows presence of defects
Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness
41/42
“Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of ?
Testing principles
Testing shows presence of defects
42/42
“Program testing can be used to show the presence of bugs, but never to show their absence!” 1970, famous quote of Dijkstra
Edsger Dijkstra (1930 – 2002) was a Dutch computer scientist. Winner of the 1972 Turing Award; creator of "structured programming."
