2
Why testing? Examples
Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander Smashed to the planet: failing to convert
between English measure to metric values
Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300 Misleading output of the tracking software
THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer
center received fatal overdoses Miss-handling of race condition of the
software in the equipment
3
Why testing? Numbers
On average, 1-5 bugs per KLOC (thousand lines of code) In mature software More than 10 bugs in prototypes
Windows2000 35MLOC 63K known bugs at the time of release 2 bugs per KLOC
$59.5B loss due to bugs in US 2002 (estimation by NIST)
It is not feasible to remove all bugs But try to reduce critical bugs
4
Answer is testing, why?
“50% of my employees are testers, and the rest spends 50% of their time testing”
---- Bill Gates, 1995
More reliable than inspection, relatively cheap Actually in the old days, when testing is expensive,
inspection was the major answer
You get what you pay (linear rewards) Compared to other 3 approaches
Inspection, static checking, formal proof
5
Testing: Concepts
Test case Test oracle Test suite Test script Test driver Test result Test coverage
6
Testing: Concepts
Test case An execution of the software with a given list of
input values Include:
Input values, sometimes fed in different steps Expected outputs
Test oracle The expected outputs of software by feeding in a
list of input values A part of test cases Hardest problem in auto-testing: test oracle
problem
7
Testing: Concepts
Test suite A collection of test cases Usually these test cases share similar pre-requisites
and configuration Usually can be run together in sequence Different test suites for different purposes
Smoke test, Certain platforms, Certain feature, performance, …
Test Script A script to run a sequence of test cases or a test
suite automatically
8
Testing: Concepts Test Driver
A software framework that can load a collection of test cases or a test suite
It can usually handle the configuration and comparison between expected outputs and actual outputs
Test Coverage A measurement to evaluate how well the testing is
done The measure can be based on multiple elements
Code Input combinations Specifications
10
Granularity of testing Unit Testing
Test of a single module Integration Testing
Test the interaction between modules System Testing
Test the system as a whole, by developers on test cases
Acceptance Testing Validate the system against user requirements, by
customers with no formal test cases Regression Testing
Test a new version with old test cases
11
Unit testing
Testing of an basic module of the software A function, a class, a component
Typical problems revealed Local data structures Algorithms Boundary conditions Error handling
12
An example: writing test cases without a test framework
Class Definition Consider a class that handles and manages a Fibonacci array Initialized the class
Fibonacci f = new Fibonacci (); Extend to a certain length
f. extend (length) Provide the number at certain index
f. get(index) Get Length
f.getLength() Provide a range of numbers
f. getRange(start, end)
13
An example: writing automatic test cases without a test framework
public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}
Not Enough!What happened if
an exception is thrown
14
An example: writing automatic test cases without a test framework
public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}
Potential test interference!
15
Common Operations
Comparison between actual results and expected results
Report error messages Catch Exceptions Re-initialize the class to be tested
16
Unit test framework
xUnit Created by Kent Beck in 1989
This is the same guy we mentioned in XP, design patterns The first one was sUnit (for smalltalk)
Junit The most popular xUnit framework There are about 70 xUnit frameworks for corresponding
languages
Never in the annals of software engineering was so much owed by so many to so few lines of code
-------Martin Fowler
17
State machine of JUnit
Initialization Of Test Class
setUptry{ testMethod} catch {…}
tearDown
Why?Try and Catch?
18
Writing a Test Casepublic class FibonacciTest extends TestCase { protected Fibonacci fTest; protected static int groudtruth[] = {0, 1,...,12586269025}; public void setUp(){ fTest = new Fibonacci(); } @Test public testInitialize(){ assertEquals(“Length of fTest after initialization”, 0, fTest.getLength()); } @Test public testExtend(){ fTest.extend(50); assertEquals(“…”, 50, fTest.getLength()); } @Test public testGet(){ fTest.extend(50); assertEquals(“…”, groudtruth[49], fTest.get(50)); } …}
19
Benefits of using test framework
Clear structure of testing Each test method for one feature Common initialization to setup
Assertion methods You can always define new assertion
methods
Reduced Interference Try catch to run all test methods Always do re-initialization in setup
21
Tips of writing test cases
Bottom Up If you know that obj.m1 -> obj.m2, test obj.m2 first Why?
You may not understand the error in m1 if you don’t know whether m2 is working or not
Externalize data Externalize expected results to files and load them
at the test case constructor Why?
Easier to maintain if you want to reuse them in several places
22
Tips of writing test cases Use development Database
Do not use real database for testing Why? Obvious
Do enough logging in the code during testing Why?
Help debugging
Testing for exceptions How?
public testException(){
try{
do(invalid); fail();
}catch(xxxException e){}
}
23
Assertions
public void testCapacity() { // a test method
….
assertTrue(fFull.size() == 100+size); //assertion
}
If assertion fails:Assertion failed: myTest.java:150 (expected true but was
false)Not so good!
Try: assertEquals(100+size, fFull.size()); //expected value firstAssertion failed: myTest.java:150 (expected 102 but was
103)Better!
Try: assertEquals(“list length”, 100+size, fFull.size()); Assertion failed: myTest.java:150 (list length expected 102
but was 103)
24
Assertions
Extend TestCase and write your own assertions AssertStringContains, AssertArrayEquals, … Use fail(String) to fail a test with certain message
Feel free to change the fixture Each test will re-setup
public void testRemoveAll() {
fFull.removeAllElements();
fEmpty.removeAllElements();
assertTrue(fFull.isEmpty());
assertTrue(fEmpty.isEmpty());
}
Tear down Consider the following test code
void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}
Problems?
void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { try { use f and b } finally { clean&close f, b }}void testBBB() { try { use f and b } finally { clean&close f, b }}
Better?
26
Tear down Consider the following test code
void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}void tearDown{ clean&close f, b}
Problems?
void setUp() { File f = open(“foo”); File b = open(“bar”);}…void tearDown{ try{ clean&close f }catch{ … } the same for b}
27
Tear down
Be careful about tear down If tear down is not complete, a test failure may
affect the following test cases Recover the changes done to global data that are
not well handled by the setup Database, files, network, global variables
Clean resources Caution of exceptions in tear down itself