Parameterized Unit Tests

Parameterized Unit Tests

By Nikolai Tillmann and Wolfram SchulteProc. of ESEC/FSE 2005

Presented by Yunho KimProvable Software Lab, KAIST

Contents

Parameterized Unit Tests, Yunho Kim, Provable Software Lab, KAIST 2/27

• Introduction

• Parameterized unit tests

• Framework

• Conclusion

• Unit testing is a method of testing that verifies the individual units of source code are working properly

From http://en.wikipedia.org/wiki/Unit_testing

• A unit test consists of a sequence of method calls with assertions

• Unit tests are often served as simple specifica-tions

IntroductionWhat Is Unit Testing?


[TestMethod]void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added}

http://en.wikipedia.org/wiki/Unit_testing

• Unit tests tools do not provide any guidance for– which tests should be written for high code coverage– how to come up with a minimal number of test cases– what guarantees the test cases provide

• What about other array lists and other objects?

IntroductionWhat Is the Problem?


[TestMethod]void TestAdd(){ ArrayList a = new ArrayList(0); // Make an array list object o = new object(); a.Add(o); // Push a value into it Assert.IsTrue(a[0] == o); // Check that value was added}

• Parameterized unit tests are generalized unit tests by allowing parameters– Parameterized test methods are specifications of the

behavior of the methods-under-tests– PUTs describe a set of traditional unit tests which can be

obtained by instantiating the parameterized test methods

IntroductionParameterized Unit Tests(PUTs)


[PexMethod]void TestAdd(ArrayList a, object o){ // Given arbitrary array // lists and objects Assume.IsTrue(a!=null); // Assume a null int i = a.Count; a.Add(o); // Push a value into it Assert.IsTrue(a[i] == o); // Check that value was added}

Contents


• Introduction


• Framework

• Conclusion

• Pex generates Unit Tests from hand-written Parameter-ized Unit Tests through Automated Exploratory Test-ing based on Dynamic Symbolic Execution.

Parameterized Unit TestsPex: a PUTs framework


From http://research.microsoft.com/pex/default-.aspx

• Symbolic execution is used to automatically and system-atically produce the minimal set of actual parameters(test inputs)

Parameterized Unit TestsTest Case Generation


Algorithms1. For each formal parameter a symbolic variable is intro-

duced2. For each code path, a path condition is built over symbolic

variables1. Loops and recursions are approximated up to a speci-

fied number of unfoldings3. After all constraints are collected, find solutions for the

constraints4. Finally, deduce what inputs cause a code path to be taken

• The number of possible paths to consider can grow expo-nentially with the number of control flow decisions

Parameterized Unit TestsReusing Parameterized Unit Tests


1 class Bag{ 2 Hashtable h = new Hashtable(); 3 4 void Add(object o){ 5 if (h.ContainsKey(o)) h[o] = (int)h[o] + 1; 6 else h[o] = 1; 7 } 8 int multiplicity(object o){ 9 if (h.ContainsKey(o)) return (int)h[o];10 else return 0;11 }12 }1 [PexMethod]2 Void AddMultiplicityTest(Bag b, objectx){3 Assume.IsTrue(b!=null & x!=nul);4 int m = b.multiplicity(x);5 b.Add(x);6 Assert.IsTrue(m+1 == b.Multiplicity(x));

During symbolic execution, ContainsKey method is symbolically run twice

• If we have axioms summarizing the hash table’s operations, we can avoid execution of the hash table

Parameterized Unit TestsReusing Parameterized Unit Tests


[TestClass, UsingAxioms(typeof(HashtableTests))]Class BagTests{ [PexMethod] void AddMultiplicityTest() { } }

[TestClass, ProvidingAxioms(typeof(Hashtable))]Class HashtableTests{ [TestAxiom] void NothingContainedInitially(object key){ } [TestAxiom] void SetImpliesContainsAndGet(){ } }

Provided axioms are re-written as con-straints

Contents


• Introduction


• Framework– Symbolic state–Constraints– Symbolic Evaluation– Axioms

• Conclusion

• Symbolic states are like concrete states except that sym-bolic states can contain symbolic expressions

• Symbolic expressions E is described by the following grammar

• ObjectId is an infinite set of potential object identifiers• VarId, TypeId and FuncId are a set of variable, type and

function identifiers respectively

FrameworkSymbolic State


E = o object ids 2 ObjectId | v variables 2 VarId | t types 2 TypeId | f( ) function application 2 Fun-cId | 8 .E universal quantification

¹E¹v

• There are two types of function symbols

• Predefined function symbols have a fixed interpreta-tion in the theorem provers used– add(x, y), equals(x, y), subtype(x, y)– null, void, 0, 1, 2, … (constants are considered as nullary functions)– Storage function symbols operating on maps

• update(m, x, y) denotes the update of map m at index x to the new value y• select (m, x) selects the value of map m at index x

• Uninterpreted function symbols represent properties of ob-jects and method calls appearing in axioms



• Uninterpreted function symbols represent properties of objects and method calls appearing in axioms– For example, type(x) denotes the type of object x, fieldf(x) denotes the

address of field f of object x

• For each method m of the program with n parameters, up to three uninterpreted function symbols are introduced, ms, mx, mr– They are used to summarize different dynamic aspects of m

• Let h be an expression denoting the state of the heap in which m( ) is called– ms(h, ) denotes the resulting state of the heap– mr(h, ) denotes the return value of the call– mx(h, ) represents the type of an exception



¹x¹x¹x¹x

• There are two kinds of heaps, ‘extensional’ and ‘intensional’ heap

• The extensional heap is represented by nested applica-tions of the map update function

• For example, the execution of the code fragment

turns a given extensional heap He into H’e assuming the lo-cations p, q hold the expressions t, u

• Extensional heap operations have fixed interpretations in the theorem provers



p.f=1; q.g=2

H’e = update(update(He, fieldf(t), 1), fieldg(u), 2)

• The intensional heap is described by a history of method invocations

• ms(Hi, ) represents the sequence of method calls followed by a call to m( )

• Consider for example the execution of the following code fragment assuming that t is the expression held in location o

• Intensional heap operations are uninterpreted functions



¹x¹x

ArrayList a = new ArrayList(); a.Add(o);

H’i = Adds (ArrayLists(Hi,a),a,t)

• A symbolic state is a 5-tuple S = (O, A, He, Hi, X) – O ½ ObjectId– A is a stack of activation records associated with a method, a program

counter, as well as a store for the formal parameters and local vari-ables

– He and Hi are expression denoting the extensional heap and the inten-sional heap respectively

– X is an object expression denoting the current exception

• S+1 is like S except that the program counter has been in-cremented

• The set of all symbolic states is called State



• A constraint on a symbolic state is a pair C = (BG, PC)

• BG is the static background only depending on the pro-gram declarations

• PC is the dynamic path condition built up during symbolic execution

• The set of all constraints is called Constraints

• A constrained state is a pair (S, C)

Framework Constraints


• The one-step relation

describes the effect of the current instruction from a given constrained state (S, C)

• Most instructions are handled in the standard way– A method call instruction pushes a new activation record onto the pro-

gram stack

• The interesting cases are object creation, conditional branch, member access, assume and assert

Framework Symbolic Evaluation


! µ (State £ Constraints) £ (State £ Constraints)

• New object creation of type T– Let o 2 ObjectId – O(S) be a fresh object identifier– (S, C) ! (S’, C Æ type(o) = T) where S’ is like S+1 except that He(S) is re-

placed with

– Where f1, , fn are the fields of T and v1, , vn default values

• Conditional branch with a condition c and label l– If (S, C Æ c) is feasible, then (S, C) ! (S’, C Æ c) where the program

counter in S’ is set to l– If (S, C Æ : c) is feasible, then (S, C) ! (S+1, C Æ : c)



update( update (h,fieldf1 (o), v1) , fieldfn (o), vn)

• Member access with target expression t and result loca-tion r – If (S, C Æ t null) is feasible, then (S, C) ! (S’, C Æ t null) where S’

is like S+1 except that the location r holds (He (S), fieldf(t))– If (S, C Æ t = null) is feasible, then (S, C) ! (S’’, C Æ type(e) = Null-

ReferenceException Æ t = null) where S’’ is like S except that the current exception X = e

• Assume.IsTrue with condition c– If (S, C Æ c ) is feasible, then (S, C) ! (S+1, C Æ c)

• Assert.IsTrue with condition c is semantically equivalent to



If (!c) throw new AssertFailedException();

• PUT can be seen as a summary, an axiom of external be-havior of method-under-test

• To summarize the set of methods M, we refine the behavior of ! relation for method calls to M

• Call to a method m 2 M– Let H’i = ms (Hi (S), ).– If (S, C Æ mx (Hi(S), ) = void) is feasible, (S, C) ! (S’, C Æ mx(hi(S), ) =

void) where S’ is S+1 except that Hi is replaced with H’i

– IF (S, C Æ mx (Hi(S), ) void) is feasible, (S, C) ! (S’’, C Æ mx(Hi (S), ) void) where S’’ is S+1 except that Hi is replaced with H’i and X = mx(hi(S), )

Framework Axioms


¹x¹x¹x

¹x¹x ¹x

• PUTs generate axiom formulas for by exploring a test axiom method like we do for test case generation

• PUTs explore the method with a modified one-step relation !‘

• !‘ is like except that Assert.IsTrue(c) in a state (S, C) is treated like As-sume.IsTrue(c) and a new axiom formula 8h, .PC(C) ! c is generated and conjoined BG(C)

Parameterized Unit TestsAxioms


¹x

[PexMethod]void TestAdd(ArrayList a, object o){ // 0 Assume.IsTrue(a!=null); // 1 int i = a.Count; // 2 a.Add(o); // 3 Assert.IsTrue(a[i] == o); // 4}

• For example




• For example

• Exploring TestAdd generates the following universally quantified formula as an axiom




8 h,a,o. (a = null Ç subtype(type(a), ArrayList)) Æ a null ! getItemr ( Adds (getCounts (h,a),a,o), a, getCountr (h,a)) = o

Conclusion


• The authors presented the concept of parameter-ized unit tests, a generalization of unit tests

• PUTs can be turned into axioms, which summarize three different aspects of the methods’ behavior– The axioms can be reused during symbolic execution

Reference


• Parameterized Unit Testsby Parameterized Unit Testsin Proc. of ESEC/FSE 2005

Documents

Parameterized Unit Tests