Parameterized Unit Testing

1

Parameterized Unit TestParameterized Unit Test

Parameterized Unit Testsserve as specificationscan be leveraged by (automatic) test input generatorsfit in development environment, evolve with the code

Traditional approach:• Writing many tests that cover the code,• making sure it does not crash.

3

Possible test case, written by HandPossible test case, written by Hand

Parameterized Unit Test, written by handParameterized Unit Test, written by hand

Test input, generated by Pex

Test input, generated by Pex

5

A unit test is a small program with assertions.

[TestMethod] public void Add(){ HashSet set = new HashSet(); set.Add(3); set.Add(14); Assert.AreEqual(set.Count, 2);}

Many developers write such unit tests by hand. This involvesdetermining a meaningful sequence of method calls,selecting exemplary argument values (the test inputs),stating assertions.

Design and specificationby example

Code coverage and regression testingconfidence in correctnesspreserving behavior

Short feedback loopunit tests exercise little codefailures are easy to debug

Documentation

Quality of unit testsDo they test what is important?

Amount of unit testsHow many tests to write?

New code with old testsEven if unit test

Hidden integration test

Coverage: Are all parts of the program exercised?

statementsbasic blocksexplicit/implicit branches…

Assertions: Does the program do the right thing?

test oracle

Experience:Just high coverage or large number of assertions is no good quality indicator.Only both together are!

Parameterized Unit TestParameterized Unit Test

A parameterized unit test is a small program that takes some inputs and states assumptions and assertions.

9

PUTs separate two concerns:1)The specification of external behavior

(i.e., assertions)2) The selection of internal test inputs

(i.e., coverage)

In many cases, Pex can construct a small test suite with high coverage !

A PUT can be read as a universally quantified, conditional axiom.

int name, int data.name ≠ null ⋀ data ≠ null ⇒ equals(

ReadResource(name,WriteResource(name,

data)),data)

Pex is a test input generatorPex starts from parameterized unit testsGenerated tests are emitted as traditional unit tests

Pex analyzes execution pathsAnalysis at the level of the .NET instructions (MSIL)Incremental analysis that discovers feasible execution pathsThorem prover/constraint solver Z3 determines satisfying assignments for constraint systems representing execution paths

Hand-written

13

// FooTest.cs[TestClass, PexClass]partial class FooTest{ [PexMethod] void Test(Foo foo) {…}}

// FooTest.Test.cspartial class FooTest{ [TestMethod] void Test_1() { this.Test(new Foo(1)); }

[TestMethod] void Test_1() { this.Test(new Foo(2)); } …}

Pex

• User writes parameterized tests• Lives inside a test class • Generated unit tests

• Pex not required for re-execution• xUnit unit tests

xUnit AttributesxUnit Attributes

Parameterized Unit Test

Parameterized Unit Test

Partial ClassPartial Class

Generated

http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx

Static analysisVerify properties for all possible executionsConservative ("over approximation")Spurious warnings ("false positives")

Dynamic symbolic execution (Pex’ test input generation technique)

Verify properties for many execution pathsFinds most errors within configured boundsNo spurious warnings (well, almost, see later)

Dynamic analysisTestingMay miss errors ("under approximation")No spurious warnings

API contracts (Eiffel, Spec#, JML, …)At the level of individual actionsGoal: guarantee robustnessProblem: abstraction is hard, e.g. how to describe the protocol

Parameterized unit testsScenarios specify functional correctnessConfidence in robustness by automated test coverage

Unit testsScenarios, spanning multiple actionsGoal: functional correctnessProblem: missing implementation coverage

Starting from parameterized unit tests as specification, we can state the testing problem as follows.

Given a sequential program P with statements S, compute a set of program inputs I such that for all reachable statements s in S there exists an input i in I such that P(i) executes s.

Remarks:By sequential we mean that the program is single-threaded.We consider failing an assertion, or violating an implicit contract of the execution engine (e.g. NullReferenceException when null is dereferenced as special statements.Since reachability is not decidable in general, we aim for a good approximation in practice, e.g. high coverage of the statements/branches/… of the program.

Exploration of all feasible execution paths:Start execution from initial state with symbolic values as inputOperations yieldterms over symbolic valuesAt conditional branch, fork execution for each feasible evaluation of the conditionFor each path, we get an accumulated path condition

For each path, check if path condition is feasible(using automated constraint solver / theorem prover)

ptrue false

C’=C⋀⌝pC’=C⋀p

C

if (p) then … else …

int Max(int a, int b, int c, int d) { return Max(Max(a, b), Max(c, d));}

int Max(int x, int y) { if (x > y) return x; else return y;}

Exploration of all feasible execution paths:Start execution from initial state with symbolic values as inputOperations yieldterms over symbolic valuesAt conditional branch, fork execution for each feasible evaluation of the conditionFor each path, we get an accumulated path condition

For each path, check if path condition is feasible(using automated constraint solver / theorem prover)

ptrue false

C’=C⋀⌝pC’=C⋀p

C

if (p) then … else …

Constraint solver cannot Constraint solver cannot reason about certain reason about certain

operationsoperations(e.g., floating point (e.g., floating point

arithmetic, interactions with arithmetic, interactions with the environment)the environment)

Constraint solver cannot Constraint solver cannot reason about certain reason about certain

operationsoperations(e.g., floating point (e.g., floating point

arithmetic, interactions with arithmetic, interactions with the environment)the environment)

Execution of programsExecution of programsthat interact with that interact with

stateful environment stateful environment cannot be forked!cannot be forked!

Execution of programsExecution of programsthat interact with that interact with

stateful environment stateful environment cannot be forked!cannot be forked!

Dynamic symbolic execution combines static and dynamic analysis:Execute program multiple timeswith different inputs

build abstract representation of execution path on the sideplug in concrete results of operations which cannot reasoned about symbolically

Use constraint solver to obtain new inputssolve constraint system that represents an execution path not seen before

TestTestInputsInputsTestTest

InputsInputs

Constraint Constraint SystemSystem

Constraint Constraint SystemSystem Execution Execution

PathPathExecution Execution

PathPath

KnownKnownPathsPathsKnownKnownPathsPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Result: small test suite, Result: small test suite, high code coveragehigh code coverage

Initially, choose Arbitrary

Finds only real bugsFinds only real bugsNo false warningsNo false warnings


InputsInputs




PathPath





Solve




a[0] = 0;a[1] = 0;a[2] = 0;a[3] = 0;…

a[0] = 0;a[1] = 0;a[2] = 0;a[3] = 0;…


InputsInputs




PathPath





Solve




Path Condition:… ⋀ magicNum != 0x95673948


InputsInputs




PathPath





Solve




… ⋀ magicNum != 0x95673948… ⋀ magicNum == 0x95673948


InputsInputs




PathPath





Solve



a[0] = 206;a[1] = 202;a[2] = 239;a[3] = 190;

a[0] = 206;a[1] = 202;a[2] = 239;a[3] = 190;



InputsInputs




PathPath





Solve




Dynamic symbolic execution will systematically explore the conditions in the code which the constraint solver understands.And happily ignore everything else, e.g.

Calls to native codeDifficult constraints (e.g. precise semantics of floating point arithmetic)

Result: Under-approximation, which is appropriate for testing

Calls to external world

Unmanaged x86 code

Unsafe managed .NET code (with pointers)

Safe managed .NET code

Most programs are not self-containedIn fact, large parts of the .NET Base Class Library are not written in .NET

void Complicated(int x, int y) { int Obfuscate (int y) { if (x == Obfuscate(y)) return (100+y)*567 % 2347; error(); } else return;}

Dynamic symbolic execution, starting from Complicated, runs the code twice:

1. Call Complicated() with arbitrary values, e.g. -312 for x, 513 for y Record branch condition “x != (100+y * 567) % 2347” error is not hitCompute values such that “x == (100+y * 567) % 2347” (using constraint solver)

2. Call Complicated() with computed values for x, y (e.g. x=(100+513 * 567) % 2347, y=513)error is hit; coverage goal is reached 29

30

Pex treats all possible exceptional control flow changes like explicit branchesDeterministic exceptions through constraint solving:

NullReferenceExceptionIndexOutOfRangeExceptionOverflowExceptionDivisionByZeroException

Non-deterministic exceptions through exception injection (if enabled)

OutOfMemoryException, StackOverflowExceptionThreadAbortException…

Add ImplicitNullCheck testRun PexHow many tests will be necessary?

Try it out with other instruction:Allocating new arrays,Accessing array indexes,Field dereference

Problem:• Constraint solver determines

test inputs = initial state of test• Most classes hide their state (private fields)• State is initialized by constructor, and can

be mutated only by calling methods• What sequence of method calls reaches a

given target state?• There may be no such sequence• In general, undecidable

Two approaches:• (Guided) exploration of constructor/mutator

methods• Testing with class invariants

Specification:

[PexMethod]public void ArrayListTest(ArrayList al, object o){ PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}

Exploration driverExplicitImplicit, in Pex configurable through attributes, e.g.

PexExplorableFromConstructorAttribute

Result: Exploration of reachable states

Only within generally configured boundsUnder-approximation

34

Write class invariant as boolean-valued parameterless method

Refers to private fieldsMust be placed in implementation code

Write special constructor for testing only

May be marked as "debug only"Constructor sets fields, assumes invariant

Result: Exploration of feasible statesMay include states that are not reachable

35

public class ArrayList { private Object[] _items; private int _size, _version, _capacity; private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; }#if DEBUG public ArrayList(object[] items, int size, int capacity) { this._items = items; this._size = size; this._capacity = capacity; if (!this.Invariant()) throw new InvalidOperationException(); }#endif}

36

void PexAssume.IsTrue(bool c) { if (!c) throw new PexAssumptionViolationException();}

void PexAssert.IsTrue(bool c) { if (!c) throw new PexAssertionViolationException();}

Executions with assumption violation exceptions are ignored, not reported as errors or test casesBoth assumption violation and assertion violation exceptions are ‚uncatchable‘

Special code instrumentation prevents catchingto avoid tainting of coverage data 37

• Revisit explicit ArrayList driver• TestEmissionFilter=PexTestEmissionFilter.All

38

If the test does not throw an exception, it succeeds.If the test throws an exception,

(assumption violations are filtered out),assertion violations are failures,for all other exception, it depends on further annotations.

AnnotationsShort form of common try-catch-assert test code[PexAllowedException(typeof(T))][PexExpectedException(typeof(T))]

DateTime.ParseMay throw FormatException

Loops and recursion give rise to potentially infinite number of execution pathsIn Pex: Configurable exploration bounds

TimeOutMaxBranchesMaxCallsMaxConditions

Number of conditions that depend on test inputs

MaxRunsConstraintSolverTimeOutConstraintSolverMemoryLimit

Code we don’t want to testWe usually don't want to re-test code that we use and trust, e.g. already tested libraries

Code not accessible to white-box test generation

No symbolic information about uninstrumented codeNo symbolic information about code outside of known execution machine (.Net for Pex)

42

Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.Integration test: exercises large portions of a system.

Observation: Integration tests are often “sold” as unit tests

White-box test generation does not scale well to integration test scenarios.Possible solution: Introduce abstraction layers, and mock components not under test

AppendFormat(null, “{0} {1}!”, “Hello”, “World”); “Hello World!”

.Net Implementation:

public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) {

if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …

44

Introduce a mock class which implements the interface. Write assertions over expected inputs, provide concrete outputs

public class MFormatProvider : IFormatProvider {

public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); }}

Problems:Costly to write detailed behavior by exampleHow many and which mock objects do we need to write?

45

Introduce a mock class which implements the interface. Let an oracle provide the behavior of the mock methods.


public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>();

return o; }}

Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 46

Extract SamplesOpen sample solutionShow AppendFormat codeShow AppendFormat testRun Pex

47

Problem: Without further work, parameterized mock objects might lead to spurious warnings.

48

Write assertions over arguments,And assumptions on results.


public object GetFormat(Type formatType) {

Assert(formatType != null);

…

object o = call.ChooseResult<object>();

PexAssume.IsTrue(o is ICustomFormatter);

return o;

}

}

(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)

49

50

API-level interface contracts (e.g. written in Spec#) can be leveraged to restrict behavior of mock objects.

Consider the following Spec# contract:

interface IFormatProvider {

object GetFormat(Type formatType)

requires formatType != null;

ensures result != null && formatType.IsAssignableFrom(result.GetType());}

TAP 2008

Go back to Max/4 exerciseAdd reference to Microsoft.Pex.GraphsRegister package:[assembly: Microsoft.Pex.Graphs.PexGraphsPackage]

Use package:[Microsoft.Pex.Graphs.SearchGraph]

Run PexView graphs in report

Motivation:Ideally, all environment interactions go through mockable abstraction layersHowever, this is sometimes not achieveable in practice

Examples:Program may (indirectly) use System.Environment.Exit, which terminates Pex' processConsole.Read blocks process

Solution:Rewriting program (at binary level) so that it will use 'friendly substitutions' for certain methods while exploring the program's behavior53

RandomTestRun PexWrite substitution for Random.NextRun Pex

54

(With Christoph Csallner and Yannis Smaragdakis)

Plug-in (using Pex’ extensibility hooks)Idea

Discover likely program invariantsBy monitoring and abstracting conditions evaluated during actual program runs

ImplementationInstead of collecting "global" path condition,trace locally derived conditions (per class, per method)Summarize conditions over all invocations

(With Christoph Csallner and Yannis Smaragdakis)

Plug-in (using Pex’ extensibility hooks)Idea

Discover likely program invariantsBy monitoring and abstracting conditions evaluated during actual program runs

ImplementationInstead of collecting "global" path condition,trace locally derived conditions (per class, per method)Summarize conditions over all invocations

For each method invocation during the test run:1. Local path condition and final state discovery

• Build terms representing values derived from method parameters

2. Class invariant derivation• Basically, conjunction of all local path conditions of that

class's methods

3. Pre- and post-condition computation• Simplify local path conditions assuming derived class

invariants• Pre-condition is disjunction of path conditions• Post-condition is conjunction of path-specific post-

conditions• Path-specific post-condition is implication of path condition to

final state assignment

(… and some heuristics to abstract loops)

ExampleLook at test sourceRun pex from console

How to use pex.exeReport has additional information

Look atOnePathInvariantTest

58

Problem: How to introduce abstraction layer to environment, and make system unit-testable in isolation?A solution (“dependency injection”, “inversion of control” pattern):

Introduce interface that describes all environment interactionsMake implementation of interface configurable

59

Most credit card companies use a “check digit” encoding scheme

Check digit: Linear combination of digits modulo 10

Additionally, each credit card company has a distinct prefix and number length

Card Type Prefix Length

Mastercard 51-55 16

Visa 4 13, 16

Amex 34, 37 15

Diners Club 300-305, 36, 38 14

Specification of “Luhn Algorithm” that performs check of linear combination of digits modulo 10,in plain English:

1. Double the value of alternate digits of the primary account number beginning with the second digit from the right (the first right--hand digit is the check digit.)

2. Add the individual digits comprising the products obtained in Step 1 to each of the unaffected digits in the original number.

3. The total obtained in Step 2 must be a number ending in zero (30, 40, 50, etc.) for the account number to be validated.

Program model checkersJPF, Kiasan/KUnit (Java), XRT (.NET)

Combining random testing and constraint solving

DART (C), CUTE (C), EXE (C), jCUTE (Java),SAGE (X86)

Combining static analysis and constraint solving

Check ‘n’ Crash / DSD-Crasher (Java)

Commercial toolsAgitarOne, …

62

Parameterized Unit Testing:Separation of concerns

Specification vs. Coverage

Pex is a Dynamic Symbolic Execution platform for .NET

Generates test inputs automatically Can digest real-world codeYou learned how to use Pex for Parameterized Unit Testing

AvailabilityReleased within Microsoft since 1/1/2007.Send emailhttp://research.microsoft.com/Pex/

More information (internals, case study): Talk on Friday 63

http://research.microsoft.com/Pex/

64

http://research.microsoft.com/Pex

Documents

Parameterized Unit Testing