103
Parameterized Unit Testing ICTSS’11 Tutorial Nikolai Tillmann Microsoft Research

Parameterized Unit Testing

  • Upload
    sanjiv

  • View
    64

  • Download
    0

Embed Size (px)

DESCRIPTION

ICTSS’11 Tutorial. Parameterized Unit Testing. Nikolai Tillmann Microsoft Research. Outline. First part: 14:00 - 15:00pm Second part: 15:30 - 17:00pm. Outline. Unit Testing Isolation Parameterized Unit Testing Data Generation by Dynamic Symbolic Execution Patterns - PowerPoint PPT Presentation

Citation preview

Page 1: Parameterized Unit  Testing

Parameterized Unit Testing

ICTSS’11 Tutorial

Nikolai TillmannMicrosoft Research

Page 2: Parameterized Unit  Testing

Outline

First part: 14:00 - 15:00pm Second part: 15:30 - 17:00pm

Page 3: Parameterized Unit  Testing

Outline

Unit TestingIsolationParameterized Unit Testing

Data Generation byDynamic Symbolic Execution

PatternsLimitations and other Details

Page 4: Parameterized Unit  Testing

About the exercises…

Interactive: http://pexforfun.com

Demos: Pex for Visual Studiohttp://research.microsoft.com/pex Requires Windows, .NET 2.0 or 4.0, ideally Visual Studio 2008 / 2010

Page 5: Parameterized Unit  Testing

Unit Testing

Page 6: Parameterized Unit  Testing

Quiz: Unit testing What is a unit test?

Page 7: Parameterized Unit  Testing

Unit Testing

void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count);}

A unit test is a small program with assertions

Test a single (small) unit of code

Let’s play a game!

Page 8: Parameterized Unit  Testing

The Code Under Test

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}

t:\myapp.iniA=BFoo=CC=D

Page 9: Parameterized Unit  Testing

Quiz: Coverage

How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not enough

Page 10: Parameterized Unit  Testing

Quiz: Coverage

How much block coverage do we need?1. 50%2. 80%3. 100%4. Block coverage alone is not

enough

Page 11: Parameterized Unit  Testing

Quiz: Coverage

Do we need more tests to get 100% cov.?

[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue();}

[TestMethod]void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue();}

Page 12: Parameterized Unit  Testing

Quiz: Assertions

Why write Assertions (or not)?1. Documentation2. Double check your program3. Please your manager4. Prevent future bugs5. Validate user inputs6. Catch errors early

Page 13: Parameterized Unit  Testing

Quiz: Assertions

Why write Assertions (or not)?1. Documentation2. Double check your program3. Please your manager4. Prevent future bugs5. Validate user inputs6. Catch errors early

Page 14: Parameterized Unit  Testing

Quiz: Assertions

Which Assertions should you write?1. Assert.IsTrue(value == “b”);2. Assert.IsTrue(value == null);3. Assert.IsTrue(String.IsNullOrEmpty(value)

)4. Assert.IsTrue(false);5. No assertions

[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

Page 15: Parameterized Unit  Testing

Quiz: Assertions

Which Assertions should you write?1. Assert.IsTrue(value == “b”);2. Assert.IsTrue(value == null);3. Assert.IsTrue(String.IsNullOrEmpty(value)

)4. Assert.IsTrue(false);5. No assertions

[TestMethod]void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????);}

Page 16: Parameterized Unit  Testing

Quiz: Coverage + Assertions What gives you confidence in the

code?1. High coverage, no assertions2. Low coverage, many assertions3. High coverage, many assertions4. Low coverage, no assertions5. I wrote it

Page 17: Parameterized Unit  Testing

Quiz: Coverage + Assertions What gives you confidence in the

code?1. High coverage, few assertions2. Low coverage, many assertions3. High coverage, many assertions4. Low coverage, no assertions5. I wrote it

Page 18: Parameterized Unit  Testing

The Code Under Test

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}

Page 19: Parameterized Unit  Testing

Quiz: Isolation

In the example, what are the external dependencies?1. Network Share2. Local Disk3. No file system, all in memory

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...}

Page 20: Parameterized Unit  Testing

Quiz: Isolation

In the example, what are the external dependencies?1. Network Share2. Local Disk3. No file system, all in memory

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); ...}

Page 21: Parameterized Unit  Testing

Quiz: Isolation

What is the problem with a Local Disk?1. Mapping already exists2. Cannot run tests concurrently3. Disk full4. Access rights

Page 22: Parameterized Unit  Testing

Quiz: Isolation

What is the problem with a Local Disk?1. Mapping already exists2. Cannot run tests concurrently3. Disk full4. Access rights

Page 23: Parameterized Unit  Testing

Quiz: Isolation

How do you mitigate the Local Disk issues?1. Always run on the same machine, same

hardware, same credentials, same time, same temperature, same solar system configuration

2. Refactoring: use Streams3. Refactoring: introduce IFileSystem4. Refactoring: pass the lines as parameter5. Change implementation of File.ReadAllLines

Page 24: Parameterized Unit  Testing

Quiz: Isolation How do you mitigate the Local Disk issues?

1. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration

2. Refactoring: use Streams3. Refactoring: introduce IFileSystem4. Refactoring: pass the lines as parameter5. Change implementation of

File.ReadAllLines

Page 25: Parameterized Unit  Testing

Quiz: Definition of Unit Test What is a Unit Test?

A Unit Test is a program that runs fast the code under test, without environment dependencies, with assertions.

What is a Unit Test Suite? A set of Unit Tests which achieves high

code coverage

Page 26: Parameterized Unit  Testing

Isolation

Page 27: Parameterized Unit  Testing

How do you change File.ReadAllLines?1. Override static method2. Changing the .NET Framework source

code(and recompiling it)

3. Rewrite application in JavaScript4. Code instrumentation

Dependency hellvar lines = File.ReadAllLines(@"t:\v.ini");

Page 28: Parameterized Unit  Testing

Replace any .NET methodwith a delegatevar lines = File.ReadAllLines(@"t:\myapp.ini");

File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn){ return new string[0];};

What if we could replace File.ReadAllLines?

Moles

Page 29: Parameterized Unit  Testing

Motivation for Moles

Why another isolation framework? Specifically designed to enable Pex

Simple, well-defined semantics▪ “Replace any .NET method”

Page 30: Parameterized Unit  Testing

Mole Types Code Generation

// System.IOpublic static class File { public static string[] ReadAllLines(string fn);}

// System.IO.Molespublic class MFile { public static Func<string, string[]> ReadAllLinesString;}// delegate R Func<T, R>(T t);

Page 31: Parameterized Unit  Testing

Injecting Detours at Runtime

// System.IOpublic static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code }}

Automatically injected

at runtime

Page 32: Parameterized Unit  Testing

MolesDemo

Start from ReadFooValue problem Create test project Add moles for mscorlib Write test using moles Run test Debug test

You can pick up project fromReadFooValueIsolation\ReadFooValueIsolation.sln

Page 33: Parameterized Unit  Testing

Quiz: Moles Usage When should you use Moles (and not)?

1. Always use Moles to solve isolation issues2. With Moles, one does not need to use interfaces

anymore3. Moles only should be used for 3rd party API, use

interfaces for isolation in your APIs4. Moles can be used in production code5. Moles lets you get away with untestable APIs6. Moles make test cases more robust7. With Moles, you do not need integration tests anymore8. Moles make tests easier to understand9. Moles is for poor programmers, real programmers rely

on interfaces

Page 34: Parameterized Unit  Testing

Quiz: Moles Usage When should you use Moles (and not)?

1. Always use Moles to solve isolation issues2. With Moles, one does not need to use interfaces

anymore3. Moles only should be used for 3rd party API, use

interfaces for isolation in your APIs4. Moles can be used in production code5. Moles lets you get away with untestable APIs6. Moles make test cases more robust7. With Moles, you do not need integration tests anymore8. Moles make tests easier to understand9. Moles is for poor programmers, real programmers rely

on interfaces

Page 35: Parameterized Unit  Testing

Exercise (Optional / homework) Step-by-step tutorial: research.microsoft.com/documentati

on.aspx “Unit Testing with Microsoft Moles” “Unit Testing SharePoint Foundation

with Microsoft Pex and Moles”

Page 36: Parameterized Unit  Testing

What you learned so far

The Definition of Unit Testing

Unit Test Isolation

1. Identify External Dependencies2. Replace each External API with your

delegate

Page 37: Parameterized Unit  Testing

Parameterized Unit Testing

Page 38: Parameterized Unit  Testing

The Recipe of Unit Testing

var list = new List(); list.Add(item); var count = list.Count;

Assert.AreEqual(1, count);}

Three essential ingredients: Data Method Sequence Assertionsvoid AddAndCount() { int item = 3;

Page 39: Parameterized Unit  Testing

Quiz: list.Add(???)

Which value matters?1. 02. 13. int.MaxValue, int.MinValue4. -15. 10006. it does not matter7. I don’t know until I read the code

list.Add(???);

Page 40: Parameterized Unit  Testing

Parameterized Unit Test = Unit Test with Parameters

Separation of concerns Specification of behavior Data to achieve coverage

Parameterized Unit Testingvoid AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

for any list,

for any item,

… adding 1 item increments Count

by 1

Page 41: Parameterized Unit  Testing

Parameterized Unit Tests areAlgebraic Specifications A Parameterized Unit Test can be read

as a universally quantified, conditional axiom.void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);}forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)

Page 42: Parameterized Unit  Testing

Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly

supported by various test frameworks .NET: Supported by .NET test frameworks

http://www.mbunit.com/ http://www.nunit.org/ …

Java: Supported by JUnit 4.X http://www.junit.org/

Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex

http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne

http://www.agitar.com/

Page 43: Parameterized Unit  Testing

Problem

We cannot execute parameterized unit tests without data.

Where does the data come from?

Random data generator Real customer data Ranges Some values hand-picked by dev/tester

Page 44: Parameterized Unit  Testing

Data Generationby Dynamic Symbolic Execution

Page 45: Parameterized Unit  Testing

45

Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible

How would you do it?

Data Generation Challenge

Page 46: Parameterized Unit  Testing

Combines ideas from symbolic execution

to capture behavior symbolically model checking

to perform systematic bounded state space exploration

testing to monitor and check concrete behavior

constraint solving to find inputs

Dynamic Symbolic ExecutionCombines many ideas

Page 47: Parameterized Unit  Testing

47

Combines concrete and symbolic execution

Algorithm:Set J := ∅ (J is set of already analyzed

program inputs)Loop

Choose program input i ∉ J (stop if no such i can be found)Output iExecute P(i); record path condition C (in particular, C(i) holds)Set J := J ∪ C (viewing C as the set { i | C(i ) } )

End loop

Loop does not terminate if number of execution paths is infinite (in the presence of loops/recursion)

Dynamic Symbolic ExecutionThe high-level algorithm• This choice decides search

order• Search order decides how quick we can achieve high code coverage!• Incomplete constraint-solver leads to under-approximation

Page 48: Parameterized Unit  Testing

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Datanull

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolveChoose next path

Done: There is no path left.

Dynamic Symbolic ExecutionExample

Page 49: Parameterized Unit  Testing

http://pex4fun.com/CoverMe

Page 50: Parameterized Unit  Testing
Page 51: Parameterized Unit  Testing

Dynamic Symbolic Execution Exercises ArrayIndexLength

Pex knows about all implicit, exception-throwing control-flow branches

ArrayHeapPex models the heap

Assert, Assert123Assertions connect code coverage and correctnesshttp://pex4fun.com/DynamicSymbolicExecutionExercises

Page 52: Parameterized Unit  Testing

Assertions vs. Coverage

What about finding faults?void AssertIsTrue(bool condition) { if (!condition) throw new AssertionException();}

Assertions induce branches Pex tries to cover branches▪ Pex tries to fail assertions -> finds

bug!

Page 53: Parameterized Unit  Testing

Create new project in Visual Studio. Insert CoverMe method below. Right-click on CoverMe method, and select “Run Pex”. Inspect results in table. Save tests, …, have fun!public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Dynamic Symbolic Executionwith Pex in Visual Studio

Page 54: Parameterized Unit  Testing

Dynamic Symbolic Executionwith Pex in Visual Studio

Page 55: Parameterized Unit  Testing

Dynamic Symbolic Executionwith Pex in Visual Studio

Generated Test Inputs

are persisted

as C# Unit Tests

Page 56: Parameterized Unit  Testing

Exercise (Optional / homework) Step-by-step tutorial: research.microsoft.com/documentati

on.aspx “Exploring Code with Microsoft Pex”

Page 57: Parameterized Unit  Testing

How to test this code?(Actual code from .NET base class libraries)

Motivation: Unit Testing HellResourceReader

Page 58: Parameterized Unit  Testing

Motivation: Unit Testing HellResourceReader

Page 59: Parameterized Unit  Testing

DemoResourceReader

[PexClass, TestClass][PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(typeof(BadImageFormatException))][PexAllowedException(typeof(IOException))][PexAllowedException(typeof(NotSupportedException))]public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}

Page 60: Parameterized Unit  Testing

Defined by execution environment / programming language, symbolic execution precision, and constraint solving Execution environment: C, Java, x86, .NET,… Precision: linear vs. non-linear arithmetic, “gods integers” vs.

bitvectors, concrete heap vs. symbolic heap., floating-point values, etc.

Solvers: lp_solve, CVCLite, STP, Disolver, Z3,… Examples of DSE implementations:

DART (Bell Labs), and also CUTE “concolic execution” EXE/EGT/KLEE (Stanford) “constraint-based execution” Vigilante (Microsoft) to generate worm filters BitScope (CMU/Berkeley) for malware analysis Sage (Microsoft) for security testing of X86 code Yogi (Microsoft) to verify device drivers (integrated in SLAM) Pex (Microsoft) for parameterized unit testing of .NET code CREST, jCUTE, jFuzz, …

Dynamic Symbolic ExecutionMany implementations

Page 61: Parameterized Unit  Testing

Constraint Solvingvs. Search Strategies

TestInputs

Constraint System Execution

Path

KnownPaths

Initially, choose ArbitraryPath-constraint

solving is just hard.

Reachability is

undecidable! (Loops)

Page 62: Parameterized Unit  Testing

Representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, …Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references to

values" Elements of arrays, memory accessed through unsafe pointers

by mutable “mapping of integers to values"Efficiency by Many reduction rules, including reduction of ground terms to

constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical

operations Patricia Trees to represent AC1 operators

(including parallel array updates)

Symbolic State Representation

Page 63: Parameterized Unit  Testing

SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to

theories SAT solver for Boolean structure Decision procedures for relevant theories:

uninterpreted functions with equalities,linear integer arithmetic, bitvector arithmetic, arrays, tuples

Model generation for satisfiable formulas Models used as test inputs

Limitations No decision procedure for floating point arithmetic

Pex uses Z3: http://research.microsoft.com/z3

Constraint Solving

Page 64: Parameterized Unit  Testing

)1()2),3,,(((2 xyfyxawritereadfyx

ArithmeticArray TheoryUninterpreted Functions

Satisfiability Modulo Theories

void Foo(int x, int y, int[] a) { if (x + 2 == y) { a[x] = 3; Debug.Assert(f(a[y - 2]) == f(y – x + 1)); }}

Page 65: Parameterized Unit  Testing

Pex uses some heuristic approaches for domain-specific problems.

Strings [TACAS’09, ICST’10] “uninterpreted functions” for string operations two-phase solving: first solve integer constraints

involving indices and lengths, then character constraints

Floating-point arithmetic Search-based approach: minimization of fitness

function … (more to come)

Domain-specific Constraint Solving

Page 66: Parameterized Unit  Testing

WritingParameterized Unit Tests

Page 67: Parameterized Unit  Testing

string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } } return null;}

ExerciseWrite PUT for ReadFooValue

Can you come up with Parameterized Unit Tests?

Page 68: Parameterized Unit  Testing

Create new solution with C# class library Write code for ReadFooValue Create Test Project Add moles for mscorlib Write parameterized unit tests.

Or get result from ReadFooValuePUT\ReadFooValuePUT.sln

ExerciseWrite PUT for ReadFooValue

[PexMethod] // attribute marks pex testspublic void Test(...) { ...}

Page 69: Parameterized Unit  Testing

ExerciseCrash Test

[PexMethod]void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue();}

Execute the code without assertions

Page 70: Parameterized Unit  Testing

ExerciseAssert Observed Results

[PexMethod]public string Regression(string[] lines) { MFile.ReadAllLines = () => lines; return Reader.ReadFooValue();}

Manually review outputs Pex inserts Assertions automatically

Page 71: Parameterized Unit  Testing

Exercise

[PexMethod]void FooExist(string value1) { PexAssume.IsTrue(value1 != null); var lines = new string[]{“Foo=“ + value1}; MFile.ReadAllLines = () => lines; var value2 = Reader.ReadFooValue();

PexAssert.AreEqual(value1, value2);}

Write and read property

Page 72: Parameterized Unit  Testing

Test Project

Test Generation Work-Flow

Code-Under-Test

Project

Parameterized Unit Tests Pex Generated

Tests

Page 73: Parameterized Unit  Testing

Patterns

Page 74: Parameterized Unit  Testing

Patternshow to create objects

Page 75: Parameterized Unit  Testing

[PexMethod]public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}

What are the test inputs? Only null An array list created by calling a constructor ... and also calling Add(…) many times … and also calling Clear() many times …

Quiz:Complex Objects As Inputs

Page 76: Parameterized Unit  Testing

[PexMethod]public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}

Other possible problems: What if… ArrayList has no public constructor ArrayList is an abstract class (or

interface)

Quiz:Complex Objects As Inputs

Page 77: Parameterized Unit  Testing

Problem: Test parameters may be classes Symbolic execution tracks constraints over

(private) fields of objects Constraint solver determines solution,

assigning concrete values to (private) fields However, in practice objects must be

initialized by constructor, and can be mutated only by calling certain methods

What sequence of method calls reaches a desired field assignment? There may be no such sequence In general, undecidable

Creating complex objectsThe Problem

Page 78: Parameterized Unit  Testing

Factory methods Parameterized factories create and configure

objects Explicit: Public static method annotated with

[PexFactoryMethod] Implicit: Pex guesses “best” constructor/setters

Result: Exploration of reachable states Reachable using factory method Reachable within configured bounds Under-approximation of possible states

Object creation:Factory methods

Page 79: Parameterized Unit  Testing

Hand-written factory method:[PexFactoryMethod(ArrayList)]public static ArrayList Create( int capacity, object[] values) { var a = new ArrayList(capacity); foreach (value in values) a.Add(value); return a;}

Pex will explore both factory method and PUT!

Factory methodsArrayList

Page 80: Parameterized Unit  Testing

Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit

Inspect generated test cases Notice constructor implicitly used by Pex Save factory method Edit factory method Explore again

Demo:Factory methods

Page 81: Parameterized Unit  Testing

Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code

Write special constructor/factory method for testing only Sets fields, and assumes invariant

Result: Exploration of feasible states May include states that are not reachable

AlternativeObject creation via class invariants

Page 82: Parameterized Unit  Testing

Patternsfor Parameterized Unit Tests

Page 83: Parameterized Unit  Testing

Pattern4A Assume, Arrange, Act, Assert[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act

// assert quiz1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }

Page 84: Parameterized Unit  Testing

Pattern4A Assume, Arrange, Act, Assert[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act

// assert quiz1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }

Page 85: Parameterized Unit  Testing

PatternRegression Tests Generated test asserts any observed

value Return value, out parameters

When code evolves, breaking changes in observable will be discovered

int AddTest(int a, int b) { return a + b; }

void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result);}

Page 86: Parameterized Unit  Testing

PatternRoundtrip For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s);

// assert quiz1. Assert.AreEqual(value, s);2. Assert.AreEqual(s, parsed);3. Assert.AreEqual(parsed, value);}

Page 87: Parameterized Unit  Testing

PatternRoundtrip For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s);

// assert quiz1. Assert.AreEqual(value, s);2. Assert.AreEqual(s, parsed);3. Assert.AreEqual(parsed, value);}

Page 88: Parameterized Unit  Testing

PatternNormalized Roundtrip For an API f(x), f-1(f(f-1(x)) = f-1(x) for

all xvoid ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);

// assert quiz1. Assert(x == intermediate);2. Assert(intermediate == roundtripped);3. Assert(normalized == roundtripped);4. Assert(x == roundtripped); }

Page 89: Parameterized Unit  Testing

PatternNormalized Roundtrip For an API f(x), f-1(f(f-1(x)) = f-1(x) for

all xvoid ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);

// assert quiz1. Assert(x == intermediate);2. Assert(intermediate == roundtripped);3. Assert(normalized == roundtripped);4. Assert(x == roundtripped); }

Page 90: Parameterized Unit  Testing

PatternReachability Indicate which portions of a PUT should be

reachable.[PexAssertReachEventually]public void Constructor(object input){ new Foo(input); PexAssert.ReachEventually(); }

Page 91: Parameterized Unit  Testing

PatternSame Observable Behavior Given two methods f(x) and g(x), and a

method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.

public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right));}

Page 92: Parameterized Unit  Testing

PatternCommutative Diagram Given two implementations f and g of the

same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.

string Multiply(string x, string y);int Multiply(int x, int y);void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2);}

Page 93: Parameterized Unit  Testing

HomeworkPatterns Read patterns paper: research.microsoft.com/Pex

/documentation.aspx “Parameterized Test Patterns for Microsoft

Pex”

Page 94: Parameterized Unit  Testing

LimitationsIt’s called Parameterized Unit Testing

Page 95: Parameterized Unit  Testing

Pex understand managed .NET code only Pex does not understand native code.

Problem if branching over values obtained from the environment Pex may not automatically detect all such

cases.

Testability

if (!File.Exists(f)) throw ...

File System?

Page 96: Parameterized Unit  Testing

Hidden Complexity

Pex analyzes every executed .NET instruction

Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different

representationsvoid Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }

Don’t do this.

Page 97: Parameterized Unit  Testing

Exploration Boundaries

Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions▪ Number of conditions that depend on test

inputs MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit

Page 98: Parameterized Unit  Testing

Multi-threaded code

Unlike test inputs, thread-interleavings can normally not be controlled

Thus, Pex can only explore single-threaded code

Related approach to explore thread-schedules (but not input parameters) by controlling thread-scheduler: CHESShttp://research.microsoft.com/CHESS

Page 99: Parameterized Unit  Testing

Lack of Test Oracle

Unit Testing != Test Generation

Page 100: Parameterized Unit  Testing

Wrapping up

Page 101: Parameterized Unit  Testing

ConclusionWhat you learned today The Definition of Unit Testing

Unit Tests are not Integration Tests Unit Test Isolation

Introduction of Abstraction LayersParameterized Unit Testing

Separation of concerns: Coverage/Specification

Patterns to write Parameterized Unit Tests Data Generation by Dynamic Symbolic

Execution

Page 102: Parameterized Unit  Testing

Availability

Pex and Moles areVisual Studio 2010 Power Tools

(academic/commercial license available) Minimum: Windows XP, .NET 2.0 Ideally: Visual Studio 2010 Professional http://research.microsoft.com/pex/

http://pexforfun.com

Page 103: Parameterized Unit  Testing

Thank you

http://research.microsoft.com/pexhttp://sites.google.com/site/asergrp/