Systematic Software TestingUsing Test Abstractions

Slides by Darko Marinov

Presented at RIO Summer SchoolRio Cuarto, Argentina

February 2011

Examples of Structurally Complex Data

<library> <book year=2009> <title>T1</title> <author>A1</author> </book> <book year=2010> <title>T2</title> <author>A2</author> </book></library>

/library/book[@year<2010]/title

XML document

web sites

s1

s2

s4

s3

s50

1

3

2

red-black tree

Java program

class A { int f;}

class B extends A { void m() { super.f = 0; }}

Running Example: Inheritance Graphs

Javaprogram

interface I { void m();}

interface J { void m();}

class A implements I { void m() {}}

class B extends A implements I,J { void m() {}}

.

I J

A

B

Inheritancegraph

Properties

1. DAG

2. ValidJava

Testing Setup

Examples of code under test Compiler or refactoring engines, input: programs Abstract data type, input/output: data structure Web traversal code, input: web topology XML processing code, input/output: XML document

codetest

generationtest

oracle

0

1

3

2

0 3

2

0 3

2

0

3

pass

fail

inputs outputs

Test Generation

tool code

0

1

3

2

0 3

2

inputsFully automaticManual

0

1

3

2

0 3

2

inputs

tool

Semi automated

input setdescription

0

1

3

2

0 3

2

inputs

Test Abstractions

Each test abstraction describes a set of (structurally complex) test inputs Example: inheritance graphs with up to N nodes

Workflow Tester manually writes test abstractions Tool automatically generates test inputs

Benefits No need to manually write large number of test inputs Useful for test generation, maintenance, and reuse

Key Questions for Test Abstractions

tooltestabstraction

0

1

3

2

0 3

2

inputs

1. Which test inputsto generate?

2. What language to use for test abstractions?

3. How to generate test inputs from abstractions?

4. How to check test outputs?5. How to map failures to faults?

Which Inputs to Generate?

Bounded-exhaustive testing Generate all inputs within given (small) bounds Rationale: small-scope hypothesis [Jackson & Damon ISSTA’96]

Found bugs in both academia and industry Java compilers, model checkers… [Gligoric et al. ICSE’10]

Refactoring engines in Eclipse/NetBeans [Daniel et al. FSE’07]

Web traversal code from Google [Misailovic et al. FSE’07]

XPath compiler at Microsoft [Stobie ENTCS’05]

Fault-tree analyzer for NASA [Sullivan et al. ISSTA’04]

Constraint solver, network protocol [Khurshid & Marinov J-ASE’04]

How to Describe/Generate Inputs?

Filtering test abstractions [SOFTMC’01, ASE’01, FME’02, ISSTA’02, OOPSLA’02, SAT’03, MIT-TR’03, J-ASE’04, SAT’05, LDTA’06, ALLOY’06, ICSE-DEMO’07, STEP’07, FSE’07, ISSTA’08, ICST’09, CSTVA’10]

searchinput

s

filters

bounds

Testers describe what properties test inputs satisfy

Tool: search

Generating test abstractions [FSE’07, FASE’09]

execute

inputs

generators

bounds

Testers describe how to generate test inputs

Tool: execution

Previously Explored Separately

filters generators

Inheritancegraph

DAG X

ValidJava X

OtherJava

properties

property 3 X

… … …

property N X

Some properties easier/harder as filters/generators Key challenge for combining: search vs. execution

UDITA: Combined Both [ICSE’10]

filters generators

Inheritancegraph

DAG X

ValidJava X

OtherJava

properties

property 3 X

… … …

property N X

Extends Java with non-deterministic choices Found bugs in Eclipse/NetBeans/javac/JPF/UDITA

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Example: Inheritance Graphs

class IG {

Node[ ] nodes;

int size;

static class Node {

Node[ ] supertypes;

boolean isClass;

}

}

Properties DAG

Nodes in the graph should have no direct cycle

ValidJava Each class has at most one

superclass All supertypes of interfaces

are interfaces

Example Valid Inheritance Graphs

I J

A

B

G2

C

G1

G3

A

CB

G4

C

D

J

I J

A B C

G5

Example Invalid Inheritance Graphs

I

A

B

G2

C

G1

G3

A

CB

G4

C

J

I J

A B C

G5

J

cycle

D

class implements

class

class extends two

classes

interface extends

classclass

extends interface

(In)valid Inputs

Valid inputs = desired inputs Inputs on which tester wants to test code Code may produce a regular output or an exception E.g. for Java compiler

Interesting (il)legal Java programs No precondition, input can be any text

E.g. for abstract data types Encapsulated data structure representations Inputs need to satisfy invariants

Invalid inputs = undesired inputs

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Filtering Test Abstractions

Tool requires: Filters: encode what properties inputs satisfy Bounds

Tool provides: All inputs within bounds that satisfy the properties

searchinput

s

filters

bounds

Language for Filters

Each filter takes an input that can be valid or invalid Returns a boolean indicating validity

Experience from academia and industry showed that using a new language makes adoption hard

Write filters in standard implementation language (Java, C#, C++…) Advantages

Familiar language Existing development tools Filters may be already present in code

Challenge: generate inputs from imperative filters

Example Filter: DAG Property

boolean isDAG(IG ig) { Set<Node> visited = new HashSet<Node>(); Set<Node> path = new HashSet<Node>(); if (ig.nodes == null || ig.size != ig.nodes.length) return false; for (Node n : ig.nodes) if (!visited.contains(n)) if (!isAcyclic(n, path, visited)) return false; return true;}boolean isAcyclic(Node node, Set<Node> path, Set<Node> visited) { if (path.contains(node)) return false; path.add(node); visited.add(node); for (int i = 0; i < supertypes.length; i++) { Node s = supertypes[i]; for (int j = 0; j < i; j++) if (s == supertypes[j]) return false; if (!isAcyclic(s, path, visited)) return false; } path.remove(node); return true;}

Input Space

All possible object graphs with an IG root(obeying type declarations) Natural input spaces relatively easy to enumerate Sparse: # valid test inputs << # all object graphs

01

32

0 32

03

03

03

03

03

03

03

03 0

3

03

Bounded-Exhaustive Generation

Finite (small) bounds for input space Number of objects Values of fields

Generate all valid inputs within given bounds Eliminates systematic bias Finds all bugs detectable within bounds

Avoid equivalent (isomorphic) inputs Reduces the number of inputs Preserves capability to find all bugs

Example Bounds

Specify number of objects for each class

1 object for IG: { IG0 }

3 objects for Node: { N0, N1, N2 }

Specify set of values for each field For size: { 0, 1, 2, 3 }

For supertypes[i]: { null, N0, N1, N2 }

For isClass: { false, true }

Previously written with special library In UDITA: non-deterministic calls for initialization

Bounds Encoded in UDITA

IG initialize(int N) { IG ig = new IG(); ig.size = N; ObjectPool<Node> pool = new ObjectPool<Node>(N); ig.nodes = new Node[N]; for (int i = 0; i < N; i++) ig.nodes[i] = pool.getNew(); for (Node n : nodes) { int num = getInt(0, N − 1); n.supertypes = new Node[num]; for (int j = 0; j < num; j++) n.supertypes[j] = pool.getAny(); n.isClass = getBoolean(); }return ig; }

Java with a few extensions for non-determinism

static void mainFilt(int N) { IG ig = initialize(N); assume(isDAG(ig)); assume(isValidJava(ig)); println(ig); }

IG0 N0 N1 N2

size s[0] s[1] isC s[1] s[0] s[1]isCs[0] isC

N1 N1 null null null null3 2 31

Example Input Space

1 IG object, 3 Node objects: total 14 fields(“nodes” and array length not shown)

null

N0

N1

N2

null

N0

N1

N2

false

true

null

N0

N1

N2

null

N0

N1

N2

false

true

null

N0

N1

N2

null

N0

N1

N2

false

true

4 * (4 * 4 * 2 * 3)3 > 219 inputs, but < 25 valid

0

1

2

3

IG0 N0 N1 N2

size s[0] s[1] isC s[1] s[0] s[1]isCs[0] isC

Input Generation: Search

Naïve “solution” Enumerate entire (finite) input space, run filter on

each input, and generate input if filter returns true Infeasible for sparse input spaces (#valid << #total) Must reason about behavior of filters

Our previous work proposed a solution Dynamically monitors execution of filters Prunes large parts of input space for each execution Avoids isomorphic inputs

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Generating Test Abstractions

Tool provides: Generators: encode how to generate inputs Bounds

Tool requires: All inputs within bounds (that satisfy the properties)

execute

inputs

generators

bounds

Example Generator: DAG Property

void mainGen(int N) { IG ig = initialize(N); generateDAG(ig); generateValidJava(ig); println(ig); }

void generateDAG(IG ig) { for (int i = 0; i < ig.nodes.length; i++) { int num = getInt(0, i); ig.nodes[i].supertypes = new

Node[num]; for (int j = 0, k = −1; j < num; j++) { k = getInt(k + 1, i − (num − j)); ig.nodes[i].supertypes[j] = ig.nodes[k]; } }}

N0

N1

N2

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Filtering vs. Generating: DAG Property

boolean isDAG(IG ig) { Set<Node> visited = new HashSet<Node>(); Set<Node> path = new HashSet<Node>(); if (ig.nodes == null || ig.size != ig.nodes.length) return false; for (Node n : ig.nodes) if (!visited.contains(n)) if (!isAcyclic(n, path, visited)) return false; return true;}boolean isAcyclic(Node node, Set<Node> path, Set<Node> visited) { if (path.contains(node)) return false; path.add(node); visited.add(node); for (int i = 0; i < supertypes.length; i++) { Node s = supertypes[i]; for (int j = 0; j < i; j++) if (s == supertypes[j]) return false; if (!isAcyclic(s, path, visited)) return false; } path.remove(node); return true;}

void generateDAG(IG ig) { for (int i = 0; i < ig.nodes.length; i++) { int num = getInt(0, i); ig.nodes[i].supertypes = new Node[num]; for (int j = 0, k = −1; j < num; j++) { k = getInt(k + 1, i − (num − j)); ig.nodes[i].supertypes[j] = ig.nodes[k]; } }}

Filtering arguably harder than generating for DAG

20+ vs. 10 LOC

However, the opposite for ValidJava property

UDITA Requirement: Allow Mixing

filters generators

DAG X

ValidJava X

UDITAinput

sgenerators

bounds

filters

Other Requirements for UDITA

Language for test abstractions Ease of use: naturally encode properties Expressiveness: encode a wide range of properties Compositionality: from small to large test abstractions Familiarity: build on a popular language

Algorithms and tools for efficient generation of concrete tests from test abstractions

Key challenge: search (filtering) and execution (generating) are different paradigms

UDITA Solution

Based on a popular language (Java) extended with non-deterministic choices Base language allows writing filters for search/filtering

and generators for execution/generating Non-deterministic choices for bounds and enumeration Non-determinism for primitive values: getInt/Boolean New language abstraction for objects: ObjectPool

class ObjectPool<T> {

public ObjectPool<T>(int size, boolean includeNull) { ... }

public T getAny() { ... }

public T getNew() { ... } }

UDITA Implementation

Implemented in Java PathFinder (JPF) Explicit state model checker from NASA JVM with backtracking capabilities Has non-deterministic call: Verify.getInt(int lo, int hi)

Default/eager generation too slow Efficient generation by delayed choice execution

Publicly available JPF extensionhttp://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-delayed

Documentation on UDITA web page

http://mir.cs.illinois.edu/udita

Delayed Choice Execution

Postpone choice of values until needed

Lightweight symbolic execution Avoids problems with full blown symbolic execution Even combined symbolic and concrete execution has

problems, especially for complex object graphs

Eager

int x = getInt(0, N);…y = x;…… = y; // non-copy

Delayed

int x = Susp(0, N);…y = x;…force(y);… = y; // non-copy

non-determinism

Object Pools

Eager implementation of getNew/getAny by reduction to (eager) getInt Simply making getInt delayed does not work because

getNew is stateful

We developed a novel algorithm for delayed execution of object pools Gives equivalent results as eager implementation Polynomial-time algorithm to check satisfiability of

getNew/getAny constraints (disequality from a set) Previous work on symbolic execution typically encodes into

constraints whose satisfiability is NP-hard (disjunctions)

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Evaluation

UDITA Compared Delayed and Eager execution Compared with a previous Generating approach Compared with Pex (white-box)

Case studies on testing with test abstractions Some results with filtering test abstractions Some results with generating test abstractions Tested refactoring engines in Eclipse and NetBeans,

Java compilers, JPF, and UDITA

Eager vs. Delayed

.

Generating vs. UDITA: LOC

Generating vs. UDITA: Time

UDITA vs. Pex

Compared UDITA with Pex State-of-the-art symbolic execution

engine from MSR Uses a state-of-the-art constraint solver

Comparison on a set of data structures White-box testing: tool monitors code under test Finding seeded bugs

Result: object pools help Pex to find bugs Summer internship to include object size in Pex

Some Case Studies with Filtering

Filtering test abstractions used at Microsoft Enabled finding numerous bugs in tested apps

XML tools XPath compiler (10 code bugs, test-suite augmentation) Serialization (3 code bugs, changing spec)

Web-service protocols WS-Policy (13 code bugs, 6 problems in informal spec) WS-Routing (1 code bug, 20 problems in informal spec)

Others SSLStream MSN Authentication

Some Case Studies with Generating

Generating test abstractions used to test Eclipse and NetBeans refactoring engines [FSE’07]

Eight refactorings: target field, method, or class Wrote about 50 generators Reported 47 bugs

21 in Eclipse: 20 confirmed by developers 26 in NetBeans: 17 confirmed, 3 fixed, 5 duplicates,

1 won't fix Found more but did not report duplicate or fixed

Parts of that work included in NetBeans

Typical Testing Scenario

Create a model of test inputs (e.g., Java ASTs) Write filters/generators for valid inputs UDITA generates valid inputs Translate from model to actual inputs (pretty-

print) Run on two code bases, compare outputs

pass

fail

=?

onecode

anothercode

<library> <book year=2001> <title>T1</title> <author>A1</author> </book> <book year=2002> <title>T2</title> <author>A2</author> </book> <book year=2003> <title>T3</title> <author>A3</author> </book></library>

/library/book[@year<2003]/titl

<title>T1</title><title>T2</title>

<title>T1</title><title>T2</title>

UDITA

bounds

filters/generators

pretty

printer

Some More Bugs Found

Eclipse vs. NetBeans refactoring engines 2 bugs in Eclipse and 2 bugs in NetBeans

Sun javac compiler vs. Eclipse Java compiler 2 reports (still unconfirmed) for Sun javac

Java PathFinder vs. JVM 6 bugs in JPF (plus 5 more in an older version)

UDITA Delayed vs. UDITA Eager Applying tool on (parts of) itself 1 bug in UDITA (patched since then)

Example Bug

Generated programimport java.util.List;

class A implements B, D { public List m() { List l = null; A a = null; l.add(a.m()); return l; } }

interface D { public List m();}

interface B extends C { public List m();}

interface c { public List m();}

Incorrect refactoringimport java.util.List;

class A implements B, D { public List<List> m() { List<List<List>> l = null; A a = null; l.add(a.m()); return l; } }

interface D { public List<List> m();}

interface B extends C { public List<List> m();}

interface c { public List<List> m();}

Outline

Introduction Example Filtering Test Abstractions Generating Test Abstractions UDITA: Combined Filtering and Generating Evaluation Conclusions

Summary

Testing is important for software reliability Necessary step before (even after) deployment

Structurally complex data Increasingly used in modern systems Important challenge for software testing

Test abstractions Proposed model for complex test data Adopted in industry Used to find bugs in several real-world applications

Benefits & Costs of Test Abstractions

Benefits Test abstractions can find bugs in real code (e.g., at

Microsoft or for Eclipse/NetBeans/javac/JPF/UDITA) Arguably better than manual testing

Costs

1. Human time for writing test abstractions (filters or generators) [UDITA, ICSE’10]

2. Machine time for generating/executing/checking tests (human time waiting for the tool) [FASE’09]

3. Human time for inspecting failing tests [FASE’09]

Collaborators from U. of Illinois and other schools Brett Daniel Danny Dig Kely Garcia Vilas Jagannath Yun Young Lee

Funding NSF CCF-0746856, CNS-0615372, CNS-0613665 Microsoft gift

Credits for Generating TA and UDITA

Milos Gligoric (Belgrade->Illinois) Tihomir Gvero (Belgrade->EPFL) Sarfraz Khurshid (UT Austin) Viktor Kuncak (EPFL)

Test Abstractions

Test abstractions Proposed model for structurally complex test data Adopted in industry Used to find bugs in several real-world applications

(e.g., at Microsoft, Eclipse/NetBeans/javac/JPF/UDITA)

UDITA: latest approach for test abstractions Easier to write test abstractions Novel algorithm for faster generation Publicly available JPF extension

http://mir.cs.illinois.edu/udita