B. Childers, M. L. Soffa, J. Beaver, L. Ber, K. Cammarata, J. Litman, J. Misurda

Presented by:

Priyanka Puri

1000676053

SOFTTEST: A FRAMEWORK FOR SOFTWARE TESTING OF JAVA PROGRAMS

STRUCTURAL SOFTWARE TESTING

• Assure quality, robust software

• Collect coverage information about the program

• E.g., branch coverage records which edges are hit

• Uses program instrumentation

..Continued..

• Automatically apply multiple test strategies

• Multiple languages and platforms

• Handle large programs

Static instrumentation approaches

• Not scalable: Instrumentation left in place• Time, memory

• Inflexible: Only certain tests, languages, platforms

• Generally not detailed

Approach

• Specification driven: User written test

• Test plans: Recipe of how & where to test

• Path specific: Instrument only what is needed

• Dynamic: Insert & remove instrumentation

TEST FRAMEWORK INSTANCE FOR JAVAWizard for creating the test specification

From test specification determine how to testTest plan uses instrumentation primitives

Source level coverage results

• Apply different tests to various code regions.

• may be applied with different coverage criteria

• A GUI for

• What tests to apply, where to apply, under what conditions

• Select code regions

• Conditions can be specified

TEST SPECIFICATION

public class Example

{

public int foo(int x)

{

if(x==200)

x = x+100;

else x = x-100;

return x;

}

}

DEFINITIONS

{

NAME: X, REGION_D,

LOCATION: FILE Example.java {

CLASS example, METHOD foo

}

}

BODY

{

DO BRANCH_TEST

ON REGION X UNTIL:90%

}

Example test specification

TEST PLANNER• Test plan: Where and how to test program

• Test table

• Where to instrument in the target machine code

• Test information (results)

• Branch coverage: when to insert & remove instrumentation, records which edges are hit/not hit

• Test payload

– What to do at an instrumentation point

– Target machine code

– Branch coverage: Update edge coverage

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 N,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 N,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 N,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 N,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,N

2 4 N

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,N

2 4 Y

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,N

2 4 Y

3 4 N

4 1 N

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,N

2 4 Y

3 4 N

4 1 Y

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,Y

2 4 Y

3 4 N

4 1 Y

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,Y

2 4 Y

3 4 N

4 1 Y

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,Y

2 4 Y

3 4 Y

4 1 Y

Instrumentation

Hit

BRANCH COVERAGE EXAMPLE

Test Table

Test Payload

Mark edge hit

Insert at next point

Remove instrumentation

1

2 3

4

Block Next Hit

1 2,3 Y,Y

2 4 Y

3 4 Y

4 1 Y

Instrumentation

Hit

Example CFG with block as its own successor

TEST VIRTUAL MACHINE• Provides functionality to insert and remove

instrumentation at run-time.

• Implements an interface for inserting and removing instrumentation with fast breakpoints*.

• API provides primitives, such as

• the placement of successor breakpoints, • storing test- specific data• removal of breakpoints, for constructing fast

breakpoints with varying payloads

TEST ANALYZER

• Displays the results of tests

• For branch coverage, the analyzer displays the CFG for a method

• CFG for target machine code

Observations• SoftTest significantly reduces the number of tests resulting in savings in

scripting effort.

• Significant impact on the acceptability of code coverage.

• Provides cost effective approach to testing.

• Low performance and memory overhead.

• For the benchmarks, the memory overhead was 178 to 822 bytes for the test table.

• The performance overhead for coverage varied from 1% to 5.5% of the total execution time.

Preliminary Results: Branch Coverage Testing

Overhead includes:

1. Test Planner

2. Instrumentation

Run-time performance overhead

[3]

REFERENCES

[1] http://veryoldwww.cs.pitt.edu/copa/eclipse.html

[2] SoftTest: A Framework for Software Testing of Java Programs

[3] http://www.cs.pitt.edu/coco/papers/SoftTest-Etx-2003

[4] http://www.softwaretestingmentor.com/types-of-testing/static-testing.php

QUESTIONS?