Tutorial / lab 2: Code instrumentation

© Dr. A. Williams, Fall 2008 - Present

Software Quality Assurance – Clover Lab 1

Tutorial / lab 2: Code instrumentation

• Goals of this session:

1. Create some tests for a Java class.

2. Measure the code coverage.



Code Instrumentation Tools

• Tools that modify production code for various purposes during code execution:

– Debuggers– Add information to allow for break points

or step-by-step execution.

– Coverage measurement– Report on the number of times various

code elements have been executed.

– Profilers– Measure amount of time spent in various

parts of the code.



Coverage tools

• Program is typically compiled with special options, to add extra source or object code.

– Additional data structures, such as a flow graph, may also be created.

• Program is run, possibly via test cases

– During execution, information is accumulated and written to an output file.

• Post-processing phase:

– User report is generated from output file.



How to measure coverage?

• Instrument the source code before compilation

• Instrument the virtual machine or object code.

– The Apache Byte Code Engineering Library (BCEL) is used by several Java coverage tools

– http://jakarta.apache.org/bcel

• Use a customized virtual machine.



Source code instrumentation

• Create a copy of the source code, instrument the copy, and compile the instrumented copy to .class files.

– This is the approach used by the Clover code coverage tool.

• Typical method:

– Set up an array of counters.

– Insert source code at points of interest that increment specific counters.

– Coverage tool calls main() method

– After main method returns, dump values of counters.



Source code before instrumentation

public final void add( final MatrixRow<T> newValues ){ if ( getNumColumns( ) == 0 ) { contents = newValues.copy( ).getContents( ); } else { contents.addAll( newValues.copy( ).getContents( ) ); }}

• A counter is inserted for each method, statement, and branch.

• Statement counters are before each statement, in case the statement throws an exception.

• Branch counters need to identify true and false branches taken.



Same code after instrumentation

public final void add( final MatrixRow<T> newValues ){ CLOVER.M[348]++; CLOVER.S[1814]++; if ( ( getNumColumns( )==0 && ++CLOVER.CT[338]!=0 | true )

|| ( ++CLOVER.CF[338] == 0 & false ) ) { CLOVER.S[1815]++; contents = newValues.copy( ).getContents( ); } else { CLOVER.S[1816]++; contents.addAll( newValues.copy( ).getContents( ) ); }}

methodcounter

statementcounters

branchcounters



Counting branches

• Count number of times that a branch condition evaluates to true and false.

if ( ( aBoolean && ++CT[338]!=0 | true )

|| ( ++CF[338] == 0 & false ) )

• If aBoolean is true, then the && is also evaluated, which will increment the CT counter. We want the expression’s overall evaluation to be true.

• If aBoolean is false, then execution switches to the right side of the || clause. Increment the CF counter, but then ensure that the entire expression still evaluates to false to take the else branch.

• Short-circuit evaluation means that as soon as the expression result can be determined, evaluation stops. This avoids:– Incrementing the true counter if aBoolean is false.– Incrementing the false counter if aBoolean is true.



Analysis

• Advantages:– Results are directly mapped to source code.– If a statement throws an exception, it will be counted

as having been executed.

• Disadvantages:– Requires keeping a second version of source code,

and ensuring that original source is not over-written.– Instrumented code must not affect original code

execution.– Management of .class files from compiling alternate

source code. – Memory usage: for statement coverage alone, every

statement requires one integer as a counter.



Byte code instrumentation

• Can be done in two ways:

– Instrument object files in advance to produce a alternate version.

– Example: create instrumented .class files for Java.

– Instrument object files as they are used.

– Example: create customized .class file loader that inserts the instrumentation at run time.

• This is the approach used by the Emma code coverage tool

– This is an open-source tool; see http://emma.sourceforge.net

– Eclipse plug-in version: http://www.eclemma.org



Analysis

• Pre-instrumentation:

– Advantages:

– source code is not modified.

– can be done as a batch process before execution

– Disadvantages:

– management of two .class file versions.

– mapping of source code statements to/from object code

– counting source code statements can be difficult

– object code may need extra information such as special compiler options

Documents

Tutorial / lab 2: Code instrumentation