Computer Science 2212a/b - UWO1 Structural Testing Motivation The gcov Tool An example using gcov How does gcov do it gcov subtleties Further structural

Computer Science 2212a/b - UWO 1

Structural Testing

Motivation

The gcov Tool

• An example using gcov

• How does gcov do it

• gcov subtleties

Further structural testing measures

• Flowcharts

• Structural testing and flowcharts

• Minimal test suites

• Path coverage

• Testing loops

Time to cover: 25-35 minutes


Structural Testing: Motivation

Consider a typical programming project:

• We have designed code based on requirements

• We have also built in some additional features

• Additional features interact with required features in various ways

• We need to test code of additional features

• We don’t remember what all of them are

• Because new features are not mentioned in requirements, testing based on requirements will not help us

Consider a function to sort an array of integers

• Requirements state only that resulting array must preserve original elements and be sorted

• Many different algorithms exist for doing this

• A test suite which tests one algorithm thoroughly may not test another


Structural Testing: Motivation continued

In both these cases, the essential problem is: How do we know we have tested all of our code?

• For all we know, some code may never have been executed by any test that we have run

• User may do something that executes this untested code • It will therefore be run for the first time by the user • Code run for the first time often has bugs!

Have we tested all of our code? • Tools exist to help us answer this question

These tools generally called “code coverage tools” • Help us to see what percentage of code has been “covered” by some

test case • These tools usually targeted to individual programming languages

Good free code coverage tools exist for C • Seems only commercial tools exist for Java • We will look at one for C


gcov: A Structural Testing Tool

gcov: GNU Project coverage utility • Based on earlier utility tcov • GNU Project:

—Supplies free software —Created gcc compiler for C —Most Linux utility programs come from GNU Project

To use gcov (overview): • Compile your C program using gcc with special switches • Run your program normally • While running, your program will write information to special “log

files”• After running test cases, run gcov • gcov will generate a coverage report listing which lines have been

executed We will look at each of these steps in more detail


Normal Compilation of a Program

In the above diagram, • Solid lines = inputs • Dashed lines = outputs

Normally, we just: • Compile a program from source code • Execute it on some selected test cases • Look at the output for correctness


Compilation Using gcov

• Compilation / execution with gcov is basically the same • However, extra switches to gcc cause extra things to happen • Switches: fprofile arcs ftest coverage


Compilation Using gcov continued

When we compile with the extra switches, gcc does the following

extra things:

• Generates “map file” of the blocks of code in our source files

• Generates object code which counts the number of times each block of code has been executed

“Block of code”:

• Any sequence of statements such that executing one of them guarantees we must execute the next one

Example (program wordcount.c):

• Instead of just “gcc wordcount.c”, we say gcc fprofile arcs ftest coverage wordcount.c

• gcc generates map file wordcount.gcno

• Exact format of the “map file” is not important to us


Running a gcov-Compiled Program

Run program as normal on test cases

First time program is run, a coverage data file appears (with extension .gcda)

• Coverage data file contains information about how many times each line of code has been executed

Example:

• After running wordcount, file wordcount.gcda is created

Coverage data is updated every subsequent time program is run

Coverage data is cumulative; e.g.

• Line 42 executed 5 times on first run;

• Line 42 executed 10 times on second run;

• Therefore, coverage data stored in .gcda file shows 15 executions of line 42


Getting Information from gcov

Run gcov with source file name as argument (e.g. gcov wordcount.c)

gcov writes some statistics to screen, creates a file with extension “.gcov” (e.g. wordcount.c.gcov)

.gcov file contains:

• On right: source code

• On left: number of times each line has been executed

• If lines have never been executed, .gcov file shows ##### to highlight it


Example wordcountProgram count the number of lines, words, chars in input file

Original Source file:

#include <stdio.h> #define TRUE 1 #define FALSE 0 main() { int nl=0, nw=0, nc=0; int inword = FALSE; char c; c = getchar(); while (c != EOF) { ++nc; if (c == ‘\n’) {++nl;} if (c==‘ ‘ || c==‘\n’ || c ==‘\t’) {inword = FALSE;} else if (!inword) {inword = TRUE; ++nw;} c = getchar(); } printf(“%d lines, %d words, %d chars\n”, nl, nw, nc);}


wordcount Example: Compiling and Running

We compile with:

gcc fprofile arcs ftest coverage o wordcount wordcount.c

We get:

• Executable in wordcount

• System files wordcount.gcno

We run wordcount • We give it as input two empty lines (two carriage returns)

File wordcount.da is created

We now run gcov wordcount.c

gcov tells us:

• 86.67% of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.


-: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:1 -: 0:Programs:1 -: 1:#include <stdio.h> -: 2:#define TRUE 1 -: 3:#define FALSE 0 1: 4:main() { 1: 5: int n1=0, nw=0, nc=0; 1: 6: int inword=FALSE; -: 7: char c; -: 8: 1: 9: c=getchar(); 4: 10: while (c != EOF) { 2: 11: ++nc; 2: 12: if (c=='\n') 2: 13: {++n1;} 4: 14: if (c==' ' || c=='\n' || c=='\t') 2: 15: {inword=FALSE; } #####: 16: else if (!inword) #####: 17: {inword=TRUE; ++nw; } 2: 18: c=getchar(); -: 19: } 1: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}


wordcount Example: Explanation of the Output

The two lines marked with # have never been executed

The other lines have been executed

For instance:

• The initial c = getchar() has been executed once (1 in left margin)

• The ++nc has been executed twice (2 in left margin)

—This is because each carriage return counts as 1 character

The declaration lines:

int nl=0, nw=0, nc=0;

int inword = FALSE;

are marked as executed because they contain initialization code

The declaration line

char c;

is not marked at all because it is just a declaration


wordcount Example: Carrying On

We now run wordcount again, this time giving just the line Zippy as input

We now run gcov wordcount.c gcov tells us: 100.00% of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.


Wordcount Example: New .gcov File -: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:2 -: 0:Programs:1 -: 1:#include <stdio.h> -: 2:#define TRUE 1 -: 3:#define FALSE 0 2: 4:main() { 2: 5: int n1=0, nw=0, nc=0; 2: 6: int inword=FALSE; -: 7: char c; -: 8: 2: 9: c=getchar(); 12: 10: while (c != EOF) { 8: 11: ++nc; 8: 12: if (c=='\n') 3: 13: {++n1;} 11: 14: if (c==' ' || c=='\n' || c=='\t') 3: 15: {inword=FALSE; } 5: 16: else if (!inword) 1: 17: {inword=TRUE; ++nw; } 8: 18: c=getchar(); -: 19: } 2: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}


wordcount Example: New Output

We have given the system an input file of 6 characters on this run (5 letters + carriage return) Along with the 2 characters we gave on the previous run, the runs of wordcount have read 8 characters Hence, ++nc line has been executed 8 times in total

• Counts on other lines have similarly been updated We have now executed the case where we have a non whitespace character

• Therefore, the lines else if (!inword) { inword = TRUE; ++nw; }

have been executed • Therefore, 100% of the lines of the program have now been

executed


Measuring the Usefulness of a Test Suite A good structural test suite will execute most lines of code

• It may not be possible to execute all lines • Executing a large percentage may be acceptable • Hence, measure of code covered (% of lines executed) is a good

measure of the usefulness of such a test suite However, with gcov:

• As we keep executing test cases, the .gcda file keeps getting updated Therefore, to measure goodness of a test suite, what we would like to do is:

• Reset all counts in .gcda file to 0 • Run test suite • Run gcov and look at % of lines covered

To reset all counts in the .gcda file: • Just remove the file!

Your compiled code will: • Assume this is the first time it has been run again • Create the .gcda file anew


How Does gcov Do It?

Each block in code numbered gcc inserts some code into your program at beginning of main, beginning of each block, end of main

• At beginning of main: —Program zeros out a big array

• During execution: —Every time block n executed, entry n of array incremented

• At end of main: —If .gcda file does not exist, program creates it —Otherwise, program reads .gcda file, increments counts in

array, writes out .gcda file again Does similar processing for calls to exit() as at end of main Map files map source line number to block

• gcov program matches block execution counts in .da file with source lines


gcov SubtletiesIf we compile and link many source files with the “gcov options”:

• Each source file gets a .gcno file (the map file) • When we run the program, each source file gets a .gcda file (the coverage data

file) Thus, we can/must get a separate report from gcov on each separate source file

• Useful if we are interested in coverage of each module of our program Example:

• We have a main program main.c and two modules, StackImplementation.c and IO.c

• On compilation, we get main.gcno, StackImplementation.gcno and IO.gcno • Every time we run the program, files main.gcda, StackImplementation.gcda,

and IO.gcda are updated • We can say gcov StackImplementation.c

—Get a report on StackImplementation.c separately —Useful if that is the module we are interested in


#include <stdio.h>

int main()

{

int x;

int y;

int z=0;

printf("\nEnter x:");

scanf ("%d", &x);

printf("\nEnter y:");

scanf("%d", &y);

z = x+y;

if (z>x)

printf("\nHi");

else

printf("\nBye");

if (z>y) printf ("\nYes");

else

printf("\nNo");

printf("\nAll Done");

return 0;

}

Let’s try it out on the gaul network:

lsgcc -fprofile-arcs -ftest-coverage junk2.c


gcov, Lines and Statements

Consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) {

pos = 1; }printf(“%d %d \n”, x, pos);

• We can execute 100% of lines in this code only if we input a positive number for x sometime

Now consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) {pos = 1;} printf(“%d %d\n”, x, pos);

• We can execute 100% of lines in this code even if we input only negative numbers for x


gcov, Lines and Statements continued

Reason:

• In second code fragment, the sub statement pos = 1 of the if is on the same line as the if

• We have executed the if in the sense that we have evaluated its condition (x>=0)

• Line is considered executed even when only some of the code on it is executed

Moral: to ensure all statements executed:

• Must put each statement on a separate line


Is gcov Enough?gcov is OK for a simple analysis of the thoroughness of a test suite

• It tells us what percentage of the lines of code we have executed However, a test suite executing 100% of lines may still not catch some problems Example: consider again the code

scanf(“%d”, &x); /* Read x from terminal */ if (x>=0) { pos = 1; }printf(“%d %d\n”, x, pos);

• If all tests in our test suite set x to a positive number, we can achieve 100% line coverage

• However, we have still not tested the case in which x is negative or zero • That case might be handled incorrectly

—For instance, maybe pos is supposed to be 1 if x is positive, and 0 otherwise

To talk about stronger measures of thoroughness, it is useful to have a graph of the program control flow


int main() {

int n1, n2, n3, ans; printf("\n\nEnter a value for n1:"); scanf("%d",&n1); printf("\n\nEnter a value for n2:"); scanf("%d",&n2); printf("\n\nEnter a value for n3:"); scanf("%d",&n3); ans = getnum(n1, n2, n3); printf("\nThe number was: %d\n”,ans); return 0;}

int getnum(int num1,int num2,int num3){ if (num1>num2) return num1; else if (num2>num3) return num2; else { if (num1==num2) return num3; else return num1+num2; } }

Consider the code to the right and answer the following questions:

• gcov would count 16 lines, which lines do you think get counted?

•What % coverage would the following test case give:n15, n24, n33

•Can we get 100% coverage? If so, list each of the test cases you would use to get that coverage, give the minimal number of test cases in your answer needed to get 100% coverage.

•Junit info

Review Questions

Documents

Computer Science 2212a/b - UWO1 Structural Testing Motivation The gcov Tool An example using gcov How does gcov do it gcov subtleties Further structural