Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Coverage testing with clover
Software Analysis and TestingCu Nguyen Duy (cunduy at fbk dot eu)Alessandro Marchetto (marchetto at fbk dot eu)Paolo Tonella (tonella at fbk dot eu)Mariano Ceccato (ceccato at fbk dot eu)
Academic Year 2011-2012
1Thursday, November 17, 11
Coverage testing (recall)
• Coverage measures describe the degree to which a program has been tested
• tell the effectiveness in terms of coverage of the test set, help improve software quality
• inform quantitatively the project manager about the progress of testing
• Many type of coverage measures
• statements
• paths
• methods, classes
• requirement specifications, etc.
2Thursday, November 17, 11
Coverage tool features
• Compile time: automatic instrument code for coverage recording
• Runtime: structure-based coverage recording
• lines, blocks, conditions, methods and classes
• Reporting
• quantitative coverage measure
• percentage of code executed
• most executed v.s. never executed code
• visual navigation
• quickly navigate to code that is not executed to improve the test set
3Thursday, November 17, 11
Coverage tools
source code in not-widely-used languages on a variety ofplatforms [30].
3.2. Instrumentation overhead
Coverage-testing tools capture coverage information by moni-toring program execution. Execution is monitored by insertingprobes into the program before or during its execution.A probe is typically a few lines of code that, when executed,generate a record or event that indicates that programexecution has passed through the point where the probe islocated. There are two kinds of overhead associated withinstrumenting a program with probes: the off-line overheadof inserting probes into the program, and the run time over-head of executing the probes to record the execution trace.
3.2.1. Off-line program analysis and instrumentationoverhead
Source code instrumentation, used by most of the toolsincluding BullseyeCoverage, Parasoft Insure!!, Intel CodeCoverage Tool, Semantic Designs and TestWork, requiresrecompilation, but provides more direct results and is moreadaptable to a wide variety of processors and platforms. Itcannot be used when the source code is not available, as isoften the case for third party code. C/C!! tools such asDynamic Memory Systems’ Dynamics, use runtime instru-mentation, which makes them feasible in a productionenvironment. They may be more efficient in terms of compi-lation time, but less portable. The Java coverage tool Koalog
Code Coverage does not require instrumentation, and there-fore no recompilation is needed [25]. It operates with the pro-duction binaries using the Java Debug Interface, which is partof the Java Platform Debugger Architecture (JPDA). KoalogCode Coverage is platform independent, but requires aJPDA compliant Java Virtual Machine (JVM). Agitar’s Agita-tor runs the code in a modified JVM, also using a dynamicinstrumentation approach. eXVantage uses source code instru-mentation for C/C!! and bytecode instrumentation for Java.As compared to the other 16 tools, it has the highest off-lineinstrumentation overhead because it analyzes the program insuch a way that it can select the least number of probes tobe inserted into the target program.
3.2.2. Run-time instrumentation overheadCompanies that provide tools for system software or embeddedsoftware tend to focus more on reducing run-time overhead, sothat their tools can be usable in real-time environments, e.g.CodeTEST [18]. TCAT claims that its TCAT C/C!!Version 3.2 maintains its overhead for execution size ratio at1.1–1.8 and execution speed ratio at 1.1–1.5 [29]; SemanticDesigns claims 1.1–1.3, varying according to language andcompiler, among the best in our survey. Clover claims thattheir execution speed overhead is highly variable, dependingon the nature of the application under test, and the nature ofthe tests. Typical execution speed ratio is 1.2–1.5. eXVantagehas different versions for different platforms, but claims aratio of 1.01 for versions optimized real-time, in some environ-ments, based on initial trials (Table 2) [24].
3.3. Additional features
Coverage testing tools can be used to assist in debugging, andsome of the coverage tools provide debugging assistance, suchas Agitar, Dynamic, JCover, Jtest and Semantic Designs. Eachuses a different solution. For example, Agitar provides a snap-shot and stack trace to help developers to track the cause ofbugs. JCover has the ability to do coverage differencing andcomparison to expose the erroneous code. Semantic Designsprovides slicing and dicing operations on test coverage datavia the GUI to allow code executed/not executed by arbitrarycombinations of test runs to be easily isolated visually.eXVantage uses a dynamic execution slicing approach. Itcreates an execution slice for each test case and reads resultsfrom a testing oracle to generate a bug localization reportautomatically whenever a failed test is detected [8].Coverage testing tools can also be used for program profil-
ing to identify heavily executed parts of programs. Profilingdata can be used in compiler optimization, program refactor-ing, performance-related debugging, etc. Many tools, includ-ing eXVantage, CodeTEST, Dynamic Code Coverage,JCover, PurifyPlus and Semantic Designs, support this feature.
TABLE 1: coverage tools and the languages to which they apply(alphabetical by tool name)
Tool name C!!/C Java Other
Agitar [14] X
Bullseye [15] X
Clover [16] X .net
Cobertura [17] X
CodeTEST [18] X
Dynamic [19] X
EMMA [20] X
eXVantage [21] X X
Gcov [22] X
Intel [23] X FORTRAN
JCover [24] X
Koalog [25] X
Parasoft (C!!test) [26] X
Parasoft (Jtest) [26] X
PurifyPlus [27] X X Basic, .net
Semantic Designs (SD)
[28]
X X C#, PHP,
COBOL,
PARLANSE
TCAT [29] X X
Page 4 of 9 YANG et al.
THE COMPUTER JOURNAL, 2007
source: doi:10.1093/comjnl/bxm021
4Thursday, November 17, 11
Coverage tools
source: doi:10.1093/comjnl/bxm021
4Thursday, November 17, 11
Coverage tools
source: doi:10.1093/comjnl/bxm021
CodePro AnalytiXhttp://code.google.com/javadevtools/codepro
4Thursday, November 17, 11
Coverage tools
source: doi:10.1093/comjnl/bxm021
CodePro AnalytiXhttp://code.google.com/javadevtools/codepro
Cloverhttp://www.atlassian.com/software/clover
4Thursday, November 17, 11
Clover
• Is a commercial coverage tool
• Integration with a wide range of development environment: Ant, Eclipse, Maven, IntelliJ
• Installation
• Eclipse update: http://update.atlassian.com/eclipse/clover/
• Get an evaluation license
5Thursday, November 17, 11
Clover license setting
6Thursday, November 17, 11
Enable Clover to an Eclipse
• Right click on the project
• Select Clover > Enable on this project
7Thursday, November 17, 11
Clover Eclipse views
8Thursday, November 17, 11
Clover Eclipse views
8Thursday, November 17, 11
Run Unit test with Clover
• In the Coverage Explorer view
• Right click on the project
• Select Run With Clover As > JUnit Test
9Thursday, November 17, 11
Clover coverage report
10Thursday, November 17, 11
Clover coverage report
report menu
10Thursday, November 17, 11
Clover coverage report
report menu
projectinfo
10Thursday, November 17, 11
Clover coverage report
report menu
projectinfo
overall coverage
10Thursday, November 17, 11
Clover coverage report
report menu
projectinfo
overall coverage
testresults
10Thursday, November 17, 11
Clover coverage report
report menu
projectinfo
overall coverage
testresults
coverage by package
10Thursday, November 17, 11
Clover coverage report
report menu
projectinfo
overall coverage
testresults
tree mapcoverage by package
10Thursday, November 17, 11
Exercise 1
• Open project: Exercise-4-CurrentAccount
• Enable Clover on this project
• Execute Unit Test
• Generate coverage report
• Report the test coverage of each class, identify which lines of code have not executed
• Task: improve the test cases to reach 100% coverage
11Thursday, November 17, 11
Exercise 2
• Similar to exercise 1, with Exercise-3-MathOperators
12Thursday, November 17, 11