Quantifying the Value of Static Analysis

Lawrence Livermore National Laboratory

William B. Oliver

LLNL-PRES-490136

Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344

Quantifying the Value of Static Analysis

Date 5/19/2011

2Option:UCRL# Option:Additional Information


What is Static Analysis

The use of tools during pre-testing to remove structural defects

Software Developer




Static analysis tools provide an in-depth analysis of source code to find defects.




Finds Defects that compilers and traditional testing miss




Defect types include but not limited to • Use of uninitialized variables




Defect types include but not limited to • Use of uninitialized variables• Memory leaks




Defect types include but not limited to • Use of uninitialized variables• Memory leaks• Null Pointer dereferences




Defect types include but not limited to • Use of uninitialized variables• Memory leaks• Null Pointer dereferences• Array Bounds Overflows (and many others)



Why Incorporate Static Analysis

Static Analysis finds additional defects

Better Code Coverage

Reduced Developer Debug Time

Advanced Tools find Defects Inter-Procedurally

Uncovers structural defects that can cause Functional Defects

Finds defects missed during code reviews/walk thrus



Inter-Procedural Analysis

File A File B

File C

foo(x) bar(y)

foobar(z)



Enhances Dynamic Testing

• Dynamic testing does not generally uncover memory leaks and other structural defects

Dr. Paul Anderson PhD

“They are best used in combination

with traditional dynamic testing

techniques, and can even reduce the

cost to create and manage test cases

for stringent run-time coverage..”

“They are best used in combination

with traditional dynamic testing

techniques, and can even reduce the

cost to create and manage test cases

for stringent run-time coverage..”

Why Incorporate Static Analysis

• Static Analysis provides 100 % code coverage

• Structural defects such as Array Bounds Overflows can cause Functional Defects



SD FD

Structural Defects vs Functional Defects

Relate to conformance to the Programming Language rules and syntax

Uninitialized Data

Memory/Resource Leaks

Array Bounds Overflows

Null Pointer Dereferences



SD FD

Structural Defects vs Functional Defects

Associated with Features, Performance, Availability etc.

Found During Dynamic TestingSome Causes Include:

Solving the wrong problem

Code Logic ErrorsSystem Integration Issues



Uninitialized Data

x

ySet of Right Answers

z

Correct Set of Values

Random Set of Values



Memory/Resource Leaks



Array Bounds Overflow



NULL Pointer Dereference

Pointer

This memory location contains the address of this memory location

Value of the contents of address pointed to by the pointer



NULL Pointer Dereference

Pointer = NULL



Typical Static Analysis Work Flow

Perform Static

Analysis

Analyze Defects

Fix DefectsTest Fixes

Add Features



Advantages for Testers

o Less Wasted Time

o Allows more time for test case development

o Better Test Cases



Assumptions About Time to Find Defects

One Million Lines of Code

Static Analysis

1000 Defects

20 Percent False Positives

800 Valid Defects

Time to Run Code Thru Tool Negligable

Ten Minutes Per Defect to Triage

Dynamic Testing

Automated Testing: 1 hour per defect

Includes Test Case Development

Test Evaluation

Test Report Generation

Manual Testing: 2 hours per Defect1000 Defects



Automated Testing

TD = Total Defects = SD + FD

Time = Time to Find SD + Time to find FD

Time/TD = 1166.67/1800 = .65 hours/defect = 39 minutes per defect

TD = 800 + 1000 = 1800

SD Time = (1000 Defects * 10 min/defect)/60min/hour = 166.67 hours

Time = 166.67 + 1000 = 1166.67 hours



Test Case: Automated Testing

Code Type:

Programming Language:

Number of Developers:

Source Lines of Code Analyzed:

Scientific Simulation

C++

4

161,880

Total Number SD found:

Total Number SD Analyzed:

Number of False Positives:

Average Analysis Time/Defect:

528

190

55

8.9 minutes



Test Case: Automated Testing

TD = Total Defects = SD + FD

FD = 297 for dynamic testing and 1 hour per defectTD = 135 + 297 = 432

SD Time = (190 Defects * 8.9 min/defect)/60min/hour = 28 hours

Time = 28 + 297 = 325 hours

Time/TD = 325/432 = .75 hours/defect = 45 minutes per defect



Just For Fun What If All 528 defects were triaged

Assuming 28 % False Positive Rate





528

528

148

8.9 minutes

Estimated number of real defects = 380

Estimated Time = (528 * 8.9) / 60 = 78 hours

TD = 380 + 297 = 677

Time = 78 + 297 = 375 hoursTime/TD = 375/677 = .55 hours/defect = 33 minutes per defect



Manual Testing

Code Type:

Programming Language:

Number of Developers:

Security Access

C#





76

35

0

3.4 minutes



Manual Testing

FD = 339 for dynamic testing and 5 hours per defectTD = 35 + 339 = 374

SD Time = (35 Defects * 3.4 min/defect)/60min/hour = 2 hoursTime = 2 + 1695 = 1697 hours

Time/TD = 1697/374 = 4.5 hours/defect



Bottom Line

Combined with dynamic testing Static Analysis results in finding more DefectsAnd the organization spends less time per defect in the process



Summary

For Static Analysis the time to Find a defect is less than or equal to 10 minutesDynamic Testing:

Automated: 1 hour per Defect

Manual: 4 – 5 hours per Defect



New Breed of Tester

Perform Static

Analysis

Analyze Defects



Questions???

Technology

Quantifying the Value of Static Analysis