Fixing the DefectCEN4072 – Software Testing
Yuriy Zaynulin, 2015 2
From Defect to Failure
How a defect becomes a failure:
1. The programmer creates a defect
2. The defect causes an infection
3. The infection propagates
4. The infection causes a failure
Yuriy Zaynulin, 2015 4
From Defect to Failure (cont.)
How to locate a defect:
1. Identify infected value that defines the failure (pic. [a])
2. Determine the possible origins of the infected value (pic. [b])
3. Check every possible origin for infection (pic. [c])
Found the earlier infection? Go to step 2To
Yuriy Zaynulin, 2015 5
Techniques for Locating Defects
Infections e.g. a failed assertion
Causes e.g. “a = 0” causes the failure
Anomalies e.g. f() executed only in failing run
Code Smells e.g. uninitialized variable
Dependencies e.g. a comes from a
Yuriy Zaynulin, 2015 6
Techniques for Locating Defects (cont.)
Assertions: ensures data sanity
failing assertion signals an infection
passing assertion needs not be considered
Anomalies - aspects of code correlated with failure
Causes - aspects of execution correlated with failure AND causing the
Yuriy Zaynulin, 2015 7
Dependences: check the backward slice for infections
start with the closest statements
anything not in the backward slice
CANNOT have caused the infection
[x] is a dependency of [!]s
Yuriy Zaynulin, 2015 8
Observation: allows a programmer to tell (or specify)
what is correct and what isn’t correct
Observe:transition from SANE to
Yuriy Zaynulin, 2015 9
1.Focus on Infections [ if an origin is already known to be faulty, else: ]
2.Focus on Causes [ if delta debugging or experimentation found the cause of a failure, else: ]
3.Focus on Anomalies [ if they are present, else: ]
4.Focus on Code Smells [ if they are present, else: ]
5.Focus on Dependences [ if nothing else worked, examine backward slices; ]
Yuriy Zaynulin, 2015 10
Is the Error a Cause?
Origin is an error, but doesn’t cause the failure?
→ Wrong track!
a = compute_value();printf("a = %d\n", a);
output is 0
we assume compute_value() produces infection
set a to 1 right before printf(...), still output is 0
→ actual problem is with printf(...)
Yuriy Zaynulin, 2015 12
Is the Cause an Error?
Origin causes the failure, but not [sure whether it is] an error?
→ DO NOT debug it into existence!
Yuriy Zaynulin, 2015 13
Devil’s Guide to Debugging
1. Find the defect by guessing:
Scatter debug statements all over
Try changing code until something works
2. Don’t waste time understanding the problem
3. Use the most obvious fix
x = compute(y); // compute() doesn’t work for y == 17, so fix it if (y == 17) x = 25.15;
Yuriy Zaynulin, 2015 14
Correcting the Defect
BEFORE: Save the code to Version Control System (VCS)
* BRIEF PERIOD OF HAPPINESS *
AFTER: Does the failure no longer occur?
Did the correction introduce new problems?
Was the same mistake made elsewhere?
Did I commit the change?
Yuriy Zaynulin, 2015 15
Not a permanent solution
Defect DOES NOT cause a failure anymore
Defect DOES still remain
→ Keep the problem open
[in the tracing system]
Yuriy Zaynulin, 2015 16
When it is appropriate to use:
Unable to change
Supplied by third party
Closed source code
Large changes throughout the system needed
Code is fine, but:
Design must undergo major revision
Yuriy Zaynulin, 2015 17
Famous workarounds: Spam filters
Flaw: anyone can easily forge email sender identity
Proper solution: redesign entire e-mail system [costs and risks]
Flaw: OS users have administrator rights, any program can gain full control
Proper solution: assign users limited rights and add proper authorization [many good programs assume administrator right for installation]
Yuriy Zaynulin, 2015 18
Learning From Mistakes
Improve test suite
Tests failed to detect the problem → extend the tests.
Set up assertions
Keep assertions used in debugging. Add more. Leave assertions active in production code.
Teach programmers about potential pitfalls and how to avoid them.
Improve the software process
Software was not tested fully? Wrong version was shipped? Critical part was not reviewed? Work on production process!
Improve analysis tools
Yuriy Zaynulin, 2015 19
Facts on Fixes
1. In the ECLIPSE and MOZILLA projects, about 30 to 40% of all changes are fixes.
2. Fixes are typically two to three times smaller than other changes.
3. Fixes are more likely to induce failures than other changes.
4. Only 4% of one-line changes introduce new errors in the code.
5. A module that is one year older than another module has 30% fewer errors.
6. Newly written code is 2.5 times as defect prone as old code.
Yuriy Zaynulin, 2015 20
To isolate infection: Work backward along infections origins
To find most likely origin: Focus on
Each origin must be an infection as well as a cause;
Use workaround if correction is too costly or risky
Yuriy Zaynulin, 2015 21
To correct the defect: know how
Your change breaks the infection chain
Will the failure no longer occur
To ensure your correction is successful: check
The failure no longer occurs
No new problems were introduced
Yuriy Zaynulin, 2015 22
CADD Manager, “When Workarounds Become Best Practices”,http://www.caddmanager.com/CMB/2010/09/when-workarounds-become-best-practices/, 2010
Graves et al., 2000
Mockus, Votta, 2000
Mockus, Weiss, 2000
Ostrand, Weyuker, 2002
Purushothaman, Perry, 2004
Sliwerski et al., 2005
Zeller A., “Why Programs Fail: A Guide to Systematic Debugging”, 2005