Testing and Inspecting to Ensure High Quality Defects in Ordinary Algorithms Defects in Handling Stress or Unusual Situations

Testing and Inspecting to Ensure High

Quality

Defects in Ordinary AlgorithmsDefects in Handling Stress or Unusual

Situations

© Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 2

Defects in Ordinary Algorithms

Incorrect logical conditions• Defect:

—The logical conditions that govern looping and if-then-else statements are wrongly formulated.

• Testing strategy: —Use equivalence class and boundary testing. —Consider as an input each variable used in a rule or

logical condition.—Break into smaller, individual logical tests.


Example of incorrect logical conditions defectWhat is the hard-to-find defect in the following code?

• The landing gear must be deployed whenever the plane is within two minutes from landing or takeoff, or within 2000 feet from the ground. If visibility is less than 1000 feet, then the landing gear must be deployed whenever the plane is within three minutes from landing

or lower than 2500 feet If (!landingGearDeployed && (min(now-takeoffTime , estLandTime-now))< (visibility < 1000 ? 180 :120) || relativeAltitude < (visibility < 1000 ? 2500 :2000) ){ throw new LandingGearException();} Class Exercise! What is the order of evaluation – for starters!!


Defects in Ordinary AlgorithmsPerforming a calculation in the wrong part of a control construct

• Defect: —The program performs an action when it should not, or does not

perform an action when it should. —Typically caused by inappropriately excluding or including the

action from a loop or a if construct.• Testing strategies:

—Design tests that execute each loop zero times, exactly once, and more than once.

- Ensure a Do…While (or While…) is false to begin with- Ensure a Perform … Until is true to begin with- Remember: Pre-tests and Post Tests

—Anything that could happen while looping is made to occur on the first, an intermediate, and the last iteration.


Example of Performing a Calculation in the Wrong Part of a Control Construct

What is wrong with the following loop?:

while( j < maximum){ k=someOperation(j); j++;}If (k == -1) signalAnError();


What is wrong with the following loop:

while( j < maximum){ k=someOperation(j); j++;}If(k == -1) signalAnError();

Must assume j has an acceptable integral value prior to entering loop.

Could say: while (j++ < maximum) but this would affect argument (j)!


Defects in Ordinary Algorithms 1Not terminating a loop or recursion

• Defect: —A loop or a recursion does not always terminate, i.e. it is ‘infinite’. —Remember in recursion, you have the ‘recursive case’ and the

‘base case’ —What do we mean by the ‘recursive case?’ Properties??—What do we mean by the ‘base case?’

• Testing strategies:

—Analyze what causes a repetitive action to be stopped.—Run test cases that you anticipate might not be handled correctly.

Count on these happening during operations!

Does recursive case converge to base case?


Defects in Ordinary Algorithms 3

Not setting up the correct preconditions for an algorithm• Defect:

—Preconditions state what must be true before the algorithm is executed. —A defect would exist if a program proceeds to do its work, even when the preconditions are not satisfied. Do we really want to ensure a < 14 is true prior to going into the algorithm?

while (a<14){ ….……} such that even if a >= 14, this loop (and perhaps other contents) continues to run appropriately.

• Testing strategy: —Run test cases in which each precondition is not satisfied.—See what happens beyond the while-loop.

Absolutely! Insert probes (Displays, printf, system.out.println…)


Defects in Ordinary Algorithms 4Not handling null conditions

• Defect: —A null condition is a situation where there normally are one or more data

items to process, but sometimes there are none. —It is a defect when a program behaves abnormally when a null condition is

encountered.—How do you test null (or non-numeric) condition in, say, COBOL, for a

numeric field? How can it happen?

• Testing strategy: —Brainstorm: determine unusual conditions and run appropriate tests. —Always test ‘no inputs’ interactively or via an OPENed file. —Always test to see if the Input or Read or … retrieved data was non-null! —Always test with only one input;

- Oftentimes code runs perfect with a single input;

—Always test with two or more.- If code runs correctly with two inputs, it’ll probably run with n inputs (if we

are dependent upon checking null or EOF conditions, etc.


Defects in Ordinary Algorithms 5Not handling singleton or non-singleton conditions

• Defect: —A singleton condition occurs when there is normally more than one

of something, but sometimes there is only one. (this is a little different than the last case)

—A non-singleton condition is the inverse. —Defects occur when the unusual case is not properly handled.

• Testing strategy: —Brainstorm: determine unusual conditions and run appropriate tests.—Can occur with incorrect number of parameters (in some

languages). - Can abend or continue, providing null values to unmatched parameters…

—Think argc and argv. Can easily test.



Off-by-one errors• Defect:

—A program inappropriately adds or subtracts one.—Or loops one too many times or one too few times. —This is a particularly common type of defect.

• Thoughts: —Results: often EOF reached; executing same instructions twice; —Very easy in C and Java: side effects of a++ and ++a; careful in using a++ or

a++ within predicate (condition) of loop as in while (a++ < b) is quite different than while (++a < b) !

—Test: if a = 1 and b = 2, which of the two preceding, if any, were true? COBOL: Perform …. Until X >= 25 or ….Until X=25! Java, C++ for (int i = 1; i < 25; i++) ….. ++i or i++ no difference here.

Always able to give correct value if ‘i’ within loop”! What is value of loop control variable when we terminate the loop?

• Testing strategy: —Develop tests in which you verify that the program:

- computes the correct numerical answer.- performs the correct number of iterations.- Always display value of loop control variables upon exiting loop!!!- Insert probes for counting!!!!


Example of off-by-one defect 7

for (i=1; i<arrayname.length; i++) { /* do something */ }Difference between length and size????What if arrayname.length = 1?

while (iterator.hasNext()){ anOperation(++val);}

Nice boolean checks are almost always benign!

Use Iterators to help eliminate these defects



Operator precedence errors• Defect:

—An operator precedence error occurs when a programmer omits needed parentheses, or puts parentheses in the wrong place.

—Operator precedence errors are often extremely obvious...- but can occasionally lie hidden until special conditions arise.

—E.g. If x*y+z should be x*(y+z) this would be hidden if z =0 and perhaps is normally zero.

• Testing: —In software that computes formulae, run tests that anticipate

such defects.—Know the operator precedence rules! —Don’t parenthesize to death! It shows you don’t know what you

are doing!



Use of inappropriate standard algorithms• Defect:

—An inappropriate standard algorithm is one that is unnecessarily inefficient or has some other property that is widely recognized as being bad.

—Can you think of any?- How about Order n**2 sorts?- How about QuickSort?- How about Address Calculation sorts (various kinds)?

• Testing strategies:

—The tester has to know the properties of algorithms and then design tests that will determine whether any undesirable algorithms have been implemented.

—There is no ‘one size fits all.’ —Knowing some of the data characteristics is a big help!


Defects in Handling Stress and Unusual Situations

Incompatibility with specific configurations of hardware or software• Defect:

—The system fails if it is run using particular configurations of hardware, operating systems, and external libraries.

— (How about browsers too?)

• Testing strategy: —Extensively execute the system with all possible configurations that

might be encountered by users.—For standard software, must anticipate (or restrict) platforms!


Defects in Handling Stress and Unusual Situations 2

Defects in handling peak loads or missing resources• Defects:

—The system does not gracefully handle resource shortage.—Resources that might be in short supply include:

- memory, disk space or network bandwidth, permissions, processors, other peripherals, ….

- Honeywell: PAT Table Overflow (< 32 files)—The program being tested should report problems in a way the user

will understand. • Testing strategies:

—Devise a method of denying resources (test missing resources, such as disk drives not available, etc.).

—Run a very large number of copies of the program being tested, all at the same time.

—May require some redesign or rethinking of the process!



Inappropriate management of resources• Defect:

—A program uses certain resources but does not make them unavailable when no longer needed.

—Memory leakage? C – free? Java? Always good? No. • Testing strategy:

—Run the program intensively in such a way that it uses many resources, relinquishes them and then uses them again repeatedly.

— Can require some sophisticated testing!— Can require heavy loading!



Defects in the process of recovering from a crash• Defects:

—Any system will undergo a sudden failure if its hardware fails, or if its power is turned off.

—It is a defect if the system is left in an unstable state and hence is unable to fully recover.

—It is also a defect if a system does not correctly deal with the crashes of related systems.

• Testing strategies: —Kill a program at various times during execution. —Try turning the power off; however, operating systems

themselves are often intolerant of doing that.—Abruptly pre-empt key processes.


Disaster Recovery

Earthquakes, Hurricanes, Tornadoes, Floods.What do we do???

Discussion of Disaster Recovery approachesData – the real valuable corporate assetBackup computing platforms

Renting / leasing resources, CPU cycles, disk space, etc.

Procedures for evacuationfire, emergency generators, travel.etc.

Documents

Testing and Inspecting to Ensure High Quality Defects in Ordinary Algorithms Defects in Handling Stress or Unusual Situations