Software Testing Fundamentals part II

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.613 Software Testing and Quality Assurance

Lecture 2, 20.9.2005

Software Testing Fundamentalspart II

Juha ItkonenSoberIT

2Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Contents

Testing terminology and basic conceptsRealities and principles of testing


Testing terminology and basic concepts


Validation and verification

Validation – are we building the right product?Implementation meets customer requirements, needs and expectations

Verification – are we building the product right?Program conforms to its specification


Functional testing

Testing conducted to evaluate the compliance of a system with specified functional requirements.

IEEE Standard Glossary of Software Engineering Terminology


Black-box and white-Box testing

Black-box (functional, behavioral, data-driven)The software under test considered as a black-box and no knowledge of the internal structure or how the software actually works is utilized in testingTesting based on inputs and respective outputsThe size of the black box can vary from one class or component to a whole system

White-box (structural, logic-driven)Testing based on knowing the inner structure of the system and the program logic


Dynamic and static methods

Dynamic methods – testing – executes codeSoftware testing in the traditional senseDynamic analysis methods

Static methods do not execute codeReviews, inspections, static analysisSome authors name these static testing


Defect Prevention

Quality assurance is focused on all activities that contribute to the quality of the final productDefect prevention is essential

Preventing defects to occur in the first placeDevelopment practices, conventions, techniques, and tools

Many practices are used to prevent bad quality deliverables

Coding conventions and standardsArchitectural patternsPair programmingEtc.

Reviews and inspectionsDetect defects in current phasePrevent defects in following phases


Defect Detection

Earlier detected defects are cheaper to fixPrevented defects can be even cheaper

Defect detection is always neededNobody is perfect, all defects cannot be preventedDetection of some bearable amount of defects can be cheaper than prevention

Detection for confidenceTesting and measuring gives information of the achieved qualityWe try to find defects in order to convince that there are none


Scripted and non-scripted testing

In scripted testing test cases are pre-documented in detailed, step-by-step descriptions

Different levels of scripting possibleScripts can be manual or automated

Non-scripted testing is usually manual testing without detailed test case descriptions

Can be disciplined, planned, and well documented exploratory testing… or ad-hoc testing


V-model of testing

Requirements

Unittesting

Coding

Moduledesign

Architecturedesign

Functionalspecification

Acceptancetesting

Systemtesting

Integrationtesting

Build Test


Implications of V-model

If you want to get something done, it helps to know what do you want, first. If you want to build something complex, it helps to have a blueprint laid out in advance. It is better to finish a given task before starting work on another task that depends on the output of the first task. It is easier to think about a product in terms of a hierarchy of blueprints.It is better to test each blueprint, as well as to use each blueprint as a basis for a test process, as the end product is being built.It is much easier to find faults in small units than in large entities.Testing of large units can be carried out more easily when the smaller parts are already tested.


Benefits of the V-model

Intuitive and easy to explainEven to people who have never heard of software development lifecycleMatches to familiar waterfall

Quite adaptable to various situationsIf the team is flexible and understands the inherent limitations of the model

Makes a good model for training peopleSimple and easy to understandShows how testing is related to other phases of the development project

Beats the code-and-fix approach on any larger project

More than a dozen or so people


Levels, types, and phases – not a same thing

Test level – a test level is a group of test activities that focus into certain level of the test target.

E.g. unit test, integration test, system test, acceptance testTest levels can be seen as levels of detail and abstraction

Test type – A test type is a group of test activities aimed at evaluating a system for a number of associated quality characteristics.

E.g. functional test, performance test, stress test, usability test

Test phase – Test phases are used to describe temporal parts in testing process that follow sequentially each other, with or without overlapping


Common testing types

Functional testingFunction testingConcurrency testingInstallation testingPlatform testingSmoke testingConfiguration testingCompatibility testingException testingInterface testingLocalization testingDocumentation testingData quality testingConversion testingProduction testingStandards testing

Non-Functional testingPerformance testingLoad testingStress testingVolume testingReliability testingSecurity testingUsability testingRecoverability testingMaintainablity testing


What is a ”bug”?

Error - a human action that produces an incorrect resultFault - a manifestation of an error in software

also known as a defect, bug, issue, problem, anomaly, incident, variance, inconsistency, feature, …if executed, a fault may cause a failure

Failure - deviation of the software from its expected delivery or service

Failure is an event; fault is a state of the software, caused by an error.


Error –> Fault -> Failure

A programmermakes an error…

…that creates a faultin the software…

…that can cause a failure in operation


What is a fault?

A software error is present when the program does not do what its end user reasonably expects it to do (Myers, 1979)

There can never be an absolute definition for bugs, nor an absolute determination of their existence. The extent to which a program has bugs is measured by the extent to which it fails to be useful. This is a fundamentally human measure. (Beizer, 1984)

A software fault is a mismatch between the program and its specification a fault can be detected if and only if the specification exists and is correct.


More accurate definition of a failure

One definition (Software Testing, Ron Patton)

The software does not do something that the specification says it should doThe software does something that the specification says it should not doThe software does something that the specification does not mentionThe software does not do something that the specification does not mention but it shouldThe software is difficult to understand, hard to use, slow, or will be viewed by the end user as just plain not right


Test Oracles – How do we know it’s broken?

An oracle is the principle or mechanism by which you recognize a problemTest oracle provides the expected result for a test, for example

Specification documentFormulaComputer programPerson

In many cases it is very hard to find an oracleEven the customer and end user might not be able to tell which is the correct behaviour

Oracle problem is one of the fundamental issues in test automation:How do we teach an automated test to recognize a defect or failure when it happens? – This is not a trivial problem.


Oracle – Is this correct?

Font size test Arial 22ptFont size test Arial 20pt

Font size test Arial 18pt— Font size test Arial 16pt

— Font size test Arial 14pt

Font size test Times 22ptFont size test Times 20pt

Font size test Times 18pt— Font size test Times 16pt

— Font size test Times 14pt


When is an issue a fault?

Consider following consistency heuristics

Consistent with History: Present function behavior is consistent with past behavior.Consistent with our Image: Function behavior is consistent with an image that the organization wants to project. Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.Consistent with Claims: Function behavior is consistent with what people say it’s supposed to be.Consistent with User’s Expectations: Function behavior is consistent with what we think users want.Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.Consistent with Purpose: Function behavior is consistent with apparent purpose.

Suorce: Black Box Software Testing, Cem Kaner


Severity and priority

Severity of a software fault refers to the severity of the consequences of a failure caused by that fault

A tester or customer is probably the best person to set the severity

Priority is the fixing order of the found faults and a result of separate prioritisation activity

A tester is probably not the best person to set the priorityPrioritisation is typically a managerial decision

business prioritiescustomer prioritiesother faults and featuresquality policyrelease strategy

Prioritisation is usually based on severity

The priority of a faultdepends on other issues


Reliability

Reliability: the probability that software will not cause the failure of the system for a specified time under specified conditions

Can a system be fault-free?Can a software system be reliable but still have faults?


Re-testing vs. regression testing

Re-tests New version of software with ”fixed fault”Re-run the same test (i.e. re-test)

must be exactly repeatablesame environment, versions (except for the software which has been intentionally changed!)same inputs and preconditions

If test now passes, fault has been fixed correctly –or has it?

Regression testsRunning a set of tests that have been run beforeAfter software or environment changes

For confidence that everything still worksFor emergency fixes (possibly a subset)

An asset (regression test suite/pack)

Standard set of tests -needs designing & maintainingWell worth automating


Depth tests

Breadth tests

Regression testing

x

x

x

x

Fault fix introduces oruncovers new faults

Test finds faultx

Re-test to check fix

Regression tests look forunexpected side-effects(but may not find all of them)


Assessing quality based on testing

Low

You thinkyou are here

High

Test quality

Software qualityLow High

Students on Software Project course claim to be here

You maybe here


Realities and principles of testing


The problem of complete testing

Why not just test everything?Complete testing would require testing

every possible valid input of the systemand every possible invalid inputand every possible valid and invalid outputand every possible combination of inputsin every possible sequence of inputs and outputsand user interface errorsand configuration and compatibility issuesand environment related failures…

There is definitely too much cases to test completely any realistic software module, not to mention a whole system


Why not just ”Test Everything”?

System has 20 screens Average 4 menus, 3 options / menu

Average: 10 fields / screen

2 types input / field

date as Jan 3 or 3/1

number as integer ordecimal

Around 100 possible values

20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests

1 second / test -> testing takes 8000 mins= 133 h = 17 working days

10 secs / test -> 34 weeks, 1 min / test -> 4 years, 10 min / test -> 40 years

(not counting finger trouble, faults orretest)


Execution paths

There are 51 + 52 + ... + 519 + 520 = 1014 = 100 trillion paths through the program.Testing would take approximately one billion years to try every path (if one could write, execute and verify a test case every five minutes ).3,2 million years if one test case per second

A

B

C

D

E

F

G

H

I

J End

< 20 loops

Begin


Testing is always prioritizing

Time is always limitedUse company’s significant risks to focus testing effort

what to test firstwhat to test mostHow thoroughly to test each featurewhat not to test (at least for now)

Most important tests first

Possible ranking criteria1. test where a failure would

be most severe2. test where failures would

be most visible3. test where failures are

most likely4. ask the customer to

prioritize the requirements

5. what is most critical to the customer’s business

6. areas changed most often7. areas with most problems

in the past8. most complex areas, or

technically critical


Realities of testing (1/2)

It is impossible to test a program completelyTesting can show the presence of defects but cannot show the absence of defects

You can report found defects and what you have tested and how

All bugs can not be foundAll found bugs will not be fixed

It’s really not a bugIt’s too risky to fixThere is more important tasks to doIt’s just not worth itThere is not enough time

Testing does not create quality software or remove defects


Realities of testing (2/2)

The critique is focused on the product, not the developerProduct specifications are never final

You cannot wait until specifications are finalThere will always be changes to specificationsThe specifications are never complete

The more bugs you find, the more bugs there areProgrammers have bad daysProgrammers often make same mistakeSome bugs are really just the tip of the iceberg

The Pesticide ParadoxPrograms become immune to unchanging tests

Software testing should be based on risksTesters are not the most popular members of a project team


Testing principles

When the test objective is to detect defects, the a good test case is one that has a high probability of revealing a yet undetecteddefect(s).Test results should be inspected meticulously.A test case must contain the expected results.Test cases should be developed for both valid and invalid input conditions.The probability of the existence of additional defects in software component is proportional to the number of defects already detected in that component.Testing should be carried out by a group that is independent of the development team.Tests must be repeatable and reusable.Testing should be planned.Testing activities should be integrated into the software life cycle.Testing is a creative and challenging task.

Burnstein 2003

Documents

Software Testing Fundamentals part II