Software Testing Fundamentals part I

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.613 Software Testing and Quality Assurance

Lecture 1, 13.9.2005

Software Testing Fundamentalspart I

Juha ItkonenSoberIT

2Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY

Contents

Why do we need testing?What is software testing?Quality and quality assurance


Why do we need testing?


Why do we need testing?

We need testing to reduce and mitigate riskAll software has faultsWe prefer to invest in testing in order to find the most important faults before the software is released to live operation

Failures that occur during live operation are usually more expensive to deal with than failures than occur during testing

Higher the risks, more we need to testNo risk – no test


Someone will always test your software

Customer or UserCompetitorA government inspector, an industrial analyst, an insurance inspectorAn editor of a wide-spread magazineChannel resellerSystems integratorPotential partnerPotential subcontractorPotential employeeSoftware tester

…and find the nastiest bugs…


Software testing case studies (1/3)

Disney’s Lion King CD-ROM, 1994-199526, December customer support phones began to ring...Multimedia CD was tested only on a few systems – did not work on most common home pc:sA lot of bad publicity, newspaper stories and TV news

Intel Pentium floating-point division bug, 1994(4195835 / 3145727) * 3145727 - 4195835 = 0Problem was detected in Intel’s own tests before the chip was released but the management decided that the problem was not severe or likely enough to fix or even publish itIn the end $400 million costs of replacing faulty chips



NASA Mars Polar Lander, 1999 Multiple testing teams – one team tested contact switch that turn the landing thrusters off after landing, and another team tested that the leg opening mechanisms worked.When the legs snapped open the vibration caused the contact switch to turn off the thrusters 1800 feet above the Mars surface -> the Polar Lander smashed to the ground.

Patriot missile defense system, 1991System failed to defend against several Iraqi Scud missiles, one of which killed 28 U.S. soldiers.Analysis found a software bug – a small timing error accumulated to the point that after 14 hours of operation the tracking system was no longer accurate.



The Y2K Bug, early 1970sA programmer was programming a payroll system for his companyThe computer had very little memory for storage and the programmer decided to use only 2 digits to store years, because this would save him a lot of memory in the system that heavily relies on date processingHe considered briefly the problems that will occur when year hits 2000, but decided that this system will surely be replaced by then

It’s estimated that several hundred billion dollars were spent to replace or update this kind of systems to fix potential year 2000 failures


Why do defects occur in software?

Software is written by human beingsWho know something, but not everythingWho have skills, but aren’t perfectWho don’t usually use rigorous methodsWho do make mistakes (errors)

Under increasing pressure to deliver to strict deadlines

No time to check, assumptions may be wrongSystems may be incomplete

Software is complex, abstract and invisibleHard to understandHard to see if it is complete or working correctlyNo one person can fully understand large systemsNumerous external interfaces and dependencies


Sources of defects

EducationDevelopers does not understand well enough what he or she is doingLack of proper education leads to errors in specification, design, coding, and testing

CommunicationDevelopers do not know enoughInformation does not reach all stakeholdersInformation is lost

OversightOmitting to do necessary things

TranscriptionDeveloper knows what to do but simply makes a mistake

ProcessProcess is not applicable for the actual situationProcess places restrictions that cause errors


What do software faults cost?

Huge sumsAriane 5 ($7 billion)Mariner space probe to Venus ($250 million)American Airlines seat booking system ($50 million)Intel pentium floating point division bug ($400 million)Y2K (estimated several hundred billion dollars worldwide)

Very little or nothing at allminor inconvenienceno visible or physical detrimental impact

Software is not ”linear”small input may have very large effect

”The national annual costs of an inadequate infrastructure for software testing is estimated to range from $22.2 to $59.5 billion.” NIST Final Report: The Economic Impacts of Inadequate Infrastructure for Software Testing. 2002


Safety-critical systems

Software faults can cause death or injuryRadiation treatment kills patients (Therac-25)Aircraft crashes Industrial process control softwareHardware controllers…


Example from Finland

”Helsingin terveysviraston tietojärjestelmä pysähtyi taas

Odotusajat venyivät, lääkärit pelkäävät potilasturvallisuudenvaarantuvan.Helsingin terveysviraston potilastietojärjestelmä saatiintoimimaan keskiviikkoaamuna, mutta jo iltapäivällä se pysähtyitaas. Uuden Pegasos-tietojärjestelmän toimintavaikeudetalkoivat viime torstaina ja tämän viikon alussa se jähmettyikokonaan. Tietojärjestelmän ja tietokannan toimittajan asiantuntijatkorjasivat vikaa koko tiistai-illan, ja keskiviikkoaamuna alkoijärjestelmän vähittäinen käyttöönotto. ”

Helsingin Sanomat 6.2.2003


What is Software Testing?


What is software testing?

Finding defectsTrying to break the systemFinding and reporting defectsDemonstrating correct functionalityDemonstrating incorrect functionalityDemonstrating robustness, reliability, security, maintainability, …Measuring performance, reliability, …Evaluating and measuring qualityProving the software correctExecuting pre-defined test casesAutomatic error detection…


The growth of software testing

- 1956 Debugging oriented periodTesting was not distinguished from debugging

1957 - 1978 Demonstration oriented periodGoals to demonstrate that software satisfies its specification

1979 - 1982 Destruction oriented periodGoal to detect implementation faults

1983 - 1987 Evaluation oriented periodGoal to detect requirements, design, and implementation faults

1988 - Prevention oriented periodGoal to prevent requirements, design, and implementation faults

(Gelperin and Hetzel. 1988. The Growth of Software Testing. Communications of the ACM)


Definition of Software Testing(Glenford Myers 1979)

Testing is the execution of programs with the intent of finding defects.


Definition of Software Testing(Ilene Burnstein 2002)

Testing is a the process of exercising a software componentusing a selected set of test cases,with the intent of revealing defects and evaluating quality.

(Testing can be described as a process used for revealing defects in software, and for establishing that the software has attained a specified degree of quality with respect to selected attributes.)


Definition of Software Testing(Pol & Veenendal 2002)

Testing is a process of planning, preparation, execution and analysing, aimed at establishing the characteristics of an information systemand demonstrating the difference between actual and required status.


Definition of Software Testing(Cem Kaner, 2004)

Software testing is a technical investigation of a product, i.e.an empirical searchfor quality-related informationof value to a project’s stakeholders


Context-driven school of testing

1. The value of any practice depends on its context. 2. There are good practices in context, but there are no best

practices. 3. People, working together, are the most important part of

any project's context. 4. Projects unfold over time in ways that are often not

predictable. 5. The product is a solution. If the problem isn't solved, the

product doesn't work. 6. Good software testing is a challenging intellectual

process. 7. Only through judgment and skill, exercised cooperatively

throughout the entire project, are we able to do the right things at the right times to effectively test our products.

http://www.context-driven-testing.com/


Testing is an integral part of development

Often, testing is seen as some separate, last phase of software development process

That can be outsourced to separate testing team That only deeds to be done just before the release – if there is any time

Testing can not be separated from rest of the software development

Testing is much more than the final acceptance testing phase

Testing has to be involved from the beginningTesters can, and should, contribute in each phase of the software development life-cycle

Testing – is not a phase


Reputation of testing and testers

Traditionally testers and their job have not been respected

Bad programmers and poor designers are put into testing teamThese people do bad job in testing -> their job is not respected

Testing is very creative work that requires as skilled and experienced professionals as any job, in order to be done wellActually, good testing requires probably more skill and experience than programming – the skills, however, are different


We need professional testers

Developers can’t find their own defectsThey have no motivation for testingThey have no skills for testingThey have no time for testing

Testers have the tester’s mindsetGoal is to find defects – break the softwareGet the defects fixed together with the developersTesters are independent verifiers, they can test the software from the customer’s or user’s viewpointTesters are professional – they have the skills, tools, and experience


Some skills of a good tester

Tester’s (destructive) attitude and mindsetExcellent communication skillsAbility to manage many detailsKnow the different testing techniques and strategies and know when and how to apply themBe familiar with the domain the software is targetedComprehensive knowledge of the software engineering discipline

Knowledge and experience of how software is specified, designed, and developedKnowledge of fault and failure types


Are testers destructive?

Finding defects is a destructive taskRequires destructive attitude and mindset

Testers’ goal is to help the development team develop high quality software

The ultimate goal is to construct software with required level of qualityTesters job is not to be destructive

Destructive attitude towards software during testing tasks – constructive, when reporting, towards people and the development project


Software Quality and Quality Assurance


Describe a high quality software system

Defect freeHigh sales figuresFast to learnNice graphicsEasy to install and upgradeRobustAdaptableCheapSimpleSmall and compactExtensibleOpen-sourceFit-for-purposeFast upgrade cycleOpen accessMaintainableAppeals majority

Feature richAward winning productFast to useLow system requirementsReliable customer supportCoolCompatibleExpensiveBig and complicatedLarge and scalableStandardizedReliable vendorMultipurposeNever crashesSecureFast time-to-marketPerfect for target segment


Definition of software quality

Quality: (1) The degree to which a system, component or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.

(IEEE Standard Glossary of Software Engineering Terminology [IEEE610.12])

Quality is value to some person(s)(Gerald M. Weinberg, 1992, “Quality Software Management”)


Users’ viewpoint differs from yours

All stakeholders have different viewpoints to qualityCustomerEnd userProgrammerProject managerTester

Note: bug free product can be unacceptable to the user

The goal of a software system is to solve customers problemRequirements should be validated Usability needs to be consideredDocumentation should be testedCustomer feedback process needs to be working


Process quality and product quality

Quality in process -> quality in productNumerous models, but has not been proven to be always true

Project: instantiated process

Quality according to ISO 9126Process quality contributes to improving product quality, which in turn contributes to improving quality in use

Process

Project


The McCall quality model

Product TransitionProduct Revision

Product Operations

PortabilityReusability

Interoperability

CorrectnessReliabilityEfficiencyIntegrityUsability

MaintainabilityFlexibilityTestability


ISO 9126 Quality characteristics

ISO/IEC9126

Functionality

ReliabilityPortability

Maintainability

Efficiency

UsabilityThe capability tobe understood,

learned, used andattractive to user.

The capability to provide appropriateperformance, relative to amount of

resources used, under stated conditions.

The capability tobe modified.

The capability to betransferred from one

environment toanother.

The capability to provide functionsthat meet stated or implied needs.

The capabilityto maintain a

specified levelof performance.


ISO 9126 Quality attributes

FunctionalitySuitabilityAccuratenessInteroperabilitySecurity

ReliabilityMaturityFault toleranceRecoverability

UsabilityUnderstandabilityLearnabilityOperabilityAttractiveness

EfficiencyTime behaviorResource behavior

MaintainabilityAnalyzabilityChangeabilityStabilityTestability

PortabilityAdaptabilityInstallabilityConformanceReplaceability


Focus on different quality attributes in testing

Traditionally emphasis on functionalityGrown emphasis on

UsabilityEfficiencyScalabilitySecurity

Everything has to be accessible from public webInternal quality attributes important for products with long life cycle

MaintainabilityPortabilityTestability…


Good Enough Quality

Defined by James Bach IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98.

To claim that any given thing is good enough is to agree with all of the following propositions:

It has sufficient benefitsIt has no critical problemsThe benefits sufficiently outweigh the problemsIn the present situation, and all things considered, further improvement would be more harmful than helpful


Quality Assurance

Application of sound technical methods and toolsFormal technical reviews and inspectionsSoftware testingEnforcement of standardsDocumentationControl of changeExtensive measurementRecord keeping and reporting of the process

QA vs. TestingDifferent attitudeQA is focused to build in quality and prevent defects to ever happen

constructiveTesting is focused to find defects and show the level of quality

destructive

(1) A planned and systematic pattern of all actions necessary toprovide adequate confidence that an item or product conforms to established technical requirements.(2) A set of activities designed to evaluate the process by which products are developed or manufactured.


Quality Assurance 1. A planned and systematic pattern of all actions

necessary to provide adequate confidence that an item or product conforms to established technical requirements.

2. A set of activities designed to evaluate the process by which products are developed or manufactured.

Application of sound technical methods and toolsFormal technical reviews and inspectionsSoftware testingEnforcement of standardsDocumentationControl of changeExtensive measurementRecord keeping and reporting of the process


QA vs. Testing

Different attitudeQA is focused to build in quality and prevent defects to ever happen

Constructive

Testing is focused to find defects and show the level of quality

Destructive

Testing can be seen as a part of QAQA includes

Measuring and trackingProcess improvementAssessing quality and reacting to variances

Documents

Software Testing Fundamentals part I