Foundation Study Guide

8/8/2019 Foundation Study Guide

1/75

Universalexams Learning Course

Foundation in Software Testing

Study Guide

Course Overview

The course is designed in modules, each one containing sections covering the majortopics of software testing. Questions covering the topics learned in the precedingmodule are provided to ensure the student has retained the required information.

The student will also be provided with practice exams to ensure that the knowledge has beentaken on board.

Copyright 2006 www.universalexams.com

Any unauthorised copying or re-selling of the contents of this document withoutpermission will constitute an infringement of copyright.

Commercial copying, re-selling, hiring, lending is strictly prohibited.


2/75

WWW.UNIVERSALEXAMS.COM

Course Index

Module 1 Fundamentals of Testing

Why is Testing necessary?What is Testing?General Testing PrinciplesThe Fundamental Test ProcessThe Psychology of TestingModule 1 Review Questions

Module 2 Testing Throughout the Software Lifecycle

Why Test Early?Software Development ModelsTest LevelsTest Types: the targets of testing

Maintenance TestingModule 2 Review Questions

Module 3 Static Techniques

Reviews and the Test ProcessReview ProcessStatic AnalysisModule 3 Review Questions

Module 4 Test Design Techniques

Identifying Test Conditions and Designing Test CasesCategories of Test Design TechniquesSpecification-based or Black-box TechniquesStructure-based or white-box techniquesExperienced-based TechniquesChoosing Test TechniquesModule 4 Review Questions

Module 5 Test Management

Test Organization & IndependenceTest PlanningTest Estimation

Test Process Monitoring and ControlConfiguration ManagementRisk and TestingIncident ManagementModule 5 Review Questions

Module 6 Tool Support for Testing

Test Tool ClassificationTypes of Test ToolsIntroducing a Tool into an OrganizationModule 6 Review Questions

Page 2



3/75


1 Fundamentals of Testing

Why is Testing Necessary?

According to news reports in April of 2004, a software bug was determined to be a majorcontributor to the 2003 Northeast blackout, the worst power system failure in North Americanhistory. The failure involved loss of electrical power to 50 million customers, forced shutdownof 100 power plants, and economic losses estimated at $6 billion. The bug was reportedly inone utility company's vendor-supplied power monitoring and management system, which wasunable to correctly handle and report on an unusual confluence of initially localized events.The error was found and corrected after examining millions of lines of code.

Unfortunately, examples such as the one above are still commonplace today. Even thesmallest mistake in a program can cause devastating results.

In April of 2003 it was announced that the largest student loan company in the U.S. made asoftware error in calculating the monthly payments on 800,000 loans. Although borrowerswere to be notified of an increase in their required payments, the company will still reportedlylose $8 million in interest. The error was uncovered when borrowers began reportinginconsistencies in their bills.

When a programmer writes hundreds, maybe thousands of lines of code, we couldnt possiblyexpect it to be 100% correct. This could be due to several factors, including:

Poor communication

Programming errors Changing requirements Software complexity

All of the above factors have human involvement, and humans make mistakes. The chancesof mistakes can also increase when pressures such as tight deadlines are present.

Through the concept of testing we can try to detect the mistake, and ensure it is rectifiedbefore the software product is released. Note the use of the word try in the last sentence.Software testing is actually a process used to identify the correctness and quality ofdeveloped software. Testing can never establish the correctness of software, as this can only

be done by formal verification (and then only when the verification process is faultless).

In July 2004 newspapers reported that a new government welfare management system inCanada costing several hundred million dollars was unable to handle a simple benefits rateincrease after being put into live operation. Reportedly the original contract allowed for only 6weeks of acceptance testing and the system was never tested for its ability to handle a rateincrease.

Testing can only find defects, not prove that there are none!

Page 3



4/75


Lets start looking at the terminology used in software testing in relation to mistakes in aprogram, as there are subtle differences which are worth knowing:

Error: A human action that produces an incorrect result

Fault: A manifestation of an error in software. A fault, if encountered may cause a failure

Failure: Deviation of the software from its expected delivery or service.

Imagine a scenario where a software program prints out a warning message when the coretemperature reaches its critical value. An explosion in the reactor could occur if no action istaken when the temperature reaches its critical value.

Program Extract:

$temperature = $temperature + $inputif $tempereture > 100 then

Print Reactor Core Temperature Critical !!!Else Print Reactor Core Temperature Normal

End:

Did you notice the spelling mistake?

If this program was actually being used, then the output would always be Reactor Core

Temperature Normal. This is because there is no such variable of $tempereture. This couldpossibly result in an explosion of the reactor, as no one would see the warning message. Wecan use the definitions previously described of an error, fault and failure with this example.

The errorwould be; the misspelling of the variable $temperature.

The fault would be; the correct variable $temperature is never used.

The failure would be; the fact that the warning message is never displayed when required.

How much testing is enough?

From what we discussed so far, it is obvious that software testing has an important part toplay in the lifecycle of the development of the software product. So why dont we testeverything in the program?

The answer is simple......We dont have enough time!

There is in fact a software testing process that does attempt to test everything. This is calledExhaustive Testing. The purpose of Exhaustive Testing is to execute a program with allpossible combinations of inputs or values for program variables.

Page 4



5/75


For example, imagine a simple program that displayed a page which contained ten inputboxes. Each input box would accept any one of ten values. You would require a test case foreach and every permutation of the entries that could be entered e.g.:10(input boxes) to the 10(values)th power

10 to the 10th power

= 10,000,000,000 test cases

Lets now look at that example a little further in practical terms. If it took only one second toperform each test case, it would take approximately 317 years to complete. So if we cant testeverything, then how do we choose what to test?

The answer is simplyRisk!

Risk and Testing

How much testing should we perform if any failures could potentially cause loss of life?

How much testing should we perform if any failures were negligible?

The amount of risk based on testing/not testing software components will often dictate howmuch time will be allocated for testing. Using the above examples, then obviously if a failurecould cause loss of life, then the amount of testing required would intensify based on thispotential risk. Testing is commonly focused on specific areas of the software product that

have the greatest potential risk of failure. Additionally, priority can be assigned to the testcases that have been chosen to be performed. This priority is often helpful when the timeallocated to testing is limited. You can simply perform the high priority test cases first, andthen any remaining time can be used to perform any low priority test cases.

Page 5


Plenty of testingtime available

Test all relevantAreas

Perform all testcases

Time isLimited

Test only highrisk areas

Perform highpriority tests first


6/75


Quality and Testing

Once the faults have been rectified, then we can say that the quality of the software producthas increased. The testing term Quality can be thought of as an overall term, as the qualityof the software product is dependant upon many factors.

In general, quality software should be reasonably bug free, delivered on time and within theoriginal budget. But, often there may be additional quality requirements from a number ofdifferent origins, such as; the customers acceptance testing, future maintenance engineers,sales people etc. and all of those mentioned may have a different view on quality.

Testing can be used as a measurement of quality!

What is testing?

To many people, when you mention the word testing, they immediately think of performing

tests. But performing the actual tests is only one of the testing activities. Testing activities

actually exist before and after the activity of performing tests. For example, planning,

reporting, reviewing and test design are all activities associated with testing. The purpose of

testing can range from, finding defects, gaining confidence and preventing defects. When

planning and designing the testing, we must take into consideration the purpose of why we

are testing in order to produce meaningful results.

It is also important to consider designing the tests as early in the project lifecycle as possible.

This helps prevent defects from being introduced into the software code before it is actuallywritten. A similar result can be achieved from reviewing documents that will be used to create

the code.

Testing objectives can also be strongly influenced by the originating environment. For

example; if you were working within a development environment, you would probably find that

the testing would be focused on finding as many defects as possible. This would be to ensure

that any problems could be fixed very early on in the development lifecycle. If you were

performing User Acceptance Testing, then the focus would be on ensuring the quality of the

product. This would provide a level of confidence in the product prior to its actual release.

A successful test is one that finds a fault

That may sound strange, as at first thought you might think that a successful test phase wouldbe one that finds no problems at all, especially when a fault is found it generally causesdelays in the product development. But if you consider that if a fault is found during testingand rectified before it is released, it could save a fortune compare to a released softwareproduct containing faults being used by a customer, and the problems that could bring.

Page 6



7/75


General Testing Principles

In order to assist a Tester, there does actually exist some general testing principles, orguidelines. These are merely suggestions, and are open to interpretation, but you often comeacross them in various testing environments, and may find them useful in givencircumstances.

Principle 1 Testing shows presence of defects

If a defect exists within a piece of software, then testing can show the defect exists. Buttesting cannot prove that no defects are present in the piece of software. What testing canprovide is decrease the probability of defects existing in the software. In simple terms; if thetests show up no defects, it doesnt mean they are not present.

Principle 2 Exhaustive testing is impossible

As we have discussed in a previous chapter, it is impossible for us to test everything. This

type of testing is called Exhaustive Testing. An alternative is to use Risk based Testing.

Principle 3 Early Testing

The earlier on in the development lifecycle testing starts, the better. By testing early we canensure the development requirements can actually be tested, and also influence the way thedevelopment proceeds.

Principle 4 Defect Clustering

By being aware of defect clustering, we can ensure that testing is focused in those areas that

contain the most defects.

Principle 5 Pesticide paradox

If we ran the same tests over and over again, we would probably find the amount of newdefects found would decrease. To avoid this, the tests should be regularly reviewed to ensureall expected areas of functionality are covered.

Principle 6 Testing is context dependant

Depending on the item being developed, the way the testing is carried out will often differ. Forexample an air traffic control system will undoubtedly be tested in a different way to achildrens story book program.

Principle 7 Absence of errors fallacy

There is no point in developing and testing an item of software, only for the end user to rejectit on the grounds that it does not do what was required of it.

Page 7



8/75


The Fundamental Test Process

Introduction

In order to perform effective testing, the testing must be planned. Once the testing has been

planned, it is also important to adhere to it. A common pitfall among testers is to create a

good test plan, but then not follow it correctly. We already know that it is impossible to

completely test everything, but with careful planning that includes the right selection of tests

and the way that they are tested, we can effectively test the software product to a high

standard.

Although the following diagram displays a logical sequence, this is by no means rigid and is

often adapted to suit individual testing requirements.

Page 8



9/75


Test Planning & Control

Page 9


Begin

Test Planning& Control

Test Analysis& design

Test Closure

Activities

End

Test Planning activities include definitionand specification of testing objectives, andverifying what and why we are testing. TestControl involves ensuring the testing is

adhering to the Test Plan.

Test Analysis involves transforming testobjectives into test designs. Test case

design reviews and consideration of testenvironments also form part of this stage.

This stage involves transforming the testdesigns into actual test cases, followed byexecuting them. The test environment willalso be set-up and used here.

The test outcomes shall be recorded in thisstage. It should be possible to establish

that the specified testing objectives havebeen met, and this forms the Exit Criteria.Test summaries can also be created in thisstage.

This final stage will check that all planned

deliverables have been met. A handover toa maintenance organisation may occurhere. It should also be possible to analyse

the testing to ensure that any problemsseen can be avoided in future testing ofsimilar products.

Fundamental Test Process

TestImplementation &

Execution

Evaluating ExitCriteria &Reporting


10/75


Test Planning basically involves determining what is going to be tested, why it is going to betested, and how it is going to be tested. It is also important to clarify what is not going to betested in the software product too. Here are some examples of Test Planning activities:

Determining Risks and objectives of testingOrganization of the equipment and people involved

Determining the approach to testingEnsuring any policies are adhered toOrganization of timescales for design, execution and evaluation stagesSpecifying the Exit Criteria

In order to meet the objectives of the testing, it is very important not only to have good plans,but also to ensure that they are adhered to. Test Control will help to achieve this, as itspurpose is to ensure that the ongoing testing progress is compared to the Test Plan. Thefollowing list shows example activities of Test Control:

Analyzing the results and feedbackPerforming actions to correct mistakes or changes

Decision makingConsistent monitoring and documentation of testing activities

Test Analysis & Design

Test objectives can come from a variety of sources. This will commonly take the form of arequirements list or specification. Once the requirements are clearly understood, it should bepossible to design tests or conditions based upon these requirements. The following listcontains examples of Test Analysis & Design activities:

Identification of conditions or tests based upon the provided requirementsDesigning the Test CasesEnsuring that all requirements can actually be testedEvaluating required tools and equipment to allow designing a test environment or setup.Review of test basis, involving requirements, architecture, design and interfaces.

Test Implementation and Execution

This stage is where the actual testing is performed. It can mean running the test casesmanually or by use of an automated testing tool. Before this can happen though, the TestCase designs need to be made into actual Test Cases. The following are all considered to beTest Implementation and Execution tasks:

Creating the Test CasesCreating any required Test DataWriting any automated Test ScriptsForming Test Suites from multiple Test CasesEnsuring that the Test Environment is setup correctlyExecution of Test CasesStoring the Results in a meaningful formatComparison of the actual results with the expected resultsReporting of any test failuresRegression Testing and repetition of tests to confirm fixe

Evaluating Exit Criteria and Reporting

Page 10



11/75


This stage is designed to ensure that any specified Exit Criteria has been met by the

performed testing activities. The Exit Criteria should have been previously specified in the

Test Planning stage. The stored Test Results can be checked in this stage against the Exit

Criteria. If the Exit Criteria has not been met, then more tests may be required, or even

changes to the Exit Criteria may be recommended. This is a good stage to create a Test

Summary. The Test Summary can be used by any interested parties to quickly ascertain thestatus of testing completeness and outcome, leading to a level of confidence in the product.

Test Closure Activities

This stage is concerned with collecting test results and test related documentation in order to

achieve a milestone prior to a release of the product. Any defects found during the testing

should have been fixed and verified fixed at this stage. An important note here is that it is rare

that all defects would have been fixed within a software development project. It is common

place for defect fixes to be deferred for future developments. This is predominantly due totime constraints. A formal handover to another department or even customer may happen in

this stage. As with most development projects there are always problems encountered, here

is a good stage to evaluate those and attempt to learn any lessons for future developments.

The Psychology of Testing

Page 11



12/75


One of the primary purposes of software tester is to find faults with software. This can oftenbe perceived as destructive to the product development lifecycle. Whereas, the purpose of adeveloper is a more creative one. This in some circumstances naturally causes frictionbetween developers and testers.

A Developer will often spend long hours working on a piece of software, sometimes for manymonths. They may take great pride in their piece of work. Then a tester comes along andfinds fault with it. You can quickly see where friction might emerge from!

A Developers attributes:

Highly valued within the companyIndustry standard qualificationsSeen as being creativePoor communicatorsSkills in a very specific area

A Testers attributes:

Rarely valued within a company

No industry standard qualificationsSeen as being destructive

Very good communicators

Multi-talented

It is often more effective for someone other than the Developer themselves to test the

product, and this is called Independent Testing. There are in fact different levels of

Page 12


Tester

Developer

A Developers perspective


13/75


independence used in testing, with the bottom of the list being the most independent:

Test cases are designed by the person(s) writing the software.

Test cases are designed by another person(s).

Test cases are designed by a person(s) from a different section.

Test cases are designed by a person(s) from a different organization. Test cases are not chosen by a person.

Developer Vs Tester

Good communication is essential on any project, whether it is verbal or through

documentation. The sooner the problem is understood by all parties, the sooner the problem

can be resolved. This is particularly the case when it comes to the relationship between a

Developer and a Tester. The way in which a Tester approaches a Developer with a problem

is important to the progression of the project. The following example is how NOT to approacha Developer:

Tester: Hey, monkey brains, I found another bug in your software

You can imagine the Developers response to that remark. How about the following more

tactful approach:

Tester: Hi, I seem be to getting some strange results when running my test. Would you mind

taking a look at my setup, just in case I have configured it incorrectly?

You and I both know that your setup is correct, but at least this way you are implying that theproblem could possibly be with your work and not the Developers. When the Developer sees

the test fail for himself, he will probably explain what he thinks is going wrong, and you will

now be in a better position to work together to resolve the problem.

Module 1 Review Questions

Page 13



14/75


1) Describe what the following terms mean:

Error

Fault

Failure

2) What are the stages of the Fundamental Test Process?

3) Try and list as many Developer and Tester attributes as you can.

4) What is termed a successful test?

5) Which stage of the Fundamental Test Process would you expect to findCollecting the test results?

2 Testing Throughout the Software Lifecycle

Page 14



15/75


Why Test Early?

So by now we know why we test software, but can we put a price on its worthwhile?

We obviously cant put a figure in monetary terms or as a percentage of potential profits. Butwe can see that the cost of a fault is dependant on when in the development lifecycle it wasfound.

Imagine a software product is released without any testing at all. Customers may find faultswhich affects their own business. The customer would never buy software products from thedevelopment company again, word would spread quickly and before long the softwaredevelopment company goes out of business. A dramatic example, but it gets the point acrossthat spending some of the development cost of testing could save money in the long run. Thecost of testing is nearly always lower than the cost of releasing a poor quality product.

As a general rule of thumb, the earlier on in the development lifecycle the fault is found, thecheaper it is to rectify.

The following table shows some examples:

When fault is found Additional work Potential Effect

Prior to testing

Faults generally found at thisstage will be documentationbased. Faults with TestSpecifications and Designdocs can pretty much beeliminated by an effectivereview process.

If the faults are not found atthis stage, the developmentwould go ahead. This couldin itself create additionalfaults.

Just prior to products release

A fault found at this stagewould probably be a softwarefault. There could be manyreasons for this, but effectivetesting leading up to thisstage should prevent thisfrom occurring.

When the fault is eventuallyfound, software re-work, re-design and additional testingwould be required wasting aconsiderable amount of time,possibly delaying theproducts release.

Found by a customer

If the customer finds the fault,additional manpower wouldbe required to resolve theproblem. Involving additional

development work probablyresulting in a patch beingcreated.

If the fault occurs after theproduct has been released,the potential cost of the fault

could be devastating.

Verification & Validation

Page 15



16/75


A common basis for testing in todays modern business environment is using a software work

product. Common software work products include CMMI(Capability Maturity Model

Integration) and the IEEEs Software Life Cycle Process. During the development of the

software work products, processes such as Verification and Validation can be carried out.

Verification and Validation is often referred to as V & V. Software Validation and Verification

can involve analysis, reviewing, demonstrating or testing of all software developments. Whenimplementing this model, we must be sure that everything is verified. This will include the

development process and the development product itself. Verification and Validation should

be carried out at the end of the development lifecycle (after all software developing is

complete).

Verification would normally involve meetings and reviews and to evaluate the documents,

plans, requirements and specifications. This can be achieved by using reviews and meetings

etc.

Validation involves the actual testing. This should take place after verification phase has been

completed.

Verification and Validation, if implemented correctly can be very cost-effective if planned

correctly.

Summary:

Verification: Are we building the product right?

Validation: Are we building the right product?

Software Development Models

Several models currently exist in relation to a Systems Development Life Cycle (SDLC).

When referring to SDLC we simply mean the activities of the software development and

maintenance. Lets look some of the most common models for testing in use today:

V & V

V Model

Spiral

RAD

Waterfall Model

Page 16



17/75


The Waterfall model is also known as the Sequential model. Each stage follows on from the

previous one. The testing is performed in block as the last stage.

Using the above example, we can see that from a testing point of view, it is simply too little,

too late. Planning or Test creation is not considered until the actual software code has been

written. This can result in problems being found much later in the project lifecycle than is

desirable.

V-Model

Page 17


RequirementsSpecification

FunctionalSpecification

TechnicalSpecification

ComponentDesign

SoftwareCoding

Testing


18/75


The V-Model is an industry standard framework that shows clearly the software development

lifecycle in relation to testing. It also highlights the fact that the testing is just as important as

the software development itself. As you can see from the diagram below, the relationships

between development and testing are clearly defined.

You will often see different names for each of the software development stages in a V-Model.You may also see more or fewer stages, as this is dependant on the individual product are

more commonly, dependant on the individual COTS (Commercial Off The Shelf) software

product.

Looking at the above diagram, we can not only see the addition of the kind of testing activities

that we would expect to be present. But also, we can see how each testing activity ties in with

each development phase, thus verification of the design phases is included. This V-Modelimproves the presence of the testing activities to display a more balanced approach.

Spiral Model

Page 18


RequirementsSpecification

FunctionalSpecification

TechnicalSpecification

ComponentDesign

SoftwareCoding

ComponentTesting

IntegrationTesting

SystemsTesting

AcceptanceTesting


19/75


The Spiral model is an incremental testing approach to both Development and Testing. This isused most effectively when the users do not know all of the requirements. From what isknown, initial requirements can be defined. Then from these the code and test cases arecreated. As time goes on, more details of the requirements are known and implemented infurther iterations of design, coding and testing phases. The system is considered to becomplete, when enough of the iterations have taken place.

RAD

RAD represents Rapid Application Development. In order to implement a RAD development,all of the requirements must be known in advance. But wit RAD the requirements are formallydocumented. Each requirement is categorised into individual components. Then eachcomponent is developed and tested in parallel. All this is done in a set period of time. RAD isconsidered to be an iterative development model.

DSDM:

DSDM represents Dynamic Systems Development Methodology. It was developed with aview to organise and control RAD developments, as RAD has very little, if any formaldocumentation.

Test Levels

Page 19



20/75


Component Testing

Component testing is also known as Unit, Module, or Program Testing. In simple terms, thistype of testing focuses simply on testing of the individual components themselves. It iscommon for component testing to be carried out by the Developer of the software. Thishowever has a very low rating of testing independence. A better approach would be to use

what we call the buddy system. This simply means that two Developers test each otherswork giving a high rating of independence. In order to effectively test the components,Developers often use Stubs and Drivers.

Stubs and Drivers:

If we need to test a low level module, then something called a driver can be used. A driver

is a high level routine that will call lower level sub-programs. The way in which a driver works

is to pass input data to the item under test and compare the output to the truth.

In order to test a high level module of a program, then we should use something called a

stub. A stub actually works by taking the place of a real routine. This saves the complexity of

actually having the real routines configured, and can efficiently show whether or not the item

under test is actually interfacing with the routine as expected, without having the real routine

present.

Example:

Suppose you have a program that plays MP3 files from a stored MP3 collection on the users

computer. The user enters the title of the artist they want to listen to, a procedure then locates

the folder containing the MP3s by that artists name, then it informs the MP3 player program

to play the MP3 files in that folder.

So what happens if the wrong MP3s are played?

You could check that the procedure that locates the artists folder is correct by creating a test

driver that simply asks for the artists name and then prints out the location of the folder

where it thinks the MP3s are stored. This will effectively check the lookup part of the

program.

If you think that the lookup part of the program may not be getting the correct input, i.e. the

artists name you entered is not being received correctly. Then you can replace part of the

program with a stub. Then when its called it could display something like:

Where are MP3s for artists named XYZ located?

You would then enter the correct folder location ABC.

The program would then send this information to the MP3 player.

The above stub would show you that the XYZ was actually the artist name you originally

entered, and if the MP3 file location was correctly passed to the MP3 player.Integration Testing

Page 20



21/75


Integration Testing is commonly termed:

Testing performed to expose faults in the interfaces and in the interaction between integratedcomponents.

In simple terms, Integration testing is basically placing the item under test in an environment

where it interacts with other complete systems. The main purpose of Integration Testing is tofind faults in the way the item under test carries out these interactions. Integration Testing canbe thought of as two separate entities which are Component Integration testing and SystemsIntegration Testing.

Component Integration Testing:

This type of Integration testing is concerned with ensuring the interactions between thesoftware components behave as expected. It is commonly performed after any ComponentTesting has completed.

System Integration Testing:

This type of Integration Testing is concerned with ensuring the interactions between systemsbehave as expected. It is commonly performed after any Systems Testing has completed.Typically not all systems referenced in the testing are controlled by the developingorganization. Some systems maybe controlled by other organizations, but interface directlywith the system under test.

The greater the amount of functionality involved within a single integration phase, then theharder it will be to track down exactly what has gone wrong when a problem is found. Itmakes good sense to increment the amount of functionality in a structured manner. This waywhen a problem arises, you will already have a rough idea of where the problem may be. Try

and avoid waiting for all components to be ready and integrating everything at the end. As thiswill normally result in defects being found late in the project, and potentially a great deal ofwork pin-pointing the problem, followed of course re-development and re-testing.

Top-Down:

Page 21



22/75


The controlling module is tested by itself, followed by testing the modules that are called bythe controlling module. This process is repeated until all modules are tested.

Bottom-Up:

Firstly, each module at the lowest level of the system hierarchy is tested, followed by testingthe modules that called the previously tested ones. This process is repeated until all modulesare tested.

Sandwich:

Page 22


L


23/75


This is basically a combination of Top-Down with Bottom-Up Incremental Testing. The middlelayer is initially targeted for testing, followed by the top and bottom layers.

Big-Bang:

This involves testing all individual modules at once as a complete system. Can be ideal for asmall system, but for large systems it is not recommended as if an error is found, it can bedifficult to track down to an individual module.

Incremental Test Strategy Comparisons

Top-Down Bottom-Up Sandwich Big-BangIntegration Early Early Early LateDevelopment Lifecycle Early Late Early LateRequires Drivers No Yes Yes YesRequires Stubs Yes No Yes Yes

Systems Testing

Page 23



24/75


Systems Testing is defined as:

The process of testing an integrated system to verify that it meets specified requirements.

Systems Testing can be thought of as two separate entities which are Functional and Non-Functional.

Requirements-based Functional Testing

Requirements-based Testing is simply testing the functionality of the software/system basedon the requirements. The tests themselves should be derived from the documented

requirements and not based on the software code itself. This method of functional testingensures that the users will be getting what they want, as the requirements document basicallyspecifies what the user has asked for.

In simple terms, Functional Systems Testing focuses on what the system is actuallysupposed to do. So how do we know exactly what it is supposed to do? It is defined in what isknown as a functional requirement.

The IEEE defines functional requirement as:

A requirement that specifies a function that a system component must perform.

An example of a requirement may be:

The system must process the user input and print out a report

Business Process Functional Testing

Different types of users may use the developed software in different ways. These ways areanalysed and business scenarios are then created. User profiles are often used in Business

Process Functional Testing. Remember that all of the functionality should be tested for, notjust the most commonly used areas.

Page 24



25/75


Non-Functional Systems Testing

Non-Functional Systems Testing is related to testing areas of the system functionality thatare not directly related to the functionality of the system. It is important to remember that non-functional requirements are just as important as functional requirements.

The following are all considered to be non-functional areas of Systems Testing:

Load Testing

Testing the ability of the system to be able to bear loads. An example would be testing that asystem could process a specified amount of transactions within a specified time period. Soyou are effectively loading the system up to a high level, then ensuring it can still functioncorrectly whilst under this heavy load.

Performance Testing

A program/system may have requirements to meet certain levels of performance. For aprogram, this could be the speed of which it can process a given task. For a networkingdevice, it could mean the throughput of network traffic rate. Often, Performance Testing isdesigned to be negative, i.e. prove that the system does not meet its required level of

performance.

Stress Testing

Stress Testing simply means putting the system under stress. The testing is not normallycarried out over a long period, as this would effectively be a form of duration testing. Imaginea system was designed to process a maximum of 1000 transactions in an hour. A stress testwould be seeing if the systems could actually cope with that many transactions in a given timeperiod. A useful test in this case would be to see how the system copes when asked toprocess more than 1000.

Security Testing

A major requirement in todays software/systems is security, particularly with the internetrevolution. Security testing is focused at finding loopholes in the programs security checks. Acommon approach is to create test cases based on known problems from a similar program,and test these against the program under test.

Useability Testing

Page 25



26/75


This is where consideration is taken into account of how the user will use the product. It iscommon for considerable resources to be spent on defining exactly what the customerrequires and simple it is to use the program to achieve there aims. For example; test casescould be created based on the Graphical User Interface, to see how easy it would be to use inrelation to a typical customer scenario.

Storage Testing

This type of testing may focus on the actual memory used by a program or system undercertain conditions. Also disk space used by the program/system could also be a factor. Thesefactors may actually come from a requirement, and should be approached from a negativetesting point of view.

Volume Testing

Volume Testing is a form of Systems Testing. It primary focus is to concentrate on testing thesystems while subjected to heavy volumes of data. Testing should be approached from anegative point of view to show that the program/system cannot operate correctly when usingthe volume of data specified in the requirements.

Installability Testing

A complicated program may also have a complicated installation process. Considerationshould be made as to whether the program will be installed by a customer or an installationengineer. Customer installations commonly use some kind of automated installation program.This would obviously have to under go significant testing in itself, as an incorrect installation

procedure could render the target machine/system useless.

Documentation Testing

Documentation in todays environment can take several forms, as the documentation could bea printed document, an integral help file or even a web page. Depending of thedocumentation media type, some example areas to focus on could be, spelling, usability,technical accuracy etc.

Recovery Testing

Recovery Testing is normally carried out by using test cases based on specific requirements.A system may be designed to fail under a given scenario, for example if attacked by amalicious user; the program/system may have been designed to shut down. Recovery testingshould focus on how the system handles the failure and how it handles the recovery process.

Page 26



27/75


28/75


Alpha Testing should be performed at the developers site, and predominantly performed byinternal testers only. Often, other company department personnel can act as testers. Themarketing or sales departments are often chosen for this purpose.

Beta Testing is commonly performed at the customers site, and normally carried out by thecustomers themselves. Potential customers are often eager to trial a new product or new

software version. This allows the customer to see any improvements at first hand andascertain whether or not it satisfies their requirements. On the flip side, it gives invaluablefeedback to the developer, often at little or no cost.

Test Types: the targets of testing

What we mean by a test type is focusing on a specific test objective. This could be a specificaction that the software is expected to do, or verifying that a defect has been fixed. Verifyingdefects have been fixed is also called Confirmation Testing. Common terms in todayssoftware development lifecycles are white box testing and black box testing. It is importantto fully understand what these terms mean.

Functional Testing

In simple terms a function is basically what the system or piece of software actually does.The functions that a system or a piece of software is required to perform can be tested byreferring to items such as a work product or a requirements specification. Functional testingis based on the external behaviour of the system and is referred to as Black-box testing.When referring to Black-box testing, we mean that knowledge of the internal workings of thesystem are not known to the tester. This method is commonly used for testing functionalrequirements, and often used in Systems Testing and Acceptance Testing by a dedicatedTester. This method can effectively be used throughout the lifecycle of the productdevelopment.

Example:

A software program designed to calculate mortgage repayments.

Test that the program does not accept invalid entries Test that the program does accept valid entries Test that the program calculates the correct results

Non-Functional Testing

While Functional Testing concentrates on what the system does, Non-functional Testingconcentrates on how the system works. Some examples of this type of testing are; stresstesting, reliability testing, interoperability testing and performance testing. This form of testingcan be performed at all stages of the development and include objectives such as responsetimes.

Example:

A software program designed to calculate mortgage repayments.

Test that the program does not crash

Page 28



29/75


Test that the program can process results within a set time period Test that the program can be upgraded

Structural Testing

Structural Testing requires some knowledge of the inside of the box to design the test cases.This can be particularly useful for creating any required test data, as you would have accessto how the data will actually be processed. Structural Testing is also known as White-boxtesting. It is commonly performed by Developers and is used predominantly for Unit Testingand Sub-system Testing. As the development lifecycle continues; the effectiveness of White-box testing decreases. This method is a prime candidate for automated test tools, and caneffectively increase the quality of the developed work. Glass-box testing is also a term in usetoday, and is a synonym for White Box Testing.

Coverage is also an important part of Structural Testing. Coverage is basically the amount ofcode that is covered by the structural tests. The aim of Structural Testing is to achieve 100%coverage. Tools are available to assist in calculating coverage.

Page 29


?

Black-Box Testing

Object under test

Performed by aDedicated tester

Systems TestingUser Acceptance Testing

Effective throughoutWhole Lifecycle

111100101010110011

00101010110111011010000111011010011011010001011101010110

1111110101000100101010101010101011100000101101100101

White-Box Testing

Object under test

Performed bySoftware Developer

Unit TestingSub-systems Testing

Effective towardsStart of project

Functional Structural


30/75


Testing Related to Change

It is imperative that when a fault is fixed it is re-tested to ensure the fault has indeed beencorrectly fixed. This is termed Confirmation Testing.

There are many tools used in a test environment today that allow a priority to be assigned to afault when it is initially logged. You can use this priority again when it comes to verifying a fix

for a fault, particularly when it comes to deciding how much time to take over verifying the fix.For example if you are verifying that a typo has been fixed in a help file, it would probablyhave been raised as a low priority fault. So you can quickly come to the conclusion that itwould probably only take a few minutes to actually verify the fault has been fixed. If, howevera high priority fault was initially raised that wiped all of the customers stored data, then youwould want to make sure that sufficient time was allocated to make absolutely sure that thefault was fixed. It is important that consideration of the possible consequences of the fault notbeing fixed properly is considered during verification.

Another important factor when it comes to testing is when there is suspicion that the modifiedsoftware could affect other areas of software functionality. For example, if there was anoriginal fault of a field on a user input form not accepting data. Then not only should you focuson re-testing that field, you should also consider checking that other functionality on the form

has not been adversely affected. This is called Regression Testing.

For example; there may be a sub-total box that may use the data in the field in question forits calculation. That is just one example; the main point is not to focus specifically on the fixeditem, but to also consider the effects on related areas. If you had a complete TestSpecification for a software product, you may decide to completely re-run all of the test cases,but often sufficient time is not available to do this. So what you can do is cherry-pick relevanttest cases that cover all of the main features of the software with a view to prove existingfunctionality has not been adversely affected. This would effectively form a Regression Test.

Regression test cases are often combined to form a Regression Test suite. This can then beran against any software that has undergone modification with an aim of providing confidencein the overall state of the software. Common practice is to automate Regression Tests.

To assist you on what to additionally look for when Confirmation Testing, it is always a goodidea to communicate with the Developer who created the fix. They are in a good position totell you how the fix has been implemented, and it is much easier to test something when youhave an understanding of what changes have been made.

Page 30



31/75


Maintenance Testing

Maintenance Testing involves testing updates to existing software. For this type of testing thesupporting documentation plays an important role. You would probably need to look at theoriginal design documentation for the software prior to the new update. This documentationcould be quite old, and may not conform to the same standard that you are used to.Effectively it could easily be a poor quality document and may not provide enough suitable

information to derive test cases from. In a worse case scenario, there may not be any originaldesign documents at all.

An important part of Maintenance Testing is Regression Testing. This is because any updateto an existing system or software could adversely affect any other existing functionality. Todetermine how much Regression we should perform, we should consider the items that havebeen updated or added, and their level of risk to the functionality of the rest of the system.This is often referred to as Impact Analysis.

When determining Impact Analysis, the following points should be considered when it comesto deciding how much testing (regression) should be carried out.

What exactly are the changes?

Which existing functionality could be affected by the changes?What are the potential risks?Who is currently using the System? (in case of a required reboot etc.)Should I back-up any part of the current system?

Page 31



32/75



1) What does RAD represent?

2) Which type of testing can be termed Testing performed to expose faults in theinterfaces and in the interaction between integrated components.?

3) Which type of Testing checks the ability of the system to be able to bear loads?

4) Which Software Development Model is also known as the Sequential Model?

5) Which type of Integration Testing involves testing all individual modules at onceas a complete system?

Page 32



33/75


3 Static Techniques

Reviews and the Review Process

So what actually is a review?

Review: A process or meeting during which a work product, or set of work products, ispresented to project personnel, managers, users or other interested parties for comment orapproval. [IEEE]

It is important to remember that any document can be reviewed. For example requirementspecifications, design documents, test specifications etc. can, and are reviewed. It is commonknowledge that reviews are cost effective. The actual cost of an on-going review process isconsidered to be approximately 15% of the development budget. This may at first sound quiteconsiderable, but compared to not performing reviews and the associated risk of producingproducts containing errors, it is obvious of the advantages that a review process can bring.

These advantages include development productivity improvements, reduced amounts ofproduct development time, and above all, a reduction in the amount of faults.

Reviews commonly find errors that are not possible to be detected by regular testing.Reviews also provide a form of training, including technical and standards related, for everyparticipant.

From testing point of view, we can use reviews to allow ourselves to be involved much earlieron in the development lifecycle. Obviously at the beginning of the project there is nothing wecan physically test. But what we can do is be involved in the review process of variousdocuments. For example, we could get involved in the review process of the productrequirements. From our involvement at this very early stage of the development lifecycle, wewould have an idea of what will be required to be tested. This would give us a head-start on

thinking about how we might approach testing the requirements too.

Goals

The goals of a review should be to validate and verify the item under review againstspecifications and standards, with an aim to achieve consensus.

Pitfalls

Page 33



34/75


The pitfalls of a review could be a lack of training, insufficient documentation, and a lack ofsupport from Management.

Review Process

You wont be surprised to hear that there is more than one type of review. Each review typehas its own characteristics, and if used in the right way can provide effective reviewprocesses in many development lifecycles.

Lets start by looking at the roles and the responsibilities of the participants of the reviews.

Manager

The Manager will be the person who makes the decision to hold the review. Managingpeoples time with respect to the review is also a Managers responsibility.

Moderator

The Moderator effectively has overall control and responsibility of the review. They will

schedule the review, control the review, and ensure any actions from the review are carriedout successfully. Training may be required in order to carry out the role of Moderatorsuccessfully.

Author

The Author is the person who has created the item to be reviewed. The Author may also beasked questions in the review.

Reviewer

The reviewers are the attendees of the review who are attempting to find errors in the itemunder review. They should come from different perspectives in order to provide a wellbalanced review of the item.

Scribe

The Scribe (or Recorder) is the person who is responsible for documenting issues raisedduring the process of the review meeting.

Page 34



35/75


Formal Review Process

Page 35



36/75


Informal Review

Page 36


Planning

Kick-off

Preparation

Meeting

Rework

Follow-up

Exit Criteria

There should be awareness of any company policies, product requirements and/or project plans that mayprovide specific requirements in order for a review to

take place. Definition of Entry & Exit criteria,personnel selection are also done at this stage.

Normally a week before the planned review, the

Moderator ensures that the item is ready for reviewand distributed. The Moderator also briefs the

attendees on their roles and responsibilities.

All review participants examine the item to be

reviewed. If standards are used, then checks fordeviation form the standards should be used.

Comparison to similar items can also be used.

The review normally lasts between 1 2 hours. All

items on the agenda/checklist are worked through. Findings are noted by the recorder. Comments are

aimed at the review item not the author.

The Moderator and Scribe document the findings in a

Review Summary and report it to the Manager. TheReview Summary will contain defects found and actions

of follow-up work to carried out.

The Moderator ensures that all additional work by the

author is checked for completeness and correctness. Anadditional review may be required; dependant on the

amount/complexity of re-work undertaken.

Exit Criteria can take the form of ensuring that all

actions are completed, or that any uncorrected itemsare properly documented, possibly in a defect tracking

system.


37/75


An informal review is an extremely popular choice early on in the development lifecycle ofboth software and documentation. Often, pieces of work in software product development canbe lengthy, whether its a piece of software or a detailed Test Specification. You dont want topresent your completed piece of work at a formal review, only to find that you have completelymisunderstood the requirements and wasted the last two months work. Think of starting ajourney at point A with an aim of arriving at point Z when your piece of work is complete.

This is where informal reviews can be invaluable. Why not have an informal review at pointB?

If, for example you are working on creating a detailed test specification which you know willtake several weeks to complete. You have just completed the first section, and you arethinking should I continue writing the rest of the specification in the same way? Then, now isthe perfect time for an informal review.

You can then ascertain whether you are travelling in the right direction. Maybe take onadditional suggestions to incorporate in the remaining work. The review is commonlyperformed by peer or someone with relevant experience, and should be informal and brief.

Summary:

Low cost No formal process No documentation required Widely used review

Technical Review

A Technical Review (also known as a peer review), is considered to be a formal type ofreview, even though no Managers are expected to attend. It involves a structured encounter,in which a peer/s analyse the work with a view to improve the quality of the original work.

The actual review itself is driven by checklists. These checklists are normally derived from thesoftware requirements and provide a procedure to the review. If the piece of work is softwarecode, the reviewer will read the code, and may even develop and run some unit tests to checkthat the code works as advertised.

The documentation from the outcome of the review can provide invaluable information to theauthor relating to defects. On the other side of the fence, it also provides information to peerson how the development is being implemented. The status of the product also be obtained byManagers from this type of review.

Summary:

Ideally led by the Moderator Attended by peers / technical experts Documentation is required No Management presence Decision making Solving technical problems

Walkthrough

Page 37



38/75


A walkthrough is a set of procedures and techniques designed for a peer group, lead by theauthor to review software code. It is considered to be a fairly informal type of review. Thewalkthrough takes the form a meeting, normally between one and two hours in length. It isrecommended that between three to five people attend. The defined roles for the walkthroughattendees would be a Moderator, a Scribe and a Tester. As to who actually attends can varybased upon availability, but a suggested list from who to pick from would be:

High experienced programmer Programming language expert Low experienced programmer Future maintenance engineer of the software Someone external to the project Peer programmer from same project

It is important for the participants of the walkthrough to have access to the materials that willbe discussed several days prior to the meeting. This gives essential time to read throughavailable documentation in order to digest it and make some notes, basically prepare them forthe meeting.

When the walkthrough starts, the person acting as the Tester provides some scenario testcases. These test cases should include a representative set of input data and also theexpected output from the program. The test data is then walked through the logic of theprogram. The test cases themselves simply assist in the generating of dialog betweenmembers of the walkthrough. In effect, the test cases should provoke the attendees intoraising questions directed towards the program itself. The aim of the walkthrough is not to findfault in the programmer but in the program itself.

Summary:

Led by the Author Attended by a peer group Varying level of formality Knowledge gathering Defect finding

Page 38



39/75


Inspection

An inspection is a formal type of review. It requires preparation on the part the review teammembers before the inspection meeting takes place. A person will be in charge of theinspection process, making sure the process is adhered to correctly. This person is called aModerator. The Moderator is normally a technical person by nature and may have Quality

Assurance experience. It is also suggested that the Moderator comes from an unrelatedproject. This is to ensure an unbiased approach, and prevent a conflict of interests.

The Moderator will be responsible for arranging the inspection meeting and inviting theattendees. An agenda will be sent out by the Moderator to the attendees containing achecklist of items to be dealt with at the inspection meeting.

At the inspection meeting the producer of the item to be review will present it to the meetingattendees. As the item is presented, items on the checklist will be addressed accordingly.Someone will be assigned the task of documenting any findings, known as the Scribe.Inspection metrics will also play a part during the meeting. These are basically a set ofmeasurements taken from the inspection in order to assist with quality prediction, andpreventing defects in the future.

When all checklist items have been addressed, the inspection meeting naturally draws to aclose. At this stage a Summary Report will be created basically outlining what has happenedduring the inspection, and what is to be done next. A follow-up stage is also a requirement ofthe inspection. This ensures that any re-working is carried out correctly. Once anyoutstanding work has been completed and checked by a re-inspection or just by theModerator, the inspection will be considered to be complete.

Summary:

Led by a Moderator

Attended by specified roles Metrics are included Formal process Entry and Exit Criteria Defect finding

Guidelines for a Successful Review

Have an objective for the review Ensure you invite the right people Views should be expressed positively to the Author Apply tried and tested review techniques All actions are documented clearly

Page 39



40/75


Training is available in reviewing techniques if required

Static Analysis

Static Analysis is a set of methods designed to analyse software code in an effort to establishit is correct, prior to actually running the software. As we already know, the earlier we find a

fault the cheaper it is to fix. So by using Static Analysis, we can effectively test the programeven before it has been written. This would obviously only find a limited number of problems,but at least it is something that can be done very early on in the development lifecycle.

Some advantages of using Static Analysis are:

Finding defects before any tests are even run Early warning of unsatisfactory code design Finding dependency issues, such as bad links etc.

The types of errors that can be detected by Static Analysis are:

Unreachable code Uncalled functions Undeclared variables Programming standard violations Syntax errors

Static Analysis is commonly performed by automatic processes. A Compiler for example issaid to perform Static Analysis when it detects problems. An example of this would be syntaxerrors.

Control Flow Graphing

Control flow graphs display the logic structure of software. The flow of logic through theprogram is charted. It is normally used only by Developers as it is a very low level formtesting, often used in Component Testing.

It can be used to determine the number of test cases required to test the programs logic. Itcan also provide confidence that the detail of the logic in the code has been checked.

Page 40



41/75


Control Flow Graph Display

Here are some examples of some common control structures:

Page 41


Node

Node

Edges

Region

Node


42/75


Cyclomatic Complexity

Cyclomatic Complexity is a software metric that is used to measure the complexity of asoftware program. Once we know now how complex the program is, we then know how easyit will be to test.

The actual complexity is computed from a graph describing the control flow of the softwareprogram. The formula for calculating the Cyclomatic Complexity is:

C = E N + P

C = Cyclomatic ComplexityE = number of edgesN = number of nodesP = number of components

This software metric does have its limitations, for example there is no way of determining thedifference between the complexities of the individual decisions within the code.

You will probably come across variations of the above formula, but they will pretty muchproduce the same end result.

When looking at a piece of code, the Cyclomatic Complexity is the number of paths through it.For example, if there were no decisions in the code (i.e. no If statements or loops). Then theCyclomatic Complexity would be one, because it has a single path. If the piece of code had asingle If statement for example, then the Cyclomatic Complexity would be two, one for eachpath. A count of five or less means that the component module is fairly simple. A count ofbetween six and ten means that you should consider simplifying it. It is thought that a

Page 42


0

1

2

IF

0

1

3

IF ELSE

2

0

1

2

WHILE


43/75


component module containing a Cyclomatic Complexity count greater than ten is prone toerrors.

For unit testing, the count can be very useful, as it gives you the minimum number of testsyou need to fully exercise the code. That means to achieve 100% code coverage!


1) The _______ is the person who is responsible for documenting issues raised duringthe process of the review meeting.

2) Which type of review is also known as a peer review?

3) What types of errors can be found by Static Analysis?

4) Which type of graph displays the logic structure of software?

5) Which type of review requires no documentation?

Page 43



44/75


For all question answers, see the Answer Sheet provided.

4 Test Design Techniques

Identifying Test Conditions and Designing Test Cases

When considering how to design test cases, we need to ensure we have appropriatesupporting documentation, such as the test basis documentation. The next step is to analysethis documentation to determine each test condition. A test condition is basically somethingthat can be verified by a test case, for example; a function or an element. A good approach isto consider traceability when designing Test Cases. We can achieve this in a number of ways,for example; referencing a requirement within the Test Case itself. This gives any reader ofthe Test Case a good idea of why the Test Case has been written, and an idea of what it istrying to achieve.

The specification of a Test Case should also clearly state the expected result. The order ofTest Cases should also be a consideration, and should be displayed in executable order. Thisis termed the Test Procedure. When referring to automated testing, the sequence of actionswill be specified within a Test Script.

The IEEE 829 document describes some guidelines of what to include when designing TestCases. Examples of what type of information to include within a Test Case are:

The exact input values that will be input and the values of any standing data that isrequired.

The exact output values and changes of value of the internal system state that areexpected.

And any special steps for setting up the tests.

Page 44



45/75


Expected Results

One of the most important aspects of any kind of testing is the expected results. If we do notknow what the expected result is liable to be prior to testing, then we cannot really saywhether the test has passed or failed. Often expected results are based on results fromprevious iterations of the test cases, this is a good way of defining expected results as we candirectly compare new results against old results.

When new test cases are created, and we are not entirely sure of the expected result, a goodplace to look would be in the design documentation related to the new software/feature.

Further information could be obtained from the design documentation author.

The way in which the expected result is actually worded is also very important. You need tobe as specific as possible, and describe exactly what is expected to happen. This is to ensurethe expected result cannot be misinterpreted and to prevent confusion.

Example of a bad expected result:

The sub-total on the graphical user interface is correct.

The above example is far too vague, as different testers may have a different idea of what iscorrect. Describe exactly want you want to be checked!

Example of a good expected result:

The sub-total displays a figure equal to the sum of all three input boxes to two decimalplaces.

Categories of Test Design TechniquesThe function of a good Test Design Technique is to identify the Test Conditions and TestCases to effectively test a product.

We commonly place Test Design Techniques in one of two categories; these are Black-boxand White-box techniques.

Sometimes we have techniques that do not clearly fit into one these categories. Althoughthese techniques have different, or overlapping qualities, you will find that the following list oftechniques will commonly be found associated with a specific category.

Page 45



46/75


Specification-based techniques ------------------------------ Black-box

Structure-based techniques ------------------------------ White-box

Experienced-based techniques ------------------------------ Black-box

Specification-based or Black-box Techniques

In this section we will focus on the techniques used by Specification-based or Black-boxTesting methodology.

Equivalence Partitioning

Imagine you were assigned the task of manually testing a software program that processedproduct orders. The following information is an extract of what is to be processed by theprogram:

Order Number: 0 1000Product code: A1 to A99, B1 to B99, C1 to C99

If you decided you wanted to test this thoroughly, it would take absolutely ages. As you wouldneed to enter every number between 0 and 1000 for the order number. You would also needto enter every possible item in the product code ranges. Imagine the combinations of all of thepossible entries.millions?

An alternative to the obviously unachievable goal of testing every possible value isEquivalence Partitioning. What this method allows you to do is effectively partition thepossible program inputs. For each of the above input fields, it should not matter which valuesare entered as long as they are within the correct range and of the correct type. This isbecause it should be handled they same way by the program (This would obviously need tobe clarified by the developer as it is an important assumption). For example, you could testthe order number by choosing just a few random values in the specified range. For theproduct code this time choose a small selection from each equivalent partition e.g A66, A91,B17, B59, C5, C77 etc.

Page 46



47/75


So the point of equivalence portioning is to reduce the amount of testing by choosing a smallselection of the possible values to be tested, as the program will handle them in the sameway.

Boundary Value Analysis

By the use of equivalence partitioning, a tester can perform effective testing without testingevery possible value. This method can be enhanced further by another method calledBoundary Value Analysis. After time, an experienced Tester will be often realise thatproblems can occur at the boundaries of the input and output spaces. When testing only asmall amount of possible values, the minimum and maximum possible values should beamongst the first items to be tested.

Using our previous example from Equivalence Partitioning. Lets now enhance the testing byadding some Boundary Value Analysis based testing.

Order Number: 0 1000Product Code: A1 to A99, B1 to B99, C1 to C99

For the order number, we would test 0 as the minimum value, and 1000 as the maximumvalue. So, those values are what the software program would expect. But lets approach thisin a more negative way. Lets add tests that are effectively out of range, i.e. -1 and 1001.This gives us confidence that that the range is clearly defined and functioning correctly.

To complete the testing scenario, we can add tests for the product code, i.e. A0, A100, andso on.

Decision Table Testing

One of the problems associated with Equivalence Partitioning is that it tests for a singlevalue. What we really need is a testing method that allows for testing combinations of values.

Cause-effect graphing can combine conditions and derive sets of test cases. This method isdisplayed a way similar to a logical circuit diagram.Creating the graph can be time consuming when there are complex operations involved.Once the graph is complete, a decision table must be created from it. From this decisiontable, the Tester can derive actual test cases. There does exist today, several tools toautomate the test case creation from the decision table.

State Transition Testing

This type of Black-box testing is based on the concept of states and finite-states, and isbased on the tester being able to view the softwares states, transition between states, and

what will trigger a state change. Test cases can then be designed to execute the statechanges.

Use Case Testing

This type of Testing is commonly based on typical scenarios from the receiver of thedeveloped product. Its purpose is to find defects by using the product in a similar way that itwill be used in the real world. For example, you may have a piece of software that processesbanking transactions. You may have tested that the software can process every expected

Page 47



48/75


type of transaction. But the customer may require that they process 10 transactions at once,so from that information we could create a Use Case Test.

Use Cases not describe the way that the product is likely to be used, but also describe theway that the processes flow through the system. Also referred to as scenarios, Use Cases area valuable way of finding defects associated with integration.

It is common for the Use Cases to form part of Acceptance Tests, and may have involvementfrom the customer or end user.

Structure-based or White-box Techniques

In this section we will focus on the techniques used by Structure-based or White-box Testingmethodology.

Statement Testing

This testing method involves using a model of the source code which identifies statements.These statements are the categorized as being either executable or non-executable. Inorder to use this method, the input to each component must be identified. Also, each testcase must be able to identify each individual statement. Lastly, the expected outcome of eachtest case must be clearly defined.

Branch Decision Testing

This test method uses a model of the source code which identifies individual decisions, andtheir outcomes. A decision is defined as being an executable statement containing its ownlogic. This logic may also have the capability to transfer control to another statement. Eachtest case is designed to exercise the decision outcomes. In order to use this method, the inputto each component must be identified. Also, each test case must be able to identify eachindividual decision. Lastly, the expected outcome of each test case must be clearly defined.

Data Flow Testing

Page 48



49/75


This White-box testing method is based on using a model of the source code to test theinteractions between individual parts of a component. The flow of data between these parts ofa component is tested along with the flow of control. In order to use this method, the input toeach component must be identified. Also, the locations of relevant variable definitions arerequired. The sub-paths of the flow of control should be provided along with the expectedoutcome of each test case.

Branch Condition TestingBranch Condition Testing uses a model of the source code, and identifies decisions based onindividualBoolean operands within each decision condition. A decision is defined as beingan executable statement containing its own logic. This logic may also have the capability totransfer control to another statement. The decision condition is a Boolean expression which isevaluated to determine the outcome of the decision. An example of a decision would be aloop in a program. In order to use this method, the input to each component must beidentified. Also, each test case must be able to identify each individual Boolean operandalong with its value. Lastly, the expected outcome of each test case must be clearly defined.

Branch Condition Combination Testing

Branch Condition Combination Testing uses a model of the source code, and identifiesdecisions based on combinationsof Boolean operands within decision conditions. A decisionis defined as being an executable statement containing its own logic. This logic may alsohave the capability to transfer control to another statement. The decision condition is aBoolean expression which is evaluated to determine the outcome of the decision. An exampleof a decision would be a loop in a program. In order to use this method, the input to eachcomponent must be identified. Also, each test case must be able to identify each individualBoolean operand along with its value. Lastly, the expected outcome of each test case mustbe clearly defined.

Modified Condition Decision Testing

Modified Condition Decision Testing uses a model of the source code, and identifiesdecisions based on individual Boolean operands within each decision condition. The testcases created using this method are designed to demonstrate that a Boolean operand withina decision condition can independently affect the outcome of the decision.A decision is defined as being an executable statement containing its own logic. This logicmay also have the capability to transfer control to another statement. The decision conditionis a Boolean expression which is evaluated to determine the outcome of the decision. Anexample of a decision would be a loop in a program. In order to use this method, the input toeach component must be identified. Also, each test case must be able to identify eachindividual Boolean operand along with its value. Lastly, the expected outcome of each testcase must be clearly defined.

LCSAJ Testing

LCSAJ (Linear Code Sequence and Jump) uses a model of the source code, and identifiescontrol flow jumps. A jump is where the flow of control does not pass to the next sequentialstatement. An LCSAJ is made up of the following three items:

Start of linear code sequence End of linear code sequence Target line to which flow of control is to be transferred

Page 49



50/75


In order to use this method, the input to each component must be identified. Also, each testcase must be able to identify each LCSAJ. Lastly, the expected outcome of each test casemust be clearly defined.

Experienced-based Techniques

In this section we will focus on the techniques used by Experienced-based testing, commonlyassociated with Black-box Testing methodology.

Why can one Tester find more errors than another Tester in the same piece of software?

More often than not this is down to a technique called Error Guessing. To be successful atError Guessing, a certain level of knowledge and experience is required. A Tester can thenmake an educated guess at where potential problems may arise. This could be based on theTesters experience with a previous iteration of the software, or just a level of knowledge inthat area of technology.

This test case design technique can be very effective at pin-pointing potential problem areasin software. It is often be used by creating a list of potential problem areas/scenarios, thenproducing a set of test cases from it. This approach can often find errors that would otherwisebe missed by a more structured testing approach.

An example of how to use the Error Guessing method would be to imagine you had asoftware program that accepted a ten digit customer code. The software was designed to onlyaccept numerical data.

Here are some example test case ideas that could be considered as Error Guessing:

Page 50



51/75


Input of a blank entry

Input of greater than ten digits

Input of mixture of numbers and letters

Input of identical customer codes

What we are effectively trying to do when designing Error Guessing test cases, is to thinkabout what could have been missed during the software design.

This testing approach should only be used to compliment an existing formal test method, andshould not be used on its own, as it cannot be considered a complete form of testingsoftware.

Exploratory Testing

This type of testing is normally governed by time. It consists of using tests based on a testchapter that contains test objectives. It is most effective when there are little or nospecifications available. It should only really be used to assist with, or compliment a moreformal approach. It can basically ensure that major functionality is working as expectedwithout fully testing it.

Random Testing

A Tester normally selects test input data from what is termed an input domain. RandomTesting is simply when the Tester selects data from the input domain randomly. In order forrandom testing to be effective, there are some important open questions to be considered:

Is the chosen random data sufficient to prove the module meets its specification whentested?

Should random data only come from within the input domain?

How many values should be tested?

As you can tell, there is little structure involved in Random Testing. In order to avoid dealingwith the above questions, a more structured Black-box Test Design could be implementedinstead. However, using a random approach could save valuable time and resources if usedin the right circumstances. There has been much debate over the effectiveness of usingrandom testing techniques over some of the more structured techniques. Most experts agreethat using random test data provides little chance of producing an effective test.

There are many tools available today that are capable of selecting random test data from aspecified data value range. This approach is especially useful when it comes to testsassociated at the system level. You often find in the real world that Random Testing is used

Page 51



52/75


in association with other structured techniques to provide a compromise between targetedtesting and testing ev

Documents

Foundation Study Guide