Software Testing: ISTQB Foundation Level CertificationWhat is Software Testing?

Software Testing is the process of finding defects in software and reporting them to the developers. Software testing only identifies the bugs in software and does not fix them.

What are the principles in software testing?

1. Testing shows the presence of bugs2. Early Testing3. Exhaustive Testing is impossible4. Testing is context dependent5. Defect clustering6. Pesticide Paradox7. Absence of Errors is a fallacy

What is the fundamental test process?

Test Planning and control Test analysis and design Test Implementation and execution Evaluating exit criteria and reporting Test closure activities

What are an error, defect and failure?

Error leads to a defect, which can cause an observed failure.

Resources triangle consist of Time, Money and quality (vertices last two)

What is defect clustering?

Any small module or functionality may contain more number of defects – concentrate more testing on this functionality

What is pesticide paradox?

If prepared test cases are not finding defects, add/revise test cases to find more defects.

What is debugging?

The process undertaken by developers to identify the cause of bugs in code and undertake corrective actions

What is static testing and dynamic testing?

Static testing deals with testing where code is not exercised. Dynamic testing exercises the program under test with some kind of test data so we speak of test execution in this context.

What is exhaustive testing?

The test approach in which all the possible data combinations are used which also includes the implicit data combinations present in the state of the software/data at the start of testing.

Note: Various reasons for defect clustering include volatile code, system complexity, development staff inexperience, Development staff experience, the effects of change upon change

Test Planning and Control

Planning and control is determining what is going to be tested. It is where we define test completion criteria.

Control on the other is what to do when the results do not match with plans.

Progress against plan

Testing analysis and Design

Analysis and design is concerned with fine detail of what is going to be tested. Test conditions

Combining the test conditions into test cases, so that a small number of cases can cover a large number of conditions

Design part requires considering the test data that will be required for the test conditions and test cases that have been drawn up

- Reviewing requirements, architecture, design, interface specifications and other parts, which collectively comprise the test basis.

- Analyzing test items, the specification, behavior and structure to identify test conditions and test data required.

- Designing the tests, including assigning priority.

- Determining whether the requirements and the system are testable.

- Detailing what the test environment should look like, and whether there are any infrastructure and tools required.

- Highlighting the test data required for the test conditions and test cases.

- Creating bi-directional traceability between test basis and test cases.

Test Implementation and execution

Test environment is checked before testing

Most important tests are run first

- Developing and prioritizing test cases, creating test data, writing test procedures and, optionally, preparing test harnesses and writing automated test scripts.

- Collecting test cases into test suites, where tests can be run one after another for efficiency.

- Checking the test environment set-up is correct.

- Running test cases in the determined order. This can be manually or using test execution tools.

- Keeping a log of testing activities, including the outcome (pass/fail) and the versions of software, data, tools and testware (scripts, etc.).

- Comparing actual results with expected results.

- Reporting discrepancies as incidents with as much information as possible, including if possible causal analysis (code defect, incorrect test specification, test data error or test execution error).

- Where necessary, repeating test activities when changes have been made following incidents raised. This includes re-execution of a test that previously failed in order to confirm a fix (retesting), execution of a corrected test and execution of previously passed tests to check that defects have not been introduced (regression testing).

Evaluating exit criteria and Reporting

Determining if more tests are needed

Writing up the test results for the clients

Test Closure activities

Ensuring that the documentation is in order

Closing down and archiving the test environment, test infrastructure

Passing over testware to the maintenance team

Noting down methods to improve the testing process also called as testing maturity

Note: Independence – Developer – Same team member – Other team member – third party member

What is bidirectional traceability? 

Bidirectional traceability needs to be implemented both forward and backward (i.e., from requirements to end products and from end product back to requirements).

When the requirements are managed well, traceability can be established from the source requirement to its lower level requirements and from the lower level requirements back to their source. Such bidirectional traceability helps determine that all source requirements have been completely addressed and that all lower level requirements can be traced to a valid source.

What is test harness?

Test Harness is configuring a set of tools and test data to test an application in various conditions, which involves monitoring the output with expected output for correctness. The benefits of Testing Harness are: Productivity increase due to process automation and increase in product quality

What is retesting and regression testing?

Retesting is testing of module once the identified bug has been fixed by the development team. Regression testing is testing the entire system if the change in software has caused any defects in other modules of the software.

Note: System testing should investigate both functional and non-functional requirements of the system.

SDLC – Life Cycles – Chapter Two

Development life cycle models include:

1. Requirements Specification2. Functional Specification3. Technical Specification4. Program Specification5. Coding6. Testing

Waterfall Model – also called as linear sequential model. Testing is carried out once the code has been fully developed.

Checks throughout the lifecycle include:

Verification: Checks that work product meets the requirements set out for it

Validation: Changes the focus of work product evaluation to evaluation against user needs

V – Model – also called as sequential development model

1. Requirements Specification – Acceptance Testing2. Functional Specification – System Testing3. Technical Specification – Integration Testing4. Program Specification – Unit Testing5. Coding

This ensures that the defects can be found as early as possible in product life cycle.

If the user requirements gets changed these issues may not be covered until the user testing is carried out

Iterative – Incremental development models

Spiral model where in the requirements design code test are all in a circular shape

The users are involved in this type development during testing which minimizes the risk of developing an unsatisfied product

Lack of proper documentation makes it difficult to test

Test driven development is used to counter this where the functional tests are first written and code is then created and tested.

Traceability is thus reduced as the project progresses.

Type of iterative development model includes RAD, Agile Software development, Rational Unified Process.

Unit testing – units are also called programs, modules or components.

Thus the test bases for integration testing can include: the software and system design; a diagram of the system architecture; workflows and use-cases.

The test objects would essentially be the interface code. This can include subsystems’ database implementations.

Big Bang Integration – All the components are linked at once

Top Down Integration – Components call other components Stubs are built in case components are yet built. A stub is a passive component, called by other components.

Bottom up integration - When these special components call other components, they are called drivers. They are so called because, in the functioning program, they are active, controlling other components.

System testing checks the end to end functionality of the system.

Functional Requirement and Non – Functional Requirement

A functional requirement is a requirement that specifies a function that a system or system component must perform. Functional requirements can be specific to a system.

Non-functional system testing looks at those aspects that are important but not directly related to what functions the system performs.

Non – Functional Requirements include installability, maintainability, Performance, Load handling, Stress Handling, Portability, Recovery, Reliability, Recovery, Reliability and Usability

System Testing: Test bases for system testing can include: system and software requirement specifications; use cases; functional specifications; risk analysis reports; and system, user and operation manuals. The test object will generally be the system under test.

Acceptance testing: The test bases can include: user requirements; system requirements; use cases; business processes; and risk analysis reports.

The test objects can include: the fully integrated system; forms and reports produced by the system.

Acceptance Testing types include UAT Testing called as User Acceptance Testing; Operational Acceptance Testing; Contract and Regulation acceptance Testing; Alpha and Beta Testing

Operational Acceptance Testing involves checking:

Back-up facilities Procedures for disaster recovery Training for end users Maintenance procedures Data load and migration tasks Security procedures

TEST TYPES

Functional Testing; Non-Functional Testing; Structural Testing; Testing after the code has been changed

To facilitate different types of testing, models may be used as follows:

Functional testing: process flows; state transition models; security threat models; plain language specifications.

Non-functional testing: performance model; usability model.

Structural testing: control flow model; menu structure model.

Security Testing and Inter-operability testing are functional testing types. Functional testing also called as specification based testing.

Behavioral aspects of the system are tested in non-functional testing.

For non-functional testing only black box testing techniques are used.

ISO-9126 – Software Engineering Product Quality

To measure how much testing is carried out structural testing is used. To check the thoroughness of the testing carried out on the system that has actually been built. A common measure is to look at how much of the actual code that has been written has been tested.

Testing related to changes

Retesting is also called as confirmation testing

Regression testing should be carried out if the test environment has been changed

Regression testing is suitable for automation testing as the tests are repeated whenever there is a change in the software code.

MAINTAINANCE TESTING

Once the system is release it may be required to make changes. Changes may be due to:

Additional features being required. The system being migrated to a new operating platform. The system being retired – data may need to be migrated or archived. Planned upgrade to COTS-based systems. New faults being found requiring fixing (these can be ‘hot fixes’)

Testing that takes place on a system that is live or in production is called as maintenance testing.

Impact analysis is analyzing the impact of the changes on the system. This would be difficult on the system that has been released as the specifications may be out of date or the development might have moved.

Sample paper Note: V –Model includes the verification of designs

Oracle assumption is that the tester can routinely identify the correct outcome of a test

We estimate the amount of Retesting required based on Metrics from previous similar projects and discussions with the development team

Test Execution provides the biggest potential cost saving from the use of CAST

Data flow analysis studies the use of data on paths through the code.

Static Testing

Static testing is the process of testing the software without executing the code.

Static testing includes testing of specifications and requirements, testing the code without actually executing it.

Reviews and Static analysis

A review is a systematic examination of a document by one or more people with the aim of finding and removing errors.

Benefits of finding defects early in the life cycle include:

Development productivity can be improved and time scales reduced

Testing costs and time can be reduced by removing the main delays in test execution

Reduction in lifetime costs

Improved communication results

Types of defects found through reviews include

Deviations from standards

Requirements Defects

Design Defects

Insufficient Maintainability

Incorrect Interface Specifications

Review objectives include

Finding Defects

Gaining Understanding

Generating Discussion

Decision making by consensus

Activities of a formal Review includes

Planning; selecting the personnel; allocating roles; defining entry and exit criteria; selecting the parts of the documents to be reviewed.

Planning; Kick off Checking entry criteria; Individual Preparation; Noting Incidents; Review Meeting; Examine; Rework; Fixing Defects; Follow-up; Checking exit criteria

Kick off involves distributing the documents, explaining the objectives, process and documents to the participants

Review Meeting approach is based in the time available; Requirements of the author; Type of review

Roles and Responsibilities

Manager – Decides what is to be reviewed; ensures sufficient time is allocated in the project plan for all the required review activities; and determines if the review objectives have been met; normally they do not get involved in the review process.

Moderator- known as the review leader; leads the review of the documents, planning the review, running the meeting, follow ups after the meeting, mediates between various points of view; makes the final decision whether to release an updated document

Author- writer or person with chief responsibility for the development of the documents to be reviewed; in most cases takes responsibility for fixing any agreed defects

Reviewers- individuals with specific technical or business knowledge who after necessary preparation identify and describe findings in the product under review

Scribe- documents all the issues and defects, problems and open points that were identified during the meeting

Types of Review

Level of formality – Low to high

Informal Review: Review may be documented but not required as many informal reviews are not documented. Purpose is to find defects and is an inexpensive way to achieve some limited benefit; review may be implemented by pair programming.

Walkthrough: Meeting is led by the author of the document under review; preparation by reviewers before the walkthrough meeting, list of findings, main purpose is to enable learning about the content of the document under review and to explore scenarios to find defects

Technical Review: These are documented and use a well-defined defect detection process that includes peers and technical experts; performed as a peer review without the management participation and is ideally led by a trained moderator who is not the author; reviewers prepare for the meeting with review report and list of findings; discussion, decision making, evaluating alternatives, finding defects, solving technical problems and checking conformance to specifications and standards.

Inspection: They are led by a trained moderator who is not the author and usually involve peer examination of a document; it is formal based on rules and checklists and uses entry and exit criteria; Pre-meeting preparation is necessary to ensure consistency; An inspection report with list of findings is prepared which includes metrics that can be used to aid improvements to the process as well as correcting defects in the document under review; formal follow up process is used to ensure that corrective action is completed and timely; the main purpose is to find defects and process improvements is a secondary purpose.

Informal

Walkthrough

Technical Review

Inspection

Success Factors for reviews

Should have pre-defined and clear agreed objective.

Any defects found are welcomed and expressed objectively

Checklists should be used

Management support is quite essential for a good review process

Emphasis should be on learning and process improvements

Static Analysis by tools

Static analysis looks for defects without executing the code. They can be used once the code has been written.

The value of static analysis is early detection of defects prior to test execution; early warning about suspicious aspects of the code or design, by the calculation of metrics, such as high complexity measure; Identification of defects not easily found by dynamic testing; improved maintainability of the code and design; Prevention of defects.

Typical defects found by static analysis tools include:

Referencing a variable with an undefined value Inconsistent interface between modules and components Variables that are never used Unreachable code Programming standard violations Security vulnerabilities Syntax violations of code and software models

Static analysis tools add greatest value when used during component and integration testing.

Developers use static analysis tools

Note: The process starting with terminal modules is called bottom-up integration

The inputs for developing a test plan are taken from project plan

Tests are prioritized so that you do the best testing in the time available

Error guessing is not a static testing technique

Data flow analysis, walkthroughs, inspections are static testing techniques

Test Design Techniques

Test development process

The design of tests comprises three main steps

Identify test conditions Specify test cases Specify test procedures

A test condition – a characteristic of software which we can check with a test or set of tests

A test case – a set of input values, execution preconditions, expected results and execution post conditions, developed for a particular objective or test condition

A test procedure specification – a sequence of actions for the execution of a test

A good test case should be traceable back to the test condition and the element of specification that it is testing

Test coverage is important because it provides a quantitative assessment of the extent and quality of testing by measuring what has been achieved.

It provides a way of estimating how much more testing needs to be done. Using quantitative measures we can set targets for test coverage and measure progress against them.

Categories of Test case Design Techniques

Black box techniques

White box test design techniques

Experienced based techniques also called as ad-hoc testing