6

Click here to load reader

On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, [email protected] The testing process of a software-producing organization can be divided

Embed Size (px)

Citation preview

Page 1: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

On Software Testing Strategy

Jussi Kasurinen, [email protected]

The testing process of a software-producing organization can be divided to severallayers. For example, the draft of software test process standard ISO 29119 [1] calls theselayers test policy, test strategy, test management and testing work. By this division, theactual tasks related to the testing are located at the bottom layer of the model, havingmanagement, strategy and policy beyond them.

The division of tasks between the upper layers is the key to researching test processes.For example, the ISTQ-B glossary [2] of software engineering terms defines testmanagement, test policy and test strategy as follows:

test management, The planning, estimating, monitoring and control of test activities.Typically carried out by test manager.

test policy, A high level document describing the principles, approach and majorobjectives of the organization regarding testing.

test strategy, A high-level description of the test levels to be performed and the testingwithin those level for an organization or program (in one or more projects).

The relation between these terms is as follows; The policy level is the high abstractionlevel of testing, only defining broad principles, approach and objectives for the testingprocess. The testing strategy is the operative plan to enforce the policy, describing thetest levels, more commonly known as phases, which the test process applies and whatsort of activities are done within those phases. Finally, the management layer isresponsible for planning, monitoring and controlling the strategy-defined activities.

Based on the definition, the test strategy plays an interesting role on the total testprocess, as it has the opportunity to both define and interpret testing activities. Thepolicy level has no practical restrictions on their definition work, but it lacks the abilityto observe details and usually leaves practical solutions open for interpretation.Similarly in test management, the ability to observe details and react to the practicalissues is immediate, but the available actions are restricted by the strategy and policies,which dictate what the management process should be doing. Based on theseobservations, it would seem that the testing strategy is the key player in the testprocess. But what should the test strategy include, if it is such an important factor? Inshort: everything testing-related, which needs to be mentioned. For example, ISO 29119[1] proposes several issues that could be addressed in the test strategy, such as

Page 2: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

•Objectives of Testing•Scope of Testing•Test Methodology•Test Process Overview•Test Responsibilities•Degrees of Independence (of testing teams)•Structure of Test Organization•Tester Education Requirements•Test Process Improvement Methods•Applied Standards•Testing Lifecycle Model•Measurements for Value of Testing•Applied Test Types•Test Techniques•Test Levels•Entry & Exit Criteria•Test Completion Criteria•Retesting and Regression Testing•Test Selection/Priorization•Test Environments•Test Asset Reuse•Test Automation•Test Tools•Other generic risks that need to be addressed

In more general definition, the test strategy is an overview document which defines thetesting phase of the software development process, and is used to inform the testprocess stakeholders of the key issues and activities. The Test Maturity ModelIntegration (TMMi) [3] also discusses the importance of test strategy, even offeringpractical approach on defining such document:

Page 3: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

1. Study test policy and set goals for testing2. Provide feedback and clarify issues in (1) in necessary3. Define strategy which is clearly linked to either policies or goals with

following topics addressed:§ Generals risks of the developed product§ Overall test model (V, incremental, prototyping)§ Test levels (unit, integration, system… )§ Test case design techniques applied on each level§ Test types carried out on each level§ Entry- and Exit-criteria for each level§ Standards that must be complied with§ Level of Independence (for testing teams)§ Environment (and testing tools) in which the tests are executed§ Test automation for each level§ Regression testing for each level.

4. Review the strategy with stakeholders5. Establish ownership and responsibilities related to the strategy6. Revise the strategy based on the feedback and observations if necessary,

e.g. annually.

The recommendations for stakeholders which should be consulted in the test strategydefinition vary between sources, but for example ISO [1] lists Head of IT, test managers,project managers and development managers as the focus groups. In some specificaspects, senior programmers, senior testers and other interest groups should also beconsulted [3].

Overall, the test strategy is a broad topic and covering all the related aspects couldbecome exhausting. However, the TMMi definition for strategy has 11 bullet points,covering the most typical aspects related to the test process as itself. In relation to theother layers, the test strategy has to comply with test policies, but can clearlydemarcate several secondary aspects outside. Aspects such as economical situation,availability of acquirable resources, marketing, market pressure, corporate levelbusiness strategies and interaction between organizations can be presented asabstractions, mentioned in the “general risks” or “other generic risks” and dealt with inthe policy level. On the other hand, secondary issues in details such as personal issueswith human resources, operability issues with tools (power outages, hardwarecomplications) and communication difficulties caused by infrastructure (mobile serviceavailability, software client conflicts) can be dealt with in the management level.

Page 4: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

Based on these observations, it seems plausible to argue that the test strategy shoulddefine following answers in all test-related software product phases: “What should betested in different phases, by whom, by which method, by what tools, in what kind ofenvironment.” By generalizing from the TMMi and ISO29119 overlaps, it would seemthat the test strategy is meant to define at least following aspects:

•testing tools (including regression and automation)•testing methods (including regression and automation)•testing personnel (responsibilities and levels of independence in the organization)•test case design and -selection methods•quality criteria (for entry- and exit criteria, including standards that has to be complied

with)

for each phase of the testing, in accordance to the test policy, defined by the uppercorporate level management, while leaving the implementation and practical details tothe test management level. One online article [4] summarizes the test strategy evenfurther, deducting that the strategy is a roadmap that answers the questions how,where, who and when. It does not have to be complete, unchangeable monolith, butrather a book or best practices and common guidelines.

As for the research approach in software test strategy, the impact of different aspectsshould be investigated. The strategy offers six major areas of interest which are tools,methods, personnel, case selection, quality and standards and the upper managementrole. One feasible approach would be to analyze the practical impact of different areasof interests (AoI) to the test process, and determine how the test strategy differs indifferent types of organizations. Especially, is there a way to determine suitable teststrategies for certain types of organizations, or is there a distinguishable group of“proto-strategies”, which can be generalized to different types of projects ororganizations?

This seems plausible, as there already exist concept of “Generic Test Strategy” [5], whichis used to define general approach on test process, and establish fixed control points.The generalization may have benefits, but developing the concept further to offer“proto-strategies” and possibly identify selection criteria for organizations should bebeneficial. This way the theoretical ISO 29119 model and practical testing could beadjoined with reference model that could offer enhancement proposals for efficiency intesting while avoiding the common process enhancement pitfalls. To achieve this goal, ithowever is imperative to first define how the different central aspects, as defined byTMMi and ISO 29119, affect the test process activities. By refining these observationsinto general strategy with variables for affecting factors, the organizations can beobserved to see how their test organization operates with different variables, and ifthere are any prerequisites to pursue the observed strategy.

Page 5: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

By establishing the effect of variables as defined by TMMi and ISO 29119, and byobserving the practical approaches the organizations use, it should be possible to definefew strong generalized strategies. These strategies are founded in the theoreticalmodels and tend to steer the test process towards efficient operating level, should theprerequisites be fulfilled.

Page 6: On Software Testing Strategy - · PDF fileOn Software Testing Strategy Jussi Kasurinen, jussi.kasurinen@lut.fi The testing process of a software-producing organization can be divided

References

[1] ISO 29119.2 ISO Software testing standard, working group 26 draft 19.5.2008.

[2] Standard glossary of terms used in Software Testing, Version 2.0 (2.12.2007),International Software Testing Qualifications Board, available athttp://www.istqb.org/downloads/glossary-current.pdf, referenced 31.08.2009.

[3] Test Maturity Model Integration version 2.0, TMMi Foundation, available athttp://www.tmmifoundation.org/html/resources.html, referenced 31.08.2009.

[4] Johnson, Karen N., Building a Software Testing Strategy, InformIT, published13.3.2008, available at: http://www.insidetech.com/training/articles/1540-building-a-software-test-strategy, referenced 1.9.2009

[5] De Grood, Derk-Jan, “Generic Test Strategy”, Testgoal, Springer Berlin Heidelberg,pages 119-123, 2008. ISBN 978-3-540-78828-7