Seven Habits

White Paper: The Seven Habits of Highly Effective Testing Organizations: Redux

Sponsored by MKS Written by Lee Copeland [email protected]

mailto:[email protected]

The Seven Habits of Highly Effective Testing Organizations: Redux

2 | S t i c k y M i n d s . c o m W h i t e P a p e r

In 2001, I wrote an article for StickyMinds.com called “The Seven Habits of Highly Effective Testing Organizations.” It was a compilation of industry “best practices” that I used in my consulting work. I recently reviewed what I wrote eight years ago, reconsidered my positions, factored in the advances in our industry, and revised my seven habits. I’ve kept six, but refocused the seventh.

1. A Separate Software Quality Assurance Organization The software quality assurance (SQA) function defines effective and efficient processes and standards for the entire software development process, including, but not limited to, testing. SQA defines the organization’s quality goals, the methods for reaching those goals, and the various roles, responsibilities, and processes of an organization-wide quality system. Typically, those processes include static testing (reviews, inspections, and walkthroughs) and dynamic testing (unit, integration, system, acceptance, regression, etc.). In addition, SQA audits the development and testing products and processes during and after the project and offers suggestions for improvement.

The focus on the key aspects of quality assurance—effective and efficient processes and standards—is just as important today as it was eight years ago. Unfortunately, even after this time, the number of organizations committed to these ideas remains very small. Few organizations specify quality goals for their products and even fewer adopt methods to meet those goals.

However, a number of changes have been adopted over the past eight years. First, the emergence and adoption of agile methods has brought better development and testing processes to many organizations even without an SQA function to drive that adoption. This speaks volumes to the effectiveness of many SQA groups, when a few developers can institute major process changes very quickly after years of ineffective efforts by SQA organizations.

One important advance has been the substantial commitment to testing, both at the unit and acceptance levels. The processes of test-driven development (TDD) and acceptance test-driven development (ATDD) have made testing the “in thing” to do during development, not that “thing that no one liked to do” at the end of development.

Another advance that has come though the adoption of agile methods is the abandonment of the creation of “Big Requirements Up Front” and “Big Design Up Front.” The agile development process has implemented the Shewhart/Deming cycle of Plan-Do-Check-Act in an iterative and incremental fashion, defining requirements and design, implementing these requirements, and then testing them in small chunks.



2. A Distinct Software Testing Organization Software testing is a recognized technical specialty within the software development process. By designating individuals as testers, we can focus their activities on the quality aspects of software. In a recent survey of attendees at the STAREAST testing conference, 90 percent of all organizations responding have a formal testing group. The placement of this group varies within the organization. While the majority of testing groups are included within the development organization, some testing groups are part of quality assurance, and in other organizations, the testing group reports elsewhere. Eight years ago, I envisioned the testing group distinct from other groups, such as developers. Today, I’ve abandoned the idea of the necessity of a distinct testing group. Agile methods have shown that integrating testers with developers can be more effective than “throwing code over the wall” to a testing organization. In hindsight, I should have stated the need for a distinct testing function. It is the function that is important, not where it appears on the organization chart. Still, we need team members who have the ability to test well—to perform effective test analysis, design, implementation, and execution. What has not changed is the need for people who are skilled at and enjoy testing; the necessity for skill building, typically through classes, conferences, and workshops; and the need for automation tools to support the testing effort. For organizations in which developers are being asked to take on increased responsibility for testing, these are particularly important. In some organizations, restrictions imposed from the “outside” will require a formal testing group on the organization chart. If that is your situation, don’t fight it—put the box on the organization chart. But remember, the box does not reduce the need for great people supported by valuable automation tools.

3. Risk-based Testing In the original version of the article, I wrote that the major components of risk-based testing are:

Risk analysis—Identify the possible risks that a project faces, estimate the loss to the organization if that risk occurs, choose risk mitigation strategies and calculate their cost, compare the cost of the loss to the cost of risk mitigation, then choose appropriately.

Test planning and estimation—Create testing strategies based on the risks that have been identified and choose sets of test cases based on those strategies.



Test tracking—Track the number of tests that have been defined, implemented, and executed; track the number of tests that have passed or failed; examine the test cases to determine the level of coverage they provide; and evaluate the efficiency and effectiveness of the testing process.

Analysis of testing’s contribution—Determine the return on investment that the test group generates by discovering defects before they impact the organization’s customers.

This is timeless advice, although today I would say “testing” rather than “test group” in determining the ROI. Unfortunately, in the organizations I visit, I see little improvement in risk analysis, formal test planning, and the analysis of the contribution that testing makes, both to products and organizations. Still, we don’t identify well what is most important to test, we don’t design test cases effectively and efficiently, and we can’t quantify the benefits we provide through testing. No wonder testing is often not more highly thought of.

4. A Defined and Integrated Testing Process A formally defined and integrated testing process includes these basic elements:

Static testing of the work products of development, including requirements, design, code, test strategies, plans, procedures, and data

Dynamic testing at the unit, integration, system, acceptance, and regression test levels

A managed testing environment The preservation of test artifacts including test strategies, test cases, test

procedures, and test data specifically for reuse on subsequent versions of a single application or sharing between applications

More timeless advice. I can’t add to this. Still, we face two problems: (1) many organizations are not aware of the processes and tools used in formal testing, and (2) organizations continue to respond that they do not have the time and resources to do better testing. I continue to be both amazed and discouraged by the number of “test professionals” who do not know the basic rudiments of their profession. One of my favorite interview questions as a test consultant is, “What is your favorite book on software testing?” The most common response is, “I’ve never read a book on software testing.”

5. Visible Defect Tracking and Reporting Effective defect tracking requires the creation and maintenance of a database describing key aspects of each defect. Each recorded defect should include a unique tracking number, a description, the cause, who detected it, how it was detected, the cost of finding it, the



estimated cost to the organization if the defect had been delivered to the customer, and the person assigned to repair it. Allow all stakeholders to examine this data. Look for defect patterns or trends that indicate that the defects are not merely random but are the result of a systemic problem within the development process.

Of all the seven habits, this one is subject to the most discussion. With the rise of agile methods, a new approach to handling defects has emerged. Rather than log defects in a database, many organizations have adopted a “fix-and-forget” approach. Ron Jeffries proposes a four-step process: (1) write a test showing the defect exists, (2) add that test to the test suite, (3) modify the system until the test passes, (4) done—no other action is necessary. Not only has the defect been repaired, but a test now exists in case of any regression failure. Mary Poppendieck stated, “In other words, stop worrying about the defect database and start worrying about why you are STILL creating code where defects are discovered a significant amount of time after the code has been written.”

Still, I like the idea of tracking defects. As famous software tester Yogi Berra once said, “You can see a lot by looking.” And looking for patterns in defects, their causes, their detection, and their solution can yield valuable information we can use to improve our development and testing processes.

6. Change Management Change management is an umbrella concept that includes three basic areas:

Requirements management—Requirements are reviewed to determine whether they are feasible and appropriate, clearly stated, consistent with each other and with the system as a whole, and testable.

Release management—Rather than develop each change to a system individually, a number of changes are collected together into a release. When the release is defined, all the changes are designed, coded, and tested together. When complete, the release is moved into production. Release management establishes a common expectation regarding the length of time required to implement a change. It provides a mechanism to evaluate and integrate changes proposed by multiple sources within the organization, and it spreads the testing overhead over a larger base, yielding a more efficient process.

Configuration management—Defines baselines, controls changes, and reports the status of application software configurations and the hardware on which they run. A library of current and past versions is maintained.

The three components of change management—requirements management, release management, and configuration management—continue to be important to producing quality software. The agile processes support these components, but in a different way than classic waterfall methods.



In agile development, requirements are typically defined and prioritized by the product owner. As individual requirements are selected for development, they are described in detail, reviewed by the team, developed, and tested.

Release management in agile is done at a number of levels—release, sprint, and daily. Configuration management is even more important in agile because of the rapid work effort.

7.a (Eight Years Ago) Visible Project Planning and Tracking Effective project planning requires a defined software lifecycle with stages of manageable size. The activities within each project are based on that lifecycle. Each stage should be planned, estimated, and scheduled before work is done. This formal plan should be visible to all stakeholders in the project. As the development progresses, the actual activities should be compared to the plan. This comparison should also be visible to all concerned. Variations between planned and actual activities should be investigated. Differences may signal a misunderstanding about the size or scope of the project, a lack of skills or resources, or that the plan was faulty in some way.

While this may sound rather waterfall-ish, it applies equally to agile methods. Processes such as Scrum and Extreme Programming are defined software lifecycles. Their iterations are planned, estimated, and scheduled before the work is done. The use of backlogs prioritized by the product owner make plans visible. The use of burn-down and burn-up charts make progress visible. The most effective charts show progress in terms of deliverable functionality that is useful to the customer rather than effort expended. Planning has been defined simply as “figuring out what to do next.” Most of us would admit that to be effective and efficient, planning is important. But when and how should that planning be done? Must it be done so very early in the project, when our knowledge is typically at its minimum? Waterfall approaches call for Big Planning Up Front, while Scrum calls for planning for releases, sprints, and days. Both call for planning of specific types at specific times. Over the years, I’ve developed a more general model, based on an analogy of the planning of plays in (American) football. When are football plays planned? Our first thought might be in the huddle before each play, but the following table shows more possibilities:

Before the game begins (the first ten plays are scripted and executed without regard to their success or failure)

Before each play (in the huddle, based on an overall game plan, field position, strengths and weaknesses, and player experience)

At the line of scrimmage (depending on the defensive setup)

Start of a play (play action—run or pass depending on the defense)

During the play (scramble when all else has failed)



Each one of these planning approaches is used by professional and collegiate teams. A general rule is that we should plan as much as we can (based on the knowledge available), when we can (based on the time available), but not before.

7.b (Today) Visible Test Planning and Tracking When I wrote the article eight years ago, I assumed that everyone would understand that project planning implied test planning. Apparently, in many organizations, this is not the case. I often see detailed plans for development and “make-it-up-as-we-go” plans for testing.

The most important work product of test planning is the test plan. By test plan I don’t mean a list of test cases. Test plans focus on higher level strategic decisions that should be made before testing begins. The IEEE 829 “Standard for Software Test Documentation” lists key elements of such a plan:

Scope of the test effort

Schedule

Resources required

Responsibilities

Test levels

Test tools, techniques, methods, and metrics

Risks and prioritization

Defect resolution and reporting

Test coverage requirements

Pass/fail criteria

Environmental needs

While the creation of the test plan is usually the responsibility of the test team, effective testing requires that this plan be communicated to, understood by, and supported by both the stakeholders and the entire project team. Without their support, testing will generally be ineffective.

What’s New and Important Since Eight Years Ago

I believe that there have been a number of innovations over the past eight years that are: affecting the testing community for good:

Agile development methods have raised the stature of testing to a level never seen before.



The rise of the context-driven school of testing reminds us that testing is to serve the project’s stakeholders, not to dictate to developers and managers. It reminds us that a “one-process-fits-all” approach is ineffective—different projects require different missions, strategies, techniques, and tools.

The advent of specialties in testing—functional, performance, security, usability, reliability, conformance, and a host of others. Not only is the technical scope of testing increasing, we realize we must specialize to improve our own competencies.

The publication of really good books on testing—management, techniques, and automation.

The creation of small testing workshops with a specialized focus and participatory style. These tend to be invitation-only, limited to fifteen to twenty participants. All participants make presentations, and the learnings from the conference are collected, published, and made broadly available.

American journalist A.J. Liebling once wrote, “Freedom of the press is guaranteed only to those who own one.” Today, with the ubiquitous Internet, we all have access to a “press.” These presses host dozens of excellent testing blogs and open source testing training.

As organizations have responded to the market’s demand for higher quality products, and as agile has freed testing from the “end” of the development process, testing has broken out of its silo to become a vital force throughout the development life cycle. In addition, complete application lifecycle management solutions are now available that integrate test management into the overall development process yielding greater transparency and productivity.

What Do the Next Eight Years Look Like? Nobel Prize winning physicist Niels Bohr (or perhaps it was Yankee baseball player Yogi Berra, the sources are not consistent) said, “Prediction is difficult, especially about the future.” And so it is—and so I won’t make anything but the most obvious predictions.

Virtualization will become ubiquitous, providing “multiple machines” on a single server. Using virtualization, multiple operating system/browser/applications can be created and stored and then loaded and run to provide different testing environments. When a defect is observed, the entire machine state can be saved for later investigation.

Cloud computing will provide immense resources for testers that are unavailable now. Imagine having thousands of servers at your command, at a very low cost, and only for the time you require. No longer will limitations on machines managed by your organization restrict the amount of testing that can be performed. Imagine the ability to create thousands of “users” for performance testing. Imagine what you could do if you had unlimited hardware resources at your disposal.



In the future—regarding the benefits of testing—some organizations will get it, but most won’t. Some will understand the great value that quality assurance and quality control activities add, will implement them, and will reap the rewards. Most will just slouch along, pennywise and pound foolish, hoping for the best. I hope you will get it.

Lee Copeland’s update of his StickyMinds.com original column, “The Seven Habits of Highly Effective Testing Organizations” is sponsored by MKS.

Breaking out of the “silo” of test management can elevate visibility of testing as a part of the software development lifecycle and also improve the accountability of testing groups. The emergence of complete application lifecycle management solutions like MKS Integrity, where test management functionality can be integrated into the development lifecycle, achieves this by delivering greater transparency and productivity. Visit http://www.mks.com/products/test or email [email protected] for more information.

http://www.mks.com/products

http://www.mks.com/products/test

mailto:[email protected]

Documents

Seven Habits