Regression Testing in an Agile World
Angela GoodbarMellanie Taylor
About Us2AngelaManual Software Test Engineer for the Sample Processing teamTeam Lead for the Sample Processing Lab sub-teamMyridian for 7 yearsTesting software for 10+ yearsEnjoy wakeboarding and being Grandma!MellanieManual Software Test Engineer for the Result Generation TeamMyriadian for 9 yearsTesting for 5+ yearsSalsa aficionadoFuture Carny
To Be Covered3WHAT is Regression Testing?WHY we should Regression TestCHALLENGES of fitting Regression Testing into our Agile processesHOW we can be more effectiveWHEN we should Regression TestWHERE we should Regression TestWHO should Regression Test
WHAT is Regression Testing?
5Regression Testing is the re-testing of a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made.
It is important to do regression testing frequently, because the code as a whole may easily "regress" to a lower level of quality after making a change.
Regression testing is necessary, even though a change appears to be working correctly and is believed not to affect the rest of the software.
When we regression test, we re-execute previously ran tests and compare current results with those previously executed.5
WHY we should Regression Test
WhatWhyHowWhenWhereWhoFix One Bug,Introduce New Ones
When you fix one bug, new bugs can be introduced.
Regression testing decreases bugs escaped to production.
The rate of defect introduction is quite high.
It is always best to identify the underlying cause and understand how the software is used rather than tackling the symptom.
Not to mention the time spent chasing and resolving bugs introduced.7
If software is deployed to production without regression testing, and new bugs are introduced, or old bugs are re-introduced, the trust the users have for developers and testers is damaged.8
Avoid Rollbacks &Emergency FixesWhatWhyHowWhenWhereWho
Regression testing decreases rollbacks and emergency fixes to production, which (admit it) is embarrassing.
CHALLENGES of fitting Regression TestingInto our Agile processes
If regression testing is executed manually, it takes more and more time testing every day, every iteration. In order for testing to keep pace with coding, either the developers have to take time to help with manual regression, or the team has to hire more testers.
In case of limited time, try to purchase more time. Consider turning the testing over for UAT while continuing regression testing.
If you are not able to get more time, inform in advance about the inability to cover possibly risky areas.11
If the team is facing a tight deadline, theres a temptation to cut corners, and the result is a missed problem.12
When iterations dont proceed smoothly and the team cant complete all of the development and testing tasks by the end of the iteration, team members may panic. It has been observed that when people go into panic mode, they fall into comfortable old habits, even if those habits never produced good results.Example: We are supposed to deliver on February 1. If we want to meet that date, we dont have time to execute all the regression tests; let alone write new ones. Well have to run what can be done in that amount of time and hope for the best. We can always write the regression tests later.This is the road to perdition. Some tests get ran, but maybe not the important tests that would have found the bug that cost the company hundreds of thousands of dollars in lost sales. Then, because we didnt finish with our testing documentation tasks, those tasks carry over to the next iteration, reducing the amount of business value we can deliver. As iterations proceed, the situation continues to deteriorate.13
Unless you're in a test role where full, complete documentation is necessary in order to be federally compliant or to keep people from dying or losing a limb, attempting to document every little thing is a fool's errand. Software changes. A lot. With constant change, what we document one day may be obsolete the next.
The overload exists right at the point where we've documented so many manual regression tests that the test suite is no longer manageable, so testers don't run the tests at all. Whatever value we're seeking to generate with all those artifacts doesn't just taper off at that point, it nosedives into oblivion.All documented manual regression tests should be ran in each major round of regression testing. Many teams have hundreds of regression tests that required hundreds of man-hours to write, yet the tests sit around collecting dust like cheap trinkets at a yard sale. What value does a so-called regression test have if wedon'tneed to run it in every major regression run?14
HOW we can be more effective
To get early and instant feedbackRegression tests onlySafety net to save time which is reinvested into manual testingNOT to replace manual testing16WhatWhyHowWhenWhereWho
Agile development cannot work without automation. Test Driven Development can help identify if a regression tests can be automated, thus eliminating the need for a manual regression test. The benefits of automated testing are significant cost and time savings. Automated tools run tests significantly faster than human testers; freeing them up to more effectively manage testing of the new components.16
Know What Was Changed
How much regression testing is needed can be effectively decided by the tester getting input from the developer about the scope, nature and amount of change. Modularization, good design, and good record keeping can minimize regression testing. It can be argued that certain modules do not need regression if they were not changed. Knowing what was changed is essential.17
Utilize Risk Based Testing
Adequate coverage without wasting time should be a primary consideration when conducting regression tests. Risk based testing can save many hours of testing something that has a low risk of failure. Testing the high risk areas give you the most value for your time. Once again knowing what was changed is essential.
Test fixes promptlyWatch for side effects of fixesWrite a regression test for each bug fix
1. Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.2. Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.3. Write a regression test for each bug fixed. Bugs commonly get re-introduced into the code.19
Build & Groom your Test Suite
Get rid of less effective testsIdentify tests that consistently pass and archive themFocus on Function, not Design20WhatWhyHowWhenWhereWho
If two or more tests are similar, determine which is less effective and get rid of it.Identify tests that the program consistently passes and archive them.Focus on functional issues, not those related to design. Focus on excessively used functions (20% of functions used 80% of the time).
Develop a suite of tests that can be run every time you build a new version of the program. The most difficult aspect involved in building a suite of test cases is determining which test cases to include. The most common suggestion from authorities in the field of software testing is to avoid spending excessive amounts of time trying to decide and err on the side of caution. Automated tests, as well as test cases involving boundary conditions and timing almost definitely belong in your suite. Some software development companies include only tests that have actually found bugs. The problem with that rationale is that the particular bug may have been found and fixed in the distant past.Groom the regression test suite often to eliminate redundant or unnecessary tests. Duplication is quite common when more than one person is writing test code. An example that causes this problem is the concentration of tests that often develop when a bug or variants of it are particularly persistent and are present across many cycles of testing. Numerous tests might be written and added to the regression test library. These multiple tests are useful for fixing the bug, but when all traces of the bug and its variants are eliminated from the program, select the best of the tests associated with the bug and remove the rest from the library.Regression test cases need to be selected very carefully so that a minimum set of test cases cover maximum functionality.Test Design should be used to focus on the most appropriate regression tests to keep; utilizing as much automation as possible.Next time you do a manual regression run, see how many tests were not run. Are these tests really worth keeping? Will they ever deliver any real value? What will you do with your answersdelete tests or spend more time regression testing next time?
Groom the test suite to ensure regression tests remain accurate and do not get out of date due to enhancements and bug-fixes.
Instead of having one large regression test, it is better to create a logical batch of unit or component test cases in a comprehensive test suite. This provides flexibility of execution in cases of urgency or time-pressure.20
Run Every Test