Understand Regression Testing

Testing Tiger Team (T Cube)Liang Gao

Agenda

• Regression testing and Regression bugs.• Regression’s When, What and How• Importance of the automated regression

testing• Managing regression testing

What is a regression testing

uncover regression bugs.

Regression Bug?

很多功能以前好好的，加入新功能以后就不Work了！

Why ？？

• No software design is 100% modulized. • IP Network is layered (ISO 7 layers), break one

layer, rest of the layers won’t work• The introduction of new code breaks the

existing code• For Example:– IPSec use Path MTU discovery – ICMP can probe the MTU value– IPSec use ICMP to change the MTU valule.– If ICMP new code breaks the MTU function, part of

the IPSec won’t work

Regression bug is bad, very bad• For a new product release, customer is ok that if the newly added function

has bugs, since the common understanding is new code is buggy. And not all customer will use the newly added function anyway.

• But customer is not OK that the previously worked function is broken in the new release, because that make things worse.

Don’t Piss Off the Customer…..

How to Catch the Regression Bug

Use your analytical skills, how to catch the regression bug?

3 Reasons A Regression Test Case Fail

• It is because of the testbed. – Testbed instabilities. – Cable connections. Switch issues. Bad hardware– Improper testbed configurations.

• It is because of the script issues– Version update of output changes. – Bad quality scripts.

• It is a software problem– Then it could be a regression bug.

Understand Clean Run• “Clean Run” means after running all your

regression test cases against a release, all failed test cases has been analyzed on the reason of failure.

• Test cases result is either “PASS” or “FAIL” with a defects ID or Issue ID (script or test bed problem).

Understand “Baseline”• Before running every regression against certain

release, always ask: What is the baseline of this release?

• The baseline release is a “Clean Run” release with each test bases be either “PASS” or “FAIL” with a defects ID. All failed test cases have been debugged and found the root cause.

• The current result will always be compared with the baseline result for delta (differences)

• The best baseline is to have high percentage of the PASS cases.

Compare with Baseline• The current release regression result has

the same result as baseline– Baseline PASS, current release PASS– Baseline FAIL, current release FAIL

• Different than baseline– Baseline PASS, current release FAIL– Baseline FAIL, current release PASS

Benefit of the Baseline• Identify regression bug quickly. • Shorten your debugging time. – Most of the case, if baseline also failed, it

should fail on the same reason.– when you run 10K+ test cases and have 5%

failure rate, baseline result speed up your debugging time greatly.

• Better test result management– Have a traceable result history

When to Start the Regression testing

• Before or after the new feature testing?• Before or after the system testing?• Before or after the solution testing?• Before or after the release?• How many times you think a regression

testing should be run?

Regression is headache• You have no choice but to do it. (don’t piss off the customer)• You have to do it whenever to release a new version• You may have to do it for every build• As more features added, regression test cases keep growing and growing

and growing….• What if your company release a new build every week, or every other

day? (Cisco does) Regression nightmare. • Your engineers are bored of doing the same old thing again and again.

Bored low quality of execution can not find bugs piss off the customer with regression bugs

Me and the Machine• I am flexible and can execute beyond the test case

scope (explore and play a little to find more problems)• I need to do different things, need to explore and to be

challenged. • I need to sleep, rest, and play 斗地主，麻将 and go to 青城山后山的农家乐 during the weekend.

• Machine will do exactly what I programmed him to do• Machine can do it over and over without complaining

with exactly the same quality• Machine can work overnight, over weekend and over

holidays.

Auto Regression is the Solution• As regression test cases exploding, it can not

possibly be run by human (Cisco IOS regression has 100,000 plus test cases, and need to be finished in 2 days)

• It same time with quality (48 hours, for me, means 6 days, for machine, means 2 days)

• It speeds up the release cycle: the faster the cycle of a full regression, the more build can be released (Cisco can release a build every 2 days, Juniper can release a build every 4 days. Keep in mind IOS has more than 6 million lines of code)

In Order to Run Auto Regression…..• I need someone to write scripts, convert manual test cases in to scripts. • I need an engine to execute thousands of scripts in batch mode, collect

pass and fail info. • I need to have libraries. So when I write scripts, I can use APIs to speed up

my development. • I need a database to store each regression cycles test report (running

results)• I need a way to debug when scripts are failed (is it script problem? Test

bed problem? Or indeed the image problem)• It would be nice if I can have a frame work that I can do all this.

Managing Regression Testing• Schedule the run based on product release

schedule. • Select suites

– If I can not cover them all, which set of features I need to pick– How long can I finish it– How much debugging time I need for failures.

• How many engineers I need to run (1 engineer per 2 test beds?

• Regression engineers time– Kick off the run– Debugging (most of the time)

Find the Way to Reduce the Regression Cycle

• Look at the whole cycle of the regression testing, which part use most of the time– Test bed management?– Debugging and defect filing?– Reporting?

• Lots of things can be automated (process automation) during a regression cycle)• In Cisco, during a perfect run, a regression engineer don’t need to do anything

– Automatic copy new image after build team finish it– Automatic load the image to router– Automatic select the features need to be run based on the image name

(different image has different feature set)– Automatic run– Automatic reporting

Regression Bug Management

• Be careful whenever filing regression bugs, need to be 100% Sure.

• How to be sure if it is a regression bug? Use your analytical skills.

Make Regression As A Train

• Pull the regression cycle out of the release cycle.

• Make regression cycle fixed (predictable). • Make it an on-going, 24/7 operation. • Benefit– Any release can schedule a time to be on the

regression train, good to arrange regression schedule.

– More regression, more stable of the product.

Regression Test Bed Management

• Standard topology test bed. • Running in parallel (multiple identical

regression test beds)• Fixed run time on each test bed on each

regression cycle

Don’t let regression team buy test bed

• Regression repository will keep expanding. • More testbeds will be needed. • Unlimited budget for regression team to buy

testbed? Probably not.

manage run time by minutes.

• Based on your release cycle, carefully control total run time on a testbed.

• Each test bed has a maximum run time, if exceed, consider build more testbeds (with in budget)

Automat Regression - Engine• If you have 1000 scripts

– How to run it in batch mode– How to know which one is fail, which one is pass at any given time– How to debug when a test case failed (pause on failure)– How to move job across test bed– Where to report the test results– Where to keep the historical running logs– How to run only certain modules– How to select cases based on different release version– How to run scripts even device is crashed

Automat Regression – Test Harness

• Need a place to store all test results• Need to have version/release info• Need to be able to compare from release to

release

Version control• 1st draft, Feb, 2008, Liang Gao