Lessons Learned on Automation

Liang Gao (liangg@gmail.com)

Lesson -1 Automation Cost Money

You need people to write scripts You need testbed to run those scripts

(lots of equipments) You need people to monitor those test

bed runs, and debug running failures. You need people to maintain those

scripts to reflect product change Can you justify all those to your upper

management?

You might if you

Show some proof in the front. Can go to your boss and say: I can

reduce major release cycle from 8 month to 5 month if I do this.

Can go to your boss and say: I can reduce customer found bugs down by 20% if I do this.

Lesson 2: Automated Testing is Dangerous

Once it is automated, chances are, it will not be manually executed anymore.

No exploratory testing can be done If your script has problem, and can not

catch bugs (output is pass even it should be fail), it will be going into darkness for ever

You may lose your chance to catch bugs for ever and you don’t know.

Lesson 3: No Framework, No Automation

If you have one couple of hundred scripts in hand, you might be fine without a framework.

You need one if you have thousands of scripts to manage. How to know which test case passed/failed? Where to get a decent running report. How to group test cases easily. How to debug

You need one if there are many people develop scripts in parallel.

Lesson 4: Use Standard Script Language

VB/TCL/Perl/Python/Ruby Customized scripts need learning

period Customized script language – Who

maintain it? No community support Hard to communicate with others –

developers, other test outside the company

Lesson 5: Separate writer and runner

Engineer should not develop script and then execute script. Script execution is a dedicated job.

Debug takes time Test bed problem? Script problem? Image regression bug?

Script integration takes time. Script execution should be a 24/7

factory, should be a machine. Script is just a by-product, it is the full version report that you want

Lesson 6: Test Bed Independent is very Important

Separate the writer and runner requires the script should be testbed independent.

Script should be able to run from testbed to testbed with minimum change to the test framework configuration, not to script itself.

Hard coded router/switch names, IP addresses, interface names are not good when switch testbeds

A handover process is needed between the writer and runner.

Develop a “script checker” tool to check the hard coded values in the script as an acceptance criteria.

Lesson 7: Manage Your Scripts the Same as Your Bugs

Script need to have states like bugs (S) – Submitted: Script is submitted to the

regression team (A) – Assigned: Script is assigned to a

regression engineer (I) – Integration: regression engineer is

putting it on the regression test bed. (P) – Production: Script is in the regression

testbed and will be run on regression testing against release

(F) - Feedback needed: Script has errors, more feedback from writer needed.

Lesson 8: Design You Scripts As Data Driven.

Script need to be data driven Different data means different test cases.

Test_case.tcl {router1 eth0} {router2 eth0} {router3 eth0 eth1} {router4 eth0 eth1 eth2} {mode 1} {phase 1} {traffic 1 speed}Can test all combinations of Mode, Phase and Traffic with one single script.

Data generation can be automated too. Can catch more bugs when vary the data

Lesson 9: Log is More Important Than Script

When fail, most of the time we only look at the logs, not scripts for debugging.

Read log like read a book More debugging info dumped when

fail.

Lesson 10: test case designer and automator separate?

Don’t use automator who doesn’t respect testing

Don’t use automator who doesn’t understand testing

C company use same tester to design, manual execute and automate the test cases. And so is J company

Lesson 11: User Standard Testbed

DUT2 DUT3 DUT4

VLAN1

VLAN2

VLAN3

VLAN4

Four Ethernet on each of the Device Under Test

1 2 3 4 1 2 3 4 1 2 3 4

被测设备 2

DUT11 2 3 4

Lesson 11: User Standard Testbed

DUT1 DUT2 DUT3 DUT4

Ethernet1 Open Open Shutdown Shutdown

Ethernet 2 Open Shutdown Open Shutdown

Ethernet 3 Shutdown Open Open Shutdown

Ethernet 4 Shutdown Shutdown Shutdown Shutdown

Logical Topology:

DUT2

VLAN1VLAN2

VLAN3

DUT1

DUT3

Lesson 12: Dynamic script allocation

Different release has different feature set

Each regression run need to pick different scripts to run

Tricks for dynamic script allocation Static file for version matching Database

Lesson 13: Use Database to keep automation result history

Database for full version report Database for regression bug

debugging

Lesson 14: Script maintenance – version per branch

Certify scripts for each new release Keep them in a separate code branch

Version control

Version 1.0, Dec 24th, 2007, Liang Gao Version 1.1 March 2008, Liang Gao