Automation Framework Design and Implementation
Liang Gao
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Why we need an automation framework Elements of automation framework Design Principles Implementation Principles. Popular automation framework case study
Agenda
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Increase productivity on repetitive testing Avoid human errors Use night and weekends Ultimate goal is to get the full version report
(auto or manual)
The Goal of Automation
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
I can develop script without any framework Each script can run itself Scripts execution results can be collected
and put into a spreadsheet I can do everything I want for automation
without a “Framework”
Automation Framework: Why?
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
杀鸡不用宰牛刀 If you have multiple developers to develop
scripts in parallel If you have thousands of thousands of scripts
need to run If your product release cycle is short and
demanding
Automation Framework: Reason
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
All we want is a full version report (manual or auto)
Remember the goal
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Control library Common library Coding guideline Execution engine Test harness Result database
Elements of an Automation Framework
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
All equipment in the network need to be controlled by control library
CLI remote control Power of the expect
GUI control Selenium
API based control Ixia, Smartbit, Ameritec, Abacus etc.
Control library
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Common APIs to use when automate command tasks
Ping library WiFi library MPLS library DHPC library ……..
Common Library
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Common style for ease of maintenance Common headers 20% + comments Enough logging and debugging Documentation
Coding Guideline
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Concept of Test Bed, Jobs and suites Controlled batch run
Exception catch (both software and hardware) Will run to the end no matter what Abortion on demand
Live status display Debugging support (not script development
debugging) Dynamic suites generation Email notification
Execution engine
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
The end product: full version test report Collection of the running results in high level Easy to present and easy to search Easy to debug base on the report Easy to compare between different version’s
results
Test Harness
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Code control Extract layer of library design Script and test bed independency Resource allocation Execution parallelization Result storage Dynamic topology Distribution mechanism
Design Principles
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Just like any product development, pick your choice
Subversion, Clearcase etc.
Code Control
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Equipment library layer to high the complexity of different equipment
Common APIs with are equipment independent
Abstract Layer of Library Design
Abacus Ameritec SIPX Others
Device dependent layer
Abstract API Layer
Scripts: A Calls B
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Any scripts can run on any test bed without change inside the script
Best case: all you need to change is the test bed name in the framework
Script and Test Bed Independency
TB1 TB2 TB3 TB4
Test Bed Map
Logical Test Bed
Scripts: A Calls B
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Each test bed can only run one job? Test bed design Dynamic resource allocation
Resource Allocation
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Per framework instance per test bed? Central framework?
Execution parallelization
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Logs need to be archived (very important) Full version report need to be stored Need a database?
Script info database? Log database? Report database? Suite-version database?
Result Storage
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Each script need different topology Fixed test bed topo vs dynamic test bed topo
Dynamic Topology
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Centralized Per engineer per copy
Distribution Mechanism
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Script language of choice Performance consideration
Implementation Principles
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
TCL Perl Ruby Python EXPECT
Script Language of Choice
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Central Framework VS Distributed
Performance Consideration
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
C company’s - largest network equipment vendor J company’s - largest network equipment vendor H-1 company’s - largest network equipment vendor H-2 company’s - largest network equipment vendor T company’s – windows based security company A company’s – Silicon valley security appliance
company N company’s – Silicon Valley startup. Sigma’s – first commercial framework in the market
Framework Case Study
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
TCL Per engineer per framework No GUI No dynamic resource allocation No dynamic topology on Framework Open source style library development CLI only Dynamic suites
C
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
TCL/Perl Dynamic topology generation by controlling
the switch Vlans. Per engineer per framework CLI only Per release code base (on scripts)
J
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Key word driven
H-1
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Key word driven Script and test bed independent IDE Static suites
H-2
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Ruby + AutoIT + Witr Strong support for GUI automation on both
MS Windows and Browser C/S structure
T
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Perl Heavily use of Selemium
A
This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.
Python All CLI
N