Automation framework design and implementation

Automation Framework Design and Implementation

Liang Gao

This material is for training purpose only. No duplication without consent of Sigma-RT is permitted.

Why we need an automation framework Elements of automation framework Design Principles Implementation Principles. Popular automation framework case study

Agenda


Increase productivity on repetitive testing Avoid human errors Use night and weekends Ultimate goal is to get the full version report

(auto or manual)

The Goal of Automation


I can develop script without any framework Each script can run itself Scripts execution results can be collected

and put into a spreadsheet I can do everything I want for automation

without a “Framework”

Automation Framework: Why?


杀鸡不用宰牛刀 If you have multiple developers to develop

scripts in parallel If you have thousands of thousands of scripts

need to run If your product release cycle is short and

demanding

Automation Framework: Reason


All we want is a full version report (manual or auto)

Remember the goal


Control library Common library Coding guideline Execution engine Test harness Result database

Elements of an Automation Framework


All equipment in the network need to be controlled by control library

CLI remote control Power of the expect

GUI control Selenium

API based control Ixia, Smartbit, Ameritec, Abacus etc.

Control library


Common APIs to use when automate command tasks

Ping library WiFi library MPLS library DHPC library ……..

Common Library


Common style for ease of maintenance Common headers 20% + comments Enough logging and debugging Documentation

Coding Guideline


Concept of Test Bed, Jobs and suites Controlled batch run

Exception catch (both software and hardware) Will run to the end no matter what Abortion on demand

Live status display Debugging support (not script development

debugging) Dynamic suites generation Email notification

Execution engine


The end product: full version test report Collection of the running results in high level Easy to present and easy to search Easy to debug base on the report Easy to compare between different version’s

results

Test Harness


Code control Extract layer of library design Script and test bed independency Resource allocation Execution parallelization Result storage Dynamic topology Distribution mechanism

Design Principles


Just like any product development, pick your choice

Subversion, Clearcase etc.

Code Control


Equipment library layer to high the complexity of different equipment

Common APIs with are equipment independent

Abstract Layer of Library Design

Abacus Ameritec SIPX Others

Device dependent layer

Abstract API Layer

Scripts: A Calls B


Any scripts can run on any test bed without change inside the script

Best case: all you need to change is the test bed name in the framework

Script and Test Bed Independency

TB1 TB2 TB3 TB4

Test Bed Map

Logical Test Bed

Scripts: A Calls B


Each test bed can only run one job? Test bed design Dynamic resource allocation

Resource Allocation


Per framework instance per test bed? Central framework?

Execution parallelization


Logs need to be archived (very important) Full version report need to be stored Need a database?

Script info database? Log database? Report database? Suite-version database?

Result Storage


Each script need different topology Fixed test bed topo vs dynamic test bed topo

Dynamic Topology


Centralized Per engineer per copy

Distribution Mechanism


Script language of choice Performance consideration

Implementation Principles


TCL Perl Ruby Python EXPECT

Script Language of Choice


Central Framework VS Distributed

Performance Consideration


C company’s - largest network equipment vendor J company’s - largest network equipment vendor H-1 company’s - largest network equipment vendor H-2 company’s - largest network equipment vendor T company’s – windows based security company A company’s – Silicon valley security appliance

company N company’s – Silicon Valley startup. Sigma’s – first commercial framework in the market

Framework Case Study


TCL Per engineer per framework No GUI No dynamic resource allocation No dynamic topology on Framework Open source style library development CLI only Dynamic suites

C


TCL/Perl Dynamic topology generation by controlling

the switch Vlans. Per engineer per framework CLI only Per release code base (on scripts)

J


Key word driven

H-1


Key word driven Script and test bed independent IDE Static suites

H-2


Ruby + AutoIT + Witr Strong support for GUI automation on both

MS Windows and Browser C/S structure

T


Perl Heavily use of Selemium

A


Python All CLI

N

Documents

Automation framework design and implementation