View
224
Download
0
Category
Preview:
Citation preview
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 1/9
© Tech Mahindra Limited 2011 © Tech Mahindra Limited 2010
Telco Grade – Test Automation SolutionProven & Cost Effective Approach
Author: Rajiv Ranjan
Feb 2011
Abstract:
Test Automation has been around but has not yet become part of the testing DNA for both Telco’sand Equipment Manufacturers. Tech Mahindra has been providing automation solution to both usingtools. It has also developed an in-house automation framework to provide a cost effective automationsolution. The learning’s of 200+ person years of experience has enabled Tech Mahindra to providevalue enhancing, vendor agnostic automation solutions.
This paper describes the approach needed to make automation a successful and cost effectivesolution.
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 2/9
1 © Tech Mahindra Limited 2011
Table of Contents
Executive Summary ......................................................................................................................... 2
Business Need .................................................................................................................................. 3
Network Automation Approach ................................................................................................. 4
Automation Assessment ............................................................................................................ 4
Arriving at the solution.............................................................................................................. 5
Design............................................................................................................................................ 6
Planning .......................................................................................................................................... 6
Benefit Measurement...................................................................................................................... 7
Conclusion ........................................................................................................................................ 8
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 3/9
2
© Tech Mahindra Limited 2011 © Tech Mahindra Limited 2011
Executive Summary
Automation is a much clichéd expression in today’s business environments. In the
networks area, automation has been notorious in soaking up expensive investments andproducing little in terms of the ballyhoo of the benefits promised. Telco’s and
communication equipment providers (CEP) therefore tread with extreme caution when the
‘A’ word is flaunted as the solution to all its problems. In the end automation in most
organizations ends up being nothing more than an ‘also-ran’ and in many cases a pariah
which is unlikely to be invoked in the testing scheme of things.
Tech Mahindra (TechM) understands this hesitation faced by the Telecom industry in
adopting automation as a viable approach in solving many of the latter’s Achilles heel.
This white paper intends to carefully describe the approach needed to make automation
succeed at Telco’s and CEP’s. Tech Mahindra’s assertion to this approach is drawn from
its expertise in providing bespoke and turnkey automation solutions to its Telecom
customers. Tech Mahindra has more than 200 person years of Test Automation
Experience which spans across various Testing tools, Network devices/equipments &
languages used for automating network test scenarios.
The stages of TechM’s automation approach covered as part of this wh itepaper include
assessment, solution formulation, and automation implementation strategy. The first stage
will probe into the need for automation assessment and will delve into TechM’s automation
methodology. The automation solution development stage will cover both the tool
evaluation & selection strategy and the automation design aspects. The planning stage will
examine the key aspects necessary to operationalize the automation plan. The narrative,
at each stage, is strengthened with examples from a successful engagement and will provide
legitimacy to the approach recommended through this white paper.
Tech Mahindra’s vendor and technology agnostic position allows it to provide value
enhancing automation solutions to its customers. Tech Mahindra, as part of the
assessment strategy, also evaluates, identifies & provides recommendations of the tools and
technologies that will bring in the desired benefits. The approach presented in this
whitepaper together with the right experience will enable unlocking some of the utopian
benefits promised by automation.
Less than 40% of Telco’s
and CEP’s use some kind
of automation currently
Less than 15% of them
have actually claimed
that automation has
provided tangible
benefits
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 4/9
3 © Tech Mahindra Limited 2011
Business Need
Telecom industry is highly dynamic both in terms of technology & competition. Changing
business needs & market compulsions make Telco’s and CEP’s to perform more and more
testing in order to ensure that the desired levels of quality and performance are met.
Time-to-market pressures often could mean deploying new features or devices into live
without adequate testing which invariable leads to increased support costs. These costs
associated with fixing problems from field are exponentially high when compared with
fixing them during pre-deployment stages. Yet, many a times Telco’s and CEP’s are forced
to cut corners during the test cycles in order to avoid missing out on bringing a product or
service to market.
In the absence of Test Automation the pre deployment testing is done manually. This
methodology has its own pros & cons. The human element involved in it brings its own
intelligence but also brings along its human limitation in terms of getting tired & bored
with repetitive work. This leads to a time consuming human incentive process which
comes out with a not-so-fully tested system which again is not-so-dependable.
Network test automation can help Telco’s and CEP’s to fasten the test cycle times
thereby enabling them to meet market opportunities while maintaining quality.
Regression tests are a ‘low-hanging fruit’ when it comes to selecting automation
candidates owing primarily to their repetitive nature and wider test coverage. Network
test automation can be used primarily for
Regression / repetitive tests for a specific test setup
Sanity Tests prior to upgrades or deployments
Stress tests to measure performance of the desired service, feature or
product
Interoperability tests to validate desired features or product behavior
1 in every 4 dollars of
R&D/Design spend goes
towards Testing
Over 45% of defects in
live is attributed to
seepage from testing
Automation could
potentially help plug this
defect seepage upto 90%
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 5/9
4
© Tech Mahindra Limited 2011 © Tech Mahindra Limited 2011
Network Automation Approach
While a lot of emphasis is seen around the execution aspects of automation, not much is
written or mentioned regarding the stages prior to execution, namely the assessment,
design & planning stages. Ironically, it is these pre-execution stages that govern the
effectiveness of the automation outcome. While there is no cookie-cutter approach to
assess and build automation solutions, a robust guiding framework will go a long way in
helping automation planners to achieve the desired benefits. The automation framework
described in the sections below follows a 3-step approach which can be applied by the
QA/Test teams to formulate their automation strategy.
A s s e s s m e n t - Existing Test
Plan & Test Bed
Analysis- Arrive at Pointsof Automation
- AssessmentReport
S o l u t
i o n- Identify
Automation Tool
- Identify clearlythe inclusions andExclusions
- Design thesolution using thetool
P l a n n i n g- Define roles and
responsibilities
- Setup testenvironment
- GenerateAutomation TestPlan
Fig 1. Automation Assessment Stages
Automation Assessment
Absence of an objective assessment before automation implementation leads to several
pitfalls. Limited understanding of the automation tools and the requirements leads to a half
baked approach resulting in a costly solution which fixes only a part of the problem. Fact
remains that automation is not the answer to all the functionalities which a Tester
performs. A detailed assessment is a must to evaluate the present state objectively and in
totality to arrive at the automation recommendations.
The assessment begins with the detailed analysis of the existing test plan & test case
documents to determine whether or not they can be automated (fully/partially automated).
Assessment involves understanding and defining the various elements of an existing system
like Base line Configuration, Interoperability, Failover, Capacity Validation, Security, Application
Test, System Upgrades, Regression Tests, New and Enhanced features, protocols etc.
In this stage, specific points of assessment (POA) are created, which are specific devices
and interfaces that currently require manual intervention and therefore need to be
assessed for automation. These points of assessment cover the three areas of test
operations: pre-execution test configuration, test case execution and test validation. If
Case Analysis
Customer Profile: Tier 1 USTelco
Business Need: E2Eautomation of its NextGeneration Mobile AccessGateway platform
Test Setup: Multiple network elements and platformsa. Network layer – RAN,
SGSN, GGSN, Cisco loadbalancer, load sharingNGG & MAG
b. Content – Web,application servers, IMgateways, MMSC, demosetup of third partyservices like MediaNet
c. Provisioning - MIPS+ R2,Radius Server, MINDLDAP subscriber database
d. User devices & trafficgenerators – 2.5G/3GHandsets, Laptop with 3Gdata card
Case Analysis
TechM identified 5 POA’s for theclient.
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 6/9
5 © Tech Mahindra Limited 2011
these points of assessment can be automated then these points are termed as points of
automation.
In addition the entire Test Bed setup is taken into consideration and evaluated. Test
Configurations are identified & analyzed for automation. Finally a summary report with a
breakup of fully automatable, partial and manual test case counts is prepared which then is
grouped logically into test categories that can be implemented as discrete test suites.
Finally, automation test case candidates are prioritized based on customer business
requirements.
Arriving at the solution
The next task, once the assessment of the current environment is complete, is to identify
the right automation tool that would fit the requirement. Three options are typically
available:
1. COTS (Commercial Off The Shelf) Tools
2. Open Source Tools
3. Bespoke or Custom Built Tools
All the options have their inherent advantages and disadvantages as shown in the table
below:
- Offers 'a l l i n o ne go ': S aves co s t & ti me - Li cens e & s u pp ort co sts a re exp en si ve
- Be tte r tra ce abi li ty a nd ma na ge abi li ty - Di ffi cul t to s ca le
- Availa bili ty of drivers & plugins for larger COTS - Integration chal lenges with other products
- Generally upto date with dependable quality - Unreliable support
- Cost effective - Tool roadmap not well defined
- Community contribution - Poor traceabil i ty and management functions
- Offers s ca la bi li ty & fl exi bi lity - Ti me cons umi ng tool devel opment cycl e
- Al lows better cost control - Robustness and stabil i ty i ssues
- Ta i l or ma de to fi t exa ct requ irements - S up po rt i s s ue on ce co re deve l op ers l ea ve
COTS Tools
Open
Source
Tools
Custom
Built Tools
Advantages Disadvantages
Fig 2. Automation Tool Analysis
Recommendation: Perform a PoC using the identified tool on a subset of automatable
features in order to validate the automation feasibility and the tool selection.
Once the right automation tool is identified, the solutioning aspect comes into play.
Besides the network or device under test, the automation environment variables need to
be chosen. These include elements such as simulators, protocol analyzers, platforms for
hosting automation suites & management applications etc. Inclusions & exclusion of the
automation are clearly stated and agreed upon. The Library procedures to be automated
Case Analysis
Tools EvaluatedHandset Automation Tools
1. AToM2. TestQuest3. Custom Application basedHandset Simulator
1. Silk Test2. Selenium (open source)3. Watir (open source)4. Custom simulators (three) Automation Platforms
1. Fanfare iTest2. TestShell
3. Quick Test Pro (QTP)
Case Analysis
Proposed Solution:
2-tiered automation solutionwas identified whereindividual automation tools,driving the POA and POS will
be centrally controlled by anautomation platform.
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 7/9
6
© Tech Mahindra Limited 2011 © Tech Mahindra Limited 2011
are identified & categorized as fully/partially automated. Response Maps (templates for
responses from the Network Elements & Test Equipment), Test Suites & Test cases are
identified & finalized during this stage.
DesignDesign considerations vary based upon the test tool selected for automation. It entails the
following:
Identification of all common procedures by examining existing test cases for
generic execution and validation steps
Building a library of common procedures
Maximizing design of analysis rules in the tool, instead of having this logic reside
on other automation elements
Ensuring that the test cases are test bed and user independent
Building modular test cases using library procedures, test beds, session profiles,
global and local variables
Design hierarchy bases test suites to allow flexibility
The Key design features of the automation solution should be
Reusability – generic & reusable library procedures reduce development time
Data Driven – input data stored in external data files eliminates need for hard
coding data in script
Variation Testing – variation testing for any number of pre-configured handset
user agents
Test Bed Portability – test bed data saved in separate configuration files for
easy script portability
Flexible Validation – user configured criterion for protocol & content validation
In addition, the design phase may involve exploring ways to integrate the automation
management system with the existing tools available in the organization for Test
Management, Defect Tracking & Reporting Tools etc.
Planning
Businesses that don't build a careful plan for automated testing may end up wasting
expensive resources: time, effort, and money. The automation planning phase defines
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 8/9
7 © Tech Mahindra Limited 2011
roles and responsibilities, project test schedule, test planning and design activities. The key
activities that should be part of this phase are:
Test environment setup
Preliminary test schedule
Define roles and responsibilities
Define performance measures
Identify risks and develop contingencies
Develop the automation acceptance criteria
As part of the test environment setup the test team schedules and track environment
activities. The planning phase should also involve installation and integration of test
environment hardware, software, and network resources. All these inputs are captured as
part of the automation test plan that will be used by the Test teams during the execution
phase. The test manager can use the plan to ensure that tests are executed according to
schedule, and test personnel are allocated and redirected when necessary to handle
problems that arise during the test execution process.
Recommendation: TechM suggests a phased approach to Test Automation
implementation to help progressively derive the benefits of Automation and also allowing
you to measure the benefits in a phased manner.
Benefit Measurement
The basic economic premise of network test automation is to automate if cost of test
automation and cost of automation execution is below cost of manual test execution. The
business case for automation will require that the benefits are measurable. A few
quantitative methods to measure automation benefits include :
Savings derived from cost of automating versus cost of manually executing test
cases in a given period of time
Savings derived from reduction in cycle time
Cost savings from avoidance of defects found in field
Case Analysis
Benefits
- Execution time per test
case reduced from 96 min
to 1.5 min
- Test Cycle time Reductionfrom 4 weeks to 8 hrs
- Enabled multiple device
testing as compared to
max of 5 device testing
earlier
Case Analysis
Planning
24 week plan covering- Dev & test
- Requirement analysis- Design of automation
test suits
- Coding and unit tests
- Perform integration and
acceptance tests
8/6/2019 Techmahindra - TelcoGrade_TestAutomationSolution
http://slidepdf.com/reader/full/techmahindra-telcogradetestautomationsolution 9/9
8
© Tech Mahindra Limited 2011© Tech Mahindra Limited 2011
Conclusion
Network test automation has the potential to significantly reduce OPEX of CEP’s and
Telco’s. Currently the network automation uptake in the telecom industry is patchy which
is attributed to inadequate assessment and planning. Detailed automation assessment in
combination with proper design and planning are a must to enable an efficient automation
strategy and to unlock the true potential of network test automation. The white paper
provides test planners with the framework required to make the right automation decision.
The white paper identified critical success factors needed for making network test
automation successful -
Shoring-up the right skills required to perform assessment of current Test
Methodology
Understanding the customers existing test management framework
Analysis & validation of test scripts & procedures- which all procedures can be
automated (if yes to what extent).
Identification of right Test Automation tool, which can interact with all
network Elements.
Designing flexibility in Test scripts, to test in the ever changing & varied
business requirements.
Understanding various interface aspects (of network elements, network test
tools, traffic simulators, existing defect and test management tools etc.).
This framework was validated by depicting instances where it was successfully applied and
the resulting benefits highlighted.
In the near future as testing in networks gets more complex, there will be an increasing
demand to improve test efficiencies and effectiveness in test cycles through automation.
The key to success here will be to adopt winning strategies realized through better
automation planning that will enable users to keep testing costs and quality under control.
How to reach us:
AmericasTech Mahindra (Americas), Inc.sales.am@techmahindra.comTel: +1 972 991-2900
Europe
Tech Mahindra Ltd.sales.eu@techmahindra.comTel : +44 01908 553400Fax : +44 01908 553499
Middle-east & AfricaTech Mahindra Ltd.sales.mea@techmahindra.comTel : +002 02 5806608Fax : +002 02 5806601
Asia-PacificTech Mahindra Ltd.
sales.apac@techmahindra.comTel: +91 80 40243000Fax: +91 80 25527027
Recommended