Upload
damian-jenkins
View
221
Download
0
Tags:
Embed Size (px)
Citation preview
A Structured Approach to Enterprise Test Automation
Venkatesh KovvuriHead of Innovation & Automation Practice
© 2012 Computer Sciences Corporation - Proprietary 2 2
Overview
» With increase in the complexity of application software and enterprise infrastructure, the
quality guidelines & standards have also mounted. Effective testing of such apps can’t be
assured by manual testing alone. By reducing human intervention during test processes
and automating them can result in an effective and efficient methodology to meet those
standards.
» A Structured Approach will provide opportunity to deal with complex application software
and enterprise infrastructure.
• Extends the usage of automation to entire software testing lifecycle in addition to conventional automated test
execution (design, data, tasks, infra etc)
• Addresses challenges around selection, prioritization and execution of automation projects in large enterprises
• Encompasses all key components essential for executing complex automation projects
• Provides an asset ‘toolkit’ for Test Automation - Process, Technology, Integration and Quality
• Supports integration with industry leading open source and commercial software testing tools
• Provides visibility into the aspects of Quality Governance on the application portfolio
© 2012 Computer Sciences Corporation - Proprietary 3
Enterprise Application Landscape
Legacy
systems
Packaged
software
External
interfacesCustom
applications
•Integration APIs (partner-specific)
•EDI•Web Services (SOAP/REST)
• Handheld interfaces
• Thick/thin custom clients
•Custom web applications
•Custom reports•OLTP Databases•Data warehouses
•Transaction processing (Mainframes and AS/400s)
•Old databases •Obsolete technologies
•ERP•Sales•CRM•HRMS•Supply chain•BI•CMS•Accounting
Maintenance Migration
New development Integration
Quality Assurance &Testing
Functionality, P
erformance and S
ecurity
© 2012 Computer Sciences Corporation - Proprietary 4
Tool support for Enterprise Testing
Test Management Tools Test Automation Tools
Test Design Tools
Data Preparation Tools
Review/Inspection Tools
Proprietary Test Runners
Custom Test Harnesses (and Drivers)
Dynamic Analysis ToolsStatic Analysis Tools
Comparators
Performance Testing ToolsSecurity Testing Tools
Lack of Unification into a Singular Platform
© 2012 Computer Sciences Corporation - Proprietary 5
Enterprise Test Automation Business ChallengesChallenge Description
Candidate Selection/Prioritization
Enterprise IT stack contains a large number (dozens to hundreds to thousands) of applications and often pose the difficulty to select and prioritize from automation perspective.
Tool/technology Proliferation Automation tools operate as vendor-locked and tightly coupled solutions with the technologies without giving the business a way to control them in a unified and standardized fashion.
Change Management Alignment
Change is the continuous mantra (maintenance, migration, integration, retirement etc) in the IT stack and often times the strategy to align automation remains out of control.
Success Criteria Definition and Measurement
While ROI remains a known metric, continuous measurement of the value delivered by automation requires a rich definition of the success criteria and continuous measurement (with corrective actions)
Future-proofing automation assets
The percentage of assets that can be used beyond the scope of the planned cycles remains an unknown and often times events like tool migration etc make some of the old assets vendor-locked and void.
© 2012 Computer Sciences Corporation - Proprietary 6
Enterprise Test Automation Technical Challenges
1. Multiple testing tools for heterogeneous technology stacks2. Multiple interfaces(GUI, Web, API, EDI, Mobile) with varied automation needs 3. Test coverage involving emerging versus legacy applications4. Test scenarios involving complexly integrated partner systems5. Test suite adequacy for controlled and emergency releases6. Capability of automated regression test automation to find new bugs in old
functionality7. Changing test priorities
1. Compliance vs. Business functionality2. Manual bug hunts vs. Automation efforts3. Functionality vs. Performance vs. Security
8. Lack of rich test data (variety &volume) that simulates the live enterprise9. Impact of Migration on Test Automation assets10. Multi-environment deployments (repetitive testing)
1. Synchronization2. Data population
© 2012 Computer Sciences Corporation - Proprietary 7
TestPlanningToolkitExecutive
QualityControl
Dashboard
TestPlanningToolkit
Reusable Automation
Assets
TestDesignToolkit
Test Data
Toolkit
Test Execution
Toolkit
Test Management
Toolkit
Packaged Application
Accelerators
TestInfrastructure
Toolkit
ExecutiveDashboard
Enterprise Test
AutomationApproach
A Structured Approach
Market Test
Automation Tools
Internal Test Automation Tools
Automation Asset Library
Automation Methods
Automation
Approach
Enterprise Applications
Test Automation
© 2012 Computer Sciences Corporation - Proprietary 8
Test Planning Toolkit
ROI Calculator
» Web enabled application to forecast returns and thereby prioritize automation projects
» Ease decision making process based upon qualitative and quantitative ROI information
Feasibility Analyzer
» Logical, quick and efficient way to identify automation candidates from the application landscape
Select, Prioritize and Plan for test automation
ROI Calculator Feasibility Analyzer
© 2012 Computer Sciences Corporation - Proprietary 9
Reusable Automation Assets
Reusable assets and Process assets to reduce time to market » Over a period of time CSC has developed lots of reusable assets and Process assets» CSC ACOE made those assets generic and can be used for any projects.» Category wise reusable assets repository has set of assets that enables to start and execute an
automation project with minimum expendable effort » Process assets has Checklists, Guidelines, Templates and process documents
Category Number of functions
Test Logging & Reporting 133
Notification Utility 25
Test Management Tools Integration 26
Ready-made Plug-In for Global Utility, Component/Reusable Functions,Events,Business Processes. 66
Exception handling & Diagnosis Information 14
Folder Management 5
File Management 38
Web Application's Control 263
Technology specific (Linux+Sieb_GlobalFuncs+Siebel_DP_Functions) 54
Total number of reusable functions 624
050
100150200250300
Series1
Check lists and Guidelines
Hand Book-ATM - Test Automation
Automated Test Script Review Checklist
Test Automation Requirements Elicitation Checklist
Test Automation Framework Guidelines
Test Automation Guidelines
Test Automation Metrics Operational Definition
Test Automation Tool Evaluation Guidelines
Test Automation Engineering Process
Test Automation Framework Selection Guidelines
Templates
Test Automation Cost Calculator Template
Test Automation Execution Report Template
Test Automation Maintenance User Guide Template
Test Automation Plan Template
Test Automation ROI Calculator Template
Test Automation Scope and Traceability Template
Test Automation Suite Release Notes
Automation Test Tool Feasibility Report Template
Test Automation Tool Evaluation Template
Reusable assets Process assets
© 2012 Computer Sciences Corporation - Proprietary 10
Test Design Toolkit
Test Case Optimizer» Proprietary implementation of Pair
wise/Orthogonal and other scientific techniques that reduce the exploded test combinations into an optimized/smaller number
» Capability to deal with large Test Input Models
» Detects up to 80% of the total defects with optimized test cases
Web Services Test Case Generator
» Provides automated generation of test cases specifically for SOA testing
» Covers both positive and negative test cases ensuring 100% test coverage
» Reduces test case authoring effort up to 90%
Optimal and efficient test case generation
Test Input Model Test Coverage Configuration
Optimized Test Cases
Test
In
pu
t Mo
del Journey mode: Return trip, One way
Preferred time: Morning, Afternoon, Evening, Late night
Leaving from: Type1_City,Type2_City,Type3_City
Going to: Type1_City,Type2_City,Type3_City
Booking type: Economy, First class
Customer type: Registered, New
Adults: 1, 2, 3, 4, 5, 6
Children: 0, 1, 2, 3, 4
Test
Cov
erag
e C
onfig
urat
ion
2-way coverage strength O
ptim
ized
Tes
t C
ases #Total Test
Configurations for 100% coverage : 8640
#Total (reduced) Test Configurations : 31
© 2012 Computer Sciences Corporation - Proprietary 11
Test Data Toolkit
Test Data Generator» Large volumes of real-looking test data is
generated.» Support for 50+ data types for US & UK (like
real names, addresses, SSNs, custom-syntax and so on)
» Support for multiple groups of fields with dependencies like referential integrity
» No restriction on volume of data generated (speed: 1 million records per minute)
Field Data Validator
» Automates negative and security test data generation for use by developers to test code under development
» Provides both command line and GUI interface options
Reduce cycle time by quickly generating large volumes of test data
Field set Metadata
Volume configuration
CSV file with real-looking data
Fie
ld s
et
Met
adat
a <?xml version="1.0"?>
<TestFields>
<Groups groupName = "sample" groupNumber ="1">
<TestField fieldName ="person name" id = "42">• <PersonName>
• <Country>US</Country>• <Gender>Female</Gender>• <Format>FirstName</Format>
• </PersonName>
</TestField>
…
Vol
ume
Con
figur
atio
n
1000000 records
Rea
l-loo
king
da
ta person name,PhoneNumber,SSN,CompanyNameTASHA,+91(0)52522-86434 ext.6072,117-93-2658,Diamond of CaliforniaJULEE,+91(0)532-3213402 ext.5136,628-62-5959,MerriamWebster IncPENNY,+91(0)8343-878361 ext.5427,622-23-8243,Komag IncBRYNN,+91(0)2462-078412 ext.7113,174-80-4445,Sweetheart Cup Company Inc
© 2012 Computer Sciences Corporation - Proprietary 12
Test Execution Toolkit
Unified Interface Automation Framework» CSC keyword driven framework for test case development
that abstracts the process enabling test case development to be performed by SMEs & BAs
» CSC Hybrid framework is the combination of Functional Decomposition & Data Driver approach.
» Tool and Technology agnostic » Integrated with QC and TEMA, Customized report
generation
Abstract test execution and simulate third party integration
Other Automation Frameworks» GSTAR – Transaction based framework for Insurance
domain» FLAME – Accelerated framework for Finance Domain» Hybrid (Functional and Data Driven) Framework
» QTP ,Selenium and Silk Test» Wiki Based framework
» Finesse + Selenium
© 2012 Computer Sciences Corporation - Proprietary 13
Test Management Toolkit
TEMA (Test Management)
» Integrated QA suite with an intuitive web-based interface for effectively managing and monitoring the Quality Assurance processes
» Offers full product life cycle support and can be used to manage multiple test cycles
» Integrated with CSC customized version of the popular BugZilla bug tracking application which is further enhanced for mandatory authorization, limited privilege access and custom reporting features
Manage test lifecycle intuitively
© 2012 Computer Sciences Corporation - Proprietary 14
Packaged Application AcceleratorsAccelerate test automation of complex systems
» Integrates market, third party and homegrown solutions to accelerate key testing activities on packaged software» Provides accelerators to readily automate plain vanilla installations like
» Key SAP processes» Key Siebel processes» Key PeopleSoft processes» Key Oracle E-business processes
Module Manual Test Cases
Automation Test Scripts
Oracle-GL 127 112 Oracle-AP 138 122 Oracle-AR 80 74 Oracle-FA 67 60 Oracle-CM 30 29 Oracle-Iexpense 13 13 Oracle- Purchasing 90 90 Oracle-I procurement 27 27 Oracle-OM 159 155 Oracle-Inv 124 118
Total 855 800 SAP-GL 42 40 SAP-AP 32 30 SAP-AR 17 17 SAP-AA 33 33 SAP-Co 22 22 SAP-BA 5 5 SAP-MM 92 92 SAP-SD 152 152 SAP-IS Utilities 78 39
Total 473 430 PeopleSoft-ESS 21 14 PeopleSoft-MSS 23 20 PeopleSoft- Payroll 17 17 PeopleSoft-WFA 66 55
Total 127 106
0
50
100
150
© 2012 Computer Sciences Corporation - Proprietary 15 15
Test Infrastructure Toolkit
AIM (CSC Infrastructure Management)» Multi-tenant inventory and infrastructure management web application» Interfaces with Test Automation tools, providing an environment for
management of un-attended test execution and results monitoring
Manage test infrastructure remotely and efficiently
© 2012 Computer Sciences Corporation - Proprietary 16
Executive Dashboard
Facilitates Go/No-Go decisions for production releases
A centralized dashboard for management teams to view quality & productivity data and reports:» Enables benchmark comparison across enterprise testing projects» Provides just in time statistics» Relieves QA team from routine capture and computation processes» Eliminates errors caused by manual processes and wipes out data manipulation possibilities» Helps take informed decisions on project performance
© 2012 Computer Sciences Corporation - Proprietary 17
Contact:
Venkatesh KovvuriMobile: 09885579911Email: [email protected]
18© 2011 Computer Sciences Corporation