Automated Continuous Test Selection for Large Scale DevOps
Marc HornbeekSenior Solutions Architect
2Spirent Communications
What is DevOps?No standard DevOps definition !
Design Right Shift Left
Development
Wants agility
Operations
Wants stability
Continuously eliminate process friction and waste
http://itrevolution.com/
the-history-of-devops/
GeneKim
8.2 million google hits (up from 4M mid September)
CI CD
2009 Flikr 10 deployments/day
21 million
CM
CT
Patrick Debois2007
“Agile Infrastructure”
3Spirent Communications
Spirent and DevOps
Continuous Deployment
(CD)
Continuous Integration
(CI)
Plug
-ins
Plug
-ins
SUTTest Tools
Lab Management
Physical, Virtual, Hybrid Lab
Analytics ALM
Orchestration
Continuous Test (CT)CCM
Test toolsCT OrchestrationLab Management
Professional Services
4Spirent Communications
Large Scale DevOps
100+ dev, multiple sites
10 MLOC+, multi-language
100+ features 10+ build targets
Complex dependencies
200+ topologies10K+ test cases10+ deployment
packages
Millions of test possibilities
5Spirent Communications
Traditional Test Selection Environment
Separate test phases each with its own fixed test suites
Large # of changes per test phase
Fixed resources and time for running test suites for each phase
6Spirent Communications
Large Scale DevOps CT Test Environment
• Many short cycles• Aggregated
changes• Large pool of tests• Orchestrated pipeline• Elastic infrastructure
CT applies at every phase and in every process
7Spirent Communications
Importance of automatic test selection for Large Scale CT
More test beds and complex topologies drives need for efficient utilization of resources
Find high severity defects early
More people testing drives need for efficiency
8Spirent Communications
Test SelectionMethods Evolution
Small scaleModules Single test phaseSnapshot analysis
Large scaleSystem-widePipeline of cyclesTrend analysis
9Spirent Communications
Common CT Test Selection Methods
Base suite always selectedRe-run last failed test casesPriority assignmentsSpecial cases for compliance
and performance tests “All-tests” policies
Static test suites are suitable but may not be optimal for small to medium scale CT
10Spirent Communications
Advanced Large Scale CT Test Selection
Methods
1) Tag code and tests2) Static code analysis 3) Build artifacts 4) Test coverage 5) Test trends
Test selections are automatically determined by…
… and run-time and resource attributes
Which tests will yield the most
critical failures?
11Spirent Communications
Advanced CT Test Selection Methods Focus
The test selection methods operate at different points of the code-build-test pipeline
Building TestingBuild Result
Coverage
CodeCoding
12Spirent Communications
Automated test selection challenges
Too many testsToo
man
y re
sult
sUncertain test coverage
Analysis time
Wasted resources
Diffi
cult
to
lear
n
Wasted time
Too many bugs
Irrelevant results
Diffi
cult
to
debu
gFalse positivesFalse negatives
Setup timeEx
ecut
ion
tim
e
Maintenance effort
Poor
inte
grat
ionDoesn’t scale
Complex tools
Dependencies
13Spirent Communications
Comparison criterion used forlarge scale CT test selection
Person-hours spent on test activities
Critical as % of total # defects
Test resource utilization
Measures(per deployment)
14Spirent Communications
Method 1: Tag Code and Tests
Descriptive attribute “tags” are manually bound to code and test modules.
Tag list is generated for changed modules.
Select tests with tags that match the changed code module tags.
15Spirent Communications
Method 1 Results: Tag Code and Tests
Heterogeneous systems
x Misses important dependencies
Easy to implementx Tag maintenance
x Too many tests and resources
16Spirent Communications
Method 2: Static code analysis
Test cases are manually associated with code modules.
Static code analysis metrics (memory leaks, uninitialized variables & pointers, etc.) identify risky code areas.
Select tests that are associated with highest risk code.
17Spirent Communications
Method 2 Results: Static code analysis
Good results for modules
x Misses important dependencies
Easy to implementx Manual code to
test and results association
More efficient than static selection
x Tool costs at large scale
Note: These ratings assume the system programming language is supported by the tool.
18Spirent Communications
Method 3: Build artifacts
Test cases are manually associated with artifacts that are generated by the build process. (image and data files)
Changes in the artifact files resulting from the build process are identified.
Changes in build artifacts are used to select tests.
19Spirent Communications
Method 3 Results: Build artifacts
Works well for changed modules
x Misses important dependencies
Improved over static selection methods
Scales well
Fairly easy to implement
Maintenance of test case associations
20Spirent Communications
Method 4: Test Coverage
Manually associate test cases to code statements.
Test coverage metrics identify risky code areas.
Select tests that are associated with least covered areas.
21Spirent Communications
Method 4 Results: Test Coverage
x Programming language dependent
x Misses important dependencies
x Low yield for critical defects
x Tests must be created for missing areas
x Tooling costx Maintenance of test
associations with code statements
x Costly to instrument code at large scale
22Spirent Communications
Method 5: Test Trends
Cumulative code-to-test verdict correlation factors (CF) are saved in a table for all code modules and tests.
CF numbers are adjusted when tests are run for a code module. (Note: Values are neutral for new tests or code set to neutral value.)
Test cases are selected which have the highest correlation to the changed code modules.
23Spirent Communications
Method 5 Results: Test Trends
Applies system wide
Program language agnostic
Identifies important dependencies
Flexible at scalex Tooling, initial setup effort
Easy maintenance Fast and relevant
results analysis
24Spirent Communications
Comparison of Test Selection Methods Applied to Large Scale CT
Method
1. Tag code & tests
2. Static analysis
3. Build artifacts
4. Coverage metrics
5. Test trends
25Spirent Communications
Test Trends - Architecture
Reference white paper:http://www.perforce.com/sites/default/files/continuous-change-driven-build-
verification-wp.pdf
The system is a simple form of predictive analytics that uses a table to keep track of code-to-test correlation (CODTEF) factors which drive test cases selection.
CODTEF values self-adjust.
26Spirent Communications
Test Trends – Results Analysis
Results analysis can be derived by “running the matrix in reverse” because the test results correlate to code modules.
Test Results
CodeOwners
CodeOwners
CodeOwners
27Spirent Communications
The test trend method can be applied to other change factors:TopologiesHardwareEnvironment factors (power,
cooling, humidity, etc.0
Test Trends – Additional Applications
28Spirent Communications
Test Trends - Implementation Approach
Break up current test suite into a small set of groups. Identify code group – test group relationships. Tooling to automatically identify group code changes, launch specific test groups, report results according to group owners
Identify code subgroups and initial test correlations. Identify and design specific tools changes.
Implement remaining tools except the advanced metrics
Determine which advanced metrics will be used by the organization (eg. reliability growth modelling)
Create metric tools using the data collected during the Change-Driven loops.
Phase 1Group Level
Phase 2Sub-Group Level
Phase 3Advanced Metrics
29Spirent Communications
www.spirent.com/clear
1) CT Blueprint white papers2) CT eBook3) CT Assessment Tool
Take-aways
http://www.perforce.com/merge/2014-sessions/automated-testing-continuous-
change-management
http://www.perforce.com/sites/default/files/continuous-change-driven-build-
verification-wp.pdf
30Spirent Communications
Best Practices Assessment Tool simplifies the assessment process.
31Spirent Communications
Summary
Of the five methods considered for large scale CT, Test Trend method was preferred
A drawback of the Test Trend method is the complexity of tooling
Test Trend method has additional benefits for analytics and system applications
32Spirent Communications
Questions anyone?
33Spirent Communications
© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name “Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved. Specifications subject to change without notice.
spirent.com
Thank youAutomated Continuous Test Selection for Large Scale DevOps
Marc HornbeekSenior Solutions Architect
[email protected]: mhexcalibur