IEEE Buenaventura cs Chapter March 9 2016 v4

Advanced Continuous Test

Automation

Marc Hornbeek

Sr. Solutions Architect

2 Spirent Communications

Agenda - Advanced Continuous Test Automation

• Evolution of automated testing

• Requirements of DevOps CT at scale

• Implementation approaches

• Summary


Sept 9th, 2015

IEEE Buenaventura Section CS chapter meeting


Etymology

“Test” (verb) – early 17th century

“Automatic” – mid-18th century

Early fully automated systems in “production”

18th century clock production

Automated Testing History – Beginnings


Automated Testing History – Mechanical mid-1700’s

1736: Sea trial of ‘Harrison 1 “ marine chronometer solving the problem

of calculating longitude while at sea


Automated Telephone

Exchange – 1891

Automated Test History - Electric-Mechanical – late 1800’s


1943

“Colossus”

Digital

Computer

Code

Breaker

1939

A. Turing’s

“Bombe”

Enigma

Code

Breaker

1946

“ENIAC”

Ballistics

Tables

and H-

Bomb

Automated Testing History - Programming mid -1900’s

1954

Mass-

produced

computer IBM

650

Debugging and testing were not considered separate concepts


• 1961 - Computer Programming Fundamentals, Leeds, Weinberg

• 1963 - Flowchart symbols standardized, Rossheim

• 1964 - First Basic program, Dartmouth College

• 1965 – IBM 360 – 1 MLOC

• 1967 - Functional Testing of Control Programs, IBM

• 1967 – “Software Engineering”, NATO

• 1968 – “Software Quality Assurance”, NATO

1960’s – Large scale software systems, SQA emerges

Scripting

C

JCL

Unix Shell

Apollo 11

145

KLOC


• 1971 - IEEE Computer Society founded

• 1972 - C language, Dennis Ritchie and Brian Kernighan

• 1974 - MIL-S-52779 Software Quality Program Requirements

• 1975 – Microsoft founded

• 1976 – Apple founded

• 1976 - Software reliability : principles and practices, Glenford Myers

• 1976 - Design and Code Inspections, Michael Fagan

• 1976 - Cyclomatic complexity metric, Tom McCabe

• 1979 - The Art of Software Testing, Glenford Myers 1979

• 1983 - IEEE 829 Standard for Software Test

1970’s – Software testing science foundations


A Network Test Environment for Packet Switching

Networks, IEEC&E, 1983

1980’s/90’s – Software and Protocol test tools

• Code coverage tools

• Static analysis tools

• Unit test tools

• Load test tools

• UI test tools

• Data protocol testers

• Advanced scripting

LINT

QTP


2004+ Test infrastructure automation

11

• Test topology automation

• User and resource scheduling

• Lab resource management

Velocity Lab Manager

2010+ Lab-as-a-Service (LaaS)

• Elastic on-demand resources

• Test infrastructure as code

• Virtualized tools

Private or public or hybrid


Waterfall Test Process - 1956

Most tests are concentrated near the end of the pipeline

Bugs have a long find-to-fix duration


Agile Test Process : 1995

Collaborative teams test each iteration of a product development in “Sprints”. Agile emphasizes test automation. Does not prescribe infrastructures for integrating test activities across the development-to-delivery infrastructure.

Reference: “Agile Testing: A Practical Guide for Testers and Agile Teams”

Extreme programming & Test Driven Development (TDD)


DevOps Continuous Testing - 2007

Design Right Shift Left

Development

Wants agility

Operations

Wants stability

Continuously eliminate

process friction and waste

13.4 million google hits (was 4M 9/2015)

CI CD

CM

CT

“Infrastructure as code”






• Summary


Continuous testing is not a phase in the DevOps pipeline

CT is implemented as part of all DevOps pipeline stages in accordance with a continuous testing strategy specific to a product and

organization.

Myth Buster: CT is NOT a Phase !


Test Automation

Versions

Dependencies

Tests

Creation methods

Type of Test

Pipeline stages

Lab Automation

Resources

Stakeholders

Administration

Topologies

Production variations

Dimensions of

Test and Lab

automation


Continuous Testing Types

• Unit tests: white box test that validates code logic

• Static analysis tests: check source code logic and consistency

• Dynamic analysis tests: test the code performance

• Code coverage tests: measure % of code tested

• Integration tests: test that code changes work after a code merge

• Compatibility tests: Test feature interactions

• Conformance tests: Test conformance to standards

• Usability tests: Test user experience

• Application tests: Test the application functions

• API tests: Test the Application Programming Interface

• Regression tests: Verify changes didn’t break something

• System test: Test system wide-features

• Performance tests: Test capacity, throughput and delay attributes

• Security tests: Test that security features operate

• Administration tests: Test administration features


Continuous Test Pipeline Stages

Pre-Flight Continuous Integration (CI) Continuous Delivery (CD)


Large Scale DevOps CT Test Environment

• Many short cycles

• Aggregated changes

• Large pool of tests

• Orchestrated pipeline

• Elastic infrastructure

applies at every phase and in every process


Sizing Test and Lab Automation


Sizing Test and Lab Automation






• Summary


Design for Automated Continuous Testing

Test Driven Development practices

Layer by layer isolation:

initiate a class, independent classes, verification

Feature toggles

GUI registration

RESTful APIs

Integrated agents

Tool plugins

Kent Beck said, "Code that isn't tested doesn't work -

this seems to be the safe assumption."

However, not all applications are easy to test.


Five Tenets of Continuous Testing

Test fast

• Design for test

• Automate at start

• Production environment

• Automated tests

• Incremental

• Infrastructure as code

• Elastic scaling

• Smart scheduling

• Rugged lab

• Analytics

• Implications for test and lab automation

CT Tenets


Best Practices

• Continuous Testing

• Test Automation

• Test scripting

• Lab automation


CT Best Practices


Test Automation Best Practices



Lab-as-a-Service Best Practices


Advanced Continuous Testing

Automated

test

selection

and analysis


http://www.slideshare.net/Spirent/automated-continuous-test-selection-

methods-for-large-scale-devops


Advanced CT Test Selection Methods Focus

The test selection methods operate at different points

of the code-build-test pipeline

Building Testing Build Result

Coverage

Code Coding


Test Trends Test Selection Method

Cumulative code-to-test verdict correlation

factors (CF) are saved in a table for all code

modules and tests.

CF numbers are adjusted when tests are run

for a code module. (Note: Values are neutral

for new tests or code set to neutral value.)

Test cases are selected which have the highest

correlation to the changed code modules.

Experiences of Automated Testing in Continuous Change Management - MERGE 2014

conference

http://www.slideshare.net/perforce/automated-testing-in-continuous-change-management

Automated Continuous Testing Optimization - Simulation

Optimum

process

• High backlog rate

• Early stages have higher fail rate

• Process time is equal = continuous flow

What pipeline of processes yields the highest backlog depletion rate,

highest number of fixes, and highest number of changes per time unit?

What to control?

• Process time

• Pass rate

• Fail rate


Consequences of not doing CT properly

Culture conflicts

False negative and false positive

test results

Test escapes

2015 software glitches

3,200 US prisoners released early

HSBC 275,000 payments failed

114,720 wrong divorce payments

London trading down for two hours

600K RBS paychecks

Nissan 1 million recall

60% of Starbucks stores closed

F-35 Fighter target error

Test and Lab Automation

Micro-Services Architecture

Test and Lab Automation Tools

Framework


CT-Ready Tools

Continuous

Deployment

(CD)

Continuous

Integration

(CI)

Plu

g-i

ns

Plu

g-i

ns

SUT Tools

Lab Management

Physical, Virtual, Hybrid Lab

Analytics ALM

Orchestration

Continuous Test (CT)

CCM

Select tools framework to match your goals for high

availability and virtual environments

Select tools with RESTful APIs, plugins

Warning !


Success Story

Metric

Major release

(#months)

Minor release

(#weeks)

# Features

Defects

Integrations / day

Tests / day

Automated tests

After

3

2

150

10

100

10

85%

Before

6

4

113

1260

0.5

0.3

5%






• Summary


The future of Automated Continuous Testing?

Self optimizing processes

• Design for test

• Self-test

• Live Test

• Analytics

• Predictive

• Automated remediation


www.devops.com Hornbeek blogs

Continuous Automated Testing

References

www.spirent.com/solutions


Summary

Advanced Continuous Test

Automation is required for

DevOps at scale

Make the effort to understand and

use best practices

Preferred implementations use

Lab- as-a-Service

The future is end-to-end process

self-optimization


© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name

“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.

Specifications subject to change without notice.

spirent.com

Thank you

Marc Hornbeek

Senior Solutions Architect

[email protected]

Twitter: mhexcalibur

https://www.linkedin.com/in/marchornbeek

mailto:[email protected]

mailto:[email protected]

Technology

IEEE Buenaventura cs Chapter March 9 2016 v4