Digital Testing: How to enable Continuous Adaptive Testing (EN)

1 © 2017 CA. ALL RIGHTS RESERVED.

Digital TestingDriving Cognitive Adaptive Testing Practices

Jonathon WrightChief Technology Evangelist DevOps, Automation & Digital Assurance (R&D)

© 2017 CA. ALL RIGHTS RESERVED.

www.slideshare.com/Jonathon_Wright


Disclaimer

Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of March 1st and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.

Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.

Copyright © 2017 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies

THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of such damages.


Abstract

In the digital economy, everyone’s in the software business, and the landscape is constantly changing. The traditional approach to providing business value through lengthy plan–build–test–deploy cycles no longer delivers value in an increasingly competitive, consumer–centric marketplace. Merely compressing these cycles by adopting iterative or agile methods doesn’t address the innovation, speed–to–market, quality and resilience challenges organisations face.

Shifting to a more dynamic, adaptive and continuous model calls for new tools and practices that accelerate communication, collaboration, integration, measurement and automation. Whether labelled this as digital, shift–left or DevOps, understanding the detail of this journey is an essential part to becoming digital–testing ready.

Join Jonathon as he explores the fundamentals of continuous adaptive testing that will enable the continuous delivery pipeline, and to provide instant, smart, cognitive adaptive technology that will power the next generation of insight and value–driven delivery capable platforms. Revolutionising how you test in the digital future will drive smart business decisions and end costly, reactive rework and errors.

Jonathon Wright

CA

Chief Technology Evangelist | DevOps, Automation & Digital Assurance (R&D)


The success of our technologies largely depend

on the fact that, while we were speculating about the

possibility of ultra-intelligence, we increasingly

enveloped the world in so many devices, sensors,

application and data that it became an IT-friendly

environment, where technologies could replace us

without having any understanding, intentions,

interpretations, emotional states, semantic skills,

consciousness, self-awareness or flexible intelligence.

Memory as in algorithms and immense datasets now

outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015

“

”

THE NEED FOR SPEED

94%of executives face increased pressure to release appsmore quickly

“

Journey to the Digital Enterprise

© 2017 CA. ALL RIGHTS RESERVED.5

Based on 2014 Vanson Bourne study commissioned by CA


VA

LUE-

DR

IVEN

TIME

1960s 1990s TODAY

Digital Enterprise - ‘Evolution, over Revolution’

1 – “The Battle for Competitive Advantage in the App Economy”, Oxford Economics, 2015

of enterprises believe that they need algorithmic business models to respond to disruption in the next 3 years.78%

First WaveLegacy is your Legacy (API)

Four WaveDigitalization of your Core (SOA)

Eight WaveCognitive Adaptive Adoption


QUALITY

2/3of business leaders say the future of their business depends on the quality of their software2

VELOCITY

94%of executives face increased pressure to release apps more quickly1

1. 2014 Vanson Bourne study commissioned by CA2. “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 3. “DevOps and the Cost of Downtime” – IDC 2014

“Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015


COST

25%of a single application’s development and operations costs is wasteful3

While Ensuring a Superior Digital Experience!

Digital EnterpriseDelivering Quality at Speed

SHIFT RIGHT


Digital EngineeringCognitive Adaptive Technology

Machine Learning

Artificial Intelligence

A

B

forces-destiny.com


9 © 2017 CA. ALL RIGHTS RESERVED.© 2017 CA. ALL RIGHTS RESERVED.9

Digital Future(ist)

Dr. James Whittaker (www.docjamesw.com) 7 stages (http://a.co/gQk9ckx)

Internet of Everything (IoT Hot Tub)


Deep reinforcement learning

Digital TechnologyCognitive Adaptive Learning

Machine learning with neural dictionaries

https://www.luis.ai

“We are not at the centre of the universe (Copernicus), of the biological kingdom (Darwin) or of the realm of rationality (Freud). After Turing, we are no longer the centre of the infosphere, we share the infosphere with digital technologies”. The ethics of artificial intelligence, Luciano Floridi


Mode 0 - Cognitive

Cognitive Adaptive Testing (CAT):

1. “How long will it take to test these new features?”2. “What is the associated risk rating of the new platform?”3. “What is the defect density and prediction for this solution?”4. “What is the current conference rating of the solution?”5. “What is the value of this capability?”

How will Machine Learning help?

Digital TestingExample Case Studies of Machine Learning


Emergent embedded AIfor neural networks

Digital TechnologyCognitive Adaptive Intelligence (iCA)

Singularity Ultra-Intelligence (2030)

“Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s


Mode 1 - Adaptive

Cognitive Adaptive Testing (CAT):

1. “Can you create me all the tests I need to fully test this system?”2. “How do I successfully test this new feature?”3. “How much testing has been done already?”4. “Have you detected any anomalies within the system?”5. “How much testing will be enough?”

How will Artificial Intelligence help?

Digital TestingExample Case Studies of Artificial Intelligence


IdeasBeliefs, Values, Culture

Heuristics

Behaviors

Value

AgileLean

Digital TestingSolution Thinking – Continuous Testing 101 – Episode I

Copyright © 2017 nuCognitive LLC. All Rights Reserved.

Methods and Skills

Ray Arell


DesignOpsCreate and deliver Ideas to Outcomes

Pervasive Security

Redefine how work is planned, managed and

executed

Agile Management

Provide a flawless app experience optimized for

performance

Agile Operations

Accelerate and streamline development, testing and

release

Continuous Delivery

Continuous Development Continuous Adaptive Delivery Continuous Release

Digital TestingShift Left introducing DesignOps


DevOps Hub

Support the pace and scale of Agile Fully Automated & Repeatable Model based Continuous Delivery supporting highly heterogeneous SDLC tool ecosystems Remove ambiguity from requirements & testing Abstraction of dependent services/APIs radically accelerate speed and quality “Pre instrumented” app releases Flexibility/portability across modern infrastructure & cloud platforms

Continuous Adaptive Delivery

Release Everywhere


DesignOps – “Shift Left” (AND Right)

CONTINUOUS ADAPTIVE TESTING

CODE

SHIFT LEFT SHIFT RIGHT

BUILD DELIVERY PRODUCTION


CODE

SHIFT LEFT SHIFT RIGHT

BUILD DELIVERY PRODUCTION


DesignOps – “Shift Left” (AND Right)


Build

Deliver

MonitorMeasure

Learn

Design

Make

Check

Think

Proactive over Reactive

Governance over

Ownership

Maturity over Mastery

Predictive over Continuous

Evolution over Revolution

Engineering over Agility

Intelligence over Things

designops.net


DIGITAL ASSURANCE

VALUE-DRIVEN DELIVERY

1Cognitive Adaptive Learning

2Cognitive Adaptive Testing

3Cognitive Adaptive

Technology

4Cognitive AdaptiveDelivery

Digital TestingCognitive Adaptive Adoption (CA-A)


Erik Simmons

Copyright © 2017 nuCognitive LLC. All Rights Reserved.

Digital TestingValue-Driven Delivery – Continuous Testing 101 – Episode II

22 © 2016 CA. ALL RIGHTS RESERVED.22 © 2017 CA. ALL RIGHTS RESERVED.

Portfolio Management

Unify long term strategy, investment and portfolio planning.

Collaboratively plan, prioritize and track work across the enterprise.

Project ManagementDevelop & Test

Synchronize efforts to dramatically speed app development & increase quality

Manage & Monitor

Make a great customer experience a competitive advantage

Release & Deploy

Control the release process, to continuously advance application quality, improve the customer experience and reduce costs.

ContinuousDelivery

Minimum Viable Experience

Minimum Viable Quality

Value-Driven Delivery

BUSINESS INITIATIVE

Complexity-Informed Organizational Change

BUSINESS VALUE

MinimumViable, Validate, ValueProduct

CUSTOMER

THINK

MAKE

Digital TestingContinuous Adaptive Adoption (CA-A)


REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO

AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS

CONTINUOUS DELIVERY MAINFRAME SECURITY


API analyticsRelease management dashboard Access incident responseReal-time app performance

Digital TestingContinuous Adaptive Delivery (CA-D)


Automate manual steps and scripts—eliminating repetitive, error-prone tasks

Design reusable deployment workflows

Integrate best-in-class tools dev through production

Gain visibility and transparency

Close feedback loops throughout the pipeline

Efficiently release multi-team, cross-app, composite releases

Define and enforce continuous delivery best practices

OPTIMIZED

AUTOMATED

CONTINUOUS

Monitor, measure and analyze release processes

Maximize quality as the velocity and volume grows

Improve business execution

Measure and improve team productivity and maturity

3

21

Digital TestingNavigate the Continuous Adaptive Delivery Journey


The ability to reliably release high-quality solutions at any time


Accelerate Deliveries

Reduce Errors

Manage Complexity

Increase Visibility

Drive Collaboration

Continually Improve

Digital TestingContinuous Adaptive Testing (CAT)


SOI

Visualization

(Big) Data

Analytics

Inference (Machine

Learning & AI)

Intelligent Process

Automation

Digital Technology Integrations

Digital TestingSystems of Intelligence



AUTOMATION MAINTENANCE

STATIC RULES BIG (un-structured) DATA

IOT, IOE

CX / DX SECURITY

STRUCTUREDAUTOMATION

Digital TestingKey Challenges Posed by Adaptive Digital Systems


AUTOMATED & SELF-HEALING AUTOMATION

DYNAMIC/ADAPTIVE RULES

BIG DATA ANALYTICS

IOT-Ops

DX-Ops SEC-Ops

INTELIGENT / ADAPTIVE / COGNITIVE AUTOMATION

+ +

+

+

+

+

+

Digital TestingWhat we need for next-gen DesignOps


Failure detection and prevention

Scripted automation

Limited process automation based on static rules

Individual metrics based on structured data in lifecycle silo

Limited insight and perspectives

Reactive: deals with “Past”

Failure prediction

Automate the automation

Intelligent process automation based on dynamic inference

Analytics based on large scale structured and unstructured data across the lifecycle

Focus on Insights

Predictive: from “What did happen?” to “What would happen?

TRADITIONAL AUTOMATION

COGNITIVEAUTOMATION

Digital TestingIntelligent Automation Characteristics


DESIGN DEVELOP TEST DEPLOY OPERATEDESIGNOPS

DATA

HUBS

DEVOPS HUB

Digital TestingCognitive Adaptive Delivery (CA-D)


AUTOMATE THE AUTOMATION

AutonomicRoot Cause Tests

Creation

Autonomic Test

Creation

Autonomic Virtual Service

Creation

Provision Configure Deploy TestBuild/CIDev

Dev Test Prod

Environments

Digital TestingCA Agile Requirements Designer + CA ArD Server + CA Automator


Please can you build tests directly into the pipeline?

Model-Driven Design of Tests from Use Cases 1

2

3

Generate ‘Automation as Code’ from Model

Add ‘CA Automator’ Task to ‘Pipeline as Code’

“Sure, just assign me the tasks and I will automatically generate the necessary artefacts within whichever release automation platform you want!”

Digital TestingTest in DevOps (TiD)


CHANGE: Model-Driven Design (Code,

Requirements & Tests)

Dynamically generated Virtual Automation

Environment Provisioning

Dynamically generated Tests based on

Cognitive Adaptive Testing

LEARN: Model-Driven Delivery (Results, Defects & Tests)

DESIGN OPSDEV TEST

Digital Testing

Plan Define Unit test Code Check in Build Provision Test Release

Infinite Probability Drive

Automation asCode

Pipeline asCode

Infinite Probability Drive Cognitive Adaptive Learning


Use Case

1. Requirement / Use case / User story (Rally)2. Generate Model-Driven Design flows (ArD)3. Optimize & Generate Automation (ArD)4. Workflow ‘Automation as Code’ (AD Automator)5. Populate ‘Data as Code’ (ArD Databuilder)6. Customise ‘Configuration as Code’ (YAML)7. Build ‘Application as Code’ (Jenkins)8. Provision ‘Platform as Code’ (Docker)9. Virtualize ‘Infrastructure as Code’ (SV, NV & NFV)10. Deliver ‘Pipeline as Code’ (RA CDE, Puppet & Chef)11. Deploy Headless Test Runners (MicroContainers)12. Deprovision ‘Environment as Code’ (GIT / Blob)13. Analyse ‘Results as Code’ (ArD Server)14. Pinpoint Failure (Optimizer NLP)15. Generate new Model-Driven Delivery flows (CLI)

Pinpoint Failure Analysis

Model Flows Optimize Generate

Workflow Code Provision

Execute Analyze PinpointIterate

Coverage


Pipeline

Automation

Configuration

Data

Code

Test

Use Case

Infrastructure

Platform

Microservices

Release

Environment

Applications

Model

Co

mm

and

Lin

e In

terf

ace

for

Test

ers

Digital Testing‘Automation of Everything (AoE)’


*CA Agile Requirement Designer - Automation Builder has native support for EggPlant, Ranorex, Selenium 3.0 and Cucumber.

Digital Testing‘Automation as Code’


Cognitive Adaptive Intelligence (CAi)

Release Management

Environment Management

ApplicationManagement

Network Management

Infrastructure Management

RepositoryManagement

Data Management

Test Data LearnProbe

CA App Experience Analytics

Digital TestingCognitive Adaptive Intelligence (CAi)


Portfolio backlog prioritization and

value analysis

Improved scheduling and

capacity

Validated requirements in

backlog

Improved estimates

Predict failures based on code

commits

Dynamic code promotion

criteria

Predict defects/failures

Optimized Tests

Scenario identification

based on trends

DX Scores

Anomaly detection

PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase

Cognitive Adaptive Intelligence (CAi)

Examples of Cognitive Adaptive

Intelligence

Examples of Outcomes

Investment decision analytics

Code quality

analytics

TestOptimization

& Defectprediction

A/B Testing

& Canary

Log & DX Analytics

DX-basedrequirements

Digital TestingCognitive Adaptive Intelligence (CAi)


> 50%large organizations globally will compete using advanced analytics & proprietary algorithms

40%enterprises' net-new analytics investment will go to predictive / prescriptive

75%technology-oriented business intelligence competency centers will evolve to focus on information value generation

Source: Gartner

Digital TestingIntelligent Analytics is the New Battleground


What were the past defect trends?

What were the root cause of the past defects?

What defects are we likely to find?

What preventive measures should we take?

Have we overlooked anything for defect prevention?

Predictive analytics

Descriptive Analytics

Diagnostic Analytics

Prescriptive Analytics

Deep Learning

Digital TestingExamples of Analytics Use cases


* CA’s Analytics Reference Architecture

Data Collectors

Dat

a A

ggre

gato

rAgents

…

Probes

Machine/Log File Data

Insight-Driven

Existing Platforms

Custom Platforms

New Platforms

Partner Platforms

Analytics Engine

Dat

a In

ject

ion

AP

I

Dat

a A

cces

s A

PI

Service

Layer

Dat

a Ex

trac

tio

nData Processing (Speed)

Data Processing (Batch)

Data Bus

‘Shift Right’ Becoming an Insight-Driven Business

Digital Testing

42 © 2017 CA. ALL RIGHTS RESERVED.© 2016 CA. ALL RIGHTS RESERVED.© 2017 CA. ALL RIGHTS RESERVED.

BRINGS TRUE ARTIFICIAL INTELLIGENCE (AI) TO LIFECYCLE VIRTUALIZATION Virtualize services without requiring any knowledge or

decoding of the service protocols

Applies a genome sequence alignment algorithm, discovers byte-level patterns in message protocols

Now virtualize a much wider range of protocols without requiring a new DPH

HIGH ACCURACY: 99.6 – 100%

Increased speed and accuracy with Entropy Weighting + Message Clustering

The more data a service observes the data, more intelligent it becomes

Perfect for performance testing where we deal with tons of data

Digital TestingCognitive Adaptive Learning (CA-L)


Digital TestingCognitive Adaptive Technology (CA-t)

1. Digital Mobile, PerfectoMobile, Chapter 22, Jonathon Wright


Where next?


Where next?

Digital Assurance - Pocketbook

Blogs - #ExcuseFreeTesting

Cognitive AdaptiveTesting

CA Technologies


Digital AssurancePocketbook

Paul Gerrard

NEW exclusive Pocketbook authored by Paul Gerrard, eminent Test-Transformation consultant, thought leader, teacher and international conference speaker.

Order your FREE copy today from http://bit.ly/digital-assurance


Digital AssurancePocketbook – World Tour

TORONTO LOS ANGELES SILCON VALLEY


Blogs #ExcuseFreeTesting – Legacy is your Legacy!

https://dzone.com/articles/legacy-is-our-legacy-1

https://dzone.com/articles/legacy-is-our-legacy-1


Q & A

Why CA: Don’t Take Our Word For it

CA is Positioned as a Leader in the Gartner MQ for ARA … Forrester Wave SV Leader

Faster time to deploy releases*

20x

Reduced testing costs & improved efficiency*

25%

Reduction in production errors*

98%

Differentiation Customer Success

5 of the top 5Telecom Companies

4 of the top 5 Retailers

4 of the top 5 Banks

Market Validation

Nominated in the “Best Use of Technology in a Project” category for

work with the London Metal Exchange

*Based on Global 500*Forrest TEI Research

51

© 2017 CA. ALL RIGHTS RESERVED. CA CONFIDENTIAL AND PROPRIETARY INFORMATION; FOR INTERNAL USE ONLY. NO UNAUTHORIZED USE, COPYING OR DISTRIBUTION.

Continuous Adaptive Delivery Ecosystem

Write user stories (requirements)

Right data for the right test environment

Dev Test QA Prod

Continuousintegration (Build-Deploy-Test)

Continuous Delivery Workflow

Build / deployment / promotion dashboard and reports

Adaptive, dynamic releases Automated & manual testing/test data Deployment workflow, rollback, pre/post tasks

Automated testkick-off, tracking

Testing coverage & progress reporting

Trigger build & deploy in Dev

Runbook

Deployment automation

Release & artifact tracking, environment mgmt, baselining

Applicationknowledge

Link application to user stories

To Do Doing Done

ManageAuthorize

Trigger release deployment

Plan Releases

Plan ReleaseTrains

Promote

Define release workflow, gates& environments

Watch progress (tracing, auditing, reporting)

Create release package with build /deploy

status

Get app definition

Release ManagerProduct OwnerBusiness UserScrum Master

EnterpriseArchitect

Dev

Developers

QA

Test/QADeployment

Manager

Customer

Implement components

Define app: Components & versions

Infra provisioning / environment mgmt / PaaS / containerization / configuration mgmt

AgileCentral

Use app

Simulate unavailable systems for full testing


© 2017 CA. All rights reserved.

Thank you!

#ExcuseFreeTesting