The Future of Testing: Continuous Enterprise Testing - 1035 - Thomas Hadorn - The Future... · Test Automation Risk-Based Test Management Browser-Based Load Testing Test-Driven Service

© 2018 by .

Thomas Hadorn

The Future of Testing: Continuous Enterprise Testing ANZTB Test Conference, Canberra, 1 June 2018

© 2018 by

Dev Ops

© 2018 by

Critical Physical Limit Technical Complexity

& Rate of Change

Years Months

Months Weeks

Weeks Days

Days Hours

Hours Minutes

Delivery Cycle Time

Perceived Disruption Software Quality

Challenge

Today

© 2018 by

Requirements Design Implementation Testing Acceptance Deployment

Inception Elaboration Construction Transition

Development Operations

DevOps

Agile Event Horizon

Digital Singularity

Incremental Rational Unified

Process

4

Agile Scrum, Kanban

2

Continuous DevOps

1

6+

Big Bang Waterfall

= wall of confusion

Flexible Framework

Rigid Rules

© 2018 by 5

Present

80%

Man

ual t

estin

g

20%

Aut

omat

ed te

stin

g

UI

Load

API

The reliance on manual testing is the

top technical challenge in app development

World Quality Report, 2015/16

Test automation requires developers

© 2018 by

Present Future

80%

20

%

Man

ual t

estin

g A

utom

ated

test

ing

Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT)

Optimize

!

© 2018 by

Present Future

remaining manual tests manual spec-driven and exploratory

80%

20

%

Man

ual t

estin

g A

utom

ated

test

ing

Progression testing

t

Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT) Optimize

!

Test the unknown. Session-Based Exploratory Testing

Explore

!

© 2018 by

Present Future


80%

20

%

Man

ual t

estin

g A

utom

ated

test

ing A

utom

ated

test

ing

+85%

Progression testing

t

Regression testing t

Higher coverage. Fewer tests. Model-based Test Coverage Design (MBT)

Optimize

!


Explore

!

© 2018 by

Present Future


automated UI tests

automated load tests UI and API

automated API tests

80%

20

%

+85%


!


Explore

!

Automating the automation framework for UI and API. Model-based Test Automation (MBTA) Automate

!

© 2018 by

Present Future


automated UI tests


automated API tests


!


Explore

!


!80%

20

%

+85%

Increase the testing window. Test-Driven Service Virtualization

Provide matching test data in time. Stateful Test Data Management

Manage

!

Integrate

!Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution

+

Virtualize data and systems key enabler for high automation

© 2018 by

risk

cove

rage

60%

80%

100%

40%

20%

0%

𝑴/𝑵 =

0

defe

ct ra

te

max

100% test cases

20% 0% 40% 60% 80%

critical defects

M N 𝟏 weight 𝟏 𝟏 𝟏 𝟏 𝟏 𝟏 … …

[%]

© 2018 by

60%

80%

100%

40%

20%

0%

risk

cove

rage

100% 20% 0% 40% 60% 80%

test cases

critical defects 0

defe

ct ra

te

max

M N 𝒘↓𝟏  𝒘↓𝟐  𝒘↓𝟑  𝒘↓𝟒  𝒘↓𝒊  𝒘↓𝑵  … … … … 𝒘↓𝒊 ≥ 𝒘↓𝒊+𝟏  weight

© 2018 by

60%

80%

100%

40%

20%

0%

Intuitive Test Design

Methodical Test Design

100% 20% 0% 40% 60% 80% Test Cases Risk

Coverage

Achieve maximum risk coverage with a minimal number of powerful test cases.

© 2018 by

choose type of vehicle

display premium

choose insurance

cover

provide insurant details

provide vehicle details

Truck

Car

Bike

Private

Commercial

Purchase no

Purchase - yes

Get a Quote

Purchase yes

Auto generate… Orthogonal Arrays Linear Expansion

Pairwise Tests All Combinations Negative Tests Boundary Tests

© 2018 by

Present Future


automated UI tests


automated API tests

80%

20

%

+85%


Explore

!

© 2018 by

Formal Testing Informal Testing

Exploratory Testing Manual Testing

Automated Testing

Ad-hoc Testing

Testing Continuum

© 2018 by

Risks Performance Issue

Usability Issue

Functional Issue

Stability Issue

Reliability Issue Scalability Issue

Coherence Issue

Understandability Issue

Testability Issue

Convenience Issue

Security Issue

Accessibility Issue

© 2018 by

Risks

I’m an automated test case.

Usability Issue

Stability Issue

Reliability Issue Scalability Issue

Coherence Issue

Understandability Issue

Testability Issue

Convenience Issue

Security Issue

Accessibility Issue

Performance Issue

Functional Issue

Automation is doing, what automation does.

© 2018 by

1 2 3 4 5 … … … … … … …

Risks

© 2018 by

I’m an automated test case. I’m the same test case executed manually.

© 2018 by

Plan as you test!

Story-Based

Motivating

Credible

Exploratory Branching The art of deciding on what to test next.

New Testing

Idea

Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation

© 2018 by

Specification Based Testing

Exploratory Testing

Analyze Potential Risks Problem vs. No Problem

Monitor Known Risks Pass Result vs. Fail Result

*That’s an illustration. Don’t confuse it with reality.

© 2018 by

Exploratory Testing « Testing as performance »

Formal Testing « Testing as artifact creation »

Monitor Known Risks Confirm what you already know (measurable things)

Intelligent Testing Create new test ideas based on what you have learned

Mechanical Testing Process pre-defined data in pre-designed steps

Analyze Potential Risks Focus on the things you don’t know

Low Information Value Repeat what you have already learned

High Information Value Learn something new

High Risk Coverage Easy to scale because it’s parallelizable

Low Risk Coverage Hard to scale continuously because it relies on humans

Change Detector Problem Detector

© 2018 by

Present Future


automated UI tests


automated API tests

80%

20

%

+85%


!

© 2018 by

100%

0%

Sprint Start

Sprint End

API

UI

Start testing early, shift left test automation Degree of

Completion

Time Efficiency

Gain

+4x Creation

+6x Maintenance

+20x Execution

© 2018 by

1

n

Test Automation Framework

(TAF)

Record Replay

Test Data Keywords …

Manually, coded framework.

Scr

ipt-B

ased

(Cod

ed T

AF)

B T

Automating the automation framework.

Mod

el-B

ased

(aut

omat

ed T

AF)

B T Automation

Model

Model-Based Test Automation

(MBTA)

Scr

ipt-B

ased

B Business Information T

Technical Steering

Information

B and/or T change requires re-recording.

Test Coverage Design (MBT)

© 2018 by

Customer Portals

Websites & CMS

Systems of Engagement

Web Server, API Gateway

Social

Mainframe

COTS

Legacy

Systems of Record

AppSrv/ESB

BI / Analytics

Open-Source

Model-Based

© 2018 by

Typical Enterprise QA Staff

Programmer Scripter Non Technical SME

Effective Velocity

Script-Based Open Source

Script-Based

Commercial

Model-Based

Typical Startup

© 2018 by

Present Future


automated UI tests


automated API tests

80%

20

%

+85%


!

© 2018 by

Browser-Level Users

Cloud Age Before Cloud

2011

60% web pages < 50 requests/page

80% web pages < 200KB JS/page

Release cycle > weeks

Memory Footprint

Browser-Level

Testing Complexity

2018

60% web pages > 100 requests/page

80% web pages > 200KB JS/page

Release cycle < days, minutes, sec

© 2018 by

15 GEOGRAPHIC

REGIONS

+36 +14& hybrid

Cloud economy of scale

© 2018 by

Present Future


automated UI tests


automated API tests

80%

20

%

+85%



Manage

!

© 2018 by

*voke, market snapshot report on service virtualization - 2012

On average, organizations require access to 33 systems for development or testing.

18 Average # of systems with

unrestricted access

96 % of testers have

restricted test lab access

Testing Present

Enterprise system landscapes are alike disease gene networks.

Welcome to the tester’s hell.

© 2018 by

Insurance App

CRM CAR CRM RATING ORDER

Business Logic API (e.g. Web-Services)

API (e.g. Web-Services)

Instable or not available applications

Test Data search & preparation

(expensive) 3rd party or Cloud based services

Decoupled, outsourced Development & coordinated

Deployment, multiple Vendors

© 2018 by

Insurance App




Virtualising systems replicates

system complexity

Virtualisation

Virtualising components creates

inconsistencies

Virtualisation is not

a fallback plan

© 2018 by

Insurance App




Orchestrated Service Virtualisation

© 2018 by

Insurance App




Test-Driven (Orchestrated) Service Virtualisation

Virtualising for automated tests eliminates

system complexity

Virtualising for automated tests solves

inconsistencies

Model-based Test Automation

Virtualisation is not a fallback plan,

it’s the first line of defence

© 2018 by

80’ and 90’ Millennia 10’…

Record/Replay Record/Replay Record/Replay

Keyword + Data Driven Frameworks

Model-Based

Automation (MBTA)

Test-Driven Service Virtualisation

Model-Based Test Design (MBT)

Exploratory Testing

Script-Based Model-Based 20’…

Machine Learning

10%

30%

80%

R I

S K

C

O V

E R

A G

E

Framework + Traditional

Commercial Automation Tool

Cucumber

+ Selenium

Systems of Record

Systems of Engagement

© 2018 by

Present Future


automated UI tests


automated API tests


!


Explore

!


!80%

20

%

+85%



Manage

!

Integrate

!Automate build to release process. Continuous Integration/Testing/Delivery (CI, CT, CD) Distributed Execution

+

Virtualize data and systems key enabler for high automation

© 2018 by

Jan 2011

Aug 2014

Dec 2015 Nov 2017

“It should be considered by enterprises that have struggled to make test automation work, and by those seeking to support agile continuous automation practices.”

© 2018 by 41 41

Model-Based Test

Automation

Risk-Based Test

Management

Browser-Based Load

Testing

Test-Driven Service

Virtualization

Model-Based Test Coverage

Design

Session-Based Exploratory

Testing

End-to-End Functional & Load Testing

Optimize Automate Explore

Optimize

10X Compared to other testing tools.

30-40% Cost Reduction

2X Risk Coverage

2-3X Release Speed

The Tricentis Tosca X-Factor Continuous Testing Platform

© 2018 by 42

APAC fast-growing customer footprint

42

© 2018 by 43

Focus Area M1 M2 M3 M4 M5

Most enterprise customers are on level M1 when they decide for Tosca.

Exploratory Testing

!

Explore

!

Risk Based Prioritization

Test Case Design

CT Integration (CI, CD)

UI Automation: Script-based

UI Automation: Model-based

API Testing

Active Test Data Management

Test Driven Service Virtualization Manage

!

Integrate

!

Optimize

!

Automate

CTMM Continuous Testing Maturity Model

© 2018 by

?Thomas Hadorn

Managing Director APAC at Tricentis +61 424 282 757

[email protected]

Documents

The Future of Testing: Continuous Enterprise Testing - 1035 - Thomas Hadorn - The Future... · Test Automation Risk-Based Test Management Browser-Based Load Testing Test-Driven Service