Why Adopt

Analytics Driven Testing

Ori Bendet | Inbound Product Manager, HPE Software |

@bendet_ori

#SrijanWW | @srijan

↓80%Reduction in regression time

7y in HPE Softwarein various managerial positionsStarted in testing

Today:Inbound Product ManagerStormRunner Functional

ABOUT ME

AGENDA • Applications Development Overview• Today’s Testing Challenges• What are the risks? • Real World Examples • How can you empower your testing by

using Analytics

The World of Software is changingMega-trends create demands on modern applications

MODERN APPLICATION DEVELOPMENT

Reduce costs

Increase customer attraction/retention

Increase the value of your brand

Get to market faster

300,000tweets

200 million+emails sent

220,000new photos posted

50,000 appsdownloaded

$80,000in online sales

72 hoursof new video content uploaded

2.5 millionpieces of content shared

Agile

Every minute*…

* Source: “The Data Explosion in 2014 Minute-by-Minute” by ACI Information Group.http://aci.info/2014/07/12/the-data-explosion-in-2014-minute-by-minute-infographic/

Mobile

Cloud

Digital

DevOps

8

What we need to test

Testing Time

=Quality

$59.5B Annual cost of software defects

How to mitigate the risk?

• Have more testers• Crowdsource• Don’t test• Let your users test for you

• Or… use Analytics!

Analytics

Example

Example • My product: 30 features• Supported Matrix: 10 environments

(OS/Browser)• My testing team: 10 testers

• Testing a feature on an environment = 1 day

• Target regression cycle: <3 days!

30 features X 10 combinations = 300 days

300 days / 10 testers = 30 days

Google Analytics • A freemium web analytics service offered by Google

that tracks and reports website traffic.

• Google launched the service in November 2005

after acquiring Urchin.

• Google Analytics is now the most widely used web

analytics service on the Internet.

• Note: I’m not affiliated with Google Analytics in

anyway * Source: https://w3techs.com/technologies/overview/traffic_analysis/all

Environment Optimization • Focus on the majority of combinations (OS x

Browser)

• Reduce your regression risk to a minimum

• Wait for customer feedback on the other areas

Let’s see it live

30 features X 10 2 combinations* = 60 days

60 days / 10 testers = 6 days

(*) with 89% confidence

Example

User Behavior • Focus on the majority of the functionality

• Target the most active areas

• Wait for customer feedback on other areas (% of escaped

defects)

• Calculated risk(!)

Let’s see it live - again

30 12 features X 10 2 combinations* = 24 days

24 days / 10 testers = 2.4 days

(*) with 76% confidence

Example – part 2

Building a risk calculator • Test everything (100%) = 300 days

• All functionality, top combinations = 60 days

• Main features, top combinations = 24 days

• Add additional levels based on your needs and

time

Building a risk calculator

Top flows

1 OS + Browser

Main Functionalities

top combinations

All Functionalities

top combinations

All Functionalities

Additional combinations

Test Everything

50%

75%

90%

100%

95%

BewareZombie tests!

Don’t have analytics? • Use market analysis and statistics

• https://netmarketshare.com/

• https://www.w3schools.com/browsers/

• Many more!

• Conduct annual user survey

• Customers Validations – before & after

A word about Test Automation

Re-use calculator for Automation

Top flows

1 OS + Browser

Main Functionalities

top combinations

All Functionalities

top combinations

All Functionalities

Additional combinations

Test Everything

50%

75%

90%

100%

95%

CI

Nightly

Full regression

Summary

Summary • Don’t test everything!

• Get to know your users

• Calculate your risk

• Measure your un-tested areas

• % of Escaped defects

• Customer Satisfaction