Drive Continuous Delivery With Continuous Testing

1

Keith Puzey

CA Technologies

Drive Continuous Delivery With Continuous Testing

Titus Fortner

Sauce Labs

August 2017

© 2017 CA. Confidential. All rights reserved. 2

Continuous Testing

How do you build better apps, faster?

© 2017 CA. Confidential. All rights reserved.

Why Many Companies Think They’re Agile…

They moved some Dev projects from waterfall to agile

They’re having daily standups

They have a scrum master

Product owner is part of the team

They are all talking and walking agile…

And are talking about Continuous Delivery

BUT…


QA is STILL a Bottleneck… Even in DevOps Shops

http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf

…of delays were occurring at

the Test/QA stage of the

cycle.

“Where are the main hold-ups in the software production process?”

32%16% 22%

63%

21% 23% 30%

63%A 2017 survey of self-

proclaimed DevOps

practitioners found that …

4

http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf


Why Quality is So Hard to Achieve?

Clear evidence that you are unlikely to enjoy

the full benefits of continuous delivery unless you

modernize your testing tools and

practices.

75% highlight continuous testing as critical or

important

76% see Continuous Testing as critical or

highly desirable to ‘shift-left’ testing activity from

downstream back into development

Traditional testing tools

are often too complex

and expensive to use

Adoption of

continuous testing is

much easier with

the right tools

Continuous testing

requires a major

shift in mindset

and culture

agree or strongly agree

63%agree or strongly agree

79%agree or strongly agree

72%

“Continuous Testing as a Digital Business Enabler”, 2017

“

5


“Continuous Testing as a Digital Business Enabler”, 2017

Tangible Evidence of the incremental value continuous testing represents for “Leaders”

more likely to have succeeded in

left-shifting testing activity to

requirements phase

more confident in quality of

output

more confident in speed

of delivery

more likely to experience

rapid revenue growth

more likely to deliver 10x faster iterations

more likely to reduce defects

by 50%+

2.5x 1.5x 1.9x

2.6x 2.4x

3.9x

SP

EE

D

QU

AL

ITY

RO

I

6


What is Continuous Testing?First need to understand what it’s not!

It’s not… Test Automation

It’s not… Continuous Integration

It’s not… Unit Testing

It’s not… done only in the Test environment

It’s not… a QA-led activity

7


What is Continuous Testing?

The practice of testing

across every activity in the SDLC to uncover and fix

unexpected behaviors as soon as they are injected


Why are Traditional Testing Activities a Bottleneck?

Enable Interfaces

(real or virtual)

Create Detailed Test Cases

ExecuteTest Cases

Identify Interfaces (internal and 3P)

Automate Tests (usually Regression)

Create Defects Fix Defects

Define an Idea or Customer Needs

Write Requirements

Create Requirements

Traceability Matrix

Read Requirements

Manual Processes

Find/Create Test Data

Testing hasn’t changed in the past 30 years

Identify Test Data Requirements


But Continuous Testing Changes the Game…

Enable Interfaces

(real or virtual)

Create Detailed Test Cases

ExecuteTest Cases

Identify Interfaces (internal and 3P)

Automate Tests (usually Regression)

Create Defects Fix Defects

Define an Idea or Customer Needs

MODELRequirements

Create Requirements

Traceability Matrix

Read Requirements

Find/Create Test Data

Identify Test Data Requirements

NOW ALL AUTOMATED


End-to-End Integrated Continuous Testing from CA

Automatically generate

test automation scripts

from requirements1

2Simulate test

environment

3Access test data on-

demand & ensure PII

compliance

4Start multi-layer /

backend “requests

& response” testing

prior to UI

5Democratize

performance testing

Integrate open source

testing tools 6

7Ensure comprehensive,

cloud-based API testing

8Built-in automated

application security

testing

9Promote artifacts when tests

pass by orchestrating &

automating the pipeline

10

Harness application insight

across the SDLC to

improve user experience

Continuous

Delivery

Director


WHAT IS SAUCE LABS?The world’s largest cloud-based platform for running automated tests

• Co-founded by Jason Huggins, the creator of Selenium; Selenium is the

de facto standard for front-end test automation (Gartner)

• One platform that spans desktop, simulators, emulators and real devices.

• Run tests across over 900 browser/OS/platform combinations & 1,000’s of

real mobile devices

• Enterprise-grade security trusted by thousands of customers

• Optimized for Continuous Testing and CI/CD


ACCELERATE YOUR PIPELINE

Faster Releases = Competitive Advantage

• Test on every build; integrate with your CI Server* and pipeline orchestration

• Run tests in parallel to speed releases

• Test more scenarios per unit of time

• Catch bugs earlier in the development cycle & fix faster

A C H I E V E C O N T I N U O U S D E L I V E R Y W I T H A U T O M AT E D T E S T I N G

*Sauce supports: Jenkins, Bamboo, Travis, Circle CI, MSFT VSTS,

Serial Testing (9 minutes)

Chrome 1 Chrome 2 Chrome 3

FF 2 FF 2 FF 3

IE 1 IE 2 IE 3

Parallel Testing (1 minute)

Chrome 1

Chrome 2

Chrome 3

FF 1

FF 2

FF 3

IE 1

IE 2

IE 3


Quality Code = Quality Experience

• Catch more bugs with increased coverage

• Higher quality keeps users from abandoning

sites and apps

• Increased coverage broadens your addressable

market

• Expand testing to GA / Beta & Dev releases

• Videos, screenshots and logs help speed bug

resolution

IMPROVE QUALITY THROUGH COVERAGEE X PA N D Y O U R T E S T C O V E R A G E W I T H O U T M O R E O V E R H E A D

Increased Quality


IMPROVE EFFICIENCY

Less Time Testing = More Time Innovating

• Enable team to focus on high value activities

instead of infrastructure

• Eliminate time spent on supporting &

maintaining an internal test grid

• Improve productivity of Devs & QE

• Increase confidence in test results

Fewer false failures

Video debugging

R E D U C E C O S T S A N D S P E E D T I M E T O VA L U E

Infrastructure

Costs

False

Failure

Ongoing

Maintenance


DEMO


Q&A


Thank

You!

Come see us:

Sauce Labs Booth #205

CA Booth #302

CI to CD in the Cloud: Orchestrate All the Things

Wed 8, 2017 12:30 - 12:45pm

Technology

Drive Continuous Delivery With Continuous Testing