W15 - Kinsbruner - AgileConnection · W15! TestAutomation$ 2019,05,0114:45$ $ $ $ $ $ $ FutureXProofingTest!Engineers!in!the EraofMLandAI!! Presentedby:$$ Eran!Kinsbruner! Perfecto$

W15 Test Automation 2019-‐05-‐01 14:45

Future-‐Proofing Test Engineers in the Era of ML and AI

Presented by:

Eran Kinsbruner Perfecto

‘

Brought to you by:

888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com

Eran Kinsbruner Eran Kinsbruner is the mobile evangelist at Perfecto and the author of The Digital Quality Handbook, as well as a monthly columnist at InfoWorld.com. He is a software engineering professional with nearly twenty years of experience at companies such as Matrix, Sun Microsystems, General Electric, Texas Instruments and NeuStar. He holds various industry certifications from ISTQB, CMMI, and others. Eran is a recognized mobile testing influencer and thought leader, as well as an experienced speaker in the major software engineering conferences. He is also a patent-‐holding inventor for a test exclusion automated mechanism for mobile J2ME testing, public speaker, researcher, and blogger. He can be found all over social media, including on Facebook, Twitter (@ek121268), LinkedIn, and his professional mobile testing blog.

4/23/19

1

1 | Future Proofing Test Engineers in the Era of AI and ML perfecto.io

Future Proofing Test Engineers in The Era of AI and ML


Today’s Speaker: Eran Kinsbruner

•  Chief Evangelist and author at Perfecto

•  Blogger and speaker

•  19+ years in development & tesCng

•  Author of “The Digital Quality Handbook” and “ConCnuous TesCng for DevOps Professionals”

•  TwiNer: @ek121268

•  Blog: hNp://conCnuoustesCng.blog

•  Website: hNp://perfecto.io

4/23/19

2


Today’s Agenda

1

2

3

Defining the main personas involved in conCnuous tesCng.

IntroducCon to codeless tesCng and its benefits

The main differences and consideraCons to be aware of between codeless and code-‐based tesCng.

4 Live demo: Perfecto Codeless!

5 Q&A


What is ConCnuous TesCng?

ConCnuous tesCng is the process of execuCng automated tests as part of the so^ware delivery pipeline in order to

obtain feedback as rapidly as possible.

4/23/19

3


The Path to ConCnuous TesCng

1 2 3 4 5

Stable automaCon Daily cycle Increase coverage Reach 95% ConCnuous tesCng

W H A T Y O U ’ L L G E T

W H A T Y O U ’ L L N E E D

•  99.9% availability lab •  Evidence collecCon •  Process integraCon

•  CI •  Defect tracking

•  ExecuCon control •  Dashboarding

•  Create scripts •  Maintain scripts •  Understand what’s

wrong

•  Skillset matched tool

•  False negaCve detecCon in reports

•  Run scripts daily •  Get results in

minutes

•  Valuable coverage •  Meaningful daily

feedback

•  Advance validaCons •  Visual automaCon •  Basic orchestraCon •  Up-‐to-‐date lab •  Scaled lab •  Analysis grouping

•  Automate all that is possible

•  Advanced orchestraCon

•  ElasCcity •  Sharding

•  Environment control

•  Scaled reporMng

•  Nightly à ConCnuously

•  Cloud execuCon •  Very high scale

D E V O P S F R I E N D L Y Z O N E


Key Personas in ConCnuous TesCng

4/23/19

4


ConCnuous TesCng: The Three Personas

B U S I N E S S T E S T E R

S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )

S O F T W A R E D E V E L O P E R

Performs exploratory and structured manual tesCng.

Creates funcConal and nonfuncConal code-‐based test automaCon scripts (via Selenium, Appium, etc.).

Creates unit and build acceptance tests.


Business Tester: Manual? Exploratory?

BDD? Codeless?

Persona-‐Based Challenges in ConCnuous TesCng

P L A N C O D E B U I L D T E S T D E P L O Y R U N

Test PlaXorm Test Framework Trigger Feedback Time

90% Emulator/Local Device 10% Cloud Device Unit API Espresso / XCUITest Exploratory Pre-‐commit Manually – On-‐demand Few minutes

70% Emulator 30% Cloud Devices API Espresso / XCUITest Commit / Few # day AutomaCc < 5-‐10 Min

100% Cloud Devices Espresso/XCUITest Appium Nightly AutomaCc < 6 Hours

100% Real Devices (S) Espresso/XCUITest 24/7 AutomaCc < 5 Min

SDET FuncConal/NF

Developer Unit/BAT

?

D E V O P S C O N T I N U O U S F E E D B A C K U N O P T I M I Z E D M O D E L U N O P T I M I Z E D M O D E L

4/23/19

5


I N T E R A C T I V E T E S T S

UI/UX manual tests

Balancing Test CreaCon for the Three Different Personas With the Right Tools

O P T I M I Z E D M O D E L

Developers & SDETs (Code-‐Based)

Business Testers Ownership (Codeless)

Business Testers Ownership


IntroducCon to Codeless TesCng

4/23/19

6


IntroducCon to Codeless TesCng

R E C O R D & P L A Y B A C K

S E L F H E A L I N G ( M L / A I )

S T A N D A R D F U N C T I O N A L I T I E S

C O N N E C T T O C I / C L O U D

C OM P L E M E N T S C O D E -‐ B A S E D T E S T I N G

Codeless tesCng – process of creaCng automated tests without wriCng a single piece of code.


Main Differences Between Code-‐Based and Codeless TesCng

4/23/19

7


•  Define manual test scenarios, BDD, and user stories style scenarios.

•  Create scripts in Java/JS from within IDEs (IntelliJ, Eclipse).

•  Define Objects using Object Spy tools, DOM viewers, etc.

•  Page-‐based test step creaCon. •  Insert visual validaCons and asserCons. •  Typically takes longer to develop, more complex

(~6 hours per test). •  Medium-‐high code development skills required.

•  Test changes are required proacCvely. Tests are more error-‐prone to changes in producCon, objects.

•  Tests are managed and maintained in an SCM tool (GIT, Perforce).

•  Tests are recorded with no coding in most cases.

•  Codeless UI tools used for creaCon. •  Objects are “learned” and generated on the fly. •  Test scenarios are o^en less structured and

more exploratory-‐based (flow-‐based). •  Time to author is shorter and can be ~1 hour

per test. •  Test reusability is easy. •  Business tester, liNle to no coding skills are

required.

•  Tools support self-‐healing with auto test correcCon. •  Local or cloud test versioning is used. No GIT

integraCon.

SDET/DEVELOPERS – Code-‐Based BUSINESS TESTERS, DEVELOPERS -‐ Codeless

Main Differences Between the Approaches

TEST AUTHORING WORKFLOW & SKILLSET REQUIRED

TEST MAINTENANCE


SDET/DEVELOPERS – Code-‐Based BUSINESS TESTERS, DEVELOPERS-‐ Codeless

Main Differences Between the Approaches

TEST EXECUTION

TESTING & APP TYPES SUPPORTED

•  Highly mature, includes samples, best pracCces, documentaCon.

•  IntegraCons exist for ALM tools, defect management, etc.

•  FuncConal, API, load, etc. •  Mobile naCve (Appium) and desktop web

(Selenium).

•  Emerging technology, less mature, no well-‐defined guidelines and pracCces (guide to shi^ from standard to ML).

•  Web is more mature than mobile codeless, basic integraCons only.

•  Mostly funcConal (E2E) and basic API. •  Most support for web, mobile is lagging behind.

•  Configured environment using tools like TestNG Data Provider.

•  ExecuCon done locally, via CI, cloud-‐based.

•  ExecuCon management is built into the codeless tools.

•  ExecuCon done locally, via CI, cloud-‐based.

TOOLS MATURITY

4/23/19

8


Code-‐Based AutomaMon Codeless AutomaMon


5 Recommended PracCces for Embedding Codeless TesCng in CT

1

2

3

IdenMfy manual and/or flaky test steps.

Mark these as “codeless” to disconnect them from the code-‐based automaCon suite (avoid duplicaCons).

Record the tests in a “SMART” manner: •  Flow-‐based and exploratory recording. •  Reusable components. •  Robust object locators. •  Tagging for reporCng and tracking.

B E S T P R A C T I C E

4 Connect codeless to CI and pipeline for conCnuous tesCng.

5 IdenMfy maintenance and examinaMon points for both suites.

4/23/19

9


Introducing Perfecto Codeless


Codeless TesCng Legacy Tools Were Painful

L E G A C Y T O O L S M A N U A L T E S T I N G

R E C O R D E R S S C R I P T I N G

•  Complex •  Time consuming

•  Inefficient use of Cme •  BoNleneck in SDLC

•  Limited capabiliCes •  Break easy

•  Coding skills •  Demanding to maintain

4/23/19

10


A.I. Object Locator

Powerful Language Simplified & Smart ValidaMons

Just Selenium •  No installaCons •  No modificaCons •  No plug-‐ins •  No extensions

•  All plazorms (Win, Mac, Linux) •  All browsers •  All versions •  Always updated, always ready

MulC-‐aNribute uniqueness matching

Over-‐Cme neural network adapCon

Framework & app awareness Run-‐Cme self-‐healing

P R O J E C T -‐ L E V E L S H A R E D A D A P T I V E R E P O S I T O R Y

•  Full browser control •  Object type awareness •  Data driven •  FuncCons

•  Rich text validaCons •  Appearance & style validaCons •  Data validaCons •  Time validaCons

•  Variables & data sets •  Branching •  JavaScript extensions

No installaCon, no setup

Record on any browser

Contextual point & click recording Object reusing InteracCve

debugger

T E S T S U I T E S

A S S E T S M A N A G EM E N T

P L A T F O RM S E T S

Integrated CI-‐connectors

S C H E D U L I N G

P R O J E C T M A N A G EM E N T

Perfecto Codeless

Web-‐Based InteracMve Smart Recorder / Debugger Test Management


Perfecto Codeless TesCng Benefits

Recorders •  Full featured capabiliCes for addiConal

complex flows •  Robust AI-‐driven bindings that takes

care of 97.4% of app changes

ScripMng •  No coding background necessary

to create automaCon scripts •  Reduce maintain cost of

automaCon

Setup •  No set up required

•  Easy to learn

Process Fitness •  Efficiently reuse automaCon scripts without the need to code

•  Accelerates the tesCng process and provides developers and

QA fast feedback

P E R F E C T O C O D E L E S S

A U T OM A T I O N

4/23/19

11


PERFECTO CODELESS SELENIUM

•  Powerful web-‐based IDE

•  No coding skills required


PERFECTO CODELESS SELENIUM DEMO

4/23/19

12



Thank You

Documents

W15 - Kinsbruner - AgileConnection · W15! TestAutomation$ 2019,05,0114:45$ $ $ $ $ $ $ FutureXProofingTest!Engineers!in!the EraofMLandAI!! Presentedby:$$ Eran!Kinsbruner! Perfecto$