AUTOMATEDTESTS

How We Learned To Love Test Automation for Localization

2

In general, localization testing is manual – as text needs to be read, layout needs to be checked, and localization needs to be verified – all of which require a person to manually perform. But, automation is possible in localization testing too, enabling the localization testers to be more efficient.

We wrote this white paper to help you understand the importance of an integrated test automation framework, and to elaborate on the idea how we learned to love test automation, and how you can learn to love it too.

This white paper will guide you through the steps we took with our clientsto build UI Automation, to make localization testing more effective and efficient.

This transition – from manual to automation testing – reduced our client’s test cases by 60%, increased test coverage by 23%, andsaved time by 97.5%.

Executive Summary

• There was lot of technical debt in the code

• There was a need for lot of manual efforts to run automation

• Silk lacked a way to run tests in parallel

• Silk lacked data driven capabilities

• Silk lacked multi-browser support and backward compatibility with IE

• Developers couldn’t reproduce bugs found in Silk, since Silk environment was complex

• Developers did not want to use point & click tools, they wanted to code

• There were 80,000 NRT test cases and a 20-day NRT cycle.

• Each of our teams took 14-20 days to run NRT, with localization tests taking the longest time

We Also Wanted to Go Agile

• GUI tests are brittle, or was there a way to make it less brittle through design?

• The goal was to have manual testing team focus on new features using exploratory testing, and automate as much of the tests as possible

• Focus on Functional, GUI, and Content tests.

• Business facing + critique product

quadrant must be manual,

or are there ways to make

manual linguistic testing

more effective?3

What We Hated About Test Automation

Testing Pyramid Testing Matrix

Solid

Brittle Tests

Automated Unit Tests

Automated API Tests

AutomatedGUI Tests

Automated Integration Tests

Automated Component Tests

FitNess REST API

Cucumber

Selenium SeLion

xUnit TestNG

Manual Session

4

Based TestingTesting Tools

Before Agile transformation, most of thetesting logic was written in silk and allbusiness logic, integration logic, and GUI logicwere tested in silk at the GUI layer. Bydefault, GUI tests are brittle, since any change

in the UI will break the tests. SeLionenforces the best practice of separating theUI elements and the UI logic, which willincrease the robustness of the GUI tests.

Agile Testing Pyramid

Symbio Automation Framework

TEP(Test Execution

Platform)

Dashboard

Framework

Jenkins

o Webo Mobile

Test ScriptsRepository

Product

JIRA

Build Deploy

Metrics &Reports Screenshots

LQA Review

Defect reporting

5

Symbio Automation Dashboard

Web

Mobile

Desktop

6

Start -> Search -> Shop -> Sort -> Select -> Buy -> Configure -> Cart

All test flow pages are captured as screenshots

Screenshot library

7

Review and Bug reporting

• Bug review window gives clear view of details with Markup highlight feature

• Bugs can be reported from Dashboard to JIRA, filling most of the data automatically

8

Selenuim is a great open source solution, however:

• All tests were flow-based vs page-based

• Selenium only supports web based testing

• Selenium didn’t support multi devices

• Selenium didn’t support multi lingual

This took us part of the way there. Luckily, Selenium gave us more options:

• Much easier, cheaper to add new core functionality

• The large Selenium community provided us with drivers to drive native apps on iOS and Android

• We could easily add whatever we were missing, then roll it out to the community for their use, in turn allowing them to add on, and improve it

Browser

Phone Tablet TV

Desktop

Multi-device

Web

Mobile

Desktop

Multi-platform

zh j a ko th h i

ar ms p t ru v i

t r p l cs f i f a

sv da n l i d ro

Multi-lingual

9

UI Test Automation with SeLion

Solution: Extend Selenium Into “SeLion”

We added what was missing, added more features, and released it into the open source community. This gave us:

• A separation of concerns:

• YAML to handle page objects + multilingual support

• Test cases focused on test logic

• YAML provided the ability to fix page changes in one place instead of having to fix in every script

• Very readable test scripts

• Testing grid, Jenkins + Selenium grid to run tests

• Testing environment component to provide data driven tests

• Added Android + iOS drivers that plugged into Selenium grid, providing the multiplatform support

10

baseClass: “com.paypal.selion.testcomponents.BasicPageImpl” pageTitle:US: “Official Apple Store” defaultLocale: “US” elements:

officialAppleStoreAppleStoreLink: locators:US: “//div[@id=’page’]/div[2]/div/div[1]/h1/a/span”

ZH: “//div[@id=’page’]/div[2]/div/div[1]/h1/a/span” searchSubmitButton:locators:

US: “//button[@class=’gh-search-submit gh-search-magnify’]”ZH: “//button[@class=’gh-search-submit gh-search-magnify’]” siteSearchQueryTextField:

locators:US: “//input[@id=’site-search-query’]” ZH: “//input[@id=’site-search-query’]”

pageValidators: [ officialAppleStoreAppleStoreLink]

How Did We Do It?

Example Test Case Script

YAML: Page Objects + Locale

Test Code

Test NG

SuiteTests

Page Object

Repository

Preprocessor

Yaml Yaml Yaml

UI in multiple

languages

- Web pages- Mobile screens- Desktop screens

Template- Page Objects- Multi-lingual support

Tests- Page flows- End-2-End test

11

SeLion Design

• Nemo combines mocha.js, the confit configuration engine, plugin architecture, and flexible parallel options to the selenium-web driver bindings for node.js.

• With all the goodies that SeLion has, Nemo is smaller and more Node.js friendly

• Allows automation testing to be part of your Node.js app’s development life cycle.

• For Node.js application

12

UI Test Automation with Nemo.JS

Why Nemo?

Nemo.JS is a open source

NPM package.

We will create functions that

fit test projects, and released

it as Nemo plugins. We will be able

to:

• Aseparation of concerns:

• Data Provider to handle

test case level and locale

default data to feed the

test.

• Test suites focused ontest

logic

• Page Object to handle

page locaters and page

level common steps.

• Less maintenance effort.

• Testing on cloud

• Test run in parallel in data

driven.

13

How Did We Do It?

Example Test Case Script

JSON: page locators, localized data

Validation Layer(Screenshots + Check Points)

Reporting Layer

14

Nemo Work Flow

JENKINSTest Suites

N

e

m

o

A

P

P

L

I

C

A

T

I

O

N

Data Provider

Case Data

Locale Data

Page Object

Test Reports

NPM

Screenshots Stack TracesEmail/Slack Notifications

•Cucumber uses Gherkin to define test cases, which is a language designed to be human readable, and easily accessible to non-technical people

•Dev and QA can help to write automation scripts easily

•Multi-languages, Multi-platforms, Multi-devices support as well as SeLion platform

UI Automation with Cucumber/Appium

Write test scripts in human language

Browser

Phone Tablet TV

Desktop

Web

Mobile

Desktop

Native/Hybrid/Webview Apps supported

15

Write Test Scripts in Human Language

We created functions and test steps that fitted test projects, and implemented to framework.

• Supported data driven tests

• Test steps were defined and centralized

• Human readable scripts

• Easy onboarding

• Supported Native/Hybrid/Webview Apps, and

• Supported multiple OS/system on devices farm

16

How Did We Do It?

Example Test Case Script

Steps pre-defined

Test cases

Testcases

Testcases

Tests

Appium/UIAutomator2/Frank

Multiple

languages cases

- Native case- WebView test - Hybrid

Run tests on multiple devices/OS

17

Cucumber/Appium Framework Design

Device Farm/Cloud

APP/Browser

JENKINS

Support Libraries

Predefined Steps

• Running tests in parallel greatly decreased total run time

• Full automated NRT cycle down to 2 days, shortening the cycle

• For localization tests, 20 days went down to 4 hours

• 60% reduction of test cases, one team went from 3000 to 800 cases

• Product and testing teams were able to develop test cases

• Every release, 90% of the tests were run using automation

• New tests per release focused on new features, and were SBTM based

• Automation cost was lower than manual testing cost, yet the coverage was much higher. The test cases were cumulative over time

$

$

$

$

$

$

$20 days 4 hours

Time Saving by 97.5%

3000 800

Test Cases Reduction by60% Cost Coverage

80%

98%

18

How We Learned To Love Test Automation?

Results

• Symbio UI Automation solution emerged out of the need for a better test automation framework

• Symbio UI Automation solution can help you to drastically improve your processes

• Symbio UI Automation solution will scale your testing and QA, and you will be able to test more and test better in a shorter period of time

• Symbio UI Automation solution will help your testing teams to be more efficient

• Symbio UI Automation solution will increase the test coverage, and will minimize test cases

• SeLion will also drastically lower the cost of testing

Multi-device

en f rdei tes f i

Multi-lingual

Manual Testing

Automated Testing

Multi-platform

Symbio Can Help You to Implement UI Automation

How can we help you?

SYMBIOAutomation

19

Symbio is the global leader in innovation, co-creation and seamless augmentation. Weprovide insight and strategy, design and research, productization, softwaredevelopment, quality assurance, globalization, R&D operations, systems managementand support operations for over 400 customers worldwide.

Visit us online at www.symbio.com or contact us directly at marketing@symbio.com

Contact Us