Testdroid: Best Practices in Mobile Game Testing

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 1

2 December 2014

Ville-Veikko HelppiTechnical Product Manager

[email protected]

Learn the Best

Practices in Mobile

Game Testing

W E B I N A R


Agenda

• Different Types of Testing and How Game

Testing Differs from Regular App Testing?

• Frameworks, Tools and Technologies – What is

Important in Mobile Game Testing Today?

• Top Priorities for Mobile Game Testing

• Infrastructure for Successful Game Testing

• Hands-on Example (Image Recognition)

• Testdroid Update

• Q&A


Let's Define 'Testing'...


Different Types of Testing?

• Black-box testing vs. White-box testing

• Methods used in Mobile Game Testing

– Functional

– Compatibility

– Performance

– Localization

– Regression

– Load

– Smoke

– Acceptance


Defining Some Ways to Test

• Functionality of the game-play, featuresFunctional Testing

• Done always when new features /regressions are includedRegression Testing

• How game runs on different configurationsCompatibility Testing

• Different languages, geo-focused materialsLocalization Testing

• Endurance test to determine if system can handle the loadSoak Testing

• Measures the capacity of the system Stress Testing

• Simplest form of performance testing, measures how system handles certain loadsLoad Testing

• Isolation of the environment (e.g. from network) to see how game worksHermetic Testing

Feature

-based

testing

Performance

testing

End-user

testing


The Old-Fashion Thinking

Deterministic systems

Identifiable objects

Tool support(s)

User-driven actions

Simple inputs/outputs

Virtualization

Random setups/results

No handle on objects

No standards

User reacts to input

Complex inputs/outputs

Real hardware

<>

<>

<>

<>

<>

<>

High-level of automation Low-level of automation


Test Automation as a Solution

Random setups/results -> Easy to generate random events

No handle on objects -> Variety of options (e.g. image

recognition)

No standards -> Variety of frameworks (cross-platform

etc.)

User reacts to input -> Test scripts can handle random

events

Complex inputs/outputs -> Test scripts can generate input/content

Real hardware -> Availability of real mobile hardwareTest Automation brings real value

with variety of options to mobile game testing


Challenges that Test Automation Tackles

• Proliferation of Devices, Brands, Operators,

Platforms

• Coupling the Testing and Development process

(Continuous Integration/Deployment/Delivery)

• Different Types of Testing (as automated)

– Stress, Load, Performance, Functionality etc.

• Test script & test asset usability

• Cons of Manual testing

– Tedious effort, Expensive human resources, Error-prone

(as done by humans) & Not reusable (done once)


OS

versions

ChipsetsCPU + GPU

Tens of

OEMs

Memory

Displays (resolutions,

physical

hw)

OEM

mods

Other

hardwar

e (connectivity,

calibration)

Relation

to other

software

Significant Benefits of Using Automation


Correct behaviour

across platforms and

browsers

Integration with web

back-ends

Typically need to

fully utilize HW

(CPU+GPU)

Resource (e.g.

battery)

consumption

OpenGL ES 2/3

Functionality

and usability

Screen

orientations,

connectivity,

user profiles

Robustness

Robustness and

security!

Brand

Compliances,

verification with back-

ends and data

Different 'Verticals' Require Different Practices


Importance of

Mobile Game Testing


Important Aspects in Game Testing

• User Interface and Functionality

– UI layouts/elements, Menus, Resolution,

Orientation, Overall visuality

• Graphics Performance

• Usability and User Experience

– Responsiveness, Real Occuring Events

• Multi-player/User

• Social Integrations

• Security and Liabilities


User Interface and Functional

• Compared to regular app testing, mobile game

testing may seem to be more challenging

– e.g. OpenGL ES content not action-identifiable

• Graphics and UI – Identical look&feel across all

devices and variants

• Usability controls in the context of game

• Navigation/game-play, Progress, Delays

between "game-stages"

• Animations, Fonts, Other graphics assets

• Screen Resolution and Orientation


Performance Testing

• Performance, Load, Stress, Spike, Soak

• Measure the real

user experience

• Execute tests with

all possible devices

(SW&HW)

• Discover the impact

of server loads, app

performance

(CPU/GPU

utilization)


• Studies: Majority of

users leave game due

sluggish, general

slowness or other bad

performance behavior

• It's very important to

understand how game

behaves on different

devices, under

different circumstances

Performance Makes +/- UX


Building the User Experience

UnderstandingHow quickly gamer assimilates what the

game is about, how to fully play with it, what

are the functionalities.

UsabilityThe main User Interface, menus, design and

how usable those are

Playing ExperienceIntuition to follow the game and go forward

with its instructions and game-play

MonetizationAre gamers willing to buy – invest – in game

• Graphics performance

capability is the most

important thing for mobile

games when forming UX

• Too many games fail to

work properly with low-

end/mid-range devices due

hardware performance

• The real understanding of

game's performance

capability can be only

figured out by testing on

real devices


• End-to-end between tests, AI or

more simple implementations

• Optimize the data communication

between server and clients

• Test simultaneously on several

(different) devices

• Check what happens if game gets

interrupted (platform)

• Location-based features – how to

properly mock up those?

• Hermetic environment – what

happens?

Multi-Player/User Features


• Important part of the after-

engagement (how gamers

share their results)

• Many of OEM branded

devices provide certain

SoMe apps pre-installed

(version?)

• Connectivity, login and

data transferring are

tested in context of game

• Hermetic environment –

what happens?

Social Integrations


Security and Liabilities


4 Top Priorities in

Mobile Game Testing


Localization


Device Diversity

By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/


Real Devices vs. Emulators

• Emulators/simulators won’t help you to test…

– User Experience and Usability

– Hardware

– Software

– Infrastructure

0 % = the percentage of your app users

that use emulator to run your app!

Why Real Devices are Must-to-Have?


Manual vs. Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

Manual Automation


Infrastructure for

Mobile Game Testing


High-Level Infrastructure


Architecture / Infrastructure

GAME

Gam

e

Engin

e

Android

iOS

...

API

&

services

Server

Database

App Middleware Platform Connectivity/Network Back-end


Isolated, Hermetic Testing

GAME

Gam

e

Engin

e

Android

iOS

...

API

&

services

Server

Database

Hermetic Environment



Compatibility & Porting Testing

GAME

Gam

e

Engin

e

Android

iOS

...

API

&

services

Server

Database

Real Devices



Smoke/Reference Testing

GAME

Gam

e

Engin

e

Android

iOS

...

API

&

services

Server

Database

Reference Environment



Back-End Performance Testing

GAME

Gam

e

Engin

e

Android

iOS

...

API

&

services

Server

Database

Massive no. of users and significant load



Continuous Integration


Game Testing Process


Hands-on Example:

Integrating Development with

Testing Environment


• Continuous Integration (e.g. Jenkins)

• Appium framework (for Android and iOS)

• No need for devices – we'll use devices on

Testdroid Cloud service

• Test scripts / test cases for game

Setup


Continuous Integration


Appium Intro

Family Tree: Android Test Automation Frameworks

Appium works for both, Android and iOS



• Uses Selenium Webdriver (W3C standard) as a

scripting framework

• Supports native Android, native iOS and mobile

web:

– Android via uiautomator (API level >=16) and

Selendroid (API level <16)

– iOS via UI Automation

– Mobile web as Selenium driver for Android and iOS

• You can write your Appium scripts on almost any

programming language

(Haskell/Go/Clojure/Java/Ruby)

Appium Basics

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

39

Test Script

Test Case

Desired Capabilities

{

“device”: “Android”,

“app”: “/Users/user/ApiDemos.apk”

“app-package”: “com.example.android.apis”

“app-activity”: “.ApiDemos”

}

Desired Capabilities / Android

Example


40

Test Script

Test Case


{

“app”: “com.bitbar.testdroid.BitbarIOSSample”

}

Example

Desired Capabilities / iOS


41

Test Script

Test Case


Appium @ Localhost

WebDriver

http://localhost_4723/wd/hubAppium Server

4723

Device

Example


Test Script

Test Case


From Localhost to Testdroid

Cloud

WebDriver

http://localhost_4723/wd/hub

*Testdroid Caps

http://appium.testdroid.com/wd/hub

Example


Test Script

Test Case


From Localhost to Testdroid

Cloud

WebDriver {

“testdroid_username”: “[email protected]”,

“testdroid_password”: “p4s$w0rd”,

“testdroid_project”: “My First Project”,

“testdroid_testrun”: “Test 1”,

“testdroid_device”: “iPad Mini 7.0.4 A1432”,

“testdroid_app”: “http://domain.com/app_v1.ipa”

.

.

“app”: “com.bitbar.testdroid.BitbarIOSSample”

}

Example


44

Example


350+ Android & 20+ iOS Devices


46

Example


47

Example


48

Example


49

Example


50

Example


51

Example


Example

Projects and Test Runs

AppiumBroker

5 Appium Ready

sessionid


53

Test Script

Behind the Scene

Device Cluster

Device 1

Device 2

Device 3

1 WebDriver Session Request@ http://appium.testdroid.com/wd/hub/

Desired Caps, .apk / .ipa

2 Configure project

3

Wait for device to become available

4 Start Appium

Session Map

WebDriver Session response6

sessionid

7 Test Run

sessionid Proxy

Appium

Device 1

Example


Image Recognition

The basic example of Image Recognition

in Mobile Game Testing: Template

Matching


Code Example


Code Example


Code Example


Code Example


• Test Script contains the following 'steps':

1. Identify the used platform (Android / iOS)

2. Search for 'Goldmine' graphical element

(using find_image to search for screen)

3. Script runs test to go to shop

4. Script runs test to buy a cannon

5. Script runs test to place the cannon (all

these three are defined in .png files)

6. Script runs test to start a battle!

7. Game is brought down and test ends.

Example: Clash of Clans


VIDEO

Image Recognition with Real Android and

iOS Devices in Local Environment


Testdroid Update


New UI in Testdroid Cloud 2.0


Testdroid Cloud's Interactive


Testdroid

PrivateCloud -

Running user-

selected devices

at any of our

hosting facilities


NEW HOSTING CENTER!Mountain View, CA

San Francisco

and

Mountain View

datacenter


Option 1:

Public Device Cloud

on-demand devices

(multitenant)

Mobile app testing on

over 400+ real

Android and iOS

devices hosted by

Bitbar

Option 2:

Private Device Cloud

reserved devices

Hosted by Bitbar in the

US and/or Europe

Devices chosen by and

reserved only for the

Customer

Option 3:

On-premise Device

Cloud

Automated mobile

app testing on real

Android and iOS de-

vices hosted by the

customer, usually

30-500 devices

Testdroid Deployment Options



Video recording & slides of this webinar will

be soon available at Vimeo and Slideshare!

Please visit at www.testdroid.com for more

information.

THANK YOU!

W E B I N A R


We operate the largest global device

cloud with over 400 devices

instantly available for developers

Devices from the US, Europe, China, Japan and Korea

– global devices for global customers