Unity Application Testing Automation with Appium and Image Recognition

Maxim MozgovoyEvgeny Pyshkin

University of Aizu, Japan

Introducing Automated Smoke Testing

• Aimed at performing basic checkups:• Program runs at all.

• Program is able to react to user input.

• Program is able to connect to perform certain basic functions.

• Can be a part of a continuous integration pipeline:• Run automated smoke tests for each new build.

• (Already integrated into the Google Play Developer Console).

2

The Case of a Mobile Game• Resource-intensive application,

may crash or run slowly.

• May behave very differently on different devices.

• Contains no native GUI controls.Contains elements that are easy to overlook with manual testing.

• Stress tests may be beneficial (such as running the app for 5 hours or with poor internet connection).

• Real mobile devices required (not emulators / simulators).

3

Problem 1: Engineering the Tests

How to design and run a GUI / smoke test on a mobile device?

De facto standard approach:• Connect mobile devices to a “test server” running some special software.

• Execute testing scenarios on a remote machine via the server.

4

Client(runs tests)

Server(operates mobile devices)

Server Software• Appium:

• Can be installed on Mac / Windows / Linux.

• Supports test scripts written in Java / JavaScript / Python / C# / etc.

• Can test Android, iOS and Windows applications.

• Calabash:• Supports Android and iOS applications via different server versions.

• Supports Ruby tests scripts only.

• Note: these systems work because target platforms explicitly support automation (via adb & UIAutomator on Android,UIAutomation & XCUITest on iOS, etc.)

5

Test Scripts

• Logically, a test script is a sequence of statements like this:wait 10 sec

tap location (100, 50)

assert that OK button appears

press OK button

• In native GUIs Appium can directly access individual controls:

6

e = appium.find_element_by_class_name('android.widget.EditText')

e.send_keys("hello")

ok = appium.find_element_by_class_name('android.widget.Button')

ok.click()

Testing Apps with Custom GUI (Unity-made Games) 7

similarity = 0.78

similarity = 0.96

Take screenshots with Appium and use OpenCV to recognize areas via matchTemplate() function. Note: 100% similarity cannotbe achieved due to screen rendering differences.

In both cases tests are quite slow (inherent limitations)

Problem 2: Maintaining a Mobile Farm• Ideally, each new build should be tested on a variety of mobile devices.

• Solution: use a cloud mobile farm provider! (AWS, Bitbar, etc.)

• Advantages: hassle-free setup, hundreds (!) of mobile devices.

• Drawbacks: still not all devices, high costs: 14-17 US¢ per minute per device, while our tests take > 1hr on each device per build(+ stress tests). So, e.g., 5 devices, 1hr ≈ 45 USD.

8

Client Server

Newbuild

Reports

9Building Your Own Mobile Farm

+

Appium + test scripts

+Test Runner

Software (in-house)

Test Runner• Gets fresh builds from TeamCity (build machine)

• Runs all tests on all devices.

• Generates HTML reports with game action logs and screenshots.

• Still under heavy development.

10

The Adventure Begins (and Never Ends…)11

First, you might want to use

a small Windows or Linux server

It will work perfect with any Android device

…but not with iPhone or iPad

The Adventure Begins (and Never Ends…)12

So you’ll have to ditch it and use a Mac

The Adventure Begins (and Never Ends…)13

Then you’ll realize that long tests drain the battery and search for powered USB hubs

…will find lots of them…

The Adventure Begins (and Never Ends…)14

…and realize that “powered” means that they simply have some dedicated charging ports

and get excited with their specs

The Adventure Begins (and Never Ends…)15

…And finally find the hub that does the job (not without quirks, but nobody’s perfect)

…then study its compatibility table:

The Adventure Begins (and Never Ends…)16

Then realize that not all devices are created equal. For example:

On Nexus 7 sometimes you need to make 2-3 attempts before an application is installed, it is a norm to get an INSTALL_FAILED_INVALID_URI error with no reason.

The Adventure Begins (and Never Ends…)17

Then realize that not all devices are created equal. For example:

On Xiaomi Redmi Note 3 Pro after a number of install / uninstall cycles you’ll get a “no storage memory left” error (a reboot fixes it till the next time)

The Adventure Begins (and Never Ends…)18

Then realize that not all devices are created equal. For example:

Kindle Fire (5th Gen) will sometimes refuse to unlock the screen to run tests (fixed by installing CyanogenMod)

The Adventure Begins (and Never Ends…)19

Then realize that not all devices are created equal. For example:

and Samsung Galaxy Tab Ewill not charge at all from anything but a wall charger.

iPad 3 will not charge fast enough and will end up with an empty battery,

The Adventure Begins (and Never Ends…)20

Running tests on iOS devices requires:• Specific version of Appium.• Specific version of Xcode and Mac OS.• Specific version of iOS

(in practice it can’t be iOS 10, and once installed, no way back)• Ability to block all iOS on-the-air updates, otherwise the “Please

update” popup will prevent the tests from running.• Understanding of numerous MacOS-specific tools.• Arcane operations with certificates and permissions.• Certain phase of the Moon.• Appium will still crash once in a while.

Conclusion21

• Automated smoke tests worked really well for us.• Despite complications of initial setup and maintenance, we now

consider them an integral part of our pipeline.• We found many bugs • More importantly, they provide a “safety net” feeling.• Appium has quirks, but there aren’t many choices around.• Image processing in general works reasonably well.• Most problems we had with tests are not related to image

processing (they are basically logical errors).• These errors are often caused with changes that happen between

the screenshot time and the test response time (e.g., appearance of a modal dialog window).