Mastering Mobile Test Automation with Appium

@isonic1 justin.ison@gmail.com

What is Appium?Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS, Windows and Android platforms.

Importantly, Appium is "cross-platform": it allows you to write tests against multiple platforms (iOS, Android, Windows), using the same API. This enables code reuse between iOS, Windows and Android test suites.

Appium is at its heart a webserver that exposes a REST API. It receives connections from a client, listens for commands, executes those commands on a mobile device, and responds with an HTTP response representing the result of the command execution.

Source: http://appium.io/introduction.html

History of AppiumDan Cuellar was the Test Manager at Zoosk in 2011, when he encountered a problem. The length of the test passes on the iOS product was getting out of hand. Less testing was an option, but would come with additional risk, especially with it taking several days to get patches through the iOS App Store Review process. He thought back to his days working on websites and realized automation was the answer.

Dan thought, what if I could get the UIAutomation framework to run in real time like an interpreter? He looked into it and he determined that all he would need to do is find a way to receive, execute, and reply to commands from within a UIAutomation javascript program. Using the utility Apple provided for executing shell commands he was able to cat sequentially ordered text files to receive commands, eval() the output to execute them, and write them back to disk with python. He then prepared code in C# that implemented the Selenium-style syntax to write the sequentially ordered javascript commands. iOSAuto is born.

Selenium Conference 2012: On the second day of the conference, Dan stepped up on stage to give the lightning talk. Jason Huggins, co-creator of Selenium, moderated the lightning talks. Dan experienced technical difficulties getting his presentation to load, and Jason nearly had to move on to the next lightning talk. At the last moment, the screen turned on and Dan jumped into his presentation. He explained the details of his implementation and how it worked, begged for contributors, and in five minutes it was over..

The Mobile Testing Summit: Jason decided that the project should be presented at the Mobile Testing Summit in November, but suggested that the project get a new name first. Many ideas were thrown out and they settled on AppleCart. A day later, while he was perusing some of Apple's guidance on copyright and trademarks, Jason noticed that under the section of examples for names Apple would defend its trademarks against, the first example was "AppleCart". He called Dan and informed him of the situation, and they brainstormed for a bit before Jason hit the jackpot. Appium... Selenium for Apps.

Appium Introduction Video

Credit: Dan Cuellar

Appium: The Pros• Crossplatform! (Native, Hybrid, Mobile Web)

• iOS

• UI Automation (< Appium 1.6)

• XCTest (>= Appium 1.6)

• Android

• UI Automator 1 (< Appium 1.6)

• UI Automator 1&2 (>= Appium 1.6)

• Windows

• WinAppDriver (>= Appium 1.6)

• Write your tests in language of choice.

• Share tests and functions across platforms.

• Tests are written in Selenium like syntax.

• Large community support.

• https://discuss.appium.io/

• It’s free!

• Parallelization with Selenium Grid.

Appium: The Cons

• It’s open source and you’re dependent on volunteers for updates or fixes.

• There can be delays getting new updates when changes occur to the underlining platform frameworks.

• Bugs can occur just like any software.

• Documentation or examples can sometimes be hard to find in your specific language bindings.

• Debugging issues can be challenging sometimes.

Writing a good test!

• Eliminate step duplication!

• Put common steps in your test setup and tear down.

• Never ever write tests that rely on another test or tests to pass!

• Write just enough steps to get from point A to point B the quickest way possible to validate your expectation.

• Eliminate steps that have no relation to the final expected result.

• We will go into depth on this with the next slide.

REST Your Tests Eliminating Steps!

• Reduced runtime.

• Reduced test steps.

• Limits test flakiness.

• Not failing on steps unrelated to the test expectation.

• Write less code.

• Execute tests more frequently.

Test Example

• Scenario: Validate shared list notification.

• Case: User B receives list invite from User A.

100% UI Example * verbose to show all test steps

Now lets see it run…

What’s wrong with this example?

• Relies on UI actions unrelated to the test expectation.

• Login x 2

• Logout

• Executes several UI steps to create and share list.

• More steps increase chances of failure or flakiness.

• Longer runtime.

Lets add some API methods

• Created method “api_share_a_list_to_me”.

• You can use SQL instead if an API is not available.

UI + API Example

Now lets run it again…

What did we accomplish?

• Eliminated over 20 UI steps.

• Eliminated steps unrelated to the test expectation.

• Reduced runtime from 74 to 29 seconds.

• We can run the test 2.5 times more now.

• Our test is much more resilient now and less flaky.

• What is the expectation or goal of the test?

• Where can I eliminate steps without losing test integrity?

• What if my application doesn’t have an API?

• Ask development to create a private test API for you.

• Develop the API yourself.

• Alternatives:

• Make updates via SQL to a database.

• Setup seed data to run prior to test execution.

• Requires maintenance.

Ask Yourself

Proxy Integration Tests

• Validate Endpoints.

• Validate JSON body.

• Validate Server Response.

• Shorten runtimes without full end to end test.

• Test your API and App during development of either.

Example• Scenario: Reset Password.

• Case 1: Validate client sends “POST” request method.

• Case 2: Validate client sends request to “/password/reset” endpoint.

• Case 3: Validate request body {“email”:”hello@seleniumcamp.com"}

• Case 4: Validate a status code “200” is returned from server.

Proxy Example

Now lets watch it run…

• Validating app settings when no UI response is displayed. e.g. Push notifications, Reset Password, User Data etc…

• Validating endpoints and server responses during API or mobile app development.

• Regression or Integration testing.

Proxy Use Cases

How will you test all of these?!The Challenge!

And each one of these cases?

• Validate every locale / languages your app supports.

• Validate every OS version your app supports.

• Validate different resolutions.

• Validate different screen sizes (phones, tablets, phablets etc..).

• Validate different device manufacturers.

Parallelization!

Selenium Grid + Appium

BFFS 4EVER

Android Setup

• Download every SDK version your app supports.• Install Intel HAXM Accelerator!

Create Android Emulators

• Create an emulator for each sdk you installed.

Single Process Test Example

• As you can see, this example was pretty slow. 48 seconds in total.

• Distributed means we spread each test to an available node. e.g. device.

• This example only has two tests (specs). So in theory we only need two devices, with each running a test.

• In this example we have one emulator (left) and one real device (right).

• Our test runtime goes from 48 seconds to 27 seconds! Almost in half.

Distributed Test Example

Allure Report Distributed

Parallel Test Example• Parallel in this case means we run

same tests on every available node. e.g. device.

• This example has two tests (specs). All tests will run on all devices/emulators.

• We have five emulators, each running a different SDK version, and one real device (bottom right).

• By testing on all SDK versions, this gives us a warm fuzzy feeling our app works correctly on all versions.

• As you saw, emulator-5554 (GRID5) crashed. Lets look at the report!

Allure Report Parallel

iOS Setup

• Enable Developer Mode on your devices.

• Pair your devices to your machine.

• Enable UI Automation in the Developer menu.

• Ideally, connect a device running each OS version your app supports. iOS is generally pretty good at backwards compatibility of OS versions.

Real devices…

Parallel iOS

That is great but reporting is key!

Manual Testing & Good Practices

• Monitor the app logs.

• A lot of time errors are occurring without visual notice.

• Video record your tests so you can you reproduce steps when errors occur.

• I’ve built a open source tool “Flick” to do this for both manual and automated testing.

• https://github.com/isonic1/flick (video record, track logs, install/uninstall apps, device info).

• Use a proxy debugging tool like Charles or Fiddler to monitor network traffic.

• Keep track of the app size of each build.

• iOS over-the-air install is 100MB.

• iOS has a 2GB cap in the App Store.

• Android has a 100MB cap in the Playstore.

• Keep track of your apps memory usage.

• iOS: Instruments Activity Monitor

• Android: Memory Monitor

Thank You!

• email: justin.ison@gmail.com

• twitter: @isonic1

• https://github.com/isonic1/appium-mobile-grid

• https://github.com/isonic1/flick