CodeFest 2014. Eduardo Bravo — Google+, A Deep Dive into Mobile Testing

Eduardo Bravo (Software Engineer in Test @ Google)

Google+, A Deep Dive into Mobile TestingCodefest 2014, Novosibirsk, Russia

Lets start with a question….. (‘cause life is full of them)

Is your product a “Mobile first” product?

Google+ became “Mobile first” overnight ..Testing teamManagers

“Yeah! Mobile first” Mobile testing ... what?

The Google+ Testing Picture

Google ApisServer

Foo backend

Bar backend

Many more

Google+ Backend

The Google+ Mobile Testing Picture

Google ApisServer

Foo backend

Bar backend

Many more

Google+ Backend

Mobile Testing

Google+ Android Mobile Test Infrastructure Unit tests

Hermetic UI tests

Monkeyrunner tests

Feature “Leak” tests

Android tooling

Android Unit TestingRobolectric

Why?Robolectric is designed to allow running android tests on the JVM, by using shadow objects and allowing resource loading. Robolectric tests can be run as normal java tests.

Advantages:● It allows us to shadow methods that can’t be mocked on the traditionally

android framework.(Exceptions apply, see powermock)● Decoupled from an android device or emulator.

Android Unit TestingRobolectric

All of a sudden your unit tests just got faster!

● There is no need to start an emulator (our tests run in the cloud)● Shorter build times● No need to install apks

Android UI TestingThe flakiness problem

Flaky tests are worse than having no tests

Android Hermetic UI TestsGoogle+ uses Espresso

Why?● Backwards compatibility: Froyo, Gingerbread, Ice Cream

Sandwich, Jelly Bean, Kit kat

● Handles Synchronization: Instrumentation tests run on a different (instrumentation) thread than UI operations (processed on the UI thread).

Android Hermetic UI TestsGoogle+ uses Espresso

Why?

● Rich debugging information, cleaner api, 95% test coverage, etc

Without synchronization of test operations with UI updates the tests will be prone to flakiness; Espresso takes care of this for us.

Android Emulator

Espresso Hermetic Tests

Fake Login Server

Normal flow Espresso + Hermetic flow

Google+ Backend

Login Server

Android Emulator

UI Tests

FakeGoogle+ Backend

Android Hermetic UI Tests

Android Monkey runnerWhat is it?● The Android Monkey is a tool that generates pseudo-

random user events such as clicks, touches, gestures, as well as a number of system-level events.

Purpose● Stress test applications

Android Monkey runnerSome details about our setup:

● 10 devices with different API levels running 24/7● Exceptions clustering● Over 100+ bugs found (yes, human reproducible ones’)

Android “Feature” Leak Testing Why?● Because people are curious and like reverse

engineering android apk’s to search for unreleased features.

Example from Android Police:[APK Teardown] Google Games, Anyone? Play Services Is Getting Real-Time And Turn-Based Multiplayer, Invitations, In-Game Chat, Lobbies, Leaderboards, And Achievements!

Android “Feature” Leak Testing Google+ 3-step strategy:1. Flags library that removes unreleased features at

compile time. (Provides separation between a developer build and a release one.)

2. Proguard every version sent to production. 3. Continuously run a leak test that reverse-engineers the

release apk, looking for “Feature” leaks.

Android Tooling● Android lint errors displayed at changelist review time.

● Continuous monitoring of APK size and alert system.

● Monitor method count (Don’t get caught by the 64K method reference limit, two days before shipping.)

iOS Mobile Test Infrastructure for Google+ app

Unit tests

Hermetic UI tests

Monkeyrunner tests

Feature “Leak” tests

iOS Unit TestingGTM + OCMock

GTM (google-toolbox-for-mac)● Unit test suite based on GTMSenTestcase (psst, we are

considering switching to xctest in the future).

Why OCMock?● Very powerful framework for mocking Objective-C code

and also very flexible.

iOS Hermetic UI TestingKIF (Keep it functional)

Why KIF?● More natural than UIAutomation (Javascript based)● Since tests are run in-process, it allows tests to be more

tightly coupled with the app under test, making the tests inherently more stable.

iOS Hermetic UI TestingHermetic environment

We swizzle methods that fire HTTP requests in the app, and instead have them read off of an in-memory NSDictionary to get response data.

@interface FakeGoogleApisServer : NSObject { …NSMutableDictionary *_sequentialResponses; …NSMutableDictionary *_fixedResponses;...}

KIF + Swizzling = Hermetic Google+ iOS tests

iOS Monkey Runner● Tool developed to randomly navigate an application in

order to test for crashes.● In house tool, but we are planning to open source it.● KIF Based.● The Google+ team runs it mostly on simulators.● More than 50+ bugs catched by monkey● Experimenting with a smarter monkey that can leverage

randomized and deterministic actions.

iOS “Feature” Leak TestingLooking for unreleased features in iOS apps is not a common practice like it is in android, we like to take precautions

We have a test that:- unzips a provided .zip or .ipa- Checks for leak terms in filenames- Uses the class-dump tool and makes sure no leak terms are found.

Backend Server TestingServer and Client have different release cyclesGoogle+ server releases new versions of services multiple times per week. Mobile clients (iOS and Android) usually twice a month.

Different mobile clients app versionsGoogle+ servers changes needs to be compatible against older mobile clients.

Backend Server TestingWe decided to test the protocolWe follow a replay-test approach. The approach simulates mobile clients requests and makes sure the Google+ backend server returns the response the mobile clients were expecting yesterday, and the response they were expecting 6 months ago.

Backend server testing

Fake Data

Спасибо

Вопросы?