Mobile Test Automation

© Utopia Solutions

Extending Test Automation To Mobile Applications

Lee Barnes, CTO Utopia Solutions

© Utopia Solutions

Agenda

• What is Successful MTA? • How Do We Achieve It? • Getting Started • Visual Based Tools • Object Based Tools • Evaluating Your Apps • Tool Evaluation • Summary & Questions

Mobile Test Automation (MTA)

© Utopia Solutions

SECTION 1 What Is Successful Mobile Test Automation?

© Utopia Solutions 4

WIFI / Carrier Networks

Mobile Test Automation – Goal

Automation Framework

Test Cases

Backend System

Single set of test cases executing across a diverse set of real devices


Test Automation Success Criteria

• Issue detection and recovery • Accurate verification • Unattended execution

Reliable

• Minimum sensitivity to application and test case changes

• Test case definition separate from automation code

Maintainable

• Test coverage expanded efficiently • Automated test cases created by non-

automation resources Scalable

© Utopia Solutions

SECTION 2 How Do We Achieve These Goals?


Test Automation Approach

Mobile Test Automation Framework

Automated Test Cases Test Results

Test Definition

Test Execution

Non-technical / Test Analysts & Subject Matter Experts

Technical / Automation Engineers

© Utopia Solutions

Mobile Test Automation Framework

Business / Application Testing Components Login

GoTo Stocks GoTo News

Add Stock Edit Stock

Remove Stock

Logout Get Position

Verify Position

…

… Object Interaction Methods

Object Exists Set Value

Get Value

Select Item Touch

Touch and Hold

Get Image Get Text

Send Key

Test Case

Login GoTo Stocks Add Stock Verify Position Logout

Mob

ile A

utom

atio

n Fr

amew

ork


Test Automation Challenges

• Defining appropriate scope • Integrating with SDLC • Addressing technical issues • Reliability, maintainability, scalability

General

• Platform diversity • Device diversity • Tool diversity • Tool immaturity • Lack of skilled resources • Rate of technology change • Rapid development cycles

Mobile Specific

© Utopia Solutions

SECTION 3 Let’s Get Started


Mobile Test Automation Tool Categories

Native Platform Frameworks

Multi-Platform Tools Visual Based Object Based

• Provided by mobile platform vendor as part of SDK

• Typically interact with the application at the UI object level

• Interacts with devices using visual methods (text and image recognition)

• Interacts directly with application UI objects using native methods and properties

• Object based interaction with app UI controls

• Greatest degree of support for native UI objects

• Control of device settings

• Supports multiple platforms

• Ability to execute tests across multiple devices

• Interact with entire device

• Supports multiple platforms • Ability to execute tests

across multiple platforms • Interacts with apps at the

object level for greater reliability

• Limited to a single platform • Requires code level access

to the app • May be limited to simulator

(e.g. Blackberry)

• Generally less reliable • No access to object

properties • May require jail-breaking /

rooting

• Interaction is limited to app under test

• Requires app to be instrumented

• Android UI Automator • iOS UI Automation

• SeeTest (Experitest) • eggPlant (TestPlant) • Perfecto Mobile • Zap-Fix (Zap Tech.)

• Trust (Mobile Labs) • TouchTest (SOASTA) • M-eux Test (Jamo Solutions)

Des

crip

tion

Pros

C

ons

Exam

ple


Start With a Plan

Scope

OS/Device Selection

Test Case Selection

Non-Testing Tasks

Technical Challenges

Platform/Device Diversity

Object Identification

Process Integration

Test Suite Maintenance

Continuous Integration


Automation Scope Selection

• Frequently tested • Predictable results • Easy to automate • Manually tedious

“Traditional” rules still apply

• Tests executed across all target devices • Avoid apps and tests that face many mobile

automation challenges (to be discussed)

Mobile specific

© Utopia Solutions

SECTION 4 Visual-based Tool Approach


Visual Based Identification

Label

Label + Boundary

Image Recognition

Label Offset

Multi-Label Offset


Cross Platform Issues – Inconsistent UI/UX

Button Image/Position

Screen Titles

Button Labels


Visual Based Techniques – Complex Interaction

iOS On/Off Switch

Function iosSwitchGetState (Label)

Define OCR parameter ranges

Search for Label on device screen

If Label not found

status = switch not found error

Else

Loop through OCR parameter range

Search for switch state(ON/OFF)

If state found

status = switch state

Else

status = state not found error

End If

End Loop

End If

Return status

End Function

Function iosSwitchSetState (Label, State)

CurrentState = iosSwitchGetState (Label)

If CurrentState = error

status = error

Else

If CurrentState = desired state

status = success

Else

If desired state = ON

Touch right side of switch

Else

Touch left side of switch

End if

End if

status = success

End if

Return status

End Function

Hidden Complexities


Visual Interaction – Reusability?

Maybe…

• Label / object offset • On / Off reversed

not

• Label offset right • No On / Off label


Object Identification - Defining Master Object Properties

Environment Object Type Attribute Value

iPhone Screen Method Unique Text iPhone Screen Label Screen name

iPhone Screen Location 20,25:40,200 iPhone Edit Label Object name

iPhone Edit Offset 0,-20

…

• Defines general rules for identifying objects by platform • Avoids adding all application objects to Object Repository • One master file per application (typically)

Master Property File


Object Identification - Define Object Map

Object Screen Platform

iOS

Login

User Name

Password

Add Stock Symbol

• Contains any screens and objects that don’t meet criteria defined in Master Property File

• Objects organized hierarchically by platform and screen

UI Object Map

© Utopia Solutions

SECTION 5 Object-based Tool Approach


© Utopia Solutions

Object Based Interaction

Tool Libraries Included in Project Build

Object Properties Visible to Tool

Class: UIButton Label: Save

Class: UITextField Label: Name


Object Based Interaction – Cross Platform Differences

Simple Interaction – “Tap” the Save/Done Button

Class: avwButton Method: Push Implementation: avwWindow(“MyStock”).avwButton(“Save”).Push

Class: iosButton Method: Tap Implementation: iosView(“EditStock”).iosButton(“Done”).Tap


Object Identification – Handling Cross Platform

Properties Class: Button Name: Save

Create Mapping of Universal Objects to Platform Specific Objects

Properties Class: avwButton Label: Save Method Name Touch: Push Object Path avwWindow(“MyStock”)

Properties Class: iosButton Label: Done Method Name Touch: Tap Object Path iosView(“EditStock”)

Step 1


Function TouchButton (Name)

Get execution platform

Retrieve platform specific object info

from Object Map

Construct statement to perform operation

Execute operation

Return operation status

End Function

Object Identification – Handling Cross Platform

Create Universal Method for All Platforms Step 2

Properties Class: Button Name: Save

Properties Class: iosButton Label: Done Method Name Touch: Tap Object Path iosView(“EditStock”)

iosView(“EditStock”).iosButton(“Done”).Tap

© Utopia Solutions

SECTION 6 Evaluating Your Apps for Potential Challenges


Analyze Business Processes for Differences

Differences • Menu always visible

on iOS • Different edit buttons • Different column

headers



Differences • Different screen titles • Different “add”

buttons • Different “save”

buttons • Different background

color on list



Differences • Cancel button

embedded in search field on iOS

• Different “expand” icons

© Utopia Solutions

SECTION 7 Handling Workflow Differences


Handling Workflow Differences – Go To Stocks

Android 1. Tap “MENU” 2. Tap “Stocks” 3. Sync on “My

Stocks” Screen

iOS 1. Tap “My Stocks” 2. Sync on “My

Stocks” Screen

Implement at the Business Process Component Level

Public Function GoToStocks Platform = GetPlatform() Select Case Platform Case “iOS” Tap (“MENU”) Tap (“Stocks”) status = WaitScreen (“My Stocks”) Case “Android” Tap (“My Stocks”) status = WaitScreen (“My Stocks”) Case Else status = INVALID_PLATFORM End Select GoToStocks = status End Function


Mobile Test Automation Tool Evaluation

Top Ten Questions to Ask Tool Vendors • Is your tool visual or object based? • Does your tool require jailbreaking or rooting? • How quickly do you support new mobile OS releases? • How strong is the company (e.g. sales, customers, years, etc.)? • How do I execute a single test across multiple platforms with

your tool? • Can (or must) your tool integrate with other scripting engines? • Do your support our mobile development platform? • Can your tool interact with devices remotely? • Does your tool integrate with our test management solution? • Will you assist me with a proof of concept?


Key Summary Points

• Set appropriate automation goals • Start with a plan

• Scope • Technical Challenges • Process Integration

• Understand how visual/object based tools work • Analyze your app for platform differences • Address differences at appropriate level in your

framework • Evaluate tools carefully • Start small and expand over time


Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

mailto:[email protected]