Testing for AndroidWhen, Where, and How to

Successfully Use Test Automation

Trent Peterson | @tdpeterson | trent@appthwack.com

Who?

AGENDAWhat is testing?

What’s the problem?Why should I care?

When: Choosing our battlesWhere: Best Practices

How: Tools & FrameworksQ&A

INTRO TO TESTING

Types of tests

• Functional• Non-functional• Integration• Unit• Regression• Smoke• Usability• Performance

• Stress• Acceptance• Load• Sanity• End-to-End• System• SecurityAnd on and on…

Testing distilled for this discussion

Functional TestingNon-functional Testing

THE PROBLEM

Building things is hard.

WHY SHOULD I CARE?

1,000,000+ apps in

2 crashes and 84% will uninstall- Compuware as reported in TechCrunch

YOURAPP

So what’s the big deal?

We’ve been here before.

CHOOSING OUR BATTLES

First, what’s our goal?

Save time! Save

money! Shrink QA!

AUTOMATE EVERYTHING

!

BE REALISTIC• Increase regularity of testing• Free up resources for other testing efforts• Enable previously impossible tests

Deciding what to automate

How often will I test this?

Is this something a human is good at?

How much effort will automation require?

Some examples

AUTOMATE• Granular

functionality (unit tests)

• Repetitive tasks (update paths, navigation, etc.)

• Performance

MANUAL• UX• Responsiveness and

“feel”• New functionality

Our options (Machine)

JVM• FAST• Unit tests

only*• Mocks

Emulators• Cheap• Catch layout

issues• Simulate

calls/SMS• SLOW• Illusion of 1:1

with real devices

Real Devices• Exactly what

end customers use

• Performance data

• Fast• Expensive:

Purchase, maintain, etc.

Our options (Human)

QA• Pros• Real

feedback• Know the

product• Expensive

Crowd• Real feedback• Might know

product• Expensive

(usually)• Slow

End Users• Don’t do this.

BEST PRACTICES

No barriers between Dev & QA

Automation is a software project

Automation is a tool, not a solution

Automate with precision

It’s all about data

Know your matrix

RISK̂

Abstraction is important

…but don’t obsess

TOOLS and FRAMEWORKS

Tool: Recorders

GOOD• Fast• Non-developers can

create tests

BAD• Fragile• Difficult to maintain• Non-developers can

create tests

BEST PRACTICE• Treat as a fancy spy

Tool: UI Exerciser Monkey

$ adb shell monkey –p my.sweet.app –v 1000

Framework: Android Testing Framework

SUMMARY• JUnit/Java• Instrumentation provides hooks into

Android SDK to control Android-specific functionality

• Very basic functionality supported

• The foundation for most Android testing. Learn it, use it, love it.

Framework: UI Automator

SUMMARY• JUnit/Java• Extends testing framework with UI

selectors and manipulators• Handle/control system dialogs, etc

• 4.1+ only

• Excellent choice if you have the luxury of supporting 4.1+.

Framework: Robolectric

SUMMARY• JUnit• Runs in JVM• Mocks Android functionality with

“shadow” objects• Super fast

• Not everything is mocked (also, mocks)

• Great for staying sane while developing

Framework: Robotium

SUMMARY• JUnit extension• Runs on emulators and real devices• Oriented towards black-box testing and

ensuring real-world outcomes

• Limited to the app under test

• Great choice for developers going beyond basic unit testing and testing UI.

Framework: Cucumber-based

SUMMARY• Human-readable tests• Runs on emulators and real devices• Cross-platform

• Complex scenarios require development• Behavior-driven design. Be careful!

• Great choice for simple flows and teams with limited developer resources.

Framework: Appium

SUMMARY• Supports many language adapters• Runs on emulators and real devices• Cross-platform

• New, and not all WebDriver concepts map intuitively to native apps

• Great choice for teams happy with Selenium and expanding to native.

Frameworks: Commercial

GOOD• Support• Added features and

integrations• Longevity?

BAD• Proprietary• Often heavy and

difficult to maintain• $$$

ADVICE• Evaluate very carefully and acknowledge lock-in.

Tool: Spoon

YOURAPP

Instrumentation Tests

Spoon

When choosing, ask…

What are we testing?

What are our automation goals?Who will write the automated

tests?

SHAMELESS PLUG

API• Continuous

Integration• IDE• Command Line

Clients• Browser-based

web app

Results• Interactive,

real-time report

• JSON data via API

AppThwack• 100s of non-rooted

devices• Parallel execution• Built-in compatibility

tests• Support for all popular

automation frameworks

APP

Q & A

Trent Peterson | @tdpeterson | trent@appthwack.com