Mobile Applications Testing: From Concepts to Practice

rial

Presented by:

Max Saperstone

rought to you by:

340 Corporate Way, Suite Orange Park, FL 32073 888‐2

MB AM Tuto4/7/2014 8:30 AM

“Mobile Applications Testing: From Concepts to Practice”

Coveros, Inc.

B

300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

http://starcanada.techwell.com/�

or almost a decade, Max Saperstone has been a test engineer focusing on test automation

ng

Max Saperstone Coveros, Inc.

Fand the continuous integration/continuous delivery process. Max specializes in open source tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automationefforts, including developing an automated suite focused on web-based software to operate over several applications. Max also headed a major project developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while developing a system to keep test data “ageless.” He is currently developing a new testiarchitecture for SecureCI to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.

http://starcanada.techwell.com/sme-profiles/max-saperstone�

Mobile Application Testing Tutorial

Student Handbook

February 2014

Copyright 2014 Coveros, Inc. All rights reserved.

Copyright © 2014 by Coveros, Inc. All rights reserved. No part or parts of this Coveros, Inc.

documentation may be reproduced, translated, stored in any electronic retrieval system,

transmitted in any form or by any means, electronic, mechanical, photocopying, recording or

otherwise, without prior written permission of the copyright owner.

Coveros, Inc.

4000 Legato Rd

Suite 1100

Fairfax, VA 22033

Phone: 703-349-6109

www.coveros.com

Mobile Application Testing Tutorial Welcome

Copyright 2014 Coveros, Inc. All rights reserved. Page 1

Welcome

Slide 1

Welcome to the Mobile Testing for Test Professionals training course. In this course, you will learn all about how to apply Mobile Testing to your Software Testing efforts.

1© Copyright 2013 Coveros, Inc.. All rights reserved.

Mobile Application Testing

Tutorial

Mobile Application Testing Tutorial Course Trainers


Course Trainers

Slide 2

Slide 3


Trainer

Jeffery Payne

Jeffery Payne is CEO and founder of Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Prior to founding Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc. Under his direction, Cigital became a leader in software security and software quality solutions, helping clients mitigate the risk of software failure. Jeffery is a recognized software expert and popular speaker at both business and technology conferences on a variety of software quality, security, and agile development topics. He has also testified before Congress on issues of national importance, including intellectual property rights, cyber-terrorism, Software research funding, and software quality.


Trainer

Alan Crouch

Alan Crouch is a Senior Software Security Specialist with Coveros, Inc. a Virginia-based firm focusing on agile, software quality, and application security. Alan has worked closely with federal agencies and private companies to advise, audit, and support IT security and governance teams. In addition to his cybersecurity experience, he has a strong background in software engineering, test analysis, test automation, and security testing. With a passion for software and security, Alan has focused his career on building secure software and developing better software security practices.



Course Trainers

Slide 4

Slide 5


Trainer

Max Saperstone

Max Saperstone has been working as a Software and Test Engineer for almost a decade, with a focus on Test Automation and the CI/CD process. He specializes in open source tools, including the Selenium Tool Suite, JMeter, AutoIT, Cucumber, and Chef. Max has lead several testing automation efforts, including developing an automated suite focusing on web-based software to operate over several applications for Kronos Federal. He also headed a project with Delta Dental, developing an automated testing structure to run Cucumber tests over multiple test interfaces and environments, while also developing a system to keep test data 'ageless.' He is currently developing a new testing architecture for SecureCI™ to allow testing of multiple interfaces, custom reporting, and minimal test upkeep.


Trainer

Richard Mills

Richard Mills has more than 20 years of experience in software engineering specializing in software development with a concentration on pragmatic software process and tools. Rich has a specific focus in Agile development methods, static and dynamic software analysis tools, configuration management, engineering, automated software build management, and continuous integration. He currently works as a Senior consultant with Coveros, Inc. Helping his customers build software better, faster and more securely. As a consultant, Rich helps customers improve software delivery by coaching and mentoring in the Agile development methodologies, automating software build, test, deployment and employing secure development techniques. Prior to joining Coveros, Rich spent 16 years with Cigital, Inc. fulfilling a wide range of roles including consulting, training, and software engineering.



Course Trainers

Slide 6


Trainer

Jeffrey Pierce

Jeff Pierce is a Managing Consultant with Coveros, Inc., a software company that helps organizations accelerate the delivery of secure, reliable software. Coveros uses agile development methods and a proven software assurance framework to build security and quality into software from the ground up. Jeff is a leader with over twenty years experience in information technology (IT) program/project management and client management in all phases of business investment and system development lifecycle. He has proven technical excellence in development and deployment of strategic web-based enterprise information systems utilizing cloud computing and continuous integration technologies. Jeff also has experience with strategy and roadmap development, policy and procedure development, program management office implementation, IT and business management, AGILE SCRUM and XP management, systems engineering, technical team building and leading large and small on-shore and off-shore full-time and contract staff with an outstanding record of accomplishment.

Mobile Application Testing Tutorial About Coveros


About Coveros

Slide 7

Coveros is a consulting company that helps organizations build better software. We provide software development, application security, QA/testing, and software process improvement services. Coveros focuses on organizations that must build and deploy software within the constraints of significant regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security & associated critical infrastructure companies, Healthcare providers, and Financial services institutions


Coveros helps organizations accelerate the delivery of secure, reliable software

Our consulting services:– Agile software development

– Mobile application development

– Application security

– Software quality assurance

– Software process improvement

Our key markets:– Financial services

– Healthcare

– Defense

– Critical Infrastructure

About Coveros

Corporate Partners

Mobile Application Testing Tutorial Agenda


Agenda

Slide 8


Agenda

Introduction to Mobile Testing

Mobile Web Applications

Native Mobile Applications

Unique Mobile Testing Considerations

Mobile Testing Platforms

Mobile Testing Techniques

Tools for Mobile Testing

Wrap-Up

Mobile Application Testing Tutorial Introduction to Mobile Testing



Slide 9

Slide 10




Agenda

What is Mobile Application Testing?

Why is Mobile Application Testing Important?

What Do You Need to Know to Be a Good Mobile Tester?

Types of Mobile Applications

Mobile Application SDLC

Mobile Application Testing Tutorial Definition of Mobile Application Testing


Definition of Mobile Application Testing

Slide 11

Companies everywhere are rapidly building and deploying mobile applications. Testing these applications must be accurate, reliable, usable and rapid – keeping up with the pace of development and changing mobile platforms. Additionally MAT will look at testing areas that have typically only been tested by hardware engineers and will need to testing focus in order for mobile applications to be built, delivered, and executed. Title Month Year


Definition of Mobile Application Testing

Mobile Application Testing is a process by which application software developed for mobile devices is tested for functionality, security, usability and consistency.

Although Mobile Application Testing is a relatively new aspect of software testing, all existing standard testing practices and techniques can be applied.

What is Mobile Application Testing?

Mobile Application Testing Tutorial Mobile Application Testing Importance


Mobile Application Testing Importance

Slide 12

App stores can reduce the time in the feedback loop, and customers can be utilized for early and frequently feedback. Without it, you may be developing an application your customers don’t want

As with all software, code quality verification is important. Users are quicker than ever to abandon an application that doesn’t work perfectly the first and every time.

Users have high expectations and are App stores also make it extremely easy to get software out to the public at a rapid pace. You want to ensure your company is putting their best foot forward to maintain a good reputation Having a consistent, predictable release process will ensure good software is released every time

The extensive choice of browsers, operating systems, devices and applications make it possible for enterprises to develop new, interactive and highly personal ways for customers and employees to access products, services and applications. Mobile devices are at the same time sources of rich information, powerful entertainment and purchasing platforms, and important work tools.

Companies that successfully implement mobile strategies engage both customers and employees in new ways, and create an environment of personal interaction that enhances sales and productivity. What makes mobile so attractive to consumers—instant access to a multitude of apps; a wide range of rapidly evolving devices and functions; a choice of operating systems; new, cloud-based services that require no storage on the device—also makes it tricky for companies to deploy effectively.

The fourth annual World Quality Report—from Cap Gemini, Sogeti and HP—reveals that only 31% of organizations surveyed currently have formal processes in place to test their mobile applications, while just 18% of those have a focus on security


App stores reduce the user feedback loop.

User feedback is both early and frequent.

Usability and quality is king, when users are quicker than ever to abandon an application that doesn’t work the first time.

Users can get updates rapidly and expect seamless updates and integration.

Mobile devices are changing quickly and constantly getting new capabilities and features.

Why Is Mobile Application Testing Important?

Mobile Application Testing Tutorial Mobile App Testing Difference


Mobile App Testing Difference

What Do You Need To Be a Good Mobile Tester?


Mobile Application Testing Tools aren’t Mature or Available– Many tools are created for specific device/OS combinations

– Cross-platform tool support is limited

– No one tool will do

Mobile Application Testing Requires Specialized Skills– UI and usability focused

– Strong testing background

– Understand networking

– Engineering/hardware focus

– Constantly learning

Cloud Components

Cost of Entry

Automation Immaturity– Automation image based and time consuming

– Automation difficult to leverage outside the framework

What Makes Mobile App Testing Different?


What Do You Need To Be a Good Mobile Tester?

History of Mobile Devices and Emerging Trends

Basic Software Testing Skills

Basic Understanding of the Mobile Software Lifecycle

Basic Understanding of Telecommunications

Awareness of Mobile Operating Systems/Platforms– Experience in virtualization for Linux, or VMWare, and data center

operations

Experience with cloud infrastructure and SaaS

Knowledge of Mobile testing tools and frameworks

Knowledge and experience with test automation tools

Mobile Application Testing Tutorial Understand the Basics of Software Testing


Understand the Basics of Software Testing

Slide 15

This course is not meant to teach you the basics of software testing, it assumes you know them and can apply them.


Understand the Basics of Software Testing

Knowledge of Software Testing basics is essential to testing any platform. Even if you just starting your career, you need to know:

– Software Testing Goals and Objectives

– Software Testing Principles

– Software Testing processes, methodologies, and techniques

If you don’t have a strong foundation in software testing, we suggest you start simple and enroll in some basic training programs.

What You Need to Know

Mobile Application Testing Tutorial Understand the Basics of Telecommunications


Understand the Basics of Telecommunications

Slide 16

Understanding the basics of Telecommunications will always give you advantages when doing testing

mobile applications. Title Month Year


Understand the Basics of Telecommunications

Mobile App Testing involves networks, not just the app or device itself.

Can you identify and understand the differences in these technologies?

– 2G, 3G, 4G, Edge, LTE, CDMA, GPRS, GSM, HSCSD, SIM, SMS, WAP, TCP, UDP, Wi-Fi

Telecom is global – …therefore, so is your app.


Mobile Application Testing Tutorial Mobile Operating Systems/Platforms


Mobile Operating Systems/Platforms

Slide 17

Slide 18


Awareness of Mobile Operating Systems/Platforms

OS Design

– Integration Points– Device Capability Usage– Security

Features/Vulnerabilities

Device Operation

– Standalone

– Offline

– Wi-Fi

– Interruptions



Awareness of Mobile Operating Systems/Platforms

Device Design

– Variable Screen Size/Resolution

– Device specific Browser Limitations/Behaviors

– Variable input/navigation methodologies Touch Screen

Keyboard/virtual keyboard

Ball/Pointer

Voice

Menu Structures

– CPU Constraints/Variability

– Memory Constraints/Variability

– Power utilization constraints

– Keypad lock


Mobile Application Testing Tutorial Emerging Trends


Emerging Trends

Slide 19


It’s important to stay on top of emerging trends!

Staying on top of emerging trends will better help you

understand what you may need to test for when your

business wants to add or integrate a new piece of

technology into your mobile application. In addition, it

will make your products more competitive in the

marketplace.

Emerging Trends

Mobile Application Testing Tutorial Types of Mobile Applications



Slide 20

Slide 21



Mobile Web Apps– Non-Responsive

– Responsive

Native Apps– Device-Specific

– Responsive

– Hybrid Apps



Web apps are really websites that, in many ways, look and feel like native applications. They are run by a browser and typically written in HTML5.

Users access them as they would access any web page: they navigate to a special URL.


Mobile Application Testing Tutorial Native Applications


Native Applications

Hybrid Applications


Native Applications

Native apps live on the device and are accessed through icons on the device home screen.

Native apps are installed through an application store and are developed specifically for one platform.

They can take full advantage of all the device capabilities and can work offline.



Hybrid Applications

Hybrid apps are application that combines elements of both native and web applications.

Hybrid apps are installed through an application store but integrate with a large amount of content from Web-based services.

They can often function without an Internet connection, but may have limited capabilities when disconnected

The majority of apps fall into this category.


Mobile Application Testing Tutorial Mobile Application SDLC


Mobile Application SDLC

Slide 24 - [Source: Harty, Julian. “Mobile Apps”]

Title

Slide 25


The Waterfall Mobile Application SDLC

Requirements

Implementation

• Design

• Code

Verification

• Unit Tests

• Internal Installation

• System Tests

Launch/Release

• Pre-Publication

• Publication

Engagement

• Search

• Trust

• Download

• Installation

• Upgrade

• Downgrade

Validation

• Payment

• Use

• Feedback


The Agile Mobile Application SDLC

La

un

ch

Rele

ase

s 1

-n

Sprint 1Sprint 0 Sprint 2

End Game Engagement

Requirements

Design

Implementation

Testing

Launch/Release

Engagement/Validation

Mobile Application Testing Tutorial Mobile Web Applications



Agenda




Agenda

– Overview

– Types of Mobile Web Apps Non-Responsive Applications

Responsive Applications

– Testing

– Technologies

Mobile Application Testing Tutorial What is a Mobile Web Application?


What is a Mobile Web Application?

Exercise


Overview

What is a Mobile Web Application?

A mobile web application is an application that is accessed by users over a network such as the Internet or an intranet on a mobile device.

Mobile Web applications are generalized for multiple platforms and not installed locally but made available over the Internet through a browser.

e.g. Online Banking Services

Mobile Web Apps can be written with HTML, CSS and JavaScript


Check your HTML 5 capabilities

With your Mobile Device, go to: – http://webdirections.org/deviceTest/

Exercise

Mobile Application Testing Tutorial Advantages


Advantages

Non-Responsive Apps


Overview

Advantages

You can use simple HTML, CSS, JS for most mobile applications

You can maintain and fix bugs in real time.

Quick upgrade and migration

Disadvantages

You can’t sell your application

Must be responsive to multiple resolutions, operating systems and browsers

Struggle with performance and testability


Types of Mobile Web Apps

Non-Responsive Apps

A non-responsive web application that responds the same way on a laptop’s web browser as it would on a mobile device.

More and more website’s are moving away from non-responsive apps, but there are many reasons why someone would choose to have a non-responsive app.

Mobile Application Testing Tutorial Responsive Apps


Responsive Apps

Responsive Web Apps - Example



Responsive Apps

Responsive Web design is an approach that suggests that an app should respond to the user’s behavior and environment based on screen size, platform and orientation.

The practice often consists of flexible screens, grids, and layouts, images and an intelligent use of CSS media queries, automatically switching to accommodate for resolution, image size and scripting abilities.




1920x1080 Screen Resolution – Windows 7

www.coveros.com

Mobile Application Testing Tutorial Responsive Web Apps - Example







1920x1080 Screen Resolution – Windows 7

www.coveros.com




1136 x 640 Screen Resolution – iPhone5

www.coveros.com

Mobile Application Testing Tutorial Testing a Mobile Web App


Testing a Mobile Web App

Slide 36


Testing

Testing a Mobile Web App

Testing a mobile web app is fairly similar to how you would test any other web application. In fact, you can often use the same tools.

Using emulators and user-agents you can often replicate resolutions and device types on your laptop.

However, there are some things you won’t be able to replicate (gestures, shakes, etc.), so don’t think that you can avoid testing on an actual mobile device.

For responsive web apps, you can often reuse tests suites for various resolutions, but no good test plan for a responsive app will have one set of identical tests for each resolution.

Mobile Application Testing Tutorial


This Page Intentionally left blank

Mobile Application Testing Tutorial Native Mobile Applications



Agenda




Agenda

Overview

Types of Native Apps

– Device Specific Applications– Responsive Applications– Hybrid Applications

Testing

Technologies

– OS API– APP Store– Device Capabilities

Mobile Application Testing Tutorial What is a Native Application?


What is a Native Application?

Device-Specific Applications


Overview

What is a Native Application?

Native applications are developed for a specific platform and installed on a mobile device.

Native applications can be pre-installed or installed through an app store on demand.

e.g. Phone app vs. Instagram.

Native applications are built in a language for a specific mobile device



Device-Specific Applications

Device-Specific Applications are native apps built for not only one platform (e.g. iOS) but also a device (e.g. iPhone).

Responsive Applications

Responsive Applications are platform-specific native apps that are built to respond to various screen size, resolutions and other built-in device capabilities based on the mobile device the application is installed upon.

Mobile Application Testing Tutorial Hybrid Applications


Hybrid Applications

Testing a Native App



Hybrid Applications

By a strict definition, Hybrid apps are technically native apps. However, they are part native apps, part web apps; they live in an app store and can take advantage of the many device features available but rely on HTML being rendered in a browser embedded within the app itself.

Hybrid apps allow an organization to have consistency between different device platforms.

They can also provide presence in the app store without having to build much of a native app.

They can provide some cross-platform development, because HTML components can be reused to reduce costs.


Testing

Testing a Native App

Testing a native app is a mix of software and hardware testing. A tester has to test the app in all the ways a user may use the app (not just standard user behavior).

Using emulators you can often replicate resolutions, gestures and device platforms on your laptop.

However, there are some things you won’t be able to replicate, so don’t think that you can avoid testing on an actual mobile device.

Since each of the selection points are determined by the developer, test that there is enough space to perform actions and that you are properly applying OS Gestures.

Mobile Application Testing Tutorial Testing a Hybrid App


Testing a Hybrid App

Slide 43


Testing

Testing a Hybrid App

Arguably, testing a hybrid app is the most complicated to plan and strategize. A good tester must know how to test both native app and mobile web app components to test a hybrid app successfully.

You can test many components like you would a mobile web app (even the same tools).

Testing the components built with the native code, should be performed just like a typical native app.

However, don’t forget about your integration points!

There are native mobile browser functionality that can have features that can cause unintended behavior in the embedded browser.

Mobile Application Testing Tutorial Unique Mobile Testing Considerations



Agenda




Agenda

User Expectations

Network

Data Storage

Operating System

Jail Breaking

Hardware/Device Capabilities

Security

Mobile Application Testing Tutorial User Expectations Challenges


User Expectations Challenges

Network Challenges


User Expectations Challenges

Global Users– Global User Community

Internationalization/Localization requirements

Instant Feedback (online ratings)

Expectations – Intuitive UI

– Performance – 60% of mobile app users will abandon app or site if load time > 3 seconds

– Seamless across multiple devices

– Seamless across various connectivity options (3g, 4g, Wi-Fi)

– Security

– Data utilization

– Power utilization

– Easy Installation and Launch

Testing Mobile Applications


Network Challenges

Networks– More than 400 mobile networks globally

Numerous Global Standards– NMT (Nordic Mobile Telephony)

FDMA (Frequency Division Multiple Access)

– GSM (Global System for Mobile Communications) TDMA (Time Division Multiple Access)

FDMA (Frequency Division Multiple Access)

– UMTS (3GSM) (Universal Mobile Telecommunications System) W-CDMA (Wideband Code-Division Multiple Access)

– IS-95 (CDMA one) / IS-2000 (CDMA 2000) CDMA (Code-Division Multiple Access)

– WIMAX / LTE (Long Term Evolution) OFDMA (Orthogonal Frequency Division Multiple Access)


Mobile Application Testing Tutorial Network Challenges


Network Challenges


Network Challenges

Multiple Carriers– Verizon, AT&T, Sprint, T-Mobile, Clearwire, US Cellular, Leap

Wireless, nTelos, etc.

– Not all 4G Networks are created equal. 4G for Verizon is not the same as 4G for AT&T

Network Hopping– Cell to Cell Hopping

– Roaming

– Cell to Wi-F

– 3G to 4G, Edge to 3G, etc.

– International Networks

Near Field Communication

Bluetooth



Network Challenges

Implementation variability and issues– Tunneling

– Web Proxies limiting site connectivity

Transcoding

http header information filtering

Several Bandwidth/Connection Modes– 1G, 2G, 3G, 4G

– Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)


Mobile Application Testing Tutorial Data Storage Challenges


Data Storage Challenges

Operating System Challenges


Small Capacity

Data Privacy / Shared Storage

Data Encryption (or lack there of)

Cloud Data Storage

Shared Data Resources – Photos

– Calendar

– Video

– Mail

– Messages

– Notifications

Data Storage Challenges




App Store Integration

OS API

Fragmentation

Backwards Compatibility

No Industry-wide standards

Capability, Capacity and Performance


Mobile Application Testing Tutorial Operating System Challenges





Interruptions are events that would interrupt normal operations

There are two types of Interruptions– Application Interruptions:

Installation, Launch, Run, Exit, Upgrade, Uninstall/Delete

– Device/OS Interruptions Incoming Call

Receiving Incoming Call

Receiving Message

Device Shutdown

Device Restore

Remove Battery

Camera Activated

Loss of Connectivity

Regain Connectivity


Mobile Application Testing Tutorial About Jailbreaking


About Jailbreaking

Jail Breaking Challenges


About Jailbreaking

Jailbreaking (iOS) or Rooting (Android) is the process of removing the limitations on mobile devices through the user of software or hardware exploits.

Jailbreaking permits root access to the operating system, allowing download of non-approved apps, extensions and themes not available otherwise.

Large populations of users utilize Jailbroken/rooted mobile devices.

Corporate policy may permit devices from being Jailbroken



Jail Breaking Challenges

Providing escalated privileges by overcoming limitations in a computer system or device by deliberating overriding security, administrative and marketing restrictions by the creator

Non-Approved Applications

Malware, Worms– First iOS Worm, named iKree, was released to affect Jailbroken

iPhones in 2009


Mobile Application Testing Tutorial Hardware Challenges


Hardware Challenges

Slide 55

Slide 56


Hardware Challenges

Screen Resolution– Size

– Clarity

– Portrait vs. Landscape

Bluetooth / NFC

Camera

Processing Power

Speaker/Microphone

Flashlight

Accelerometer

Magnetometer



Hardware Challenges

Touch Screen– Single Touch Inputs

– Multi-Touch Inputs

– Virtual Keyboard

– Multiple Inputs Simultaneously

Physical Inputs– Keypads

– Track Ball

GPS / Location Services

Power Consumption

Memory Usage– Launch, Background, Foreground, Exit

– Simultaneous Applications

– Memory Sweeping


Mobile Application Testing Tutorial Security Challenges


Security Challenges

Slide 57

Slide 58


Security Challenges

Encryption– Data at Rest

On Device

On Middle-Tier Servers

On Databases

Cached Content

– Data in Transit

Privacy– Application Privileges

Features

Data

Capabilities

– 3rd Party Application Integration

User Configurable Trust Settings



Security Challenges

Operating System– Malware

– Critical Bugs

– Out of Date (Upgrading)

– Support ends after two years typically

Network– No Firewalls

– Few devices limit Internet connections

– Unsecure Communication Channels Bluetooth Discovery Mode

Unsecure Networks

Unauthorized Modification, Jail Breaking, Rooting


Mobile Application Testing Tutorial Mobile Testing Platforms



Agenda




Agenda

Introduction

Desktop Browsers

Mobile Emulation

Cloud Testing

Physical Device Testing

Remote-Device Testing

Recommendations

Mobile Application Testing Tutorial Choosing the Right Platform


Choosing the Right Platform

What is possible with Desktop Browsers?


Introduction

Choosing the Right Platform

There are many different device platforms, that are often fragmented and grow at different paces

iOS, Android, Blackberry, Windows phone, Symbian, etc.

Multiple form factors and screen resolutions

Users get new devices every 24 months

There are often carrier restrictions on devices (beyond network considerations)


Desktop Browsers

What is possible with Desktop Browsers?

Desktop browsers can be extremely useful when developing and testing mobile web apps.

Desktop browsers like Chrome and Firefox have many tools to effectively use user-agents to simulate multiple mobile browser types with little or no cost.

Leveraging desktop browsers does not replace device testing but they can provide robust debugging options and can be used to simulate multiple screen sizes.

Mobile Application Testing Tutorial User-Agent Switcher for Chrome


User-Agent Switcher for Chrome

Slide 63

https://chrome.google.com/webstore/detail/user-agent-switcher-for-c/djflhoibgkdhkhhcedjiklpkjnoahfmg/reviews?hl=en-US

You can also add user-agent strings as necessary (highly recommended) Title Month Year


Desktop Browsers

User-Agent Switcher for Chrome

Example: iPhone Agent on Google Chrome in Incognito Mode

Mobile Application Testing Tutorial What is Mobile Emulation?


What is Mobile Emulation?

iPhone Emulator


Mobile Emulation

What is Mobile Emulation?

Emulators offer the less expensive testing option, but they have many limitations for mobile testing and may not give an accurate depiction of the mobile user experience.

However, as part of the mobile testing process the use of emulation has an important role. There is certain testing that you can conduct with emulators that provides value. For example, testing for screen size or visual form factor.

However, emulation is often an approximation of the rendering and can only get you so far.

Leveraging emulators is a positive step towards real device testing, however, not sufficient for comprehensive testing needs.


Emulator

iPhone Emulator

Mobile Application Testing Tutorial iPad Emulator


iPad Emulator

Android Emulator


Emulator

iPad Emulator


Emulator

Android Emulator

Mobile Application Testing Tutorial What is Cloud Testing?


What is Cloud Testing?

Sauce Labs


Cloud Testing

What is Cloud Testing?

Cloud Testing platforms allow testers and developers to run their applications on cloud-based emulators and simulators in virtual environments.

This eliminates the need to maintain emulator, simulators for all your testing/development teams.

You can easily test on all types of OS versions and device models.

Some cloud-based tools allow mobile teams to receive real-time alerts against any measurement criteria, allowing them to address issues before they impact end-users

There are often APIs that allow automated testing through these platforms.


Cloud Testing

Sauce Labs

Mobile Application Testing Tutorial What is Device Testing?


What is Device Testing?

Crowdsourcing


Physical Device Testing

What is Device Testing?

Testing on a real, physical device with full functionality to the tester.

Physical device testing will always provide the most accurate testing results, but is also the most costly.

Without access to the real processor and hardware to understand the quirks that some real device have, testers may not discover some real life issues that do not present themselves in emulators


Crowdsourcing is the practice of obtaining testing services by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees, outsourcing or suppliers.

It combines the efforts of numerous self-identified volunteers or part-time workers, where each tester of their own initiative adds a small portion to the greater testing efforts.

Crowdsourcing is different from outsourcing in that the work comes from an undefined public rather than being commissioned from a specific, named group.

Crowdsourcing

Mobile Application Testing Tutorial What are Remote Device Access (RDA) Service?


What are Remote Device Access (RDA) Service?

Which Platform is Best for My Mobile Testing?


Remote Device Access

What are Remote Device Access (RDA) Service?

RDA services enable access to a live device over the Internet. As a Mobile Apps Tester, you should be aware of such services and should suggest your managers about the capabilities of such services.

Due to the large number of devices available in the market, it is not feasible to buy a new device every time. That makes RDA a much more cost effective solution.


Recommendations

Which Platform is Best for My Mobile Testing?

Type of Testing Browser Emulator Device RDA Cloud Emulator

Debugging x x x

Feature Testing x x x

Usability Testing x x x

Exploratory

Testing

x x x x x

Performance

Testing

x x x x x

Automated Testing x x x

UAT x x

Security Testing x x x x x

Pre-Prod Testing x x

Mobile Application Testing Tutorial Mobile Testing Techniques



Agenda




Agenda

– Exploratory Testing

– Device Access

– Security Testing

– Performance Testing

– Compatibility Testing

– Submission Guideline Testing

Mobile Application Testing Tutorial Mobile Testing Techniques



Exploratory Testing



Test Management

Exploratory Testing

Service/API Testing

Device Access

Security Testing

Back-End Testing

Mobile UI Automation

Performance Testing

Submission Guidelines

Testing

Compatibility Testing


Exploratory Testing

Exploratory Testing is still the most widely used mobile testing technique

Necessary for some validation

Frequent key in outsourcing and crowdsourcing scenarios

Advantages– Tester can assess performance of application under actual

operating conditions Determination/Evaluation of Performance

Determination/Evaluation of Ease of Use

Identification of unexpected behaviors and errors

Does it feel right?

– Test-As-You-Fly A Goddard Space Flight Center Technical Standard which essentially means

that you test the system under test as it is intended to be used.


Mobile Application Testing Tutorial Standard Mobile App States


Standard Mobile App States

Interruption Testing


App Store– Upload– Download– Update Broadcasting– Monetization scenarios

Installation and Removal– Multiple Modes– Verification of App Logo and Splash

Operation– Start/Stop– Pause/resume– Credentials– Auto-Start

Operational Modes– Standalone– Offline– Wi-Fi– Cabled– Airplane Mode

Standard Mobile App States

Exploratory Testing


Incoming Call

SMS Received

Voicemail

Email

Social media notifications

Alarms

Calendar reminders

Battery low

Battery Critical

Power Sleep Cycle

Battery Removed

Battery Charging

Low Memory

Device Switched Off

Device Reboot

Network outage and Recovery

Bluetooth Connection request

Phone/System Updates

Data Transfer Cable Insertion

Camera Activation

Home Button

Search Button

Volume Control Changes

Screen Lock

Interruption Testing

Exploratory Testing

Mobile Application Testing Tutorial User Interface Testing


User Interface Testing

User Inputs


Layouts, alignment, colors, fonts

– Views/Tables

– buttons, Menus, Links, Pickers, Toolbars

– Alerts/Messages

– Themes, Color Schemes, Styles

– Brightness Settings

UI Elements– Position

– Size

– Inputs

– Actions

Images and Symbols– Clarity

– Alignment

– Density (load performance)

Error Message Displays

Consistency– Across the application

– Across various devices

User Interface Testing

Exploratory Testing


Gestures– Tap

– Press/Hold

– Double Tap

– Drag

– Pinch

– Spread

– Swipe/Flick

– Rotation

– Shake

– Tilt

User Inputs

Exploratory Testing

http://uxdesign.smashingmagazine.com/2011/10/06/not-your-parents-mobile-phone-ux-design-guidelines-smartphones/

Mobile Application Testing Tutorial Accessibility and Usability Testing


Accessibility and Usability Testing

Slide 82

Slide 83


Usability– Efficiency

– Accuracy

– Recall

– Emotional Response

Accessibility– Screen Size

– Lighting Conditions and brightness settings

– Multi-screen capability and compatibility

– Touch Interface size


Exploratory Testing


Workflows– Information Presentation

– Ease of Task Completion

– Collaborative Task Completion (Sequential and Parallel)

Application Integration and Interactions– Native Application/Device

– Web Application/Device

– Hybrid Application/Device/Web App


Exploratory Testing

Mobile Application Testing Tutorial Mobility


Mobility

Connectivity


Geo aware (GPS, Tower Triangulation)

Accelerometer (Motion, Gestures)

Magnetometer (Compass)

Traversing zones/cells/networks– GPS Signal fade/interference

– WiFi Signal fade/interference

– Cellular Signal fade/interference

Mobility

Exploratory Testing


Near Field Communication– Android Beam

– Bluetooth

– Wi-Fi

Networks– Variable Networks Data throughput support

– Variable network connectivity protocol types 4g/3g/analogue

1G, 2G, 3G, 4G

Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)

– Signal Strength No Network

Intermittent connectivity

Strong Signal

Varying signal strength

– Trust level settings (Security/Data Exposure)

Connectivity

Exploratory Testing

Mobile Application Testing Tutorial Usability: Deadly Application Failures


Usability: Deadly Application Failures

Testing Device Sensors


Exploratory Testing

Usability: Deadly Application Failures

• Insufficient features and functionality

• Consumption of resources (Battery, CPU, Storage)

• Failure to operate well under sub-optimal connectivity situations

• Slow Performance

• Interoperability• doesn’t integrate with native device features

• causes errors in other existing applications

• Difficult to use


Rotation/Tilt– Not supported (user expectations problem)

– Could cause screen res/layout issues (switching from Portrait to Landscape mode)

– Other unexpected results

Movement/Shake Inputs– Not supported (user expectations problem)

– Unexpected results Does jogging on a treadmill cause song over-shuffling on iPod?

Geo-Location Capabilities– Verify location mapping correct

– Verify location capabilities while in motion

– Verify lack of freeze-ups while in motion

– Verify re-location upon loss/re-gain of signal

Magnetometer/Rotation– Verify Compass direction correct

Light– Automatic brightness adjustments

Heat– Device function shutdown, reduced performance

Testing Device Sensors

Device Access Testing

Mobile Application Testing Tutorial Security Testing


Security Testing

Performance Testing


Security Testing

Network Attacks– Traffic Analyzers

Server Side Attacks– API/Service Attacks

– Back-end Server

– Database Attacks

Client-Side Attacks– Cross Site Scripting

– Data Privacy Exploitation

– Buffer Overflows

Injection and Data Flow Analysis

Jail-breaking/Rooting

Penetration Testing



Performance Testing

Mobile apps are often supposed to scale to massive volumes. We have to delve into lots of different realistic scenarios for performance over more than just WI-FI.

Performance testing should cover real world mobile networks (Not all networks are created equal).

– 4G LTE, 3G, 2G, EDGE, GPRS

Don’t forget to test middle-tier and back-end for performance

If you’re not testing your mobile app out in the real world like a client would experience, your performance testing is not complete


Mobile Application Testing Tutorial Performance and Stress


Performance and Stress



Low resources– Optimization Modes

– Power saving Modes

– Variable Battery Levels (Full, Low, Critical, Charging)

– Memory/CPU utilization Launch, Background,

Foreground, Exit

Simultaneous Applications utilization

Continuous Keypad entry (across various keypad controls/events)

Memory Sweeping

– Storage Space Installation

Application Utilization

– Battery Consumption

Server Load– Stress testing

Multi-user

Multi-functional

– Backups

Crowd-Source Testing

Network Testing – Internetwork

– Network Roaming

– Wife Cell

Performance and Stress



Testing for Compatibility:-Testing the compatibility of your application with native device features (i.e. To make sure your application is not hampering native device functionality)

Platform

Device

Other Applications

Application backwards compatibility

Browsers

Networks



Mobile Application Testing Tutorial Submission Guideline Testing


Submission Guideline Testing

Slide 92

Just an excerpt from the Apple Guidelines:

2. Functionality 1. Apps that crash will be rejected

2. Apps that exhibit bugs will be rejected

3. Apps that do not perform as advertised by the developer will be rejected

4. Apps that include undocumented or hidden features inconsistent with the description of the app

5. will be rejected

6. Apps that use non-public APIs will be rejected

7. Apps that read or write data outside its designated container area will be rejected

8. Apps that download code in any way or form will be rejected

9. Apps that install or launch other executable code will be rejected

10. Apps that are "beta", "demo", "trial", or "test" versions will be rejected

11. iPhone apps must also run on iPad without modification, at iPhone resolution, and at 2X iPhone

12. 3GS resolution

13. Apps that duplicate apps already in the App Store may be rejected, particularly if there are many

14. of them

15. Apps that are not very useful or do not provide any lasting entertainment value may be rejected

16. Apps that are primarily marketing materials or advertisements will be rejected



Google Play Store– Quick review

– Simple standards to meet

– http://play.google.com/about/developer-content-policy.html

Apple App Store– Notoriously difficult standards

– Long review time

– Stringent Requirements

– https://developer.apple.com/appstore/guidelines.html


Mobile Application Testing Tutorial Submission Guideline Testing


17. Apps that are intended to provide trick or fake functionality that are not clearly marked as such

18. will be rejected

19. Apps larger than 20MB in size will not download over cellular networks (this is automatically

20. prohibited by the App Store)

21. Multitasking apps may only use background services for their intended purposes: VoIP, audio

22. playback, location, task completion, local notifications, etc.

23. Apps that browse the web must use the iOS WebKit framework and WebKit JavaScript

24. Apps that encourage excessive consumption of alcohol or illegal substances, or encourage

25. minors to consume alcohol or smoke cigarettes, will be rejected

26. Apps that provide incorrect diagnostic or other inaccurate device data will be rejected

27. Developers "spamming" the App Store with many versions of similar apps will be removed from

28. the iOS Developer Program Title Month Year

Mobile Application Testing Tutorial Android Core Quality Requirements Categories


Android Core Quality Requirements Categories

Apple App Store Requirements Categories


Functionality– Permissions– Installation– Audio– UI and Graphics– User / App State

Performance and Stability– Stability– Performance– Media– Visual Quality

Google Play– Policies– App details page– User Support

Android Core Quality Requirements Categories



Functionality

Metadata

Locations

Push Notifications

Game Center

iAds

Trademarks and trade dress

Media Content

User Interface

Purchasing and Currencies

Scraping and Aggregation

Device Damage

Personal Attacks

Violence

Objectionable Content

Privacy

Pornography

Religion, Culture, and ethnicity

Contests, sweepstakes, lotteries, and raffles

Charities and Contributions

Legal Requirements

Apple App Store Requirements Categories


Mobile Application Testing Tutorial Tools


Tools

Agenda


Tools


Agenda

Introduction

Multi-platform Testing Tools

Android Testing Tools

IOS Testing Tools

Blackberry testing tools

Windows Mobile Testing Tools

Remote Device Access Services

Outsource/Crowd Source Testing Services

Mobile Application Testing Tutorial Multi-platform Test Tools


Multi-platform Test Tools

Multi-Platform Test Tools


Selenium 2 (WebDriver)

Can run on emulators and devices

Native support exists for Android, iPhone and iPad but is deprecated

Third party drivers have been developed to fill in these gaps– iOS-Driver

– Selendroid

– Appium

Support for all major browsers– Browsers downloaded and installed on devices to supplement

native browsers can be fully tested with WebDriver.

Multi-platform Test Tools

Testing Tools


Gomez, mobiReady, and W3C mobileOK Checker

Tests web based apps and evaluates their mobile performance.

Examines mobile best-practices and usability, and provides steps to correct

Web-based testing, no download or install needed

Multi-Platform Test Tools

Testing Tools

Mobile Application Testing Tutorial Android Tools


Android Tools

Android Debugging Tools


Monkey Runner – Runs on simulator or device

– Generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events.

– Can run an automated start-to-finish test of an Android application. You provide input values with keystrokes or touch events, and view the results as screenshots.

Robotium– Support for native and hybrid applications

– Can write function, system and acceptance test scenarios, spanning multiple Android activities

– Integrates smoothly with Maven or Ant to run tests as part of continuous integration.

Android Tools

Testing Tools


Dalvik Debug Monitor Server (DDMS)– Port-forwarding services

– Screen capture on the device

– Thread and heap information on the device

Logcat – Process, and radio state information

– Incoming call and SMS spoofing

– Location data spoofing and more

TaskKiller– A utility application used to kill process

– Useful to skip device reboot.

Android Debugging Tools

Testing Tools

Mobile Application Testing Tutorial iOS Tools


iOS Tools

Blackberry Tools


UIAutomation– Lets you run an automated set of tests against an application– Can be used to perform automated tests on real devices and on the

iPhone Simulator– It enables you to quickly track regressions and performance issues

NUnit– Unit level testing for iOS apps

Test Studio for iOS– Lets developers test native iOS apps as well as hybrid and Web apps– The tool can test common iOS controls and record test scripts for

gestures such as taps, multiple finger gestures such as zooms, swipes and taps.

iPhone test automation– Fonemonkey– UISpec– Bromine (not recommended)

iOS Tools

Testing Tools


Testing Tools

BlackBerry device simulator – “Fledge”

Jamo Solutions – “M-eux Test”

Borland Silk Mobile

Blackberry Tools

Max1

Mobile Application Testing Tutorial Windows Tools


Windows Tools



MSDN Hopper Test Tool– Simulates input stress on Windows Mobile devices

– Hopper will stress all applications that are available through the menu system by rapidly sending keystrokes and screen taps in a random fashion

MSDN Device Emulator– Mimics the behavior of Windows Mobile

– Provides a virtual hardware platform that you can use to test applications on multiple virtual devices

– Runs code compiled for ARM microprocessors

MSDN Cellular Emulator

Windows Tools

Testing Tools


DeviceAnywhere

Perfecto Mobile

Keynote

Paca Mobile Center

pCloudy


Testing Tools

Mobile Application Testing Tutorial Security Testing Tools


Security Testing Tools

Performance Testing Tools


Kryptowire (Native App Testing Tool)

OWASP ZAP (Mobile App Testing Tool)

For more information and training on Security Testing Tools, check out the Security Testing Course from SQE.

Additional resources at owasp.org

Security Testing Tools

Testing Tools


NeoLoad

BlazeMeter

SOASTA

Mobitest

Performance Testing Tools

Testing Tools

Mobile Application Testing Tutorial Crowd Source Testing Services


Crowd Source Testing Services

Outsourced Mobile App Testing Services


GetJar

Mob4Hire

uTest

Ubertesters

App Lover

Apphance

Beta Bait

Bugcrowd

Crowd Source Testing Services

Testing Tools


Coveros

Mobica

Intertek

RapidSoft

Belatrix

Ranorex

HP

TestQuest

Utopia Solutions

Wipro

Outsourced Mobile App Testing Services

Testing Tools

Mobile Application Testing Tutorial Wrap-Up


Wrap-Up

Agenda


Wrap-Up


Agenda

Wrap-Up

Recommendations

Questions

Mobile Application Testing Tutorial Mobile Testing Books we recommend


Mobile Testing Books we recommend

Questions?


Mobile Testing Books we recommend

“Tap Into Mobile Application Testing” Kohl, Jonathan

“A Practical Guide to Testing Wireless Smartphone Applications” Harty, Julian

“Mobile Application Security” Dwivedi, Himanshu

“Mobile Developer’s Guide To The Galaxy”

http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Mobile_DevGuide_13.pdf


Questions?

Contact Information:

[email protected]

Mobile Application Testing Tutorial Notes

Copyright 2014 Coveros, Inc. All rights reserved.

Notes

Technology

Mobile Applications Testing: From Concepts to Practice