Share this eBook!

Share this eBook!

TABLE OF CONTENT

WHY YOU NEED AN IN-HOUSE TEST LAB 4

HOW TO CHOOSE DEVICES AND HOW MANY OF THEM 9

REQUIRED INFRASTRUCTURE 14

ENTERPRISE GRADE RELIABILITY 20

OPERATING YOUR IN-HOUSE TEST LAB 24

Share this eBook!

IntroductionBuilding a large-scale in-house test lab from scratch can be a big challenge. It takes time and efforts to figure out the right equipment and tools that are needed for a successful deployment. In addition, it can be trickier if something is not properly configured.

At Testdroid, we understand how daunting and time-consuming it can be. And we have experienced it ourselves when we were building Testdroid Cloud. To help customers with different needs, we have been building various in-house test labs ranging from 20 devices to several hundred devices with Testdroid Enterprise.

Here, we’ve gathered and put together a guide that discusses about why you need to build an in-house test lab, what and how many devices you need, what infrastructure is required, how to create enterprise grade reliability, and how to operate your test lab without failure. In fact, the tips we cover in this ebook are mainly influenced by our own setup at Testdroid Cloud.

With the help of this ebook, you will be able to create the best offline test lab with the right setup that maximizes your return on investment, instead of spending tons of time correcting the wrong configuration back and forth.

Share this eBook!

Chapter 1

The Reasons for an In-House Test Lab

Share this eBook!

Top Reasons for In-House Test Labs

The very first thing you should tackle is probably to make sure whether or not you need to set up one in-house test lab. Before simply jumping into its construction, you must know the reasons why in-house test labs, rather than cloud based test automation services, are the best options to fulfill your needs. The most typical reasons you should take into account are:

PREMARKET DEVICES

Surprisingly many of top tier developers work with pre-market devices. Typically you are work-ing on a significant enough app that will ship with a device and you have to make sure that your app works even when the actual device soft-ware changes from release to another. Typically you are working inside or together with a device manufacturer, carrier or one of the TOP100 app companies that do get access to pre-market de-vices from top manufacturers.

CORPORATE SECURITY POLICIES

Like many other companies, you probably have strict security policies regarding up-loading development versions of your un-released applications outside of the corpo-rate firewall. Quite often your company is also very professional with your testing and setting up your own in-house device lab makes a lot of sense to get the scalability and time to market benefits of test automation on real devices while maintaining required security levels.

Share this eBook!

AVAILABILITY AND USAGE LEVEL

Sometimes your QA teams need to run very long soak tests that take all night or there are tens of development teams in your company relying on the availability of the test results every day so the shared in-frastructure of public cloud based services does not fit well to your usage profile. If you are running a lot of tests (potentially from every code commit), your tests take a long time to execute or you are relying on having the test devices available when you need them an in-house device lab is the way to go.

SPECIAL SETUPS

Your company might also require specific hardware or environment setup to be able to fully test your soft-ware. For instance some spe-cial Bluetooth or WIFI enabled accessory needs to be physical-ly in the same location with the test devices or the devices need to be in some specific location or in some specific carrier network. Many of the environment related characteristics can be simulated but not all and therefore this sort of spe-cial cases need to be run in specific in-house labs.

As you can see the reasons for setting up in-house device labs can be as varied as the companies that are set-ting those up. An in-house lab is by far the most flexible and secure setup and it accommodates for wide spectrum of different hardware setups as well as wildly different usage profiles from 24/7 continuous testing to providing remote access to specific device prototypes.

An in-house lab is by far the

most flexible and secure setup

and it accommodates for wide

spectrum of different hardware set-

ups as well as wildly different usage

profiles from 24/7 continuous test-

ing to providing remote access to

specific device prototypes.

“ “

Share this eBook!

One essential tool for mobile application developers to use in order to automate their testing effort is continuous integration software –

for example Jenkins, which is currently the leading open source continuous integration (CI) server monitoring executions

of repeated jobs. With continuous integration and testing, you can reap the following benefits.

VERSION CONTROL AND BUILD REVISION

This rather old practice in software industry is the same with mobile application development. The quicker you can ensure your additions and regres-sions to builds work properly, the more quickly you can move forward. On the other hand, if you face problems or find any issues, all attention to fixing those bugs can be drawn there right away. Fixing a bug today costs much less than fixing the very same bug tomorrow.

BUILD AUTOMATION, INSTANT EXECUTION AND TESTING

Build Automation refers to the ability to either automat-ically trigger or build a clean version of your application.

You can take this further and make your systems built from source code just committed to version control. This typically

shortens the build time significantly, but also makes the appli-cation ready for execution and testing instantly. This is one of the

greatest benefits that continuous integration can bring to you.

The Core Benefits of Continuous Integration & Testing

Share this eBook!

ALL TYPES OF TESTING ENABLED

By automating mobile application testing you can ensure that your ap-plication behaves correctly on every mobile OS version & build (30+ with Android, 10+ with iOS), which will reduce costs and increase efficiency of the overall testing process. Not only the basic unit and GUI testing, but testing in continuous integration environment provides a way to adopt any test frameworks, methods, complementary technolo-gies and so on, for your continuous testing and development environment.

FREQUENT COMMITS, CODE CONSOLIDATION -> FAST BUILDS

Time is money. And fast builds give you that time. Fast builds allow you to receive test results (= bug report) faster.

CONTINUOUS INTEGRATION/TESTING IS THE RIGHT WAY TO GO FORWARD

By adopting continuous integration you are not only improving the quality of your code and process, but also making your development and testing effort much faster, more consistent and easier to measure. These are significant benefits to be gained out from simple process change and adopting something that already exists in processes.

By adopting continuous

integration you are not only improving the quali-

ty of your code and process, but also making your

development and testing effort much faster, more

consistent and easier to measure.““

Share this eBook!

Chapter 2

Devices forIn-HouseTest Lab

Share this eBook!

Today the number of different device variants, OS ver-sions, resolutions and form factors is really large and to make matters worse a lot of these different devices have shipped tens of millions units making each of them signif-icant in their own right. This problem is not limited to only Android anymore but also on iOS. Prob-ably you have already over 10 unique hard-ware and resolution combinations and 3-4 active OS versions, making the iOS device matrix fairly sizeable as well.

So how many devices you need to provide wide enough market coverage? For Android you take a great data published by OpenSignal and from here you can see that for in-stance if you want to cover 50% of the global market you will need about 60 devices, about 20% global market cov-erage can be reached with 12 devices and if you want to go over 50% you will need significantly more devices.

If you focus mainly on US market you can find great and fresh information published by app development company Double Encore. Ac-cording to this data with well-selected 25 Android de-vices you can cover two-thirds of the Active users and for 90% US market coverage you will need 128 devices.

For estimating the number of needed different iOS de-vices, you can take a look at the data published by Apple. In May 2014 there are 8 unique iPhone models, 7 unique

Decide the Size of Your Device Pool

Share this eBook!

iPad models and 5 generations of iPod Touches, totaling at 20 unique devices.

Given Apple’s user base you can potentially safely leave out iP-hone 2G, 3G and 3GS as well as the first three generations of iPod Touch, since the number of these devices in active use is most likely fairly small and especially these were introduced 5-7 years ago and have not even been in production for several years. This leaves 15 unique and still significantly represented iOS devices in the market.

On iOS you also need to remember that while most of the iOS us-ers do always update to the latest version, there are always some users who don’t. For instance now about nine months after the introduction of iOS 7, there are still 12% of users who have not updated to the latest version. Although this is nothing close to the situation in Android land, it is advisable that you keep some iOS devices with the second latest iOS version as well.

Share this eBook!

Select the Right Devices forMaximum Coverage

Now that we know how many devices you need for both Android and iOS, let’s take a bit more detailed look at which devices are the correct ones. Assuming your target market is North America we can again use Double Encore’s data as a good yardstick. So the list of 25 most used Android devices looks like this:

1. Samsung Galaxy S3 (17%)2. Samsung Galaxy S4 (12.3%)3. Motorola Droid Razr (3.7%)4. Samsung Galaxy Note 3 (3.7%)5. Samsung Galaxy Note 2 (3.3%)6. Asus Nexus 7 (3.2%)7. HTC One (1.9%)8. Samsung Galaxy S2 (1.8%)9. Samsung Galaxy Tab2 10.1 (1.6%)10. Droid Razr HD (1.4%)11. Droid Razr M (1.4%)12. Samsung Galaxy S2 Epic (1.3%)

13. Samsung Galaxy Tab 2 7.0 (1.3%)

14. Samsung Galaxy Note 10.1 (1.2%)

15. LG G2 (1.1%)

16. Samsung Galaxy S (<1%)

17. Motorola Droid X (<1%)

18. Motorola Droid Bionic (<1%)

19. Samsung Galaxy S5 (<1%)

20. Samsung Galaxy Nexus (<1%)

21. LG Nexus 5 (<1%)

22. Motorola Moto X (<1%)

23. Samsung Galaxy Tab 3 10.1 (<1%)

24. LG Nexus 4 (<1%)

25. HTC Thunderbolt

Share this eBook!

The Frequency of Updating YourDevice List

Once you have your lab up and running, you need to think how often you should add new devices to keep your market coverage. For iOS devices it is simple, as you only need to add new devices when Apple announces new models (twice a year). But on Android it is a bit more complex than that. A good rule of thumb is to check every quarter what are the latest blockbusters (especially any new Galaxy devices) and add those to your lab. You cannot really substitute the older ones because, shown in the TOP25 Android device list, even 2-3 year old number one devices (think Samsung Galaxy S2 and Sam-sung Galaxy S) have such a huge installed base that they will hang on the list of most used devices for years.

For Testdroid Cloud, we always expand our device pool with the latest Android and iOS device models when they are available on the market

Respectively if you want to cover the most used iOS devices, the list looks like this:

1. iPhone 5S2. iPhone 5C3. iPhone 54. iPhone 4S5. iPhone 46. iPad Mini7. iPad Mini 2nd generation

8. iPad Air9. iPad 4th generation10. iPad 3th generation11. iPad 2nd generation12. iPad 1st generation13. iPod Touch 5th generation14. iPod Touch 4th generation

Share this eBook!

Chapter 3

Required Infrastructure

Share this eBook!

After selecting your target devices, you need to take a look at other parts of a reliable infrastructure. Regardless of the technology you use to build your device lab, you will need some sort of device control node servers that will take care of managing devices and possibly executing tests. Additionally these servers can collect, process and store test results and do other test execution related tasks. At Testdroid we use either generic PC-hardware running Ubuntu Linux for controlling Android devices or Mac Minis for controlling iOS devices. Because we use the OS vendor provided meth-ods to control the devices (Android Debug Bridge (ADB) on Android and Instruments on iOS) the control nodes/servers need to have decent number of free USB ports as each device will be connected to the device servers with USB cable.

Device Control Servers

Share this eBook!

In order to have stable test execution infrastructure you need to take following things into account when selecting the

hardware for device control servers:

HARD DISKS

You need to have enough disk space for all test artifacts (logs, screenshots, videos, performance data, memory dumps, net-work dumps etc.) and make sure that there is a mechanism that either deletes the data created by previous test runs or trans-fers it to somewhere where there is enough storage space. Running out of disk space is one of the most common failure points in large-scale test automation. You also may want to consider investing in SSD disks to make sure that writing the test results to storage does not become a bottleneck under heavy load.

RAM

You need enough RAM to run several virtual machines in one physical device control server as that is the best way to ensure that each test run starts from identical starting point. Also run-ning virtual machines is required to run several iOS Instruments sessions on one Mac.

ENERGY EFFICIENCY

If you are building a large device lab, you will potentially running tens of device control serv-ers as well. So keep in mind that the more pow-er your hardware draws, the more heat those will generate and more cooling you will need.

Share this eBook!

USB Hubs

As all devices will be connected to the device control servers over USB, you need to pay attention to the quality and capabilities of your USB hardware. This is one of the most error-prone parts of the system. While there are no fixed hard limits on how many devices you can plug into one USB, our experiments show that you should not con-nect over 16 devices per USB hub. If you connect more than 16, the devices may start disconnecting randomly especially when a lot of data is being transferred either at the beginning or end of each test.

The other important reason why good quali-ty USB hubs are essential is that many devic-es (especially ones with a large display) have problems with charging if the charging cur-rent is not high enough. The USB specification mandates that the output current is limited to 500mA when the USB port is on data mode but the best USB hubs on the market allow you to programmatically switch between data and charging modes allowing you to implement au-tomatic charging cycles for devices that cannot be charged in USB data mode.

QUICKTIPS!To make sure that there are as few USB related problems as possible we prefer professional USB hub hardware by company called Cambrionix. Those are sold under several names and at least ones sold under Datamation brand are great. Just be careful when selecting those as some models are

just for charging and some models are just for iOS device.

Share this eBook!

WIFI Infrastructure

WIFI infrastructure is another very important area that is often overlooked when creat-ing large-scale test lab. You can get to about 10 maybe even 15 devices with any WIFI in-frastructure without any problems. But as the number of devices in your WIFI network adds up, so do the problems. Quite often all devices are able to connect to the WIFI without problems but the problems appear when all of those start transferring data at the same time. Most WIFI access points are not designed for this kind of throughput and you will start observing different kinds of timeouts on server responses etc. Net-work throughput on typical WIFI router is around 100-150 Mbit/s. But when you have several dozens of wireless devices trying to download at tens of Mbits/s, you can easily see where the bottleneck is.

QUICKTIPS!There are several brands on the market selling enterprise grade WIFI access points. At Testdroid, we have tried at least Aerohive, Cirrus and Cisco Meraki. Our current favorite is Cisco Mer-aki MR24 which has three separate ra-dios each capable of working on either 2.4Ghz or 5Ghz fre-quency band and each of them can transfer up to 300 Mbit/s giving whopping 900 Mbit/s from your devices to the back end servers. Another nice thing about Cisco Meraki WIFI infrastructure is their cloud based management UI that has features like throttling the speed of indi-vidual WIFI connection which can be use-ful for instance when simulating different network conditions.

Share this eBook!

Integration to Your Build ProcessThe next important consideration is how you integrate your in-house device lab to your build process. Most likely you have already a separate Continuous Integration server in place and you have to make sure that without any manual interaction your test infra-structure can be controlled by your CI server. This means that everything needs to be controllable over an API so that your CI jobs can seamlessly both trigger test runs and use the test results from your in-house test lab to trigger further actions such as deploy-ing your app to production.

Access to Your OrganizationsCentralized Access Control System

Last but not least, if you are building your in-house device lab in an enterprise setting you need to be able to connect to your company’s centralized user right management system in order to validate, grant and revoke user rights for each user in your test auto-mation system. This is just to ensure that any unauthorized users are not able to access the data and binaries that they are not supposed to and to synchronize these access rights when people leave the organization, change roles or simply do not anymore need access to such data. Most common protocols for doing this are LDAP (Lightweight Di-rectory Access Protocol) and OAuth. And your infrastructure should validate each user login through such centralized system.

Continuousintegration

applicationserver

Source Management USER INTERFACE

UIAPI Test execution

DEVICES repor-ting

Testmanage-

ment

users &rolesadmin moni-

toring

DATABASE

API

Share this eBook!

Chapter 4

Enterprise Grade Reliability

Share this eBook!

Always Run Tests on Clean Devices

In addition to hardware infrastructure, you need a reliable software con-figuration for avoiding any test failures. One of the most common causes for flaky test results is that the environment where you run your tests i.e. your test device is not exactly in the same state during each test session. This means that there may be different processes running, different applications or background services installed, different amount of free memory or different amount of storage space available. In order to avoid test flakiness, you should make sure that there is a proper cleaning cycle between every test session that uninstalls all unwanted apps, sets the device storage to the same state as it was before the test and reboots the devices so that there are no processes running from the previous test session.

Create IntelligentRetry Mechanisms Some of the test execution related failures are not actually related to the application under test but are caused by some test infrastructure related reason. The good news is that this type of failures are easy to identify and tests that have failed because of such reasons can be automatically retried as many times as needed to have a test run that is free from infrastructure related failures. The most common reasons for test infrastructure related failures are: The connection between the device control server and device fails, the connection between the device and your back end server fails, either the device or device control server runs out of storage space, device runs out of power in the middle of test session, and etc. All of these can be identified during the test session and the tests can be automatically retried so that the only remaining failures are real failures related to the application under test or the test scripts.

Share this eBook!

Check and automatically reconnect the USB and Wireless data connections

Losing either the USB or wireless data connection at some point is unfortunately very common on both Android and iOS devices. To make matters worse quite often, the

device reports that it has live connection over USB, WIFI or Wireless data but no data is moving. You need to be able to automatically verify that both

USB connection and WIFI connection are actually trans-ferring data and if that’s not the case, you need to automatically disconnect & reconnect the connec-

tions to get them back up again.

Automate All Configuration Changes

When you have hundreds of devices and tens of device control servers in your test lab, you cannot (and should not) really make any changes to settings or configurations man-ually because the risk of not doing the changes identically on all devices or servers is simply too high. We automate everything we can by using Opsworks Chef and we have implemented our on-device services that take care of changing the device settings etc. Even if we have been very systematic with this, every now and then there are problems because some small setting was done by hand and it was not replicated identically on all devices or servers. This is one area where you cannot afford to cut corners if your goal is 99.99% reliability of your in-house test lab.

Share this eBook!

Use Professional Monitoring

The last but not by any means least important way to ensure the robustness of your in-house test lab is to set up a professional monitoring for all aspects of your test lab infrastructure. The most important areas to monitor are:

DISK SPACES

– Automated tests create a lot of data in the form of logs, screenshots, videos, memory dumps, and network dumps. When you are running tens or even hundreds of devices sometimes 24/7, you may run out of disk space just out of the blue.

NETWORK CONNECTIONS, LATENCIES, PACKET LOSS RATES

– These have an impact on everything in your test lab and once you start troubleshoot-ing the strangest and most randomly appearing problems, most often the root cause is networking related.

POWER

– Disruptions on power supply (even very short ones) can cause very strange problems when the system has lots of moving parts and all of them are dependent on steady power supply. Use UPS backed power whenever you can and also make sure that your servers close down and come up in a controlled way when longer power disruption happens.

You can use any generic monitoring software for this. For Testdroid Cloud we are cur-rently using AWS monitoring, Loggly and New Relic to handle our monitoring.

Share this eBook!

Chapter 5

OperatingIn-HouseTest Lab

Share this eBook!

Once you’ve set up

hardware and software infrastruc-

ture, operating your device lab is the

next thing you need to take care of. The

first aspect of operations is to save all op-

eration logs. Operation teams should doc-

ument all bits and pieces that happen to

the environment, devices and test runs.

By doing great homework with docs,

teams will be able to maintain and

upgrade the environment when

needed easily.

Everything is about power nowadays. The power and

its consumption is the most import-

ant thing to contribute the temperature

of the system. Even smaller changes in

temperature can cause that lots of devices

malfunction (This should never be under-

estimated!) and go offline. When you set

up the device pool, make sure you leave

enough space for those devices to “breath”.

You naturally need to make

sure your devices get enough power

to perform. Once those devices are 24/7

plugged in, maybe surprisingly, those devic-

es need to be constantly monitored if they get

enough power. The battery as hardware compo-

nent doesn’t always allow full charging and easily

gives up when it is constantly kept in charging

mode. Make sure you monitor those devices!

Operational Log

Keep Your Devices Cool

Monitor the

Charging of Devices

Share this eBook!

Like with any software de-

velopment, staging environment can

help you a lot verify even smaller chang-

es in critical environment. In staging you

check that changes work as expected and

validate all new configurations in that con-

text. It is also a good idea to roll in new

hardware (even devices) through

staging if devices are ‘exotic’.

It is always a good idea to use the same hardware when setting up or configuring a larger in-

house lab. This means that if you use certain device setup for servers, it makes your operational team’s life easier. The very same hardware is constantly used across the environment. Those devices naturally differ, but then again you can

use the same ways to connect, con-trol and maintain those.

Find the maximum number of

devices that can be connected to each

server. Generally, and from our experience,

the typical number of devices connected to one

server for robustness is 15-20. Sometime connect-

ing devices in different USB slots in the same server

may give much better stability.

v So it definitely pays off to

find out what USB ports work

the best. Also, there is one

minor but very important

trick in case of huge setups:

Use color-coded USB ca-

bles and you know right

away in what devices

they connect with.

Use Staging Environment

Standardize Your Test Lab Hardware

USB Tips

Share this eBook!

ConclusionBuilding a large-scale in-house test lab is not a one- or two-day work. It needs all appro-priate devices, infrastructure and software as basic ‘weapon’ for your on-premise test-ing environment. More than that, you also need to make sure that your test lab is well developed and reliable enough to support your team running tests 24/7 with no flaws.

At Testdroid, we ourselves have been hosting an over 300-device cloud-based test lab – Testdroid Cloud for the past years. All of the tips and tricks mentioned above are exactly what we have experienced when hosting Testdroid Cloud. When you decide and start to build your own in-house test lab, it is of great significance to take into account all of the aspects (hardware, software and operations) mentioned above.

The purpose of this ebook is to give you a complete guide for creating offline test lab and help you inspect potentially weak parts in your current in-house test lab settings if you already have one in place. In addition, with plenty of experiences on hosting a large-scale test lab, we are also able to deliver to you such a testing solution –Testdroid Enterprise. Building your in-house test lab with Testdroid Enterprise will ease you at any point and you can just sit back and relax.

Share this eBook!

Build Your In-House Test-Lab with

interested?

Learn More About Testdroid Enterprise HERE

Leave us your MESSAGE and we will get back to you soon.