Scripted Exploritory Automated Testing

7/27/2019 Scripted Exploritory Automated Testing

1/16


2/162TestComplete

by SMARTBEAR

Combining Your Scripted,

Exploratory and Automated

Mobile Testing

Contents

Introduction ............................................................................................3

Why Testing Mobile Apps Is Different Than Any Other Platform............4

Rule #1 - Make sure your mobile app does what the user is

expecting it to do. ............................................................................................. 5

Rule #2 - Concentrate on the devices where the majority of

your target market will be and camp out there. ................................................. 5

Asking the Right Questions ...................................................................6

GPS .................................................................................................................. 6Data .................................................................................................................. 6

Gestures ........................................................................................................... 7

GUI.................................................................................................................... 7

Processing Power ............................................................................................. 8

Memory ............................................................................................................. 8

Battery............................................................................................................... 8

Fragmentation of devices.................................................................................. 8

Scripted, Exploratory and Automated Testing in the Mobile Space .......9Automated Testing .......................................................................................... 10

Scripted Testing .............................................................................................. 12

Exploratory Testing ......................................................................................... 13

The Mobile Testing Balancing Act ........................................................14

Conclusion ...........................................................................................15
http://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.smartbear.com/testcomplete


3/163TestComplete

by SMARTBEAR

Introduction

SoLoMo. You know the mantra. Social, local, mobile. Its the way the

Internet is heading. Its the direction that the world is heading. We have

been hearing it for years.

The difference with that clarion cry is that we are way past the hype

phase. No longer do we wonder if this is the Year of Mobile.

Mobile has arrived and it is past the point of novelty. Whether it is busi-

ness or personal use, mobile is getting more and more ingrained in all

aspects of our lives each and every day. As a result, users are more

savvy than ever before and their patience with poorly designed and

delivered mobile apps is very limited.

This shift in how the world accesses information and uses data has

forced the mobile world to grow up faster than it may have been ready

for. No longer can an app be almost there and be released into the

market. Today, if your mobile app is even the slightest bit buggy or

crashes from time to time or doesnt work on as many platforms as pos-

sible, you stand to lose big.

Its for this reason alone that the practice of mobile app testing is a

critical step in the life cycle of an app. Very few companies are allowed

any leeway in how their apps perform. A perfect example of this is

Facebook. Its hard to get more ubiquitous than Facebook, yet when thecompany went public the doubts about their mobile capabilities ham-

pered the stocks performance.

Now, if Facebooks mobile presence doesnt meet the high expectations

of the marketplace the entire brand suffers.

Granted, not all businesses face being put under this kind of harshinterrogation, but that day is not far off if we continue to progress at the

current pace. A mobile app that has even the smallest imperfection will

be quickly pushed aside in search for another to serve an end users

needs.


4/164TestComplete

by SMARTBEAR

Why Testing Mobile Apps Is Different Than Any Other Platform

That said, it is critical to address the testing aspect of your app develop-

ment with the utmost concern. In this eBook, we will take a look at the

different ways mobile app testers need to test.. Scripted testing alone

will not be enough. Exploratory testing alone will not be enough. Auto-

mated testing alone will not be enough.

Instead, todays testers of mobile apps will be required to nd the blend

of all three types of testing in order to feel condent that their apps are

ready for a rigorous marketplace.

We will explore the unique elements of the mobile world and how soft-

ware testers will need to nd the right mix of testing so that, when their

app gets to the end user, they wont immediately be cast aside.

Mobile is like the wild west of the digital era. While in some ways its not

much different than other areas, what separates it is that the wide range

and array of users and available devices create a myriad of combina-

tions. Because of this never ending series of combinations of user

types, platforms and other variables, it is likely to be an impossibility to

test for everything.

Accepting this fact from the onset will be important just so any work thathas been done can actually see the light of day in the marketplace.

To start, well rst take a look at a few of the elements that make the mo-

bile environment so challenging especially from a testing perspective,

and some general rules to help you deal with them.


5/165TestComplete

by SMARTBEAR

Before you go through the hard work of

putting an app through its paces with a

variety of tests, you better make sure it is

accomplishing the desired business task.

Rule #1 - Make sure your mobile app does what the user is

expecting it to do.

Before even getting near the intricacies and complexities of the mobile

environment, it has to be determined whether the app itself simply ac-complishes what it was originally intended to do.

Why is this more difcult in the mobile world? Its quite simple as to why,

but its a whole other issue as to what to do about it.

The why its hard is due to how

end users behave in the mobile

space. It is a very new area of

exploration for everyone and

almost everything that is seen is

some variation of rst use case,

which means that there is little or no historical data or experience to fall

back on for validation.

With this in mind, before you go through the hard work of putting an app

through its paces with a variety of tests, you better make sure it is ac-

complishing the desired business task.

Rule #2 - Concentrate on the devices where the majority of your

target market will be and camp out there.

There may not be a more difcult hurdle to overcome in the mobile app

world than the issue of device acceptance.

iOS, Android, Windows, and BlackBerry are difcult enough to test for

on their own. Then, get into the various versions of each mobile OS and

you can quickly feel like the better alternative is to simply scream and

hope it goes away. But add on to this the many and various operating

systems of Android, and the mobile landscape begins to seem innite

so it is rarely enough to only develop for iOS anymore. Certainly the

demographics of the iOS mobile device user are more attractive than

any other, but even that metric is becoming less dependable with the


6/166TestComplete

by SMARTBEAR

advances that Samsung and HTC have made with their Android models.

Many kids these days are walking around with a Galaxy S4 and not an

iPhone 5S. Since its difcult to be all things to all people, if you think

that most of your users are going to be a combination of iOS and the

most popular avor of Android / device, you may need to be developing

and testing just for those targets.

We are going to make a quick run through a variety of elements that will

be critical to consider as you test your mobile apps. Take note of some

of the questions you should be asking of each element.

GPS

How important are geolocation elements to your app? The merefact that you are working in the mobile environment implies that

you know the location of the user will be a critical piece of your

mobile app puzzle.

Does your app play with the major map platforms?

Will your app integrate properly with Google Maps? Apple

Maps?

Who do you use? Whom do you partner with?

Who gives the best data and user experience?

How are the various APIs?

What are the limitations and costs?

Data

How much data? What data? What are your privacy policies?

What are you accessing on their device?

What is your end users tolerance for privacy encroachment?

Asking the Right Questions


7/167TestComplete

by SMARTBEAR

Network Availability How does your app perform on various carrier networks as

opposed to Wi?

Will your app be a data hog that causes users to watch their

bandwidth usage as a result?

Is the performance on Wi so markedly improved as to

discourage the end user from using the app in any other

environment?

Gestures

Does your app give the user gesture options that are confusingor that create conict with other apps?

If they are not willing to adapt to the gestures you have

incorporated in your app is the app navigable without them?

GUI

Lets get down to brass tacks here. Is your app simply butt-ugly? Does your app have a command line feel to it or is it appealing

to even the most non-technical of users (who will be a vast

majority of most app audiences anyway)?

Portrait and Landscape Views

What happens to your app when your end user turns his or herdevice on its side or on its head?

Does it respond quickly?

Is it responsive at all?

Are there any elds that suddenly become impossible to

navigate because of the limited space due to the keyboard

expanding for use?


8/168TestComplete

by SMARTBEAR

Processing Power

The last thing you want your app to be associated with ismaking your users device so slow because it is a tremendous

resource hog. If within those rst few hours of downloading yourapp your customer notices that nothing is working as well on

their device as it did before they downloaded your app guess

what happens.

Memory

Lets be honest. Most mobile device users are pushing theirmemory limits to the nth degree on any given day.

How will your app impact device memory?

Battery

What impact will your app have on the battery life of a mobiledevice? If an end user downloads your app then suddenly sees

their battery life fall through the oor you may be viewed as a

troublemaker. Not good.

Fragmentation of devices

We alluded to this earlier, but its the simple truth about themobile space. There are an ever growing list of devices, screen

sizes, form factors, etc. that make up the mobile marketplace.

Its not cheap to make design alterations for every make and

model. Also, once you make accommodations for one you may

end up breaking something with another.

Will you need to pick and choose which devices are going to

represent the largest portion of your target market then go from

there? This is a critical decision that must be taken seriously.


9/169TestComplete

by SMARTBEAR

There is one more important element in this process. The element is

time. Everyone gets the same amount of it every day and it is a re-

source that cannot be grown or added to. The results, as it relates to

your mobile app development and testing, is that you have to place time

limits and parameters on the testing environment. Therefor, it becomes

even more critical to use the right type and combination of testing meth-

odologies to ensure that your mobile app is ready for its introduction to

the marketplace.

Now that we have explored the various elements of the mobile environ-

ment in at least a cursory fashion, its time to look at what types of app

testing will help you put your best mobile app foot forward.

To set the stage lets take to heart this quote from a Mobile Marketer

article

Mobile is different, and when companies dont treat mobile ap-

plications as a key part of their business by investing in a great

user experience, consumers are disappointed in the applications,

said Peter Galvin, senior vice president of marketing at Soasta,

Mountain View, CA.

Our research has shown that many mobile applications are not

properly tested before they are delivered to consumers, he said.

In the physical world marketers test their advertising, store con-

cepts and messaging to ensure they resonate with consumers.

If that is the case, then why is testing in the mobile space not priori-

tized properly by many? Simply put, testing effectively can be a daunt-

ing task. Effective app testing in much more stable and less variable

environments is hard enough, but add the reality of testing in a world ofconstant change and it becomes a challenge that has to be faced with

appropriate expectations.

Scripted, Exploratory and Automated Testing in the Mobile Space


10/1610TestComplete

by SMARTBEAR

What are those expectations? Well, with a fair amount of planning, a

concentrated effort and a little luck, one should expect that they could

cover most of the bases when testing mobile apps.

What is probably unrealistic is the expectation of testing for every situa-

tion, under every potential scenario and uncovering every little ailment

of an app before it gets to market. Especially since the uid nature of

the mobile space changes the rules so often, you can never take care of

every angle.

That doesnt mean that it cant get close though. The best way to get

result that are going to set your efforts apart from your competitions is

by utilizing various testing methods in the right mix.

As with any software or app testing, the process is as much art as it is

science. Flexibility will win the day, so loosen your collar as we delve

into the testing options that are available and help you decide what is

the right formula for your needs.

Automated Testing

We are going to start off this conversation about automated testing with

this caveat. Automated testing in the mobile environment should not

be an end unto itself. In other words, the majority of mobile app testing

should occur in the manual (scripted) and exploratory space, while auto-

mated testing serves as a support system for those two approaches.

Now that we have established that, we can look at automated testing of

mobile apps in the proper light. Just the mere mention of automation in

the mobile space can make many queasy because automation implies

process. Process implies repetition. Repetition implies that the mobile

experience is something that is rote and repeatable and doesnt neces-

sarily require human intelligence or creativity. Well, its not really true,

especially given the strategy required in understanding how and when

to use automated testing.


11/1611TestComplete

by SMARTBEAR

So, what can automated tests actually cover? The best-case scenario is

to test aspects that require no user interaction. That might include:

Did this API function return the correct value? Did this function calculate the correct result? Did we successfully send a le over the network with the correct

checksum?

Did this callback function actually re?(Source: Corona Labs)

The automated piece of mobile app testing is more about the backend

functionality of the app. In other words, if something were to be triggered

by a user (regardless of how that trigger was activated) how does the

app respond?

When looking into automated testing for a mobile app, it is critical to

remember that anything regarding UI and UX requires some form ofhuman interaction in order to truly see what happens. Relegating this

interaction of end user and app to an automated process exposes the

test process to the risk of missing all the nuance of having a real person

generate an action in the app.

Automation will likely only investigate a few options as to how something

may occur in the app.

Its these limitations in the automated testing process that keep this form

of testing in a supporting role of the other testing methods. It is impor-

tant to realize the limitations of automated testing and making sure it

stays within those limits.

Now that we have established where automated testing plays best inthe mobile space, lets turn our attention to the methods that are going

to produce the most important feedback.
http://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.smartbear.com/testcompletehttp://www.coronalabs.com/blog/2011/08/08/automated-testing-on-mobile-devices-part1/http://www.coronalabs.com/blog/2011/08/08/automated-testing-on-mobile-devices-part1/http://www.smartbear.com/testcomplete


12/1612TestComplete

by SMARTBEAR

Scripted Testing

In manual testing, scripted tests are the next step up from automation.

There is much more exibility in the testing, but the fact that it is scripted

implies limitations. Thats ne. Developers, business stakeholders,and other critical input sources have areas of concern that need to be

vettedand scripted tests allow for those areas of concern to be ad-

dressed.

Using Personas

Developing user personas (different proles of those who the app is

really being built for) is a key component in increasing the effectiveness

of a mobile app. Hopefully, the various personas that make up the target

market of any mobile app were considered in the design and develop-

ment of the app from the onset.

There can be scripted testing that is built around personas. This will help

to uncover potential issues that a specic subset of customers might

encounter based on:

Who they are

How they function

What they like

Why they would use the app, etc.

The number of personas tested will depend on how many there are for a

particular app and what kind of resources are available (read: time and

money) to test as many variations of a theme as possible. The use of

scripted tests from that personas point of view can do well to uncover

potential issues.


13/1613TestComplete

by SMARTBEAR

Exploratory Testing

Its at this point, however, that we quickly reach the limitations of

scripted testing. No matter how much an end user ts the persona, a

scripted test will never allow for the inevitable variations on a theme. In

other words, human beings rarely, if ever, truly follow a script.

With that said, it becomes imperative for those conducting the tests of

an app to start to go off script and into an area that might more ac-

curately represent reality. The means to get there is through exploratory

testing.

Exploratory testing is simply a more free owing methodology of testingthat, through experimentation, builds on itself and allows testers to get

into real world situations that may never be accounted for in automated

or scripted tests.

This mindset becomes even more valuable in the testing of mobile

apps because the situations an end user nds him or herself in change

constantly due to the fact that they are mobile. They are on the move.

Performing a task on a mobile device while sitting at a desk may look

very different than performing the

same task while walking down a

crowded sidewalk. Or at a trafc

light.

Exploratory testing ts perfectly

with mobile app development

because the very nature of many

mobile apps is exploratory in nature. People are doing things on the

move and are often in discovery mode themselves. Each time the app

is openeven though the function that is being called upon in the app is

the samethe reason for its use and its ultimate utility could be differentalmost every time.

In other words, in the mobile world things that might look like they are

repeated events are only repeated at a surface level. The intent and

Exploratory testing ts perfectly with mo-

bile app development because the very

nature of many mobile apps is explor-

atory in nature. People are doing things

on the move and are often in discovery

mode themselves.


14/1614TestComplete

by SMARTBEAR

ultimate goal of the user might be completely different each time that

particular function is called upon, which may create a different interac-

tion with the same function based on the external variables the end user

is experiencing.

To test how an app responds to such a dynamic environment requires

freedom. Exploratory testing that builds on previous experience or on

a new experience can allow the test to ex and move in directions that

best mimic that of a real life end user. As a result, the real discovery

regarding the strength and weakness of an app can occur.

It is critical to make sure that in this exploratory process there is ac-

curate data collection. This is a difculty of the exploratory model that is

compounded when used in the mobile space. How you require testers

to report their ndings could mean the difference between a jumble of

interesting tidbits and a result-set that impacts the performance of an

app.

Now that we have looked at the three areas of testing, their limitations

and their place individually, it is critical to make sure that we use the

right blend of all the available techniques to produce the strongest pos-

sible testing environment and the ensuing results.

To accomplish this takes much more than just crossed ngers and a

few Hail Marys. There needs to be real intentionality in the blending of

different testing options so that the entire scope of the mobile app can

be tested thoroughly enough to give greater go-to-market condence to

the business stakeholders. How this is accomplished is strategic plan-

ning on the upfront along with the willingness to be truly agile during the

process. This agility and sensitivity to the very nature of mobile comput-

ing will make the difference in how your tests perform.

The Mobile Testing Balancing Act


15/1615TestComplete

by SMARTBEAR

Obviously, this kind of balancing act is best performed by those with

experience in these matters. If they are not available, or affordable, this

might be the time to consider outside help in the form of products andservices designed to get you most of the way there.

In the end, testing mobile apps is something that needs special attention

and a renewed approach from traditional software testing. Over time, it

is likely that new methodologies will be developed that are more appro-priate for the mobile space.

Right now though, at this early stage in the game, we need to work with

the existing testing types like automated and scripted testing but allow

them to be avored by a sense of adventure, so to speak.

The mobile space is new and exciting. It is also relatively unmapped ascompared to other places in the digital geography. With that in mind, ef-

fective mobile app testing will require a spirit of true exploration.

Testing mobile applications so that you may be able to create the best

possible experience for your end users is a true work in progress.

As you move through it, mixing and matching testing procedures and

techniques will be the key to giving your efforts the best chance to truly

make an impact in the mobile world.

Are you ready? Are you exible? Are you adventurous?

Conclusion


16/16

About SmartBear Software

More than one million developers, testers and operations profession-

als use SmartBear tools to ensure the quality and performance of their

APIs, desktop, mobile, Web and cloud-based applications. SmartBear

products are easy to use and deploy, are affordable and available for

trial at the website. Learn more about the companys award-winning

tools or join the active user community at http://www.smartbear.com, on

Facebook or follow us on Twitter@smartbearand Google+.

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915

+1 978.236.7900 www.smartbear.com 2013 by SmartBear Software, Inc. Specications subject to change.
http://www.smartbear.com/http://www.smartbear.com/https://www.facebook.com/smartbearhttps://twitter.com/SmartBearhttps://plus.google.com/112549653237106886127/postshttp://www2.smartbear.com/tc-trial-sb-lazy-superhero.htmlhttp://www.smartbear.com/https://plus.google.com/112549653237106886127/postshttps://twitter.com/SmartBearhttps://www.facebook.com/smartbearhttp://www.smartbear.com/

Documents

Scripted Exploritory Automated Testing