Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
AboutThiseBook
ePUBisanopen,industry-standardformatforeBooks.However,supportofePUBanditsmanyfeaturesvariesacrossreadingdevicesandapplications.Useyourdeviceorappsettingstocustomizethepresentationtoyourliking.Settingsthatyoucancustomizeoftenincludefont,fontsize,singleordoublecolumn,landscapeorportraitmode,andfiguresthatyoucanclickortaptoenlarge.Foradditionalinformationaboutthesettingsandfeaturesonyourreadingdeviceorapp,visitthedevicemanufacturer’sWebsite.Manytitlesincludeprogrammingcodeorconfigurationexamples.To
optimizethepresentationoftheseelements,viewtheeBookinsingle-column,landscapemodeandadjustthefontsizetothesmallestsetting.Inadditiontopresentingcodeandconfigurationsinthereflowabletextformat,wehaveincludedimagesofthecodethatmimicthepresentationfoundintheprintbook;therefore,wherethereflowableformatmaycompromisethepresentationofthecodelisting,youwillseea“Clickheretoviewcodeimage”link.Clickthelinktoviewtheprint-fidelitycodeimage.Toreturntothepreviouspageviewed,clicktheBackbuttononyourdeviceorapp.
Hands-OnMobileAppTesting
AGuideforMobileTestersandAnyoneInvolved
intheMobileAppBusiness
DanielKnott
NewYork•Boston•Indianapolis•SanFranciscoToronto•Montreal•London•Munich•Paris•MadridCapetown•Sydney•Tokyo•Singapore•MexicoCity
Manyofthedesignationsusedbymanufacturersandsellerstodistinguishtheirproductsareclaimedastrademarks.Wherethosedesignationsappearinthisbook,andthepublisherwasawareofatrademarkclaim,thedesignationshavebeenprintedwithinitialcapitallettersorinallcapitals.
Theauthorandpublisherhavetakencareinthepreparationofthisbook,butmakenoexpressedorimpliedwarrantyofanykindandassumenoresponsibilityforerrorsoromissions.Noliabilityisassumedforincidentalorconsequentialdamagesinconnectionwithorarisingoutoftheuseoftheinformationorprogramscontainedherein.
Forinformationaboutbuyingthistitleinbulkquantities,orforspecialsalesopportunities(whichmayincludeelectronicversions;customcoverdesigns;andcontentparticulartoyourbusiness,traininggoals,marketingfocus,orbrandinginterests),pleasecontactourcorporatesalesdepartmentatcorpsales@pearsoned.comor(800)382-3419.
Forgovernmentsalesinquiries,[email protected].
ForquestionsaboutsalesoutsidetheU.S.,[email protected].
VisitusontheWeb:informit.com/aw
LibraryofCongressCataloging-in-PublicationDataKnott,Daniel.Hands-onmobileapptesting:aguideformobiletestersandanyoneinvolvedinthemobileappbusiness/DanielKnott.pagescmIncludesindex.ISBN978-0-13-419171-3(pbk.:alk.paper)—ISBN0-13-419171-41.Mobileapps—Testing.I.Title.QA76.76.T48K642015006.3—dc232015009688
Copyright©2015PearsonEducation,Inc.
Allrightsreserved.PrintedintheUnitedStatesofAmerica.Thispublicationisprotectedbycopyright,andpermissionmustbeobtainedfromthepublisher
priortoanyprohibitedreproduction,storageinaretrievalsystem,ortransmissioninanyformorbyanymeans,electronic,mechanical,photocopying,recording,orlikewise.Toobtainpermissiontousematerialfromthiswork,pleasesubmitawrittenrequesttoPearsonEducation,Inc.,PermissionsDepartment,200OldTappanRoad,OldTappan,NewJersey07675,oryoumayfaxyourrequestto(201)236-3290.
ISBN-13:978-0-13-419171-3ISBN-10:0-13-419171-4
TextprintedintheUnitedStatesonrecycledpaperatRRDonnelleyinCrawfordsville,Indiana.Firstprinting,May2015
Formywife,Sarah.Thankyouverymuchforyoursupportand
encouragementwhileIwrotethisbook.
Contents
Preface
Acknowledgments
AbouttheAuthor
Chapter1:What’sSpecialaboutMobileTesting?UserExpectationsMobilityandDataNetworksMobileDevicesMobileReleaseCyclesMobileTestingIsSoftwareTestingSummary
Chapter2:IntroductiontoMobileDevicesandAppsOverviewofMobileNetworksMobileDevicesandtheYear2007TheBigTwoWhatTypesofAppsExist?NativeAppsHybridAppsWebApps
BusinessModelsofMobileAppsFreemiumPaidTransactionChoosingaBusinessModel
AppStoresSummary
Chapter3:ChallengesinMobileTestingTheCustomer
CustomerSummaryMobilePlatformsandFragmentationMobileDeviceGroupsMobileDeviceLabs
SensorsandInterfacesAmbientLightSensorProximitySensorAccelerationSensorGyroscopeSensorMagneticSensorPressure,Temperature,andHumiditySensorsLocationSensorTouchlessSensorSensorsSummaryTouchscreenMicrophonesCamera
SystemAppsInternationalization(I18n)andLocalization(L10n)MobileBrowsersSummary
Chapter4:HowtoTestMobileAppsEmulator,Simulator,orRealDevice?ManualTestingversusAutomatedTesting“Traditional”TestingMobile-SpecificTestingMobileFunctionalTestingMobileUsabilityTestingAccessibilityTestingBatteryUsageTestingStressandInterruptTesting
PerformanceTestingStandbyTestingInstallationTestingUpdateTestingDatabaseTestingLocalStorageTestingSecurityTestingPlatformGuidelineTestingConformanceTestingCheckingtheLogFilesBeCreative!
Checklists,Mnemonics,andMindMapsChecklistsMnemonicsMindMaps
HowtoFileMobileBugsBugIDDescriptionStepstoReproduceExpectedResultActualResultWork-aroundReproducibleOperatingSystem,MobilePlatform,andMobileDeviceMobile-Device-SpecificInformationBrowserVersionSoftwareBuildVersionNetworkConditionandEnvironmentLanguageTestDataSeverity
BugCategoryScreenshotorVideoLogFilesTesterWhoFoundtheBugThreeMorePoints
AppQualityAllianceSummary
Chapter5:MobileTestAutomationandToolsTheFlippedTestingPyramidTheMobileTestPyramid
DifferentTypesofTestAutomationToolsImageRecognitionCoordinate-BasedRecognitionOCR/TextRecognitionNativeObjectRecognitionCaptureandReplayToolTypeRecommendation
WhatShouldBeAutomated?Emulator,Simulator,orRealDevice?Emulator/SimulatorProsEmulator/SimulatorConsRealDeviceProsRealDeviceConsWheretoAutomate?
HowtoSelecttheRightMobileTestAutomationToolSelectionCriteriaforaTestAutomationTool
CurrentStateofToolsAndroidToolsiOSToolsMobileTestAutomationToolsSummary
ContinuousIntegrationSystem
BetaDistributionToolsSummary
Chapter6:AdditionalMobileTestingMethodsCrowdTestingPrivateCrowdTesting
MobileCloudTestingPrivateCloud
Cloud-BasedTestAutomationSummary
Chapter7:MobileTestandLaunchStrategiesMobileTestStrategyDefineRequirementsTestingScopeDefineTestLevelsandTestTechniquesTestDataSelectTargetDevicesandTestEnvironmentManualandin-the-WildTestingMobileChecklistsandToursTestAutomationProductRisks
MobileLaunchStrategyPre-Launch—ChecktheReleaseMaterialPost-Release—WhatHappensafteranAppLaunch?CommunitySupportReviewsCrashReportsTrackingandStatistics
Summary
Chapter8:ImportantSkillsforMobileTestersSkillSetofaMobileTester
CommunicationCuriosityCriticalThinkingTenacityConstantLearnerCreativityCustomerFocusProgrammingandTechnicalSkills
HowtoImproveYourMobileTestingSkillsLearnfromOtherAppsCrashesonSpecificDevicesObserveTakePartinCompetitionsandTestCyclesTheMobileCommunityandtheMobileWorldValuableSources
Summary
Chapter9:What’sNext?AndFinalThoughtsInternetofThingsConnectedHomeConnectedCar
WearablesSmartWatchesandFitnessWristbandsSmartGlasses
HealthAppsFinalThoughtsFiveKeySuccessFactors
Summary
Index
Preface
Mobilephoneshavebeenaroundsincethemiddleofthe1970s.Thedeviceshaveofcoursechangedprofoundlysincethen,butthebiggestchangecamein2007whenApplepresenteditsfirstiPhone.Fromthatmomenton,themobilesmartphonemarkethasknownonlyonedirection—UP!Eightyearslater,touchdevicessuchassmartphonesandtabletshavebecomeubiquitous.Morethantwomillionappsareavailablefordownloadinthestoresofthebiggestvendors,andthisnumberisstillrising.1Thereareappsforeveryaspectofourlives,rangingfromphotosandmusic,toofficeapplicationsandgames,andontofitnessandhealth.Butwhataboutthequalityofthoseapps?Aretheyreliable,trustworthy,easytouse,welldeveloped,andtested?
1.iOSStorenumbers,www.engadget.com/2014/06/02/apples-wwdc-2014-in-numbers-40-million-on-mavericks-and-more/;AndroidPlayStorenumbers,www.appbrain.com/stats/number-of-android-apps.NumbersarefromJune2014.
Thisbookisapracticalguidetomobiletestingforanyonewhoworksinthemobilebusiness,butitisespeciallyaimedatmobiletesters.
WhyIWroteThisBookItallstartedin2010whenIhadtheopportunitytoworkonmyfirstmobileproject.ThemobileteamIworkedinwasresponsiblefordevelopingamobileWebapp,anativeAndroidapp,andanativeiOSapp.Thiswasthecompany’sfirstmobileprojectandacompletelynewtestingenvironmentforthequalityassurancedepartment.Togetherwithacolleague,Ihadthechancetobuildamobiletestingstrategyfromscratch.Weevaluatedseveraltestautomationtoolstoseewhichonefitbestinoursoftwaredevelopmentlifecycle.Atthattime,mobiletestingtoolswerefewandfarbetween,andataveryearlydevelopmentstage.Wethentriedseveraltestingapproachesandtools.Ofcoursewefailedwithsomeofthem,butintheendthewholeteam,thecompany,andourcustomerswerehappy.AnotherreasonwhyIwrotethisbookwasbecauseofmyblog,
www.adventuresinqa.com.Istartedbloggingin2011aftergivingapresentationattheAgileTestingDaysinPotsdam,Germany.Thiswasmyfirsttalkatamajortestingconference,andIwastheonlyspeakerontheagendawhospokeaboutmobiletesting.AftermypresentationIwasverybusyfortherestoftheconferenceasalotofpeopleapproachedmetoaskaboutmobiletesting,theapproachesIuse,whatkindoftoolsIuse,andsoforth.Thehugeinterestinand
thelackofknowledgeaboutmobiletestingconvincedmetostartwritingablog.Thegoalwastosharemyknowledgeofmobiletestingandtoexchangeviewsandideaswithothermobiletesters,whilealsoimprovingmywrittenEnglishskills.SofarI’vewrittenabout90postscoveringmobileappsandtesting,andIneverexpectedsomanypeoplefromaroundtheworldtotakeaninterestinmyblog.ThefeedbackI’vegottensofarhasbeengreat,anditconvincedmetotakethenextstep.Thatstepiswhatyou’rereading:abookaboutmobiletestingthatcapturesmy
practicalexperienceandknowledgeforanyoneinvolvedinthemobilebusiness.Ihopeyouenjoyreadingthisbookandcanlearnsomethingnewaboutthemobiletestingbusiness.
WhoShouldReadThisBook?Thisbookisaimedatanyonewhoisinterestedinmobileappsandmobiletesting,rangingfromjuniortoexpertmobiletesterswhoarealreadyinvolvedinmobiledevelopmentteams.Thisbookisalsoidealforsoftwaretestmanagerswhoneedtomanagemobile
testingteamsortoselectamobileteststrategy.It’salsogreatforsoftwaretesterswhoarenewtothistopicandwanttoswitchtomobiletechnologies.Softwaredeveloperswhowanttoknowmoreaboutmobiletestingandtesting
theirmobileappshavealsocometotherightplace.Thisbookisalsointendedforproductmanagerslookingtogainfurther
insightsintothechallengingjobofmobiletesting.
TopicsCoveredinThisBookThisbookcontainsthefollowingchapters:
•Chapter1:What’sSpecialaboutMobileTesting?Thefirstchapterfocusesonthespecialcharacteristicsofmobiletesting.Itprovidesanintroductiontomobileuserexpectations,mobiledatanetworks,mobiledevices,andwhymobiletestingissoftwaretesting.•Chapter2:IntroductiontoMobileDevicesandAppsChapter2introducesmobiledatanetworksandwhatisimportanttoknowaboutthem.Thechapteralsodescribesthemobiledeviceevolutionfromdumbphonestothecurrentsmartphones.Furthermore,thischapterintroducesthedifferenttypesofappsandpossibleappbusinessmodels.•Chapter3:ChallengesinMobileTestingChapter3isallaboutmobile
testingchallengesandhowtohandlethem.Therearechallengessuchasthecustomer,devicefragmentation,sensorsandinterfaces,systemapps,andmobilebrowsers.Eachsectionofthechapterprovidessolutionsforhandlingthosechallengesinyourdailybusinessasamobiletester.•Chapter4:HowtoTestMobileAppsChapter4isallabouthowtotestmobileapplications.Thischapterexplainsthedifferencesamongemulators,simulators,andrealdevices.Italsoexplainswheretotestamobileapp.Furthermore,thischapterprovidesseveralfunctionalandnonfunctionalapproachestotestingamobileapp.Inaddition,thischapterpresentsmobiletestingmindmaps,mnemonics,andcheckliststoimproveyourmobiletestingefforts.•Chapter5:MobileTestAutomationandToolsChapter5coversthetopicofmobiletestautomation,whichisaveryimportantone.Thechapterintroducesthedifferenttestautomationtooltypesandapproaches.Itprovidesideasforhowtoselecttherightmobiletestautomationtoolforyourtestenvironment.Additionally,thechapterprovidesanoverviewofthecurrentstateofmobiletestautomationtoolsforAndroidandiOS.•Chapter6:AdditionalMobileTestingMethodsChapter6providesanoverviewofadditionalmobiletestingmethodssuchascrowdandcloudtesting.Bothmethodsareexplained,includingtheprosandconsandwhereitmakessensetousetheminyourmobiletestingapproach.•Chapter7:MobileTestandLaunchStrategiesChapter7dealswiththetopicofmobiletestandlaunchstrategies.Itisveryimportantfordevelopersofmobileappstohavebothinplaceinordertodevelop,test,andlaunchamobileappwithhighquality.Thischapterprovideslotsofideasaboutandexamplesofhowtoestablishmobiletestandlaunchstrategies.•Chapter8:ImportantSkillsforMobileTestersChapter8describestherequiredskillsetofamobiletester.Furthermore,thechapterprovidesideasandsolutionsonhowtoimprovetheskillsofamobiletester.•Chapter9:What’sNext?AndFinalThoughtsChapter9isthefinalchapterofthisbookanddealswithpossibletopicsthatsoftwaretestersmayhavetohandleinthenearfuture.ThechaptercontainstopicssuchastheInternetofThings,connectedhomes,connectedcars,andwearables.Attheend,fivekeysuccessfactorsareprovided.
Eachchapterfocusesonthepracticalsideofmobiletesting.Sure,therewillbesometheoreticalparts,butmostofthecontentisbasedonreal-lifeexperience
asamobiletester.
HowtoUseThisBookThisbookisapracticalguidetomobiletesting.Youcanreaditfromfronttobacktogetanoverviewofmobiletesting,oryoucanjumpstraighttothechaptersyou’remostinterestedin.There’soneimportantpieceofadviceyoushouldbearinmindwhilereadingthisbook:makesureyouhaveatleastonemobiledevicenexttoyousoyoucantryoutthethingsyouread.Ifyouwanttogetstartedwiththetestautomationtoolsmentionedinthis
book,nowwouldbeagoodtimetogetyourcomputer.
Acknowledgments
ThankstoTobiasGeyerforbeingmydedicatedreviewer.Withoutyourcontribution,encouragement,help,feedback,andcriticalquestionsthisbookwouldnothavebecomewhatitis.ThankstoDominikDaryforyourawesomefeedbackonmobiletest
automationandhelpingmeshapethecontentofseveralchaptersinthisbook.ThankstoRudolfGrötzforyourgreatideasaboutseveralgraphicsand
images.Thankyouverymuchforyourhelpandcontributiontothecrowdandcloudtestingchapter.ThankstoDagmarMatheswhogavemetheopportunitytobepartofthe
mobiletestingbusiness.Thanksforyourtrust,support,andencouragement.ThankstoSergejMudrukandChristophWielgusforyoursupportinreading
mybookandprovidingmewithveryusefulfeedback.ThankstoAndrewRennisonforbeingmygreatcopyeditor,foryourhelpand
constantcontributiontomytext.Andlastbutnotleast,Iwanttosaythankyoutoallsoftwaretestersoutthere
forsharingyourknowledgeofvarioussoftwaretestingandmobiletestingtopics.Thanksforyourcontributiontothecommunityandallofthetimeandeffortyouputin.
AbouttheAuthor
DanielKnotthasbeenworkinginthefieldofsoftwaredevelopmentandsoftwaretestingsince2003.HestartedhiscareerasatraineeatIBMwherehewasinvolvedinenterprisesoftwaredevelopmentandtesting.AfterhistimeatIBM,DanielstudiedcomputerscienceattheUniversityof
AppliedSciencesinWiesbaden,Germany.Softwaretestingbecameapassionduringhistimeatuniversityandisthereasonhechoseacareerinthefield.DanielhasworkedatseveralcompaniesinvariousindustrieswherehewasresponsiblefortestingWeb,desktop,andmobileapplications.DuringanumberofprojectshedevelopedfullyautomatedtestingframeworksforAndroid,iOS,andWebapplications.Danielisawell-knownmobileexpert,aspeakeratvariousconferencesinEurope,andablogauthor(www.adventuresinqa.com).Furthermore,Danielisthefounderandorganizeroftwolocalsoftwaretesting
usergroupsincentralGermany.OneistheSoftwareTestUserGroupRheinMain(www.stugrm.de)andtheotheristheRheinMainMobileQualityCrew(www.meetup.com/Rhein-Main-Mobile-Quality-Crew).
Chapter1.What’sSpecialaboutMobileTesting?
BeforeIstartdescribingtheuniqueaspectsofmobiletesting,I’dliketoshareatruestorywithyou.What’sspecialaboutmobiletesting?Someoneaskedmethisexactquestion
severalyearsagowhileatatestingconference.Istartedtalkingaboutmobiletechnologies,apps,howtotestthem,andwhat’sspecialaboutmobiletesting.Theguysimplysmiledatmeandsaid,“Butit’ssoftwarejustonasmallerscreen.There’snothingspecialaboutit.”Hewasreallyarrogantanddidn’tseethechallengespresentedbymobiletesting.NomatterwhichargumentsIusedtoconvincehim,hedidn’tbelieveintheimportanceofmobiletechnologies,apps,andtesting.Imetthesameguyagainin2014whileatatestingconferencewherehe
talkedaboutmobiletesting.Hespokeabouttheimportanceofappsandhowimportantitistotestthem.Asyoucansee,it’sveryeasytounderestimatenewtechnologies.Asa
softwaretesterit’sespeciallyhelpfultobecuriousaboutlearningsomethingnewandexploringnewtechnologiestobroadenyourskills.Solet’scomebacktotheinitialquestion:What’sspecialaboutmobiletesting?
IthinkIcanassumeyouhaveatleastonemobiledevice,namely,asmartphone.Ormaybeyouhaveatablet,orevenboth.Ifyoulookatyourdevice(s),whatdoyousee?Justasmallcomputerwithlittleshinyiconsonitsscreen?Ordoyouseeaverypersonalcomputerwithlotsofsensorsandinputoptionsthatcontainsallofyourprivatedata?Pleasetakeaminutetothinkaboutthat.Mysmartphoneandtabletareverypersonalcomputersthatholdalmostallof
mydata,beite-mails,SMS,photos,music,videos,andthelike.IcanaccessmydatanomatterwhereIamandusemysmartphoneasanavigationandinformationsystemtofindoutmoreaboutmysurroundings.ForthatreasonIexpectmyappstobereliable,fast,andeasytouse.InthosethreesentencesIdescribedmypersonalexpectationsofmobile
devicesandapps.Butyoumayhaveentirelydifferentexpectations,asdoesthenextperson.Andthisbringsmetothefirstspecialcharacteristicoruniqueaspectofmobiletesting:userexpectations.
UserExpectationsInmyopinion,theuserofanappisthemainfocusandmainchallengefor
mobileteams.Thefactthateveryuserhasuniqueexpectationsmakesitdifficulttodevelopanddeliverthe“right”apptocustomers.Asseveralreportsandsurveyshaveshown,mobileusershavefarhigherexpectationsofmobileappsthanofothersoftwaresuchasbrowserapplications.1Themajorityofreportsandsurveysstatethatnearly80%ofusersdeleteanappafterusingitforthefirsttime!Thetopfourreasonsfordeletionarealwaysbaddesign,poorusability,slowloadingtime,andcrashesimmediatelyafterinstallation.Nearly60%ofuserswilldeleteanappthatrequiresregistration,andmorethanhalfofusersexpectanapptolaunchinundertwoseconds.Iftheapptakesmoretime,itgetsdeleted.Again,morethanhalfofusersexperiencecrashestheveryfirsttimetheystartanapp.Anaverageusercheckshisorhermobiledeviceeverysixminutesandhasaround40appsinstalled.Basedonthosenumbers,youcandeducethatmobileusershavereallyhighexpectationswhenitcomestousability,performance,andreliability.Thosethreecharacteristicswerementionedmostoftenbyfarwhenuserswereaskedabouttheirexperiencewithmobileapps.
1.http://offers2.compuware.com/rs/compuware/images/Mobile_App_Survey_Report.pdf
Currentlytherearemorethantwomillionappsavailableintheappstoresofthebiggestvendors.Alotofappsperformthesametask,meaningthatthere’salwaysatleastonecompetitorapp,whichmakesitveryeasyforconsumerstodownloadadifferentappasit’sjustasingletapaway.Herearesomepointsyoushouldkeepinmindwhendevelopingandtestingamobileapp:
•Gatherinformationaboutyourpossibletargetcustomergroup.•Askyourcustomersabouttheirneeds.•Yourappneedstosolveaproblemfortheuser.•Usabilityisreallyimportant.•Yourappneedstobereliableandrobust.•Appperformanceisreallyimportant.•Appsneedtobebeautiful.
Thereare,ofcourse,aplethoraofotherthingsyoushouldtakeintoaccount,butifyoupayattentiontothesepoints,yourusersarelikelytobehappy.You’veprobablyalreadyheardoftheKISSprinciple.2KISSisanacronymfor
KeepItSimple,Stupidandisalwaysausefulreminder—especiallyforsoftwareprojects—tonotinflatethesoftwarewithjustanotherfunctionoroption.Keepingitsmall,easy,andsimpleisbestinmostcasesandislikelytomakeyourcustomershappy.InspiredbyKISS,Icameupwithmyownprinciplefor
mobileapps:KIFSU(seeFigure1.1).Thisabbreviationisagoodmnemonictohelpyoucovercustomerneedsandaconstantremindernottoinflateappswithuselessfunctions.
2.http://people.apache.org/~fhanik/kiss.html
Figure1.1KIFSU
MobilityandDataNetworksAnotherchallengemobileappshavetodealwithmorethansoftwarerunningoncomputersisthefactthatusersaremovingaroundwhiletheyuseapps,whichoftenrequiresanInternetconnectiontofetchdatafromthebackendandservetheuserwithupdatesandinformation.Mobileappsneedtobetestedinreallife,inrealenvironmentswherethe
potentialuserwillusethem.Forexample,ifyou’retestinganappforsnowboardersandskiersthataccessesslopeinformation,onethatisabletorecordthespeedofthecurrentdownhillrunandmakesitpossibleforuserstosharerecordsdirectlywiththeirfriends,youneedtotestthesefunctionsonaslope.Otherwiseyoucan’tguaranteethateveryfeaturewillworkasexpected.Ofcourse,therearepartsofanappthatyoucantestinalabsituation,suchas
slopeinformationavailabilityorwhetherornottheappcanbeinstalled,butwhataboutrecordingaperson’sspeed,theweatherconditions,ortheInternetconnectionatthetopofamountain?Theweatherconditionsonamountain,inparticular,canbeverydifficultto
handleastheycan,ofcourse,rangefromsunshinetoasnowstorm.Insuchscenariosyouwillprobablyfindlotsofbugsregardingtheusabilityanddesignofanapp.Maybeyou’llalsofindsomefunctionalbugsduetothetemperature,whichmayhaveanimpactonyourhardwareand,inturn,yourapp.AsIalreadymentioned,thespeedandavailabilityofInternetconnections
couldvaryinsuchregions.Youwillprobablyhaveagoodnetworkconnectionwithhighspeedatthetopofthemountainandareallypooronedowninthe
valley.WhathappensifyouhaveabadornoInternetconnectionwhileusingtheapp?Willitcrashorwillitstillwork?Whathappensifthemobiledevicechangesnetworkproviderswhiletheappisbeingused?(Thisisacommonscenariowhenusingappsclosetoaninternationalborder,suchaswhensnowboardingintheAlps.)Allofthesequestionsareveryhardtoanswerwhentestinganappinalab.
Youasamobiletesterneedtobemobileandconnectedtodatanetworkswhiletestingapps.Asyoucansee,it’simportanttotestyourappinreal-lifeenvironmentsandto
carryouttestsindatanetworkswithdifferentbandwidthsasthebandwidthcanhaveahugeimpactonyourapp;forexample,lowbandwidthcancauseunexpectederrormessages,andtheswitchbetweenhighandlowbandwidthcancauseperformanceissuesorfreezes.Here’sanexerciseforyou.Takeanyappyouwantandfindthreeusage
scenarioswheretheenvironmentand/ornetworkconnectioncouldcauseproblems.
MobileDevicesBeforeyoucontinuereading,pickupyourmobiledeviceandlookatit.Takeyourdeviceinyourhandandlookateverysideofitwithoutturningiton.Whatdoyousee?Youwillmostlikelyseeadevicewithatouch-sensitivescreen,adevicewith
severalhardwarebuttonswithacharger,aheadphoneconnection,andacamera.That’sprobablyit—you’renotlikelytohavemorethanfivehardwarebuttons(exceptforsmartphoneswithaphysicalkeyboard).Inanerawhenthewordscellphonehavebecomesynonymouswith
smartphone,it’simportanttorememberthatthereusedtobeothertypesofcellphones,so-calleddumbphonesandfeaturephonesthathavelotsmorehardwarebuttonsformakingacallortypingamessage.Withaconventionaldumbphoneyouareonlyabletomakeacall,typeamessage,orstoreacontactlist;they’renotusuallyconnectedtotheInternet.Themoreadvancedones,thefeaturephones,havegames,acalendar,oraverybasicWebbrowserwiththeoptiontoconnecttotheInternet.Butallthesephonesarereallybasicintermsoffunctionalityandexpandabilityasusersaren’tabletoinstallappsoreasilyupdatethesoftwaretoanewerversion,ifitall.Bothtypesofphonesarestillavailable,especiallyinemergingmarkets,butsince2013moresmartphoneshavebeensoldworldwidethandumbphonesorfeaturephones,3andthistrendis
likelytocontinueastimegoeson.Infact,inthenextcoupleofyearsdumbphonesandfeaturephoneswillbeathingofthepast.
3.www.gartner.com/newsroom/id/2665715
Thephonesweusenowadaysarecompletelydifferentfromthe“old”ones.Currentsmartphonesareminisupercomputerswithlotsoffunctionalityintermsofhardwareandsoftware.They’repackedwithvarioussensorssuchasbrightness,proximity,acceleration,tilt,andmuchmore.Besidesthat,allmodernsmartphoneshavebothfront-andrear-facingcameras,variouscommunicationinterfacessuchasBluetooth,nearfieldcommunication(NFC),andGlobalPositioningSystem(GPS),aswellasWi-FiandcellularnetworkstoconnecttotheInternet.Dependingonthemobileplatformandmobilemanufacturer,youmayfindanarrayofotherhardwarefeatures.Fromasoftwarepointofview,smartphonesofferlotsofapplication
programminginterfaces(APIs)formanufacturers,developers,anduserstoextendsmartphonecapabilitieswithapps.Ifyoujustfocusonthemajormobileplatforms,iOSandAndroid,thereare
plentyofhardwareandsoftwarecombinationsthatmobiletestershavetodealwith.Thefactthattherearesomanycombinationsisknownasfragmentation.Mobiledevicefragmentationisahugetopicandyetanotherchallengewhenitcomestomobiletesting.Youcan’ttestyourappwitheverypossiblehardwareandsoftware
combination.Andthefactthatyoushouldtestyourappinarealenvironmentmakesitevenmoreimpossible.Mobiletestersneedtofindastrategytodownsizetheeffortoftestingondifferentdevicesandtofindawaytotestontherightdevices.Buthowcanthatbeaccomplished?Bytestingonjustonemobileplatform?
Bytestingonjustthelatestdevice?Bytestingwithjustthelatestsoftwareversion?Beforeyoudefineastrategy,youshouldkeepinmindthateveryappis
unique,hasuniquerequirements,hasotherproblemstosolve,andhasauniqueuserbase.Withthesepointsinmind,youcanaskyourselfthefollowingquestionstofindthe“right”mobiledevicesfortesting:
•Whoismyuserbase?•Howoldistheaverageuser?•Howmanymenorwomenareinmytargetusergroup?•Whichplatformisusedmostamongthatuserbase?
•Whichdeviceisusedmost?•Whichsoftwareversionisinstalledonmostofthephones?•Whatkindofsensorsdoesmyappuse?•Howdoestheappcommunicatewiththeoutsideworld?•Whatismyapp’smainusecase?
Ofcourse,therearelotsmorequestionstoask,butifyouanswermostoftheonesIsuggest,thelistofpossibledevicesyoushouldconsidertestingismuchshorter.InlaterchaptersIwilldescribeothertechniquesforselectingtherightdevices
formobiletesting.
MobileReleaseCyclesNowthatyouknowhowtofindtherightdevicesfortestingyourapp,itdoesn’tmeanthattheprocessisover.Tobehonest,it’snevergoingtoend!Themainmobilemanufacturersreleaseanewflagshipphonewithmore
featureseveryyear.Inandaroundthosereleasestheybringoutotherphonesfordifferentuserscenariosandusergroups.ThisisespeciallytrueintheAndroidworldwhereeverynewphonecomeswithanewversionoftheoperatingsystempackedwithnewfeatures,designs,orAPIs.Therearemultiplesoftwarereleaseswithinthecourseofayear,rangingfrombugfixestofeaturereleases.Youasamobiletesterneedtobesurethatyourappwillrunonthelatesthardwareandsoftware.Buthowshouldyouhandlethesesituations?Bybuyingeveryphonethat
appearsonthemarket?Byconstantlyupdatingtothelatestoperatingsystemversion?Again,themostimportantfactorsareyourtargetcustomergroupandtheapp
you’retesting.Whenyouknowthatyourtargetgroupalwaysusesthelatestandfastestphonesonthemarket,youneedtobuythosephonesassoonastheyappear.Regardlessofwhetherornotyourtargetgroupisup-to-date,youshouldalwaysmonitorthemobilemarket.Youneedtoknowwhenthemainvendorsareduetoreleasenewflagship
phonesthatalotofpeoplearelikelytobuy.Youalsoneedtoknowwhentheoperatingsystemsreceivepatches,newfeatures,ornewdesignpatterns.Sotheanswertothequestionofwhetheryouneedtobuyeveryphoneand
constantlyupdatetheoperatingsystemsisyesandno.Ofcourseyoudon’tneedtobuyeveryphonethat’sonthemarket,butyoushouldconsiderupdatingtothe
latestoperatingsystemversion.Whendoingso,keepinmindthatnoteveryuserwillinstalltheupdate.Manypeopledon’tknowhowtodothat,ortheydon’tcareaboutnewversions.Youneedatleastsomephonesthatarerunningolderversionsoftheoperatingsystemtoseehowtheappreactsinthatenvironment.Olderversionsoftheoperatingsystemarealsoneededtoreproducereportedproblemsandbugs.Agoodwaytomanageallthisistostickwiththesameoperatingsystem
versiononthephonesthatyouhaveandbuynewphoneswiththelatestsoftwareversion.Thisofcourseleadstoanotherproblem—it’sreallyexpensive!Noteverymanagerwantstospendsomuchmoneyonmobiledeviceswhenaphoneisgoingtobeusedforonlyacoupleofmonths.Asolutionforthatistorentdevices.ThereareseveralprovidersandOpenDeviceLabswhereyoucanrentadeviceforacertainperiodoftime(alistofproviderscanbefoundinChapter3,“ChallengesinMobileTesting”).Anotherwaytorentdevicesisthemobiledevicecloudasthereareanumberofproviderswhogivemobiletestersexclusiveaccesstothephysicaldevicestheyhavemadeavailableinthecloud.Justuseyoursearchengineandcheckthemout.InthemobileprojectsI’veworkedon,wealwayshadthetoptento15devices
usedbyourtargetusergroupindifferentvariationsfordevelopingandtesting.Thiswasagoodnumberofdevicesthatcoverednearly90%ofourtargetgroup.Withthosetento15deviceswewereabletofindmostofthecriticalbugs;theremaining10%ofdeviceswedidn’thavewereofnomajorconsequencetotheprojectoruserexpectations.Inordertohandlethefastpaceofmobilereleasecycles,youshouldkeepthe
followingthingsinmind:•Monitorthemobiledeviceandsoftwaremarket.•Knowwhennewphoneswillberolledout.•Findoutaboutthenewfeaturesoftheoperatingsystems.•Keepaneyeonyourtargetcustomergrouptoseeifnewdevicesareshowingupinyourstatistics.•Thinktwicebeforeupdatingaphonetothelatestoperatingsystemversion.•Buynewphoneswiththelatestoperatingsystemversion.•Ifbuyingisnotanoption,rentthedevices.
Updating,buying,andmaintainingallofyourdevicesisachallengingtaskandshouldnotbeunderestimated!Atsomepoint,dependingonthenumberoftestdevicesusedwithinaproject,thiscouldbeafull-timejob.
MobileTestingIsSoftwareTestingLet’scomebacktothestoryItoldatthebeginningofthischapterwhentheguyattheconferencedidn’tbelieveintheimportanceofmobiletesting.Hehadtheattitudethatmobiletestingisnotrealsoftwaretesting.Inhisopinion,mobileappswereonlysmallprogramswithlessfunctionalityandnorealchallengeswhenitcomestosoftwaretesting.Butthisisdefinitelynotthecase.IfyoulookatthetopicsIdescribedinthischapter,youshouldhaveaninitialimpressionaboutthechallengingjobofamobiletester.MobiletestingistotallydifferentfromtestingsoftwareapplicationssuchasWebordesktopapplications.Withmobileapps,physicaldeviceshavefarmoreinfluenceoverthesoftwarethatisrunningonthemwhencomparedtoothersoftwaresuchasWebapplications.Becausetherearesomanydifferentsmartphonesavailableonthemarket,mobiletestersneedtofocusalotmoreonhardwareduringthetestingprocess.Inaddition,usersmovingaroundandusingdifferentdatanetworksforcemobiletesterstobeonthemovewhiletesting.Besidesthehardware,userexpectationsplayanimportantpartinthedaily
businessofamobiletesterandneedtobetakenseriously.Therearemanymoretopicsandissuesmobiletestersneedtoknowaboutin
ordertohelpthewholeteamreleaseasuccessfulapp.Therestofthechaptersinthisbookwillcoverthefollowingtopics:
•Morechallengesformobiletestersandsolutionstothosechallenges•Howtotestmobileappssystematically•Howtoselecttherightmobiletestautomationtool•Thedifferentconceptsofmobiletestautomationtools•Howtofindtherightmobiletestingstrategy•Additionalmobiletestingmethods•Requiredskillsformobiletesters
Keepthetopicsfromthischapterinmindasastartingpoint.Keepyourappsimpleandfast(rememberKIFSU).Testwhileyou’reonthemove,andtestondifferentdevicesbasedonyourtargetcustomergroup.
SummaryThefirstchapterofthisbookmentionedsomeveryimportanttopicsfromthemobiletestingworld.Asyouhaveseen,mobiletestingiscompletelydifferentfromtestingonothertechnologiessuchaslaptopsordesktopcomputers.Thebiggestdifferencebetweenmobileandothertechnologiesisthatthemobileuser
isonthemovewhileheorsheisusingyourproduct.Therefore,itisveryimportanttoknowaboutthedifferentdatanetworksandthedifferenttypesofmobiledevices.Thischapteralsoprovidedafirstoverviewofmobileusers’highexpectations.
ItisreallyimportanttokeepKIFSUinmindwhendesigning,developing,andtestingamobileapp.Itwillhelpyoutofocusontheimportantelementsandnotwastetimeonunnecessaryfeaturesthatyouruserswon’tuse.Andlastbutnotleast,thischaptershouldremindyoutoneverunderestimatea
newtechnology.Beopen-mindedandcurioustoimproveyourdailyworklife.
Chapter2.IntroductiontoMobileDevicesandApps
BeforeIdivedeeperintothetestingchapters,I’dliketointroduceyoutothehistoryofmobiledevicesandcellularnetworks.Thismaysoundabitboring,butasyouwillseebytheendofthischapter,it’sreallyimportanttoknowthebackgroundofthemobileworldandtohavethoroughknowledgeofformermobiletechnologies.LaterinthischapterIdescribethedifferentmobileapptypesandtheappbusinessmodelsandprovideyouwithashortoverviewofthecurrentstateofmobileappstores.Let’sstartwiththewordmobile.ItcomesfromtheLatinwordmobilis,which
itselfisderivedfromtheLatinverbmovere,“tomove”—tobeabletomovearoundfreelyandeasilybywalking,driving,orflying.Thisdefinitionsoundsreallysimpleandcomprehensible,andI’msureyou
hadsomethingsimilarinmind.Ifyoulookatthewordmobilefromatechnologicalpointofview,it’snotquiteassimpleduetothevastchangesinthewaypeoplehavebeguntousemobiletechnologiesoverthepastfewdecades.Solet’sgobackafewdecadesintime.
OverviewofMobileNetworksBeforewecancommunicatewithanykindofmobiledevice,acommunicationinfrastructuremustbeavailable.Themobileinfrastructureiscurrentlyinitsfourthgeneration,knownaptlyas4GorLTE(Long-TermEvolution).1Beforethatwesawthegenerations0G,1G,2G,and3G,andeachgenerationwasamilestoneatitstimeofintroduction.
1.www.etsi.org/technologies-clusters/technologies/mobile/long-term-evolution
Thezerogeneration—theearlypredecessors—includedjustanalogradiocommunicationandwasmainlyusedinthe1960s.ItisalsoknownastheMobileRadioTelephoneSystem.Communicationatthistimewashalfduplex,meaningthatonlyonepersonwasabletotalkatatimewhiletheotherlistened.ThezerogenerationconsistedofdifferentmobiletechnologiessuchasMobileTelephoneService(MTS),MobileTelephonySystemD(MTD),AdvancedMobileTelephoneSystem(AMTS),andOffentligLandmobilTelefoni(OLT).Thecellphoneswerereallyheavyandwereinstalledmostlyintrucks,trains,andothervehicles.Thephoneconsistedoftwoparts,thetransceiver(transmitterandreceiver)andthehead.Thetransceiverwasresponsibleforestablishingtheconnectiontothelocaltransmitterstations,andtheheadwas
wiredtothetransceiverandconsistedofdialkeys,adisplay,andahandset.Thisgenerationhadalotofproblemswithconnectivityandhadlimitednumbersofsubscribers.Thefirst-generation(1G)cellularnetworkwasanimprovementoverthezero
generationandwasintroducedinthe1980s.1GstillusedanalogradiosignalstotransmitinformationusingtheAdvancedMobilePhoneService(AMPS)orNordicMobileTelephone(NMT)technology.ThefirstnetworkswerelaunchedinJapan,followedbyDenmark,Finland,Norway,Sweden,andtheUnitedStates.Acoupleofyearslaterothercountriesbuiltuptheir1Gnetworkinfrastructure.Thebiggestadvantageoverthezerogenerationwasthat1Gwasabletoaccommodateuptotentimesmoreusersbydividingthelocalareaintosmallercells.Thisgenerationhaditsdrawbackswhenitcametosecurityasuserswereabletolistentosomeoneelse’sconversationandhackthesystemtomakefreecalls.Thebiggestimprovementinmobilecommunicationnetworkswasintroduced
withthesecondgenerationofcellularnetworks.2Gwasfirstlaunchedin1991inFinlandusingtheGSM(GlobalSystemforMobileCommunications)standard.2AcoupleofyearslaterCDMA(CodeDivisionMultipleAccess)waslaunchedintheUnitedStates.3Thesenewstandardsformedthebasisoftoday’smobilecommunicationinfrastructureandofferedthreemainadvantagesovertheirpredecessors:
2.www.etsi.org/technologies-clusters/technologies/mobile/gsm3.www.etsi.org/technologies-clusters/technologies/mobile/w-cdma
•Forthefirsttimeever,communicationwasdigitalandencrypted.•2Gwaswaymoreefficientandprovidedbetterglobalcellphonecoverage.•Dataserviceswereintroduced,themostcommonlyknownonebeingtheSMS.
The2Gnetworkwasbuiltmainlyforvoiceandtextcommunicationandonlyhasslowdatatransmission.Afterthe2Gnetworkswereestablished,mobileservicesusageincreasedanddatatransmissionbecametooslow.Toachievehigherdatatransferrates,the2GnetworkwasextendedwiththeGPRS(GeneralPacketRadioService)4andEDGE(EnhancedDataratesforGlobalEvolution)5standards.GPRSisalsocalled2.5GandEDGE,2.75G.Bothtechnologieshavehigherdatatransferrates(GPRS=56Kbit/supto115Kbit/s,EDGE=upto236Kbit/s)thanthenormal2Gnetworkandarethepredecessorsofthe3Gnetwork.
4.www.etsi.org/index.php/technologies-clusters/technologies/mobile/gprs5.www.etsi.org/index.php/technologies-clusters/technologies/mobile/edge
Thethirdgenerationofmobilenetworks(3G)hasbeenaroundsince2001andisanevolutionoftheexisting2Gnetworks.ThethirdgenerationusestheUMTS(UniversalMobileTelecommunicationsSystem)6andCDMA2000standards.3Goffershigh-speeddatatransferratesupto21Mbit/sdependingontheuser’scurrentlocation.Thishighdatatransferrateallowssmartphone,tablet,orcomputeruserstomakevideocalls,watchmobileTV,andsurftheInternetwhileonthemove.The3Gnetworkswiththeirhigh-speeddatatransferrateshavehadamajorinfluenceonthesuccessofmobiledevicesandapps.
6.www.etsi.org/technologies-clusters/technologies/mobile/umts
Thefourthgenerationofmobilecommunicationnetworksaccommodatedthehugeamountsofdatanowbeingtransferredoverthenetworkbyincreasingdatatransferratesevenfurther.The4Gnetworkisbasicallydividedintotwostandards:WiMAX(WorldwideInteroperabilityforMicrowaveAccess)7andLTE.WiMAXoffersadownloadtransferrateofupto128Mbit/sandanuploadrateofupto56Mbit/s.LTEoffersdownloadratesofupto100Mbit/sandanuploadrateofupto50Mbit/s.Ifbothstandardsarefullyimplemented,thedownloadspeedcanincreaseto1Gbit/s.
7.www.wimaxforum.org/index.htm
Dependingonyournetworkproviderandcountry,yoursmartphoneisconnectedtoeitheraWiMAXorLTEnetwork.Ifyourphonesupports4Gnetworks,youwillseealittleLTEor4Giconinthestatusbarofthephone.Thefifthgenerationofmobilenetworksiscurrentlyunderdevelopment.
Severalresearchgroupshavebeenformedtodescribeanddevelopthenextgenerationofmobilecommunicationandarchitecture.However,thisstandardwillnotbeavailablebefore2020.8
8.http://europa.eu/rapid/press-release_IP-13-159_en.htm
Thiswasahigh-leveloverviewofthetechnologybehindmobilenetworks,buteventhishigh-levelviewisimportantwhenyoustarttestingyourappindifferentnetworksusingdifferentstandards.It’simportanttoknowwhatkindofnetworkstandardsareavailableandwhichnetworkspeedsareprovided.Furtherinformationonhowtotestthedifferentdatanetworkswillbeprovidedinlaterchapters.Ihighlyrecommendthatyouusethisknowledgeinyourdailyapptesting.
MobileDevicesandtheYear2007Priorto2007mostphoneswereso-calledfeaturephones(asdescribedinChapter1,“What’sSpecialaboutMobileTesting?”)whosefunctionalitycould
notbeextendedwithsoftware(apps).NotallofthemwereabletoconnecttotheInternet,evenwhenthemobilenetworkswereabletohandledatatransmission.AtthattimeNokia,Motorola,BlackBerry(ResearchinMotion),andsomeothermobilemanufacturersdominatedthemobiledevicemarket.Mostofthedeviceshadasmallscreenandaphysicalkeyboardandweregoodjustformakingacallortypingatextmessage.However,thosedeviceswerenofuntousewhentryingtosurftheWeborsearchforcontactswithinthephone.ThesmartphonerevolutionstartedinJanuary2007whenApplelaunchedits
firstiPhone.SteveJobspresentedthefirstgenerationoftheiPhonewiththefollowingsentence:“Today,Appleisgoingtoreinventthephone.”AndSteveJobswasright.Themobiledevicemarkethasofcoursechanged
dramaticallysince2007.Justoneyearlater,GooglepresentedthefirstAndroidsmartphonedevelopedbyHTC,theHTCDream(alsoknownasT-MobileG1).InthefollowingyearslotsofothermanufacturersbuilttheirownAndroidsmartphoneswithdifferentAndroidsoftwareversions.WhenAppleannouncedtheiPhone,onlyGooglewasfastenoughtoadaptand
buildanothermobileplatform.However,MicrosoftandBlackBerry(ResearchinMotion)haveupdatedtheirmobiletechnologiestoclosethegapbetweenthemselvesandAppleandGoogle,butuntilnowtheystillhaven’tmanagedtocatchup.Since2007,smartphoneshaveconstantlybeenimprovedwithnewhardware
andsoftwarefeatures,andsometimesit’shardtofollowallthenewfeaturesthatarerolledoutintothemobilemarket.Youneedtoknowwhat’sinsideasmartphonetogetanoverviewofits
physicalpartsandwhateachofthemdoes.Thisknowledgewillhelpyouunderstandthedevicesyouwanttotest,whichinturnwillhelpyoutestmoreeffectively.Ifyoulookatthemobiledevicenexttoyou,youwillseeasmall,thin,and
flatorcurvedpieceofglass,plastic,ormetal.Allthehardwarethatisneededtobringthislittlethingtolifeispackedintothesmallcaseandisnotseenbytheuser.Butwhat’sinside?Atypicalsmartphoneconsistsofthefollowinghardwarecomponents:•Mainboardorlogicboard•CPU(centralprocessingunit)•GPU(graphicsprocessingunit)•Memory
•Differentantennasandinterfaces:•Cellularnetworkchipstoconnectto2G,3G,or4G•Wi-Fi•NFC(nearfieldcommunication)•GPS(GlobalPositioningSystem)•Bluetooth
•Varioussensors(notnecessarilyalloftheonesinthislist):•Ambientlightsensor•Proximitysensor•Accelerationsensor•Gyroscope•Magneticsensor•Pressuresensor•Temperaturesensor•Humiditysensor
•Battery•Vibrationmotor•Slotforadditionalmemorycard•SlotforSIMcard
Herearethepartsyoucanseewhenyouhaveyoursmartphoneinyourhand:•Smartphonecase•Touchscreen•Hardwarebuttons(powerbutton,volumeupanddownbutton,navigationbuttons)•Headphonejack•Speakersandmicrophones•Charger/USBconnector•Frontandrearcameras•Flash
Ifyouwanttogetmoredetailedinformationaboutaspecificphone,justusetheInternetandsearchfor“teardowns”ofthedeviceorcheckoutthedevicemanufacturer’sWebsite.ThehardwareIlistedalsoappliestotabletswithsome
variations.Asamobiletester,it’sveryimportantthatyouknowallthepossible
componentswithinamobiledevice.Thisknowledgeisneededtoidentifyandnarrowdownproblemsorbugsthatcouldberelatedtothedevicehardwareoryourapp.
TheBigTwoAsmentionedinthepreviouschapter,othermobileplatformssuchasWindowsPhoneandBlackBerryhaveverylittlemarketsharecomparedtoAndroidandiOS.9AsofFebruary2015,marketshareswereasfollows:Android,55.26%;iOS,23.82%;WindowsPhone,2.32%;andBlackBerry,1.66%.Themissing17%includesdevicesrunningSymbian,Series40,andotheroutdatedmobileoperatingsystems.
9.Exactfiguresonmarketsharecanbefoundhere:http://gs.statcounter.com
ImportantThesefigureswillofcoursevarydependingongeographicallocation,butthey’reagoodindicatorofthebigpicture.
SinceBlackBerryandWindowsPhonehaveverylittlemarketshare,I’llfocusonthebigtwofortherestofthechapter:iOSandAndroid.Whatarethedifferencesbetweenthetwooperatingsystems?Whatdothey
haveincommon?Table2.1comparestheoperatingsystemsbasedoncertaincriteria.
Table2.1ComparisonofAndroidandiOS
Asyoucansee,bothplatformshavelotsoftechnologies,functions,andappsincommon,suchasappsforvoicecommands,maps,videochats,e-mail,acalendar,andmuchmore.Butifyoulookatthesourcemodelandprogramminglanguages,you’llnoticethemaindifferences.Partsoftheopen-sourceAndroidoperatingsystemarewritteninCandC++.AndroidappsarewritteninJava.iOSisalsowritteninCandC++andisclosedsource.iOSappsarewritteninObjective-CorSwift.iOSusestheopen-sourceDarwinoperatingsystemasasystembasis,butthefinishediOSversionasweknowitisclosedsource.Anotherobviousdifferencebetweenthetwoisthemanufacturers.Apple
producesiOSdevicesonitsown,whereasGooglegivesothermanufacturerstheopportunitytobuildtheirownhardwaredevicesbasedontheAndroidoperatingsystem.ThemanufacturersareabletoextendtherawAndroidoperatingsystemandbuildacustomizedAndroidversiontailoredtothehardware.Mobiletestersneedtoknowallaboutthetailoringofthedifferenthardware
manufacturersbecausetheuserinterfacesonAndroidcandifferalotandthereforeinfluencethebehaviorofanapp.ThefragmentationofAndroiddevicesandsoftwareversionsisoneofthemainchallengesthatmobiletestersneedtohandle.However,thereisalsoapureAndroiddeviceavailableonthemarketasGooglecooperateswithsomehardwaremanufacturerstobuildtheirNexusdevices.Last,Iwanttopointoutthedifferencesbetweentheuserinterfaces.Both
platformsprovidetouchinterfacesthathavelotsofgesturesincommonsuchasswiping,tapping,pinch,andzoom,buteachhasitsownUIanddesignpatterns.ForcompleteinstructionschecktheAndroiddesignguide10andtheiOSdesignguide.11Witheachnewversionoftheoperatingsystem,checkthoseguidelinesagain,becausechangesarelikely.Thesepatternsmustbefollowedinordertopublishanapp.Ifyourappdoesn’tfollowtheguidelines,itmaywellberejectedfromtheappstore,whichismorelikelytohappenintheAppleAppStorethanintheAndroidappstores.Togetanoverviewofpossiblereasonswhyanappisrejectedfromanappstore,lookatthe“CommonAppRejections”pageprovidedbyApple.1210.https://developer.android.com/design/index.html11.https://developer.apple.com/design/12.https://developer.apple.com/app-store/review/rejections/
Anothergoodreasontofollowthedesignprinciplesistomakecustomershappybecausetheyknowhowtousetheplatform-specificfeaturessuchasswipingfromlefttorighttoswitchviews,ortopulldowntorefreshthecurrentview.Whenthedevicehasbooted,bothplatformspresentahomescreentotheuser
thatissimilartoacomputerdesktop.WhiletheiOShomescreencontainsonlyrowsofappiconsspreadoverseveralhomescreens,Androidgivestheusertheabilitytocustomizethehomescreenwithappsandwidgets.13Widgetsareabletodisplaymoreinformationonthedesktop,suchascontentfromthee-mailapp,Twitter,ortheweather.Widgetscanberesizedandplacedwherevertheuserwantstohavethem.SinceiOS8,usersareabletoplacewidgetsintheiOSnotificationcenteraswell.
13.https://developer.android.com/guide/topics/appwidgets/index.html
Bothhomescreenshaveadocksectionatthebottomofthescreentopinthemostimportantappsthatareavailableoneveryhomescreen.Also,bothplatformshaveastatusbarthatisavailableatthetopofthescreenanddisplaysdevice-relevantinformationsuchasthebatterystatus,thenetworkstrength,thecurrenttime,andanynotificationsfrominstalledapps.Again,onAndroidthestatusbarcandisplaymoreinformationsuchasreceivede-mails,messages,phonecalls,orreminderslinkedtoinstalledapps.Ifyou’refamiliarwithonlyoneplatform,youshouldbuyorrentthe
unfamiliaronetolearneverythingaboutit.Youwillneedtoknowasmuchaspossibleaboutthemajorplatformstosucceedinthelongrunasamobiletester.
WhatTypesofAppsExist?Ifyouunlockyourphonetogettothehomescreen,youwillseeyourinstalledapps.Butwhichtypesofappsdoyouhaveinstalled?Aretheynative,hybrid,orWeb-basedapplications?Ifyoujustlookattheappicon,youprobablycan’ttell.Maybeyoucanfindoutbytappingonanappicontoopenit.Let’stryitout.Getyourmobiledevice,unlockit,andopenanappofyour
choice.Whatdoyousee?Isabrowserwindowshown,oristheappvisibleinfull-
screenmode?Ifyouseeabrowserwindow,yourappisaWeb-basedapplication.Butcanyouseethedifferencebetweenahybridandanativeapp?Thatwill
dependonhowwellthehybridappisdevelopedandoptimizedforyourphone.InthefollowingsectionIwilldescribethedifferentmobileapptypesandlist
theprosandconsofeachofthem.
NativeAppsNativeappsareprogrammedwithaspecificprogramminglanguageforthespecificmobileplatform.Forexample,AndroidappsaredevelopedinJava,whereasiOSappsarewritteninObjective-CorSwift.Nativeappshavefullaccesstoallplatform-specificlibrariesandAPIsinordertotakeadvantageofallthefeaturesamodernsmartphonehastooffer.Assumingtheuserhasgrantedthenecessarypermissions,theapphasdirectaccesstothecamera,GPS,andalltheothersensors.DevelopersareabletobuildappsthatmakeuseofsystemresourcessuchastheGPUandCPUtobuildpowerfulapps.Nativeappsgenerallyexhibitexcellentperformanceandareoptimizedformobileplatforms.Inmostcases,nativeappslookandfeelgreatandareabletosupportevery
possiblegestureonthetouchscreen.Appdistributionisalsoquitesimpleasyoucanuploadyournativeapptothe
appstoresofthedifferentvendorsandstartsellingit.Someappstorevendorshaveanapprovalprocess,meaningthatitcantakesometimeuntilyourappisavailable.Thesameprocessapplieswhenanalreadyreleasedappisupdated,whichcanbeaproblemespeciallywhenyouwanttofixareallyurgentbuginyourapp.
Pros:•Nativeappshavefullaccesstoplatform-specifichardwareandsoftwarefeatures.•Nativeappshavegoodperformancebecausetheyareoptimizedforthespecificmobileplatform.•Nativeappshaveagoodlook-and-feel.•NativeappsoffergoodusabilityiftheplatformUIguidelinesaremet.•Nativeappshavefullaccesstoalltouchgestures(ifimplemented).•Nativeappdistributioniseasy.Userscansearchforyourapp.•Nativeappscanstoredataoffline.
Cons:•Theamountofdevelopmentworkincreaseswitheachsupportedplatformbecauseeachplatformneedsitsowncodebase.•Theapprovalprocesscanbequitelong.•Updatingareleasedappmaytakesometime(whichisannoyingwhenitcomestourgentbugfixes).•Developmentcostscanbehigher.•Youmustshare30%ofyourapprevenueswiththeplatformprovider.
HybridAppsHybridapps,asthenamesuggests,areappsthatconsistofdifferentWebtechnologiessuchasHTMLorJavaScript.OncetheWebparthasbeenbuilt,developersareabletocompilethiscodebasetothedifferentnativeformats:Android,iOS,WindowsPhone,orBlackBerry.TocompiletheWebcodeintonativemobilecode,developersneedtouseahybriddevelopmentframeworksuchasPhoneGap.14SuchframeworksofferAPIstoaccessthedevice-specifichardwarefeatureswithintheWebpartoftheapp.14.http://phonegap.com/
Howdoessuchaframeworkwork?
ImportantThedescriptionhereisaverysimplisticviewofhybridmobileframeworks.
Theframeworkbuildsaso-calledbridgetotheWebcodeviaanHTMLrenderingengine.AsmallpartoftheapprunsonthenativeoperatingsystemandcommunicateswiththeWebcodeintherenderingengineviathebridge.Withtheaidofthisbridge,theWebcodecanaccesssomeofthenativehardwarefeatures.TheHTMLcontentorcomponentsofhybridappscanbehostedonaserver.
Thisapproachmakesitveryeasytomakesmallupdateswithoutupdatingthewholeappthroughtheappstoresubmissionprocess.Storingtheinformationandelementsontheserverhasonebigdrawback;however,thecontentandelementsdon’tworkwhenthephoneisoffline.Thesepartsareavailableonlyifthedeviceisconnectedtoadatanetwork.However,youcanputallthecontentandelementsintotheappforfullofflinesupport,butthensmallonlineupdatesarenolongerpossible.Ifyourteamisdevelopingahybridapp,keepthosepointsinmind.
Pros:•Thereisonecodebasefordifferentmobileplatforms.•Frameworksofferaccesstohardwarefeatures.•Smallupdatescanbeperformedontheserver.•Appdistributioniseasy.•Userscansearchforyourapp.
Cons:•Performanceisbadwhenthecontentandcomponentsareaccessedfromtheserver.•Meetingthedesignguidelinesofthedifferentmobileplatformsisnoteasy.•Platform-specificfeaturescan’tbedeveloped,becausetheymaynotbeavailableontheotherplatforms.•Theapprovalprocessmaybelong.
Anicecomparisonofdifferentmobiledevelopmentframeworkscanbefound
onthe“MobileFrameworksComparisonChart”Website.1515.http://mobile-frameworks-comparison-chart.com/
WebAppsAmobileWebappisaWebsitethatcanbeaccessedfromthedevice’sWebbrowser.SuchWebsitesareoptimizedformobilebrowserusageandareindependentofthemobileplatform.MobileWebappsaredevelopedwithWebtechnologiessuchasHTMLandJavaScript,particularlywithHTML5,16CSS3,andJavaScript.1716.http://dev.w3.org/html5/html-author/17.www.w3.org/Style/CSS/
HTML5offersdevelopersthecapabilitytoimplementmobileWebsiteswithanimatedandinteractiveelements.Theycanintegrateaudioorvideofilesandusepositioningfeaturesaswellassomelocalstoragefunctionality.TheuseofHTML5,CSS3,andJavaScriptmakesiteasytodevelopmobileWebapps.Furthermore,mobileWebappsrequirenoappstoreapprovalandcanbeeasilyandquicklyupdated.However,mobileWebappshavesomedrawbacks.Forexample,theyoffer
onlyverylimitedtonoaccesstothedevicehardwarefeaturessuchasproximityoraccelerationsensors.MobileWebappshavenoaccesstothecamera,compass,microphone,oranykindofnotifications.Theytendtobeslowerthannativeorhybridappsbecausetheyneedtodownloadalltheinformationthatisshownonthescreen.Dependingonthemobilebrowser,mobileWebappscanworkandbehave
differentlybecausenotallmobilebrowserssupportthefullstandardsofHTML5,CSS3,andJavaScript.ThiscanhaveamajorinfluenceonthemobileWebapp,meaningthatdifferentmobileWebbrowsersneedtobecoveredduringthetestingprocess.Tosummarize,Webappshavethefollowingadvantagesanddisadvantages:
Pros:•Populartechnologiesareusedfordevelopment.•Webappsarefasterandcheapertoimplementthannativeandhybridapps.•Theyaremobileplatformindependent.•ThereiseasyaccesstothemviaaWebbrowser(noinstallationrequired).
•Noappstoresubmissionisrequired.•Theupdatingprocessisfastandeasy.
Cons:•Thereislimitedaccesstohardwarefeatures.•Thereislimitedofflinebehavior.•Largemediafilessuchasimagesorvideoscantakealongtimetodownload.•TheyhavedifferentWebbrowsersupportfortheHTML5,CSS3,andJavaScriptstandards.•Theyarenotasconvenienttouseasnativeapps.•Thereislimitedusageoftouchgestures.•Userscan’tfindtheappintheappstores.
BusinessModelsofMobileAppsWhere’sthemoneyinmobileapps?Andhowcanyoutesttomakesuremoneyisbeingearned?Thesetwoquestionsareimportantwhendevelopingandtestinganapp.Nearlyeveryappdeveloperandcompanywantstomakemoneyfromin-apppurchases!Youthereforeneedtotestthepaymentmodelstobesurethatthemobileapp
isgeneratingrevenues,whichinturnmeansthatyouneedtobeawareofcurrentbusinessmodels:
•Freemium•Paid•Transaction
FreemiumThefreemiummodelisdesignedtoreachasmanyusersaspossible.Oncetheappisinstalled,thereareseveralwaystogeneraterevenuesoutof
thefreeapp:•Themostwidelyusedapproachisthefreeversionoftheapp.Thefreeversionislimited,suchasintermsoffunctionalityorcontent.Ifauserwantstohavefullfunctionality,heorshecandownloadtheenhanced(paid)versionoftheapp.Thisapproachisthemostwidelyusedfreemiumappmodel.
•Thesecondmostusedapproachforgeneratingrevenuesissellingadvertisementswithintheapp.Therearedifferentkindsofadvertisementformatsthatcanbeaddedtoanappinordertogeneraterevenues.Advertisementsarepartofnearlyeveryfreeappandcanoftenbereallyannoyingandfrustratingfortheuser.Thinkverycarefullybeforeaddingadstoanappasyouruntheriskoflosingcustomers.DeveloperscanimplementadframeworkssuchasAdMob18oriAd.19
18.www.google.com/ads/admob/19.http://advertising.apple.com/
•Thethirdapproachforgeneratingrevenuesisthe“in-apppurchase.”Thisisoftenusedwithingamingappswherenewlevelsormoretoolscanbeboughttohavemorefunwiththegame.Lotsofnewspaperappsalsoofferin-apppurchasestogetthelatestversionoftheirdailynews.Someappscanbecomeadfreeiftheuserpaysforit.
PaidThepaidbusinessmodelisquitesimple:beforeuserscandownloadtheapp,theyhavetopayforit.ThisisacommonusecaseforgamingappsorappsthatfulfillaspecialtasksuchasapplyingfilterstoimagestomakethemlooklikePolaroids.
TransactionInthetransactionbusinessmodeltheuserpaysonlyaftercompletingatransactionwiththeapp.AnexampleofatransactionalappisGoogleWallet,whereusersareabletosendmoneytoanotheraccountusingtheircreditordebitcard.20Oncethetransactioniscomplete,asmallfeeispaidthatdependsonthetransactionamount.20.www.google.com/wallet/
ChoosingaBusinessModelResearchcarriedoutbyGartnershowsthatfreemiumappscontainingin-apppurchasesaredownloadedthemost(approximately90%)fromtheappstores,whereaspaidappsaredownloadedfarlessfrequently.21Developersthereforeneedtoputsomethoughtintotheirapp’sbusinessmodelandprice.21.www.gartner.com/newsroom/id/2592315
Whenchoosingabusinessmodelforyourapp,makesureyoukeepthedifferentapptypesinmind.Noteverymodelcanbeappliedtoeveryapptype.
Forexample,ifyouwanttodevelopapaidmobileWebapp,yourappneedsalogintoidentifytheuser’ssubscriptionsoastogainaccesstothepaidcontent.Theloginfunctionmaynotbenecessarywithinanativeorhybridappbecausepaymentismadewithintheappstore.
AppStoresAppstoresformthecoreofthemobileworldwhereappscanbedownloadedandreviewed.Withouttheappstores,smartphoneswouldn’tbeasintelligentandfunctionalaswewantthemtobe.Atthetimeofwriting,theappstoresofthebiggestmobileplatforms—Google22andApple23—containmorethantwomillionapps.Sofarmorethan100billionappdownloadshavebeencounted.Thesearehugenumbersthatareboundtoincreaseinthefuture.22.GooglePlaystore,https://play.google.com/store23.AppleAppStore,https://itunes.apple.com/us/genre/ios/id36?mt=8
BesidesthebigtwoappstoresfromAppleandGoogle,thereareotherstoressponsoredbydevicemanufacturersandnetworkoperators.Thefollowinglistisnotcompletebutcontainssomeotherappstoresforthedifferentmobileplatforms:
•Amazon•AT&T•ChinaMobile•Mozilla•Samsung•T-Mobile•Vodafone
Butwhyaretheresomanydifferentstores?EspeciallyintheAndroidworldthereismorethanoneappstoreavailable,suchastheAmazon24andSamsung25stores.Theanswertothisquestionissimple:everyappstoreproviderwantstomakemoneyoutofthemobileappbusiness!24.www.amazon.com/mobile-apps/b?node=235014901125.http://apps.samsung.com/
Let’staketheSamsungstoreasanexample.SincethelaunchofitsGalaxydeviceseries,SamsunghasgrowntobecomeoneofthebiggestandmostsuccessfulAndroiddevicemanufacturersandhassoldandcontinuestosellmillionsofdevicesaroundtheworld.HavingitsownappstorepreinstalledoneverySamsungphoneisahugeadvantagebecauseitdirectspotentialmobile
appcustomersstraighttoSamsungandawayfromGoogle.Ifmillionsofusersusethisstore,thestoregeneratestraffic,whichmeansitcansellads.Ontopofthat,appsalesviatheappstoreallowSamsungtogenerateadditionalrevenues.Inmostoftheappstores,30%ofthesalepricegoestotheplatformprovider,andthesameappliestoin-apppurchases.Ithinkthisisindicativeofwhytherearesomanydifferentmobileappstores.
IfyousearchtheInternet,I’msureyou’llfindevenmoreofthem.Theothermobileplatformsalsohaveappstores.BlackBerryappscanbe
downloadedfromtheofficialBlackBerryWorld,26andWindowsPhoneappscanbedownloadedfromtheMicrosoftStore.2726.BlackBerryWorld,http://appworld.blackberry.com/webstore/?
d=android&o=m&countrycode=US&lang=en27.MicrosoftStore,www.windowsphone.com/en-us/store
Beforeamobileteamdistributesanappinthebiggeststores,theyshouldthinkaboutthegoalsoftheapp.Maybethereareotherstoresthatarebettersuitedtotheirappthanthebigplayers.Forexample,somestoresofferabetterrevenuesharethantheusual70%(developerrevenues)/30%(storeproviderrevenues)split,ortheyofferbetterapptargeting,forexample,indifferentmarketssuchasAfricaorAsia.However,uploadinganapptooneofthestoresrequiresonethinginparticular
—knowledgeofthestore’sreviewandpublishingprocess.Yourappneedstofulfillthereviewandpublishingguidelinesofthevariousvendorsandappstores;otherwiseyourappisverylikelytoberejected.Knowledgeofthedifferentreviewguidelineswillallowyoutobettersupport
yourteamwhiledevelopingandreleasingyourapp.ThereviewandpublishingguidelinesofthemajormobileappstorescanbefoundonthefollowingWebsites:
•AmazonAppstoreGuidelines(https://developer.amazon.com/help/faq.html)•AppleAppStoreGuidelines(https://developer.apple.com/app-store/review/)•BlackBerryWorldGuidelines(http://developer.blackberry.com/blackberryworld/vp_checklist.html)•GooglePlaystoreGuidelines(http://developer.android.com/distribute/googleplay/publish/preparing.html)•SamsungAppStoreGuidelines
(http://developer.samsung.com/distribute/app-submission-guide)•WindowsPhoneStoreGuidelines(http://msdn.microsoft.com/en-us/library/windows/apps/br230835.aspx)
SummaryChapter2coveredtheevolutionofmobiledatanetworksandmobiledevices.Itisveryimportanttoknowthedifferencesamongthedatanetworks,theirspeed,andtheirtechnologies.Thisknowledgeisrequiredwhentestingamobileappwhileyouareonthemove.Besidesthedatanetworks,knowledgeaboutmobiledevicesandtheir
evolutionisalsoveryimportant.Amobiletestermustknowallthehardwareandsoftwarecomponentsofamodernsmartphoneinordertotestmobileappsinvarioussituationsandwithdifferenthardwareandsoftwarecombinations.InalatersectionofthischapterIexplainedthedifferentapptypesthatare
currentlyavailableonthemarket.Thedifferencesamonganative,hybrid,andWebappaswellastheirprosandconsshouldbeclearbynow.Thedifferentbusinessmodelsformobileappswereexplained.Theclosing
sectionofthischapterdealtwiththedifferentmobileappstoresthatareavailableforeachplatformandwhatisimportanttoknowaboutthem.
Chapter3.ChallengesinMobileTesting
InChapter1,“What’sSpecialaboutMobileTesting?”Idescribedtheuniqueaspectsofmobiletesting,coveringuserexpectations,datanetworks,mobiledevices,andmobilereleasecycles.Butthereare,ofcourse,othertopicsthatmakemobiletestingaspecialandchallengingjob.Thischaptercontainsmoremobiletestingchallengestogetherwiththeir
possiblesolutions.
TheCustomerAsImentionedinChapter1,customersandtheirexpectationsareoneofthemainchallengesformobiledevelopersandtesters.Tosatisfycustomers,it’sreallyimportantthatyougatherinformationabout
yourpossibletargetcustomersandtheirneeds.Ifyoureleaseanappwithoutanykindofknowledgeofyourtargetgroup,theappwillmostlikelynotbeinstalledoritwillreceivereallybadreviews.Thisleadstofewerdownloads,andcustomersmayevendownloadanappfromyourcompetitor.Inordertohandlethecustomerchallenge,youneedtogatherasmuch
informationaboutyourpotentialusersaspossible.This,inturn,meansthatyouneedtoincorporatethespecificsofyourtargetgroup,suchasage,gender,andgeographicalbackground,duringthedevelopmentandtestingprocess.Youneedinformationsuchasthefollowing:
•Gender•Age•Monthlyincome(importantforpossibleappbusinessmodels)•Educationalbackground•Geographicalbackground(e.g.,dotheyliveinacity?)•Whatappstheyuseandwhatkind•Smartphonehabits(howoftentheychecktheirsmartphone)•Whethertheyuseacompetitor’sappsandifso,how,andaretheyhappywiththem•Whatdevicestheyuse
Important
Becarefulwhenaskingpeoplepersonalquestionsasyoucouldendupinfringinguponprivacylaws.
Anotherwayofgettinginformationaboutyourtargetgroupistoconductinterviews:inviteyourcustomerstoyourofficeandaskthemabouttheirmobileusageandhabits.Ifyouencounterproblemsgettingtheanswerstoallofthesequestionsorarenotallowedtoaskyourpotentialcustomersdirectly,youcanuseservicessuchasMobilePersonas1togetgeneralinformationaboutthebehaviorofmobileusers.
1.www.mobilepersonas.com/
Ifyougather,analyze,andgroupthatkindofinformation,youwillmostlikelyhavemorethanonetypeofpersontheappwillbemadefor.Tosatisfythedifferenttypesofcustomers,itishelpfultocreateso-calledpersonas2torepresenttheirvariousneeds.PersonaswereintroducedbyAlanCooper3in1998inhisbookTheInmatesAreRunningtheAsylum.
2.www.usability.gov/how-to-and-tools/methods/personas.html3.www.cooper.com/journal/2008/05/the_origin_of_personas
Personasarefictionalcharactersthatarerepresentativeofyourrealcustomers.Personasareacommonandveryusefulwaytoidentifyusermotivations,expectations,problems,habits,andgoals.Personascanhelpyoutomakedecisionsaboutyourapp,suchasfeatureset,
possiblegestures,anddesign.Personascanhelpamobileteamgetafeelfortheircustomers’needsandproblems.Youasamobiletestercanalignyourdailyworkflowtothepersonadescription.Table3.1presentsanexampleofatypicalpersona.
Table3.1PossiblePersonaDescription
Youcanalsowriteapersonadescription,suchasthisone:Martinisa28-year-oldtech-savvymalewithamaster’sdegreeincomputerscience.HelivesinNewYorkandisafrequentsmartphoneuserwhocheckshissmartphonearound150timesaday.MartinusesappslikeTwitter,Facebook,Spotify,andTumblr.Hismonthlyincomeisabout$3,000.Martinisafriendly,smart,andpolitepersonwholikestomeetfriends.
Withtheaidofpersonasandknowledgeoftheirhabits,it’sabiteasiertotestamobileappasthetestingprocessismorefocusedoncustomerneedsthanonthetester’sexpectationsorhabits.Personasareagoodwaytohandlethecustomerchallengewhenitcomesto
mobiletesting.Oncethepersonasareinplace,themobileteamshouldtrytofindrealcustomerswhomatchthepersonas.Onceyou’vefoundsomecustomers,talktothemandaskthemquestionsaboutyourappanddiscusspossibleadditionalfeatures.Inviteuserstoausabilitytestingsessioninyouroffice.Thiswaytheusersfeelconnectedwiththecompanyandyourapp.Anotherwayofgettinguserfeedbackorengagementisabetatestingphase.Thisisacommonapproach;somebigmobileappvendors4invitecustomerstotestbetaversionsofnewappstocollectfeedbackatanearlydevelopmentstagesotheycanimprove
theappbeforegoinglive.4.www.sonos.com/beta/screen/
Othervaluablesourcesofinformationaboutyourcustomersaretheappstoresofthedifferentvendorsandtheratingsforandcommentsaboutyourapp.Readthecommentscarefullyandlearnfromthem.Ofcourse,theremaybelotsofcommentsthatdon’tdeliveranyusefulinsights.However,thereareuserswhocomplainaboutusability,bugs,orproblemsyouhaveneverheardof,andthosecommentsareextremelyusefulforyouandyourteam.Moreinformationaboutappstorereviews,ratings,andcommentscanbefoundinChapter7,“MobileTestandLaunchStrategies.”
CustomerSummaryDothefollowinginordertomaximizethechancesthatyouwillsatisfyyourcustomers:
•Gatheruserinformation.•Createpersonas.•Usepersonaswhiletesting.•Invitecustomerstousabilitytests.•Interviewcustomersaboutyourproduct.•Invitecustomerstobebetatesters.•Checktheappstorereviewsandlearnfromtheusefulcomments.
MobilePlatformsandFragmentationInthepreviouschaptersIexplainedthedifferentmobilevendorsandmobileplatforms,soyounowknowwhat’sinsideamobiledevice.Youalsoknowthatforsomemobileplatformsthereismorethanonemobiledevicemanufacturer.Fragmentationisahugeprobleminthemobileworldandespeciallyinthe
Androidworld.BasedonthenumbersfromOpenSignal,5nearly19,000Androiddevicesareavailableonthemarket.It’ssimplynotpossibleand,asyouwillsee,notnecessarytotestonallofthosedevices.Thisproblemisn’tjustlimitedtoAndroid;othermobileplatformssuchasiOS,WindowsPhone,andBlackBerryarealsoaffectedbyfragmentation.Thepossiblehardwareandsoftwarecombinationsonthoseplatformscanalsobeaproblem.
5.http://opensignal.com/reports/2014/android-fragmentation/
Thenextseveralsectionsofthischapterpresentsomesolutionsforhandlingfragmentationwhiletesting.
MobileDeviceGroupsOnesolutionforhandlingdevicefragmentationinyourmobiletestingprojectistogroupyourmobiletestdevices.Ifthedevicesaregrouped,youcantestononlysomedevices(onetothree)fromeachgroup,whichhelpsyoudownsizetheamountoftestingwork.I’veadoptedthisapproachinallofmymobileproj-ectsandithasprovenveryefficient.Basedonyourtargetcustomergroup,youcancreatethreedevicegroups(thisexampleassumesthatthetargetgroupisreallytechsavvy).Thefirstgrouphasthehighestpriority:A.Devicesinthisgrouparemost
likelytobenewdeviceswithpowerfulhardwareandabigscreenwithahighresolutionandpixeldensity.Theyalsousuallyhavethelatestoperatingsystemversioninstalled.Devicesinthisgroupmustbefullysupportedbyyourappintermsoffunctionality,design,andusability.
•Group1,PriorityA:•High-enddevices•Dual/quad-coreCPU•RAM>=2,048MB•Displaysize>=5″•Retina,fullHDdisplay•Latestoperatingsystemthatisavailableforthedevice
Thesecondgrouphasmediumpriority:B.DevicesinthisgrouparemidrangedeviceswithaveragehardwaresuchasasmallerCPU,screenresolution,andsizethanthedevicesingroupA.Theoperatingsystemversionisprobablylessthanoneyearold.Thedevicesinthisgroupshouldfullysupporttheappintermsoffunctionalityandusability.Thedesigndoesn’tneedtobeperfectforthisgroupduetothesmallerscreens.
•Group2,PriorityB:•Midrangedevices•Dual-coreCPU•RAM<2,048MB•Displaysize<5″•NoRetinaorfullHDdisplay•Softwarelessthanoneyearold
Thethirdgrouphaslowpriority:C.DevicesinthisgrouphaveasmallCPU
andasmallscreenresolutionanddensity.Thesoftwareversionismorethanoneyearold.Devicesinthisgroupstillhavetofullysupporttheappintermsoffunctionality,butthedesignandusabilitymaydifferfromtheothergroupsbecausethehardwaremaybetooslowtoprovidesufficientresponsiveness.
•Group3,PriorityC:•Slowdevices•Single-coreCPU•RAM<1,024MB•Displaysize<4″•Lowscreenresolution•Operatingsystemmorethanoneyearold
Onceyouhavedefineddevicegroups,youneedtomakesureyoukeepthemcurrentbymonitoringthemobiledevicemarketfornewdevicesthatmatchyourtargetgroups.Ontheotherhand,youcanremoveolderdevicesfromyourdevicegroupsifyourcustomersdon’tusethemanymore.Andlastbutnotleast,youneedtocheckyourgroupcriteriafromtimetotimetomakesuretheystillsufficientlycoveryourcustomerspectrum.Withtheaidofsuchdevicegroups,you’llfinditmucheasiertohandledevice
fragmentationandhavetherightdevicesfortesting.AniceWebpageprovidedbyGoogleis“OurMobilePlanet,”6whereyoucan
getinformationbasedonthecountry,theage,thegender,andthebehaviorofusers.Suchinformationcanbeusedwhenthetargetcustomerisunknown.
6.http://think.withgoogle.com/mobileplanet/en/
ImportantDevicegroupsmayvarygreatlyfromprojecttoproject!
MobileDeviceLabsDependingonthemobileappproject,youmayneedlotsofdevicesfortesting,whichisofcourseveryexpensiveandtime-consuming.Agoodalternativetobuyingallofyourtestingdevicesistorentthem.Youcanusemobiledevicelabsordevicecloudstorentthetestingdevices
youneed.However,beforeyourenttestdevices,keepthedevicegroupinginmindtodownsizetheamountoftestingrequiredforallthosevirtualandphysicaldevices.
Currentlythereareplentyofmobiledevicelabprovidersthatoffermobiletestdeviceswithinthecloud.Mobiledevelopersandtestersareabletouploadtheappfiletothecloud,selectthedevices,andstartmanualorautomatedtesting.Themainadvantageofsuchservicesisthatyoudon’thavetoworryabout
buyingnewdevicesandmaintainingthem.However,beforechoosingamobilecloudprovider,youshouldcheckwhethertheproviderofferstestingonrealdevices,simulators,oremulators.Amobilecloudtestingprovidershould:•Testonrealdevices•Beabletouseemulatorsorsimulatorsaswell•Notusejailbrokenorrooteddevices•Beabletotestondifferentmobileplatformsifpossible•Beabletowritetestautomationscriptsindifferentprogramminglanguages•Offercontinuousintegration•Havesomeperformancemonitoringinplace•Generateatestreportattheendofthetestcycle
Testingyourmobileappinthecloudonemulatorsorsimulatorscanbeveryannoyingduetoperformanceissues.Usingemulatorsorsimulatorsonalocalmachineisoftenalreadyfairlyslow;usingthemontheInternetviaaWebGUIcanbeevenslower.Eveniftheproviderofferstestingonrealdevices,it’sstillpossiblethatthe
performancemaynotbethesameaswhentestingonalocalphysicaldevice.Thelatencymaybetoohigh,whichcanleadtoveryslowreactionsonthedevice.Scrolling,tapping,orjustloadingtheappcantakealongtime,whichmayhaveabigimpactonyourtestingandthesubsequentresults.Anotherthingtolookoutforwhenchoosingacloudprovideristomakesure
youhaveexclusiveaccesstothephysicaldevicesandthatyourappisdeletedoncethetestsessioniscomplete.Checkthattheprovideroffersaprivatetestcloud;otherwiseyourappmaystillbeinstalledonthetestdevicesforsubsequentcustomerstosee.Discoveringsuchanissuewhileevaluatingaproviderisaclearwarningsignthatshouldnotbeignored.Ifsecurityandprivacyareimportanttoyou,considerpickinganotherprovider.Thefollowingprovidersofferamobiledevicetestcloud:•AppThwack(https://appthwack.com/)
•CloudMonkeyLabManager(www.cloudmonkeymobile.com/labmanager)•KeynoteMobileTesting(www.keynotedeviceanywhere.com/)•MobileLabs(http://mobilelabsinc.com/)•PerfectoMobile(www.perfectomobile.com/)•SauceLabs(https://saucelabs.com/)•Testmunk(www.testmunk.com/)•TestObject(http://testobject.com/)•TouchTest(www.soasta.com/products/touchtest/)•XamarinTestCloud(http://xamarin.com/test-cloud)
ImportantThislistisbynomeanscomplete.JustbrowsetheInternettosearchformoreprovidersthatfitinyourenvironment.
Ifyoudon’twanttobuyeverytestingdeviceforyourcompanyordon’twanttouseacloudprovider(ormaybeyou’renotallowedtouseone),thereisanotherwayofgettingphysicalmobiledevices:ODLs,orOpenDeviceLabs.OpenDeviceLabswereintroducedbythemobilecommunitywiththeaimof
establishingphysicalstoreswheremobiletesters,developers,andanyoneinvolvedinthemobilebusinesscanobtaindevicesfortestingpurposes.TheprimarygoaloftheseOpenDeviceLabsisthateveryonecanborrowdevicesforfree!Inordertomakethisideasuccessful,OpenDeviceLabsneeddevicedonationsfromindividualsorcompanieswhowanttosupportthemobilecommunity.Dependingonyourgeographicalregion,youmighthaveanOpenDeviceLabnearby.TheWebsite7providesanicemapthatshowsyouwhereyournearestOpenDeviceLabislocated.Youshoulddefinitelytrythem!
7.http://opendevicelab.com/
Chapter7includesmobiletestlabsandOpenDeviceLabsaspartofthemobiletestingstrategyandalsoprovidesfurtherinformationonthistopic.Inthispartofthechapteryouhavelearnedthattherearethreepossible
solutionstotheproblemofhowtohandlemobilefragmentation:•Groupingthedevicesyouneedfortesting•Mobiledevicelabsinthecloud•UsingphysicaldevicesfromanOpenDeviceLab
SensorsandInterfacesEverysmartphonehasavarietyofsensorsandinterfacesthatcanbeaccessedbytheinstalledappstoprovideusefulfeaturestousers.Theactualsensorsandinterfacesuseddependontheapp’sspecificusecase.Theinstalledsensorsandinterfacesarereallycomplexandarepotentiallysusceptibletoelectricalinterference.Asamobiletesterit’syourresponsibilitytoensurethattheimplemented
sensorsandinterfacesareusedcorrectly.It’salsoimportanttoverifythatfailingsensorsdon’taffecttheappinanegativeway.
AmbientLightSensorTheambientlightsensorisabletodeterminehowmuchlightisavailableinthecurrentlocationandautomaticallyadjustthescreen’sbrightnessbymeansofsoftwareinordertoprolongthedevice’sbatterylife.Ifyourappmakesuseoftheambientlightsensorinanyway,youshouldtest
yourappindifferentlocationswithdifferentprevailinglightsituations.Forexample,testyourappinadarkroom,outsideinthesunshine,oratyourworkplacetoseeiftheapprespondscorrectly.Becreativewithyourtestingenvironments.Differentlocationscouldinclude:•Adarkroom•Yourworkplacewithadesklampon•Outsideinthesunshine•Roomswithdifferentlightsontheceiling
ProximitySensorAnotherlittlehelperistheproximitysensor,whichdetermineshowclosethephoneislocated,suchastoafaceorsurface.Itcanbeusedtoautomaticallyturnoffthedisplaywithoutanyphysicalcontact.Thispreventsyoufromaccidentallyclickingortouchingbuttonsonanactivescreen.It’salsousefulforsavingbatterylife.Makesureyoualsotesttheusageoftheproximitysensorindifferent
locationswithdifferentlightsituationsaswellaswithdifferentkindsofsurfacestoseewhethertheappisusingthesensorcorrectly.Differentlocationsandsurfacescouldincludethefollowing:
•Adarkroom•Yourworkplacewithadesklampon•Outsideinthesunshine•Yourhandmovingoverthedisplay•Yourhandclosetothedisplay•Thedevicemovingtowardyourface•Thedevice’sdisplaymovingtowardamirrororpaneofglass
AccelerationSensorTheaccelerationsensordetectschangesinthedevice’smovement.Themostcommonusecaseforthissensoriswhenthedeviceisrotatedbetweenportraitandlandscapemodes.Thissensorisusedinnearlyeveryappifthedevelopersimplementedaportraitandlandscapeview.Ifbothmodesaresupported,youshouldchangetheorientationofthedevicealotduringtesting.Bydoingso,youwillprobablyfindlotsofUIglitchesbecausetheUIelementscouldbemovedtoadifferentposition.Itisalsolikelythattheappwillcrash,forexample,whendataisfetchedfromthebackendwhileaUIrefreshisbeingperformed.
GyroscopeSensorThegyroscopesensorisusedtoeithermeasureormaintaintheorientationofthedevice.Unlikeanaccelerometer,whichmeasuresthelinearaccelerationofadevice,agyroscopemeasuresthedevice’sexactorientation.Thismeansthatthedeviceisabletodetect360-degreemotion.Thankstotheaccelerometerandgyroscopesensors,thedeviceisabletooperateonsixaxes—leftandright,upanddown,forwardandbackward—andtrackroll,yaw,andpitchrotations.Thecombinationofbothsensorsismostlyusedingamingappssuchasflight
simulationstocontrolaplanewithrealphysicalmovements.Keepthesixaxesinmindwhentestingyourapp,andtesteachaxisseparatelyaswellasmultipleaxesatoncetobesuretheappisrespondingcorrectly.
MagneticSensorAmagneticsensorisabletomeasurethestrengthandorientationofmagneticfieldsaroundthedevice.Thissensorismostlyusedbyappsthatrequirecompassinformation,suchasfornavigation.Withtheaidofthissensorthedeviceisabletodeterminethedirectioninwhichit’sfacing:west,east,north,orsouth.Ifyourappusesthemagneticsensor,youshouldalsotestitindifferent
locations.Forexample,ifyou’reusingtheappinabuildingwhereyou’resurroundedbylotsofmetal,themagneticsensormaydeliverfalseinformationordata,whichcanleadtoadversesideeffectsinyourapp.Differentlocationscouldincludethefollowing:•Insideabuilding•Outsideonthestreet•Whenstuckintraffic•Nearbuildingswithlotsofmetal
Pressure,Temperature,andHumiditySensorsThesethreesensorsarenotincludedineverysmartphoneyet,buttheywillbesoon.Allthreesensorscanbeusedtocollectmoredataabouttheuser’scurrentlocationtoprovideappswithusefulinformationsuchasthecurrenttemperature,currentaltitudebasedonatmosphericpressure,andhumidity.Thesesensorsareused,forexample,byoutdoororweatherapps.Again,make
sureyoutestthesesensorsindifferentlocationswithdifferenttemperatures,pressures,altitudes,andhumidities.Youcan,ofcourse,simulatepressureorhumidityinatestlabsituationtocheckthatthesensorsareworkingcorrectly,buttestingtheminrealsituationscouldproducesideeffectsthataren’tlikelytohappeninalab.
LocationSensorYoucertainlyknowthelocationsensor,betterknownasGPS.ThankstoGPS,appscandeterminethecurrentdevicelocation.GPSisusedinlotsofdifferentkindsofappssuchasmapapps,cameraapps,andsocialmediaapps.Userscansharetheircurrentlocationwithappsandsendtheircurrentposition,forexample,tofriendstoletthemknowwheretheyare.IfyourappusesGPStodeterminethedevice’scurrentposition,besurethat
theGPSsensorisswitchedoffafteruse.Otherwisethedevice’sbatterywillbeemptyverysoon.GPSfunctionalityshouldofcoursebetestedinvariouslocations,suchasin
thecountrysideorindowntownsprawlwithlotsofhugebuildings.InbothscenariosyoumayfindthatthecurrentlocationisincorrectornotfoundbytheGPSsensor.Thisisespeciallytrueincitiesduetointerferencecausedbysurroundingbuildings.IftheGPSsignalisnotstrongenoughtodeterminethecurrentlocation,checkthattheWi-Fiornetwork-basedlocationfindingfeature
ofthephoneisactivatedtolocatethephone.AlowGPSsignalcanalsocauseperformanceproblemswithinyourappduetodifficultiesincalculatingthecurrentposition.WhentestingtheGPSfunctionofyourapp,keepaneyeonthetemperatureofyourdevice.UsingGPSwillheatupyourdevice,consumelotsofbattery,andthereforepossiblyinfluencetheperformanceofyourapp.IalsorecommendthatyouturnGPSofftoseehowyourappreactstothisscenario.
TouchlessSensorAnothersensorthatisnotbuiltintoeverysmartphoneisatouchlesssensor.Inmostcasesthissensorisbuiltintothefrontofthedevicetoaccepttouchlessgesturesfromafingerorahand.Whatthismeansisthatyoucanswipebetweenphotosinyourphotogallerysimplybywavingyourhandoverthescreen.Oryoucanansweraphonecallbyholdingthedevicenexttoyourearwithouttouchingtheanswerbutton.Ifyourappsupportstouchlessgestures,besureeverygesturealsoworksat
differentangles.
SensorsSummaryMostofthesensorsjustdescribedareinfluencedbytheuser’scurrentlocationandsurroundings.Anykindofmovementisveryimportantwhiletestingyourappandthedevice’ssensors.It’salsousefultotestthesensorsincombinationwithoneanothertocheckthatyourapprespondscorrectlytochangesofsensordata.Possiblesensortestscouldincludethefollowing:•Walkwhileusingtheappasthissimulatesback-and-forthmotion.•Whilewalking,suddenlystopforafewmomentsbeforecontinuingonyourway.•Shakeandrotatethedevice.•Simulatevibrationsasifyou’reonatrainorinacar.•Performextrememovementssuchasspinningorrotatingthedevice.•Usetheappoutdoorsinsunny,cloudy,anddarklocations.•Usetheappindoorswithnormallightorunderalamp.•Waveyourhandsinfrontofthedevice.•Testtheedgecases,forexample:GPScoordinates0,0;temperatureat0degrees;waterproofdevicesinareaswithhighhumidity.
•Checktoseeiftheappcanbeusedondevicesthatdon’thavebuilt-insensorsandinterfaces.
Whileperformingthesetasks,watchoutforappcrashesorfreezes.CheckthattheUIelementsarevisibleanddisplayedcorrectlyonthescreen,especiallyindifferentlightsituations.Keepaneyeontheperformanceoftheappandthesmartphone’sbatterylifetimeaswell.
TouchscreenThebiggestandmostimportantinterfaceinsmartphonesisthetouchscreen.Thankstothetouch-sensitivescreen,userscanuseoneormorefingersonthedevicetocreategesturesthatthephoneisabletoconvertintocommands.
ImportantThedescriptionoftouchscreentechnologieshereissimplified.Ifyouwanttogetmoredetailedinformationaboutthem,pleasedosomeonlineresearch.
Therearebasicallytwokindsoftouchscreentechnologiesavailable.Thefirstistheresistivescreenthatismadeoutofvariouslayersandreactstopressure.Usuallythiskindoftouchscreenismadeforusewithastylus.Wheneveryouhavetosignforaparceldelivery,youprobablysignonaresistivescreen.Thistechnologyhasonebigdrawback:itdoesn’tsupportmultitouchgestures.Thatisthereasonwhythesecondtechnology,capacitivetouchscreens,isused
insmartphones.Capacitivescreensreacttotouchratherthanpressureandsupportmultitouchgestures.Capacitivescreensconsistofaninsulator,whichinmostcasesisglasscoated
withatransparentconductorlikeindiumtinoxide.Sincethehumanbodyisalsoanelectricalconductor,touchingthecapacitivescreenresultsinadistortionofthescreen’selectrostaticfield.Thisdistortionisthenconvertedintodatathatthedevicehardwareisabletounderstand.Thefollowinggesturesarepossibleonacapacitivetouchscreen:•Touch:Touchthescreenwithafingertip.•Longtouch:Touchthescreenforalongertime.•Swipe:Moveyourfingertipoverthescreen.•Tap:Brieflytouchthescreenwithafinger.•Doubletap:Brieflytouchthescreenwithafingertiptwice.
•Drag:Moveafingeroverthescreenwithoutlosingcontactwiththescreen.•Multitouch:Usetwoormorefingersonthescreenatthesametime.•Pinchopen:Touchthescreenwithtwofingersandmovethemapart.•Pinchclose:Touchthescreenwithtwofingersandbringthemclosertogether.•Rotate:Usetwofingersonthescreenandrotatethem.Someapps,suchasmapapps,willrotatethecontentintheapp.
Thevarietyofpossibletouchgesturesposesaspecialchallengewhiletestingamobileapp.Youshouldkeepallthepossibletouchgesturesinmindandusethemwhiletesting.Agoodwaytoseeifanappcanstanduptotouchgesturesistousemultiplefingersonthetouchscreenatthesametime.Youshouldalsoperformseveraldifferenttouchgesturesveryquicklyonthescreentoseehowtheappreactsandhandlestheinputs.AlsowatchoutforperformanceissuesorUIglitchesthatcanhappenwhileusingmultitouchgestures.Oneimportantfactortonoteabouttouchscreensisthecurrentweather
conditions.Theycanaffectyourfingersandpreventtouchesfrombeingregisteredproperly.Therefore,it’sagoodideatoalsousethedifferentgesturesindifferentweatherconditionssuchasoncoldorhotdaysoronadaywithhighorlowhumiditytoseehowtheappreactstoyourgesturesonthetouchscreen.
MicrophonesAnotherwayofcommunicatingwithyourappisbyusingyourvoiceorsound.Mostsmartphoneshavemorethanonemicrophoneinstalled.Usuallythereareuptothreemicrophones,oneatthefront,oneattheback(nearthecamera),andoneatthebottomofthedevice.Thethreemicrophonesensureverygoodvoicerecordingfromallpossibleanglesandregardlessofthephone’sposition.Whentestingsoundinputsviathemicrophone,makesureyoudothe
following:•Testvoiceinputindoorswithnormalsoundandinnoisysituations.•Testtheappindoorswithmorebackgroundnoisesuchasinanofficeorrestaurant.•Testtheappoutdoorswithbackgroundnoisescomingfromastreetorcars.•Checkthattheappcanhandletheusermutingandunmutingthephoneorthemicrophone.•Startotherappsthatgeneratesounds,suchasmusicapps,andcheckthe
behaviorofyourapp.•Usethevolumeupanddownbuttonstoincreaseordecreasethesoundlevelandcheckthebehavioroftheapp.•Checkthatvoiceinputsareprocessedcorrectly.•Ifthevoiceinputisstoredonthephone,checkthatplaybackisworkingproperly.•Testthevoiceinputinareal-lifeenvironment(thepurposeoftheapp).
Whiletestingthesoundinputsinallthedifferentsurroundings,watchoutforinputdelaysordistortionintherecordingorplaybackmode.Keepaneyeontheperformanceoftheappandthephonetoseeifthedevicefreezesorcrashesduetoalackofhardwareresources(especiallywhensavingtheinputs).Anddon’tforgettocheckthedigitalrightsmanagement(DRM)whenworkingwithvoiceorsoundinputs.
CameraAtypicalsmartphonehasatleastonecamera,usuallylocatedatthebackofthesmartphone.However,mostoftoday’ssmartphonesalreadyhavetwocameras:arear-facingandafront-facingcamera.Therearcameraisusedtotakehigh-resolutionpictures,andthefrontcameraismostlyusedforvideochattingwithalowerresolution.Also,mostoftherear-facingcamerashaveaflashincluded.Camerasareusedinavarietyofmobileappsfortakingpicturesorvideos.
SomeappsusethecameraasascannertocaptureinformationusingOCR(opticalcharacterrecognition)orotherkindsofshapes.GoodexamplesofthisincludeQR(QuickResponse)codes,scanningapps,orappsthatscanbusinesscardstotransformthemintodigitalcontactsinyourphone.SomeappsjustusetheLEDflashtoactasaflashlight.Ifyourappusesthecamera,testitwithavarietyofmobiledevicesthatmatch
yourtargetcustomergroup.Everysmartphonehasauniquecamerawithauniquelens,flash,andresolution.Thedifferentresolutionsofthecamerahaveanimpactontheimagesize.Thehighertheresolution,thebiggerthepicturesare.Besuretotestthecamerafunctionwithdifferentresolutionsandcameratypesinordertoseeiftheappisabletoprocessbothsmallandlargeimages.Besidesthat,youshouldchecktheperformanceoftheappwhilethecamera
modeisactivated.Thecameratakesupalotofthedevice’shardwareresources,whichcanhaveanadverseimpactontheappandcauseittocrashorfreeze.Heavycamerausecanheatupthecamerasensorandcancausehardwaredamage,sobesurethatthisisnotpossiblewithinyourapp.Finally,don’tforget
totestthattheappcanuseanimagestabilizertopreventthecamerafromtakingblurrypictures.Thesameappliestovideocapturingwithinyourapp.
SystemAppsMostpreinstalledappsfromdevicemanufacturersaresystemapps.Thereareoftenpreinstalledappssuchasacontactapp,aphoneapp,acalendarapp,andthelike.However,onmostofthemobileplatformsusersareabletoinstallappsthatreplacethesystemapps.Themainreasonsusersdothisisbadusabilityorthelackoffunctionalityonthepartofthesystemapps.Ifyouchecktheappstoresofthedifferentvendorsforcalendarapps,forinstance,you’llfindplentyofappswithfarmorefeaturesthanthepreinstalledones.Areallyinterestingexampleofthisiskeyboardappsonsmartphones.Such
appsarelikelytohaveanimpactonyourapp.OnAndroidphonesandtablets(and,sinceiOS8,alsooniOSdevices),usersareabletoreplacethepreinstalledkeyboardappwithspecialkeyboardsthatofferatotallydifferentwayoftypingandkeyboardlayouts.Thereareconventionaltapkeyboards(likeQWERTY),keyboardswithdifferentlayouts(multiplesymbolsperkey),keyboardswithtap-sliderfunctions,andkeyboardsthatofferaswipingmethodtoinserttext.Changingthekeyboardcanhaveaninfluenceonyourapp.Forexample,the
keyboardmaybemuchtallerthanthestandardoneandthereforehideUIelementsthatareimportanttointeractwith,orthescreenmightnotrecognizethatitneedstoscroll.Youmaycomeacrossimportantkeysthataremissingfromthekeyboard,thuspossiblyrenderingyourappunusableinsomecases.Itisalsopossibleforsettingstogetlostwithinyourphone,andyourappmightcausefreezesorcrashes.Thisisjustoneexampleofhowreplacedsystemappscanhaveanimpacton
yourapp.Ifyourappinteractswithoneofthepreinstalledsystemapps,keepinmindthatusersareabletoreplacethemwithotherapps.Youneedtohaveanoverviewofpopularappsthatreplacesystemappsinordertotesttheirintegrationandinteractionwithyourapp.
Internationalization(I18n)andLocalization(L10n)Anotherchallengethatneedstobehandledduringthemobiledevelopmentprocessistheinternationalization(I18n)andlocalization(L10n)ofmobileapps.I18nistheprocessofdesigningsoftwareapplicationsinsuchawaythattheycanbeadaptedtovariouslanguagesandgeographicalregionsthroughouttheworld,withoutchangingthecodebaseandwhilethesoftwareisrunning(some
appsandsomemobileplatformsrequirearestart).L10nistheprocessofadaptingtheinternationalizedsoftwaretoaspecific
languageorregionbyaddinglocale-specificelementsortranslatingthetext.It’sreallyimportanttotestbothI18nandL10nifyourappisdesignedtobe
usedindifferentcountries.YoumustbesurethatthedifferentlanguageswillnotbreakyourUIelementsorhaveaninfluenceontheapp’susability.
ImportantManylanguageshavetheirowncharacterset,andonewordcanhaveaverydifferentwidthandheight.
Havealookattheexamplewordlogout.IfyoucomparethewordwithtranslationsfromGermany,France,Turkey,andRussia,youwillseelotsofdifferenceincharacters,width,andheight:
•Logout(English)•Ausloggen(German)•Déconnexion(French)•Çıkışyap(Turkish)•(Russian)
BoththeGermanwordAusloggenandtheFrenchwordDéconnexionaremuchlongerthanLogout.TheTurkishversionevenconsistsoftwowords.Asasample,Asianlanguagescanbeusedas“short”languages,whileGermanandPortugueseareconsidered“long”languages.AlltranslationscanleadtoaUIglitchorevenbreakyourdesignrules.Whentestinganappindifferentlanguages,checkthateverytexttranslationwillfitintotheUIelementsandthateveryscreenhasthesamelook-and-feel.Alsocheckthatthedifferentcharactersandfonttypescanbestoredinthelocaldatabase.Thesameappliestodateformatsthatareusedwithinapps.Dependingonthe
countryorregioninquestion,differentdateformatsarealsoused.Youneedtocheckthatthecorrectformatisshowndependingontheregionsettingsofthephone.Youshouldalsotesttheparsingfromonedateformattoanother.Thefollowingdateformatsarecommonlyused:•DD/MM/YYYY(Day/Month/Year)•DD.MM.YYYY•DD-MM-YYYY
•MM/DD/YYYY(Month/Day/Year)•MM.DD.YYYY•MM-DD-YYYY•YYYY/MM/DD(Year/Month/Day)•YYYY.MM.DD•YYYY-MM-DD
Ifyourappisgoingtobeusedindifferentcountriesorregionsintheworld,makesureyouaddtheimportantlanguagesbasedonyourtargetcustomergroupinordertoprovideagooduserexperience.Whentestingtheapp,youneedtocheckitineverylanguage.Youalsoneedtocheckthatthelanguageswitchisworkingtobesurethatonlyasinglelanguageisshowndependingonthedevice’slanguagesettings.Alsocheckthefallbacklanguageofyourapp.Totestthis,changethedevicelanguagetoonethatisnotsupportedandcheckthattheappusestheimplementedfallbacklanguage.Languagesanddateformatsshouldneverbemixedupwithintheapp.Ifyouwanttotestiftheusedlanguageiscorrect,youshouldaskanativespeakerinordertoavoidusingmisleadingtranslationsortextthatissimplywrong.
ImportantThedifferentlanguagesanddateformatsmustbeconsideredduringthedesignphaseoftheappsothatdesignerscanplanthelook-and-feeloftheappinalloftheprovidedlanguages.Latechangesduetotranslationissuescandelaythereleasedateoftheappornegativelyaffecttheapp’sdesign.
MobileBrowsersIfyou’retestingamobileWebapp,youwillofcourseneedtodosoinamobileWebbrowser.MobileWebbrowsersareoptimizedtodisplayWebcontentsuchthattheycanbeaccessedonsmallerscreens.Unfortunately,thereismorethanonemobileWebbrowseravailableindifferentversionsofthedifferentmobileplatforms.Besidesthat,thebrowsersusedifferentlayoutenginessuchasthefollowing:
•Blink(www.chromium.org/blink)•Gecko(https://developer.mozilla.org/en-US/docs/Mozilla/Gecko)•Presto(www.opera.com/docs/specs/)
•Trident(http://msdn.microsoft.com/en-us/library/aa741312(v=vs.85).aspx)•WebKit(www.webkit.org/)
Dependingonthelayoutengine,browsersettings,andbrowserversion,mobileWebappsmaylookandbehaveverydifferently.Thisisespeciallythecaseforthedifferentbrowserlayoutengines.EverybrowserlayoutenginehandlesstandardslikeHTML,CSS,andJavaScriptdifferently.Noteverybrowserhasimplementedthecompletefeaturesetorthelatestversionofthevariouslanguages(HTML,CSS,JavaScript),whichcanleadtodifferencesinbehavior.TobesurethatyourmobileWebappworksondifferentbrowsers,testiton
differentmobileplatformssuchasAndroid,iOS,WindowsPhone,orBlackBerrytogetherwithdifferentbrowserversions.Thisisofcoursethesameproblemnativeappshave:fragmentation.However,testingmobileWebappscanbemoredifficultbecausenowyouhaveanothervariableaddedtoyourtestingmatrix:thedifferentWebbrowsers.Thismeansthatyouhavetofocusonthedifferentbrowserversions,mobileplatforms,andoperatingsystemversions.ThemobileWebbrowserslistedinTable3.2areavailableforthedifferent
mobileplatforms.Toseewhichbrowserlayoutengineisusedbywhichbrowserversion,checkthebrowservendorWebsites.
Table3.2OverviewofMobileBrowsers
ImportantNoteverybrowserisavailableoneveryplatform.Table3.2showsthemostusedmobileWebbrowsers.8Thetableisnotcomplete.
8.http://akamai.me/1EQZbP0
Asyoucansee,testingamobileWebappondifferentmobilebrowserscanalsobearealchallenge.TokeeptheamountoftestingworkneededformobileWebbrowserstoaminimum,youshouldcreatemobilebrowsergroupsoraddmobilebrowserrequirementstothedevicegroups.Ifyoucreatebrowsergroups,prioritizethembasedonyourtargetcustomergroupandonlytestyourWebapp
withinthesegroups.
SummaryChapter3wasoneofthefirsthands-onchaptersofthisbook.Youarenowabletocreateyourownpersonasbasedontheinformationaboutyourtargetcustomergroupandtheusageofyourapp.Thosepersonaswillhelpyoutofocusyourdevelopmentandtestingeffortsonyourtargetgroupandnotwastetimeonunnecessaryfeatures.Inadditiontopersonas,younowknowhowtohandledevicefragmentationin
themobileworld.Testingyourapponeverymobiledeviceisnotpossibleoreconomical.Basedonyourtargetgroup,youcandefineso-calledmobiledevicegroupstotestonlyonthedeviceswithineachgroup.Thiswillhelpyoutodownsizethetestingeffortdramatically.Anotherapproachtohandlingdevicefragmentationistousemobiledevicelabs.Asyouknow,mobiledevicesarepackedwithlotsofsensorsandinterfaces,
andifyourappusesthem,youneedtotestthemaswell.Testingthedifferentsensorsandinterfaceswasanotherbigtopicinthischapter.Foreachsensorandinterface,testingideaswereprovided.Whenyourappisavailableindifferentcountriesandsupportsdifferent
languages,thesection“Internationalization(I18n)andLocalization(L10n)”shouldhelpyouremembertotestyourappagainstthoselanguagesandsettingsrelatedtolocationsfromaroundtheworld.Thechapterclosedwithanoverviewofthedifferentmobilebrowsersthatare
currentlyavailableonthemarket.
Chapter4.HowtoTestMobileApps
Inthepreviousthreechaptersyoulearnedalotaboutthemobileappworld,rangingfromthedifferentnetworktypes,apptypes,businessmodels,appstores,customerexpectations,andchallengesformobiletesterstodevice-specifichardwareelements.Inthischapteryouwilllearnhowtotestmobileapps.Thisisahands-onchapter,andIsuggestthatyouhaveatleastonedevicewithanappofyourchoicenexttoyouwhilereadingit.Useyournewlyacquiredknowledgetotesttheappandseeifyoucanfind
bugsorotherdiscrepancies.
Emulator,Simulator,orRealDevice?Beforeyoustarttotestanapp,there’soneimportantquestiontoanswer:Areyougoingtotestitonarealdevice,inasimulator,orinanemulator?Mobiledeviceemulators,suchastheAndroidEmulator,1aredesktop
applicationsthattranslatetheinstructionsofthecompiledappsourcecodesothattheappcanbeexecutedonadesktopcomputer.Theemulatoractsexactlylikethemobiledevicehardwareandoperatingsystem,thusallowingthedeveloperandtestertodebugortesttheapplication.Sincetheappisexecutedonacomputer,notallofthemobile-specifichardwareelementssuchasthesensorsortouchgesturescanbeemulated.However,emulatorscanbeveryusefulatanearlystageofthedevelopmentprocessinordertoobtainquickfeedbackabouttheimplementedfeatures.
1.http://developer.android.com/tools/help/emulator.html
Simulators,suchastheiOSsimulator,2arelesscomplexsoftwareapplicationsthatsimulateasmallsubsetofthedevice’sbehaviorandhardware.Incontrasttoemulators,simulatorsareonlysimilartothetargetplatformandsimulatetherealdevice’shardware,makingthemmuchfasterthanemulators.Itisalsonotpossibletotestdevice-specifichardwareelementswithsimulators.However,simulatorsareusefulatanearlystageofthedevelopmentprocessinordertoobtainfeedbackabouttheimplementedfeatures.
2.https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html
Thebiggestdifferencebetweenasimulatorandanemulatoristhatasimulatorattemptstoduplicatethebehaviorofthemobiledevice,whileanemulatortriestoduplicatetheentireinnerarchitectureofthemobiledeviceandistherefore
closertothetargetplatform.Dependingonthemobileplatform,vendorsprovideeitherasimulatororanemulator.AppleandBlackBerry(ResearchinMotion)offerasimulator;GoogleandMicrosoftprovideanemulator.Asyouhavelearnedinthepreviouschapters,mobiletestingrequires
movementanddifferenthardware,meaningthatyouneedtotestyourapponphysicaldevicestobesurethateverythingworkstogetherinreal-lifesituations.Emulatorsandsimulatorsshouldbeusedonlyforverybasictestssuchas
simplefunctionality(isthebuttonclickable?)ortomakesurethelook-and-feeloftheappisOK.
ManualTestingversusAutomatedTestingThere’sanotherimportantdecisiontomake:Areyougoingtotesttheapppurelywithautomatedtests,purelywithmanualtests,orwithacombinationofboth?Thisdecisiondependsonyourapp.Simplyperformingtestautomationwillnotworkandisnotsufficientfor
severalreasons.Noteverymobile-device-specificfunctioncanbeautomated—forexample,locationdata—andotherenvironmentalsensordataisreallyhardtotestinalabsituation.Becauseoftheselimitations,yourappwillmostlikelycontainlotsofbugsandproblemsthatyourcustomerswillfind.Onlyperformingmanualtestingcanwork,butit’salsonotsufficient.You
shouldperformonlymanualtestsofyourappifitmeetsthefollowingcriteria:•Yourappisverysimpleandbasic.•Yourapphasonlyverylimitedfunctionality.•Yourappisavailableforonlyalimitedtimeintheappstores.
Inallothercasesyoushouldcombinemanualandautomatedtesting.Beforeperformingtestautomation,youshouldalwaysdomanualtesting.Everynewfeaturemustbemanuallytestedsystematicallyondifferentdevices.Onceyou’vecompletedmanualtesting,youcanthendefinethepartsoftheappthatrequiretestautomation.InChapter5,“MobileTestAutomationandTools,”Idescribethedifferent
mobiletestautomationconcepts,explainhowtoselectamobiletestautomationtool,andprovideyouwithanoverviewofpossibletools.
“Traditional”TestingMobileapplicationsaresoftwareapplications.Besidestestingthemobile-specificfunctionsandelements,itisstillnecessarytotestmobileappsinthe
samewayyoutestWebordesktopapplications.Youstillneedtodesignthetestcases,managethetestdata,andofcourserunthetests.IfyoulookatFigure4.1,youcanseethetypicalactions/stepsthatshouldbe
takenduringthesoftwarequalityassuranceprocess.Therearetwotypesofsoftwarequalityassurancemeasures:productfocusandprocessfocus.Theproductfocusphaseisusedtofindbugs,whereastheprocessfocusphaseaimstoprotectthesoftwarefrombugs.
Figure4.1Softwarequalityassurancemeasures
Theprocessfocusphaseofsoftwarequalityassurancecoversthemethods,languages,tools,guidelines,standards,andprocesseswithwhichthesoftwareisdeveloped.Inthisphase,softwaretestersareresponsibleformakingsure,forexample,thattheguidelines,developmentmethods,andtoolsaswellasthestandardsaremetduringthewholedevelopmentprocess.Theyshouldguideandhelpdevelopersandotherteammembersinfulfillingthedefinedprocessesinordertopreventanybugsfromarising.Theproductfocusphaseisdividedintostatictestinganddynamictesting.Instatictesting,thesoftwareisnotexecuted.Testersanddevelopersshould
performreviewsduringthisstageofthesoftwaredevelopmentprocess,eithercodereviewsbeforecommittingthecodetoarepositoryorreviewingthedocumentsandspecificationsbeforedevelopmentstarts.Duringthestaticphase,theapplicationcodewillbecheckedwithtoolstoseeifitmatchescoding
guidelinesorcontainsanymemoryleaksorraceconditions.ThefollowinglistcontainssomeJavastaticanalysistools:•Checkstyle(http://checkstyle.sourceforge.net/)•FindBugs(http://findbugs.sourceforge.net/)•Lint(http://tools.android.com/recent/lint)•PMD(http://pmd.sourceforge.net/)
ImportantTherearestaticanalysistoolsavailablefornearlyeveryprogramminglanguage,justhavealookforthemontheInternet.
Indynamictesting,theapplicationcodeisexecutedtoseehowtheapplicationbehaveswhilebeingrun.Dynamictestingisagainsplitintotwotypes:whiteboxtestingandblackboxtesting.Whiteboxtestingisawayoftestingsoftwarewithknowledgeoftheinternal
structureofmethodsandclasses.Whiteboxtestingisusuallydonebydeveloperswhotesttheircodeonaunitlevel,inmostcasesusingtoolssuchasJUnit,3TestNG,4XCTest,5orsimilarunittestingtools.
3.http://junit.org/4.http://testng.org/doc/index.html5.https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/Introduction/Introduction.html
Whiteboxtestingtechniquesincludethefollowing:•Statementcoverage•Pathcoverage•Branchcoverage•Decisioncoverage•Controlflowtesting•Dataflowtesting
Blackboxtestingisawayoftestingsoftwarewithoutanyknowledgeoftheinternalstructureofthemethodsorclasses.Thisapproachisusuallydonebysoftwaretesterswhoneedtoknowwhatthesoftwareshoulddoanddon’tneedtoknowhowthesoftwareactuallydoesit.Writingtestcasesandplanningtestingactivitiesareimportantpartsofblackboxtesting.Therearetestdesigntechniquesthatshouldbeconsideredinordertodeveloptherighttestcases,for
example,boundaryvalues,decisiontables,andequivalenceclasspartitioning.Boundaryvaluetestingwasdevelopedfromtheexperiencethatbugsoccuron
theboundariesandedgesofconditionsandlists.Thesebugsmaycausethesoftwaretocrash,freeze,orbehaveincorrectly.Decisiontableswillhelpyoutosplitcomplexapplicationsandconditionsinto
smallersections,whichpreventsyoufromforgettingimportantfeatures.Theyhelpyouobtainanoverviewofthecriticalelementsofthefeatureset.Equivalenceclasspartitioningisatechniqueusedtodefinetestcasesthat
uncovererrorclasses.Equivalenceclasspartitioninghelpsyoureducethenumberoftestcasesthatneedtobedevelopedduringyourtestingactivities.Thisapproachistypicallyusedtotestpossibleinputsoftheapplication.Moreexamplesofblackboxtestingtechniquesfollow:•Cause-effectgraph•Errorguessing•Statetransitionanalysis•All-pairstesting•Exploratorytesting
ImportantThissectiongivesjustaverybriefoverviewofpossibletestingmethodsandtechniques.Iroughlysummarizedthefoundationsofsoftwaretestingandanalysis,soifyou’renotfamiliarwiththedifferenttesttechniques,methods,andapproaches,IhighlyrecommendthatyoureadsomesoftwaretestingbooksortakeatestingclasssuchasISTQB6foundationlevelorBlackBoxSoftwareTesting.7
6.www.istqb.org/7.www.testingeducation.org/BBST/
Mobile-SpecificTestingInthispartofthechapteryouwilllearnhowtotestyourappindifferentcategoriesandensurethatitisrobust,stable,usable,andasfreeofbugsaspossible.PleaseusetheknowledgeyougainedinChapter3,“ChallengesinMobileTesting,”aswellwhentestingyourapp.Keepthesensors,networks,gestures,customers,languages,andsystemappsinmind.
Beforeyoucontinuereading,getyourmobiledeviceoutofyourpocketandstartanappsothatyoucantryoutthethingsyou’reabouttoread.
MobileFunctionalTestingTestingthefunctionalityofyourappshouldbethefirstthingyoudo.Executethedesignedtestcaseswithyourmobileapptomakesurethatthefeaturesandrequirementshavebeencorrectlyimplemented.Besidestestingyourtestcases,itisimportantthatyoualsotestthedefinedacceptancecriteria.Testtomakesurethatallthefunctionsperformastheyshould,forexample,
inputs,outputs,tappablebuttons,navigation,anddataprocessing.Testthefunctionalityindifferentmobileuserscenariosandenvironments.Keepthesensorsandinterfacesofamobiledeviceinmindwhiletestingthefunctionality(seeChapter3again).Furthermore,usethequalityassurancemeasurestotestthefunctionalityof
yourappinastaticanddynamicway.
ImportantTestingthefunctionalityisoneofthemostimportantaspectsofeverysoftwareproject.Dependingonyourapp,functionaltestingwillbedifferentineveryproject.
MobileUsabilityTestingUsability,especiallymobileusability,isahuge,complex,andimportanttopic.Withinyourmobiledevelopmentteamyoushouldhaveausabilityexpertwhocoversthispartofthedevelopmentprocess.Goodusabilityrequireslotsofrefining,intenseuserresearch,andevenmoretestingwithrealusers.Itisimportantthatyourappbeeasytouse;otherwiseitmightendupwithlowratings,whichcausedamagetotheapp’sandpossiblyyourcompany’sreputation.Toavoidsuchreputationdamage,mobiletesterscanperformtheactionsdescribedinthispartofthechapterinordertohelptheteamdevelopausableapp(keepKIFSUinmind).Thefollowinglinksareagoodstartingpointifyouwanttolearnmoreabout
mobileusability:•Usabilityheuristics(www.nngroup.com/articles/ten-usability-heuristics/)•Mobileusability(www.nngroup.com/articles/mobile-usability-update/)•GoogleBestPractices(www.google.com/think/multiscreen/#mobile-best-
practices)•Usabilityprinciplesandtechniques(www.usability.gov/what-and-why/index.html)•Mobileuserinterfaceandusabilitydesignprinciples(www.neobytesolutions.com/mobile-user-interface-and-usability-design-principles/)
Oryoucanrefertothefollowingbook:•MobileUsabilitybyRalucaBudiuandJakobNielsen(www.nngroup.com/books/mobile-usability/)
LessIsMoreWhentestingtheusabilityofanapp,thedesignprinciple“Lessismore”isagoodstartingpoint.Checktheappforuselessnavigationelementsandbuttons.Ifyoufindany,talktoyourteamortheusabilityexpertandgetthemremoved.Thesameappliestotextandwords.Removethemiftheyarenotrequiredtointeractwiththeapp.Trytofocusontheprimarycontentoftheapp.Keepthegoalandtheproblemtheappshouldsolveinmind,andgetridofanythingthatisdistracting.
Self-ExplanatoryCheckiftheappisself-explanatory.Askyourselfthesequestions:Istheuserabletoseetheelementswithwhichheorshecaninteract?Iseveryelementonthescreenclearandunderstandable?Thisaspectofusabilitytestinggetsmoredifficultovertimebecausethemoreyouworkwiththeapp,themoreblindyouwillbecometothingsthatarenotself-explanatory.Youwillmostlikelybeusedtopotentialusabilityissues.Inthatcaseitmakessensetoaskcolleaguesnotinvolvedintheappdevelopmentprocessoryourcustomersforearlyfeedback.
PayAttentiontoPatternsEveryappshouldfollowausabilityanddesignpattern.Ifsuchapatternisinplace,verifythatyourappfollowsthoserules.Ifthereisnopatternavailable,keepaneyeonthelook-and-feeloftheapp.Sametype-elementssuchasbuttonsortextlabelsshouldhavethesamespacings,sizes,andcolors.Checkthatalloftheelementsareaccessible,forexample,thatbuttonscanalsobetappedbyapersonwiththickerfingersandondifferentscreensizesanddensities.Alsocheckthedefaultpatternsthatareprovidedbythemobileplatformsasyourappshouldatleastfollowthosepatterns.
AllowMistakesInordertoprovideatrulyusableapp,yourappmustallowuserstomakemistakes.Ifauser,forexample,missesarequiredinputfield,providefriendlyandusefulerrormessages.Checkthattheerrormessagesarenottoogenericandthattheydescribetheerrorinawaythatpeoplewithoutatechnicalbackgroundcaneasilyunderstand.Furthermore,niceerrorhighlightingshouldbeinplacetoshowtheuserthemistakeheorshemade.Keepyourtargetcustomergroupinmindwhenverifyingerrormessagesasthiswillhaveanimpactonthewording.Besidesthat,yourappshouldprovideundoandgobackactionsinordertooffertheuseraneasywaytocorrecterrors.Anotherapproachmayinvolveuserinputrecommendationsorso-calledtype-aheadsearches,whichprovidetheuserwithpossibleinputsolutions;forexample,whenheorsheistyping“NewY,”theappwillsuggestNewYorkasapossibleinput.
CheckWorkflows,Details,andNavigationIfyourappfollowsaspecialworkflow,checkthatitfollowsthesameworkflowineverysection.Keepaneyeonthedetailsofthoseworkflows.Besurethatbuttons,labels,andotherelementsarebigorsmallenoughtobeused.Screendivisionisimportant,sochecktheapponsmallerandbiggerscreens.Don’tforgettochecktheappforimportantdetails.Aretheyprominentenough?Istheuserabletofindthem?Alsokeepaneyeonvisualtransitionssuchasanimationsorelementsthatfadeinorout.Isthetransitionofthoseelementssmoothordoesitfeelchoppy?Makesureyoualsouseslowdevicestocheckthatallofyouranimationsaresmoothenough.Checkthatthemainnavigationandsub-navigationoftheappareeasytouse.
Arethereanyunnecessarytapsthattheusermustperforminordertoreachhisorhergoal?Checkthatthenavigationhasnounnecessarysteps,andchecktoseeifsomeofthestepscanbecombined.Isthenavigationdoablewithonehand?Askpeoplewhohavedifferenthandsizesandareleft-orright-handedtoperformthistask.Havetheplatform-specificnavigationpatternsbeenmet?
ChecktheWordingVerifythatthetextusedwithintheappisclearandeasytounderstand.Askdifferentpeopleiftheyunderstandthewordingandthefeaturebehindit.Ifyourappcontainstext,itmustbefreeoftyposastheycanbeveryembarrassingandhaveanimpactonyourstorereviewsandratingsaswellasyourreputation.Ifyourappsupportsmorethanonelanguage,besurethetextfitsintoeveryUIelementandthatthetranslationiscorrect.Don’tforgettotestyourappfor
placeholdertextslike“Loremipsum”oranyotherdefaulttextsfromdevelopersordesigners.
CheckConsistencyAreallyimportanttaskwithinmobileusabilitytestingistocheckforconsis-tency.Verifythatyourappisconsistentineverysectionorview.AsIalreadymentioned,theUIelementsmusthavethesamelook-and-feel,thesametext,spacings,colors,andimages.Furthermore,itisreallyimportanttocheckthatyourappisconsistentoverallofthesupportedmobileplatformssuchasiOS,Android,WindowsPhone,orBlackBerry,notjustintermsofnavigationorpatterns(eachmobileplatformhasitsownrules),butbyensuringthatthetexts,colors,andimagesarethesame.MakesureyoualsochecktheconsistencyofanyofferedWebordesktopapplicationsthatprovideadifferentviewtoyourapplication.Allofyourapplicationsmustfollowthecorporateidentityguidelinesofyour
companyinordertolendafamiliarlook-and-feeltoyourproduct;otherwisetheusermaygetconfused.Askforyourcorporateidentityguidelines,whichwillincludetopicssuchascolors,fonts,logos,images,andtext.
WriteItDownWhiletestingyourapp,itisimportantthatyouwritedowneverythingthatfeelsdifferentorsomehowjumpsoutatyou,nomatterhowtrivialitmayseem.Detailedobservationsarereallyimportantandwillhelpimprovetheapp.Youcancreatescreenshotsoftheentirescreenusingthemobile-platform-specificbuttoncombinationorscreencapturevideoswithaWebcamwhileusingtheapptoshowthese“wrong”thingstoyourproductownerordevelopersothatyoucandiscussthem.Animportantpointis:don’tgiveupwritingdownsuchthings,evenifthe
productownerorthedeveloperrejectsyourfindingstimeandtimeagain.Fightforthosethingsandkeepraisingthem.However,dobecarefulwithyourbugremindersasitislikelythatyourcomplaintswillbeignoredifyouconstantlyraisethesameissues.Isuggestthatyoucollectandcategorizerejectedbugsandpresentthemasanimprovementstorytoyourproductmanagerordeveloperforoneofthenextreleases.Anotherapproachwouldbetoincludefixesofoldbugsineveryreleasetoimprovethemobileappandcleanupthebugpool.Talktoyourteamandfindthesolutionthatworksbestwithyourapp.Testersneedtobecuriousandtenacious,andyourcustomerswillloveyoufor
that!
AccessibilityTestingAsyouhavelearnedfromthepreviouschapters,lotsofpeoplearoundtheworldareusingmobiledevicesandmobileappsintheirdailylife.Appsareusedforawholerangeofdifferentreasons,suchasnavigation,hotelbookings,ticketpurchases,listeningtomusic,orplayinggames.Whendevelopingandtestinganapp,didyouthinkaboutpeoplewith
disabilities?Therearemanypeopleouttherewhoarevisually,audibly,physically,and/or
cognitivelyimpaired.Makingmobileappsaccessibleisnoteasyandit’salotofwork,butitdoesn’tjustbenefitpeoplewithdisabilities;italsoimprovesyourcompany’sreputationandincreasesyouruserbase.Itisthereforeveryimportantthateveryoneinvolvedinyourmobileapp
developmentprocessbeawareofaccessibilityandknowabouttheaccessibilityrequirementsofmobileapps.Therearefourtypesofimpairmentthatarerelevanttomobileappdevelopment:visual,auditory,physical,andcognitive.
VisualImpairmentPeoplewithvisualimpairmentmaybeblind,havelimitedvision,becolor-blind,orhaveacolordeficiency.Duetotheirvisualimpairment,theymaynotbeabletoseeallofthebuttons,labels,text,andotherelementsinyourappandarethereforeunabletoaccessorcontroltheappcorrectly.Toadaptyourmobileappforvisuallyimpairedpeople,youcanimplementthe
followingfeatures:•Makeallelementsavailableforscreenreaders.Nearlyeverymobileplatformofferstheoptiontouseabuilt-inscreenreadertoreadoutthecontentandtheelementsofthemobileapp.Iftherearenobuilt-inscreenreadersavailable,youcaninstallthemfromthird-partyvendors.•Everymobileplatformoffersabuilt-infeaturetoenlargethefontsizeoftheentirephoneandapps.Implementyourappinsuchawaythatthefontsizecanbeadjustedwhileretainingapleasantlayout.•Everymobileplatformoffersafeaturetochangetheforegroundorbackgroundcolorofthescreen,ortochangethecontrast.Therefore,youshouldusecolors,brightness,andcontrastoftheUIelementsthatworkwiththedifferentforegroundandbackgroundsettings.•Built-inscreenmagnifierscanbeusedtoenlargetheelementsonthescreentomakethemeasiertoread.
•Anotherwayofhelpingvisuallyimpairedpeopleisthroughtheuseofvoicerecognition.Thisallowsmobileuserstocontroltheentiremobileappthroughvoicecommands.
AuditoryImpairmentSomepeoplehavetroublehearing,meaningthattheydon’thearifsomeonecallsthemoriftheyreceiveaudionotifications,instructions,oranyotherkindofaudiocontent.Tohelppeoplewithhearingproblems,youcandothefollowing:•Usethebuilt-invibrationorvisualnotificationtoinformuserswhentheyreceiveacall,message,oranyothernotification.•Ifyourappreliesonvideocontent,usesubtitlestoinformtheuseraboutthecontentofthevideo.•Implementadjustablesoundorvolumecontrols.Thisisreallyimportantforpeoplewithahearingaid.•Ifyourappgeneratesanysounds,implementafeaturetoswitchthedevicetomonoaudio.Thiswillhelppeoplewhocanhearwithonlyoneear.
PhysicalImpairmentPhysicallyimpairedpeoplehaveproblemsperforminggestureswiththeirhands.Theyarenotabletousegesturessuchaspinch,tap,orswipetocontroltheelementsonthephoneorwithinthemobileapp.Tohelpthemuseyourapp,youcandothefollowingthings:•Usevoicerecognitionasthisallowsmobileuserstocontroltheentiremobileappthroughvoicecommands.•Implementafeaturetoincreaseordecreasethespeedofpressingbuttonsasthiswillmakeiteasierforphysicallyimpairedpeopletocontrolyourapp.
CognitiveImpairmentCognitivelyimpairedpeoplemayhaveproblemsrelatedtoattention,communication,memory,computation,andreadingskills.Thosepeopleexperienceproblems,forexample,withunderstandingcomplexnavigationsorinstructionsthattheyneedtofollowwithinanapplication.Tohelpthemuseyourapp,youcandothefollowingthings:•Implementaneasyandintuitiveuserinterface.Makeitassimpleas
possibletounderstandanduse.•Usescreenreadersincombinationwithsimultaneoustexthighlighting.•Implementanauto-textfeatureforinputfieldssothatpreloadedanddefinedtextelementsarealreadyinplace.•Implementyourappinsuchawaythatcognitivelyimpairedpeoplehaveenoughtimetocompleteanoperation.
AccessibilityGuidelinesAsyouhaveseen,accessibilitytestingisareallychallengingtask,andyouandyourteamneedtofindawaytosupportsuchfeatures.Ifyourappisaccessible,itisveryimportantthatyoutestitwithpeoplewhohavesuchimpairmentsinordertobesurethattherequirementshavebeencorrectlyimplemented.Providinganaccessibleappwillincreaseyouruserbaseandimproveyourreputation.TheW3Ccreatedanaccessibilityinitiativethatcoversmostofthe
accessibilityconcerns.Youcanfindinformationatwww.w3.org/WAI/orwww.w3.org/WAI/mobile/.Togetdetailedmobileaccessibilityinformationforthedifferentmobileplatforms,checkthemanufacturers’Webpages.TheguidelinesforAndroidcanbefoundhere:•Androidaccessibilityguide(http://developer.android.com/guide/topics/ui/accessibility/index.html)•Androidaccessibilitytesting(http://developer.android.com/tools/testing/testing_accessibility.html)
TheguidelinesforiOScanbefoundhere:•iOSaccessibilityintroduction(https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/iPhoneAccessibility/Introduction/Introduction.html•AccessibilityoniPhone(https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/iPhoneAccessibility/Accessibility_on_iPhone/Accessibility_on_iPhone.html•iOSaccessibilitytesting(https://developer.apple.com/library/ios/technotes/TestingAccessibilityOfiOSApps/TestingtheAccessibilityofiOSApps/TestingtheAccessibilityofiOSApps.html
TheguidelinesforWindowsPhonecanbefoundhere:•WindowsPhoneaccessibilityguide(www.windowsphone.com/en-us/how-to/wp8/settings-and-personalization/accessibility-on-my-phone)
TheguidelinesforBlackBerryarehere:•BlackBerryaccessibilityguide
(http://ca.blackberry.com/legal/accessibility.html)
BatteryUsageTestingWhiletestingamobileapp,youneedtotestbatteryconsumptionduringuse.Ifyourappconsumestoomuchpower,userswilldeleteitandmoveontoanotherone.Totestyourapp’sbatteryusage,youcanrunthroughthetwoscenarios
describedinthefollowingsections.
FullyChargedBatteryInthefirstscenario,youhaveafullychargedbattery.Afterinstallingandstartingtheapp,leaveitopenandputthedeviceinstandbymode(yourappisnowrunningintheforeground).Nowyouhavetowaitacoupleofminutes(orhours)toseewhethertheappisconsuminglotsofbattery.Checkthebatteryusagefromtimetotimeandnotethebatterylevel.Youshoulddothiswithdifferentdevicestogetbetterresults,anddon’tforgettocloseanyotherappsthatarerunningonthedevicetoeliminateanysideeffectstheymaycause.Thesametestshouldbedonewhentheappisrunninginthebackground.Toputtheappinthebackground,juststartitandthencloseit(bypressingthehomebuttonorclosebutton).Onmostmobileplatforms,theappwillnowruninthebackgroundormultitaskmode.Again,checkthedevicefromtimetotimetomonitortheapp’sbatteryusage.Usethesmartphonehardwarefeatures(ifusedwithinyourapp)suchasGPS
orothersensorsandcheckthebatterylevelofyourdevicewhileusingthosefeatures.It’simportantthatyourappswitchoffdevicefeaturesoncetheyarenolongerneeded.Ifyourappdoesn’tdothis,itwillconsumelotsofbattery.Checkthattheappisnotsendinganyunnecessaryrequeststoabackendsystemwhilerunninginthebackgroundorforeground.Unnecessaryrequestswillhaveanimpactonbatterylife.Tocheckthis,useaproxytoolsuchasCharles8orFiddler9tosendallyourrequestsandcommunicationsfromthedevicethroughyourworkstationtothebackendsystem.
8.www.charlesproxy.com/9.www.telerik.com/fiddler
Whileusingthedeviceandtheapp,checktoseeifthedevicegetshot.Insomecasesdevice-specificfeatureswillcausethedeviceandbatterytoheatup,whichwillhaveanimpactonyourdevicehardwareandmayultimatelydamageit.
Alsocheckthatyourappreallyisclosedafterremovingitfromthemultitaskthread.Toverifythis,openthemobile-specificappsystemsettingsandcheckthatyourappisnotcurrentlyrunningthere.
LowBatteryThesecondscenarioyoushouldtestiswhenthebatteryofthephoneisnearlyempty,whenithas10%to15%ofitscapacityleft.Inthisbatterystate,mostmobiledevicesswitchofffeaturessuchasnetworkconnections,GPS,andsensorsinordertoprolongthebatterylife.Ifyourappusesanyofthosefeatures,checkhowithandlesthislowbatterystate.Watchoutforfreezes,crashes,andperformanceissues.Onsomedevicesyoucanactivateapower-savingmodethatswitchesoff
sensorsorInternetconnections.Again,don’tforgettocheckhowyourapphandlesthispower-savingmode.Areallyimportanttestistoseehowyourapphandlesacompletelyempty
battery.Useyourappuntilthephonebatterydies.Thenpluginachargerandstartthephoneagain.Oncethephonehasbootedup,checkyourappfordatalossordatacorruption.Theappshouldworkasexpectedandnodatalossorcorruptionshouldoccur.Youshouldalsotestyourappinthetransitionfromagoodtoabadbattery
state.Mostdeviceswillpopupanalertinformingtheuserthatthebatteryisnearlyempty.Thistransitioncanhaveanimpactonyourapp,socheckitforfreezes,crashes,andperformanceissues.Thesameshouldbedonetheotherwayaround.Useyourappwhilethebatteryischargingasthiscanalsoimpacttheapp.
BatteryConsumptionToolsSomemobileplatformsofferabatteryusagestatistictoseewhichappisconsumingthemostbattery.Android,forexample,providessuchanoverviewtoseethecurrentstateofthebattery(seeFigure4.2).Usethistoolforyourbatterystatetesting;it’sreallyusefulformobiletesters!
Figure4.2Androidappbatteryusageoverview
Ifyouoryourdeveloperswanttoconstantlymeasurebatteryconsumptionduringyourproject,therearetoolsavailableforthat.ForAndroidthereisatoolcalledJouleUnit.10JouleUnitisanopen-sourceenergyprofilingtoolforAndroidappsthatfindsanyunnecessarybatteryusageofappswhentheyarerunning.ItalsomeasurestheusageoftheCPU,Wi-Fi,ordisplaybrightness.ThetoolbasicallyhasthesamestructureasJUnittestsandisreallyeasytointegratewithinyourdevelopmentenvironmentsoyoucangetearlyfeedbackaboutthebatteryconsumptionofyourapp.10.https://code.google.com/p/jouleunit/
iOSallowsyoutousetheenergyusagefunctionwithinInstruments11toprofiletheenergyusageofyourapp.InstrumentsispartoftheXcodedevelopmentenvironmentandisfreefortestersanddevelopers.11.
https://developer.apple.com/library/mac/documentation/developertools/conceptual/instrumentsuserguide/Introduction/Introduction.html
ThereareacoupleofusefuldocumentsonlineaboutenergyusagewithinInstruments:
•EnergyUsageInstrument(https://developer.apple.com/library/ios/documentation/AnalysisTools/Reference/Instruments_User_Reference/EnergyUsageInstrument/EnergyUsageInstrument.html
•LoggingenergyusageiniOS(https://developer.apple.com/library/ios/recipes/Instruments_help_articles/LoggingEnergyUsageinaniOSDevice/LoggingEnergyUsageinaniOSDevice.html
Batterytestingiseasytodoandshouldbepartofyourtoolchain.Combinethemanualwiththeprofilingapproachtogetgoodresults.
StressandInterruptTestingStressandinterrupttestingisanimportantpartofthemobiletestingprocess.Withtheaidoftools,mobiletestersareabletodetermineanypotentialperformanceorstabilityissuesexhibitedbyanapp.Totestyourappagainstinterrupts,youcanmanuallytriggerlotsofnotificationstothedevicewhileusingtheapp.Notificationscanbeincomingmessages,calls,appupdates,orpushnotifications(softwareinterrupts).Furthermore,pressingthevolumeupanddownbuttonsoranyotherkindofhardwarebuttonisalsoaninterrupt(hardwareinterrupt)thatcanalsohaveanimpactonyourapp.Doingallofthesetasksmanuallyisalotofworkandverytime-consuming.In
mostcases,thesetestscenarioscan’tbedonemanuallybecauseitisveryhardtosimulatefastandmultipleuserinputswithoneortwohands.Butitcanbedonewiththeaidoftools,anditisreallyeasytointegratethemintothedevelopmentandtestingprocess.ForAndroidapps,atoolcalledMonkey12canbeusedwhichispartofthe
AndroidSDK(SoftwareDevelopmentKit).Monkeycanrunoneitheraphysicaldeviceoranemulator.Whilerunning,itgeneratespseudo-randomusereventssuchasatouch,click,rotate,swipe,mute,Internetconnectionshutdown,andmuchmoretostress-testtheappandseehowithandlesallthoseinputsandinterrupts.12.http://developer.android.com/tools/help/monkey.html
ThepackagenameoftheAndroid.apkfileisneededtobeabletorunMonkey;otherwiseitwillexecuteitsrandomcommandstotheentirephoneinsteadofjusttheappundertest.Withaccesstotheappcode,thepackagenamecanbefoundinthe
AndroidManifest.xml.Ifonlythecompiled.apkfileisavailable,mobiletesterscanusetheAndroidAssetPackagingTool13(aapt)togetthepackagenamefromtheapp.aaptislocatedinthebuild-toolsfolderoftheinstalledAndroidSDKversion.13.http://elinux.org/Android_aapt
Thepathtoaaptmaylooklikethis:Clickheretoviewcodeimage
/../daniel/android/sdk/build-tools/android-4.4/
Withthefollowingcommand,thepackagenamecanbereadoutfromthe.apkfile:Clickheretoviewcodeimage
./aaptdbadging/daniel/myApp/myApp.apk|grep'pack'
...package:name='com.myApp'versionCode=''versionName=''...
Whenthepackagename(inthiscasecom.myApp)isavailable,executeMonkeywithadb14(AndroidDebugBridge):14.http://developer.android.com/tools/help/adb.html
Clickheretoviewcodeimage
./adbshellmonkey-pcom.myApp-v2000
Thenumber2000indicatesthenumberofrandomcommandsthatMonkeywillperform.Withanadditionalparameter-sforseed,Monkeywillgeneratethesamesequenceofeventsagain.ThisisreallyimportantforreproducingabugthatmayoccurwhenrunningMonkey.ForiOSappsthereisasimilartoolcalledUIAutoMonkey.15UIAutoMonkey
isalsoabletogeneratemultiplecommandstostress-testaniOSapp.TouseUIAutoMonkey,aUIAutomationInstrumentstemplatemustbeconfiguredwithinXcode.Afterthetemplateisconfigured,aJavaScriptfileneedstobewrittentotellthetoolhowmanyandwhichcommandsshouldbeexecutedduringthestresstestingsession(seeListing4.1).15.https://github.com/jonathanpenn/ui-auto-monkey
Listing4.1UIAutoMonkeyScript
Clickheretoviewcodeimage
config:{numberOfEvents:2000,delayBetweenEvents:0.05,//Inseconds
//EventsthatwillbetriggeredonthephoneeventWeights:{tap:30,drag:1,flick:1,orientation:1,clickVolumeUp:1,
clickVolumeDown:1,lock:1,pinchClose:10,pinchOpen:10,shake:1},
//ProbabilitythattoucheventswillhavethesedifferentpropertiestouchProbability:{multipleTaps:0.05,multipleTouches:0.05,longPress:0.05}},
Ifthescriptiswritten,itcanbeexecutedwithinXcode.Attheend,bothtoolsgenerateanoverviewofpossibleerrorsandproblemswithintheapp.
ImportantPleasevisitthetoolmanufacturer’sWebsiteforinstallationinstructions.
Asyoucansee,thesetoolsmakeitsimpletostress-andinterrupt-testamobileapplication.Besidesthat,usingthemisahugebenefitformobiletestersasithelpstheteambuildareliableandrobustmobileapp.Bytheway,it’susefultocombinebatterytestingwithstressandinterrupttestingtoseehowthebatteryisusedwhenlotsofinterruptsanduserinputsaretriggeredthroughouttheapp.
PerformanceTestingPerformancetestingisoneofthekeytestingareasineverysoftwaredevelopmentprojectandespeciallyformobileapps.IfyourememberthehighuserexpectationsdescribedinChapter1,“What’sSpecialaboutMobileTesting?,”mobileperformancetestingisareallyimportantandcriticalpartofappdevelopment.Mobileusersexpectanapptostart/loadwithintwoseconds;otherwisetheyareunhappyandmaydeleteit.Testersanddeveloperscanuseperformanceteststodiscoverpotential
bottlenecksintheirsoftwareapplication.Normally,performancetestingisdoneonserversorbackendsystemstocheckhowthesystemsorsoftwarecanhandlehugenumbersofrequests,tomeetacceptableresultsfortheusers.Performancetestsmustbeexecutedinadefinedstateoftheapplicationwith
hardwareresourcesthatareequaltothelivebackendenvironment.Thecollected
datamustthenbeanalyzedtofindpossiblebottlenecksandproblemsinthesoftware.Performancetestingisacomplextopicthatshouldneverbeunderestimatedorpostponedtotheendoftheproject;itshouldbedoneasearlyaspossible.Keyperformancefiguresshouldbepartoftherequirementsphaseofappdevelopmentinordertostartplanningrightfromthebeginning.Formobileapps,performancetestsaremorecomplexandneedtocovermore
systemsinordertogetmeaningfulresultsthatimproveperformance.Thetypicalmobileappreliesonabackendsystem(server)withwhichit
communicates.Theappsendsrequeststotheserver,whichthenhandlesthoserequestsandsendsbackaresponse.Tosendasimplerequesttoabackendsystem,therearethreecriticalperformanceareasthatneedtobecovered:
•Theserver•Themobiledatanetworks•Themobiledeviceandappitself
Totesttheperformanceofmobileapps,youneedtoperformance-testatleastthebackendsystemandtheappitself.Itisnotpossibletotesttheperformanceofthemobiledatanetworks,anditmakesnosenseforyouasamobiletester.Thenetworkspeedandlatencycanbesimulatedduringthetestingphase,butthenetworkspeedwillbetotallydifferentwhileauserisonthemoveinrealdatanetworks.Inthischapter,Iwanttofocusonperformancetestingofthemobileappitself.
KeepthebackendsystemperformanceinmindandlookforfurtherinformationontheInternet.Thereareplentyofgoodtoolsavailabletohelpyousetupaperformancetestingsuite.Ifyouwanttodoperformancetestingwithinyourprojectandusetoolstodoso,keepthefollowingstepsinmindastheywillhelpyoutodefineaclearstrategy:
1.Planperformancetestsearlyinthedevelopmentphase.2.Designtheperformancetestsforyourapp.3.Preparetestdataandinfrastructure(theseshouldbethesameasfortheliveenvironment).
4.Executetheperformancetests.5.Collect,analyze,andevaluatethedata.6.Findpotentialbottlenecksandrefinetheapp.7.Monitortheapplicationchangesagaintoseeiftherefinementsweregoodenough.
MobileAppPerformanceTestingWhentestingtheperformanceofamobileapp,keepitsimpleandfocusontheUIoftheapp.Forexample,useastopwatchandmeasuretheapplicationlaunchorthedelaysbetweencertainoperations.Measuretheloadingtimeofthecontentsuchasimages,text,oranimationsthatneedtobecreatedorprocessedbytheapp.Performthosetestsseveraltimesandnotetheperformedstepsinordertoreproduceandtrackpossibleperformanceissues.Also,writedownhowoftentheproblemoccurred.Ifithappensallthetime,that’sfine;thatisaneasyfix.Butsomeproblems,especiallyperformanceproblems,mayhappenonlythreeoutoftentimes.It’sthereforeimportanttofindthebugandreproducethebehavior.Whiletestingtheapp,writedowneverythingthatseemsslowasitcouldbea
potentialperformanceissue.Manualperformancetestingshouldbedoneonseveraldeviceswithdifferenthardwarespecificationsinordertogetmeaningfulresults.Anothertestthatshouldbedoneisacomparisonbetweenthecurrentapp
versionthatisliveandavailablefordownloadintheappstoresandthenewreleasecandidateapp.Comparebothversionsintermsofapplaunchtimeandallotherareas.Thenewreleasecandidateshouldnotbeslowerthanthecurrentversion;otherwisetheappwillgetbadfeedbackfromusers.
ImportantThecomparisonshouldbedoneonthesamehardware;otherwisetheresultswilldiffer.
Ifyourappcontainsthird-partyelementssuchasadvertisingornewsfeeds,checkthatthoseelementshavenoimpactontheperformanceoftheapp.Asanexample,youcanuseaWebproxytoollikeFiddlertosendthethird-partyrequesttoatime-outtoseethatithasnoimpactontheapp’sperformance.Anotherwaytotesttheperformanceoftheappistoprofileandmeasurethe
processandoperationtimeofthecode.Thereareplentyofprofilertoolsavailabletochecktheappcodeforpotentialbottlenecksandperformanceissues.Thistaskshouldbedonebythedevelopers,soencouragethemtousesuchtools.Tosummarizethesimplemobileappperformancetests:•Measuretheapp’slaunchtime.•Checkfordelaysduringoperationsoruserinteractions.
•Measuretheloadingtimeofthecontent.•Noteeverythingthatseemsslow.•Testondifferenthardware,especiallyonslowerphones.•Comparetheliveappversionwiththenewreleasecandidate.•Checkforthird-partyelements.•Useprofilingtoolstomeasuretheprocessandoperationtimeofmethodsandclasses.
Asyoucansee,thesearesimplestepstotesttheperformanceofamobileapp.Ifyouwantmorecomplexanddetaileddataabouttheperformanceoftheapp,thebackend,andthedatanetworks,you’llneedtouseaperformancetestingtoolorsolutionthatcoversallparts.Thereareplentyofmobileappperformanceprovidersavailable,sosearchforthemontheInternetandseewhichonebestfitsyourdevelopmentandtestenvironment.
StandbyTestingStandbytestingisreallyeasyandsimpletodobutcanshowsomenicecrashes,freezes,andUIglitcheswithintheapp.Whiletheappisrunning,putthedeviceintostandbymodebypressingtheoffbuttononce.Dependingontheappundertest,wakeupthedeviceafteracoupleofseconds,minutes,orhourstoseehowtheappreactsuponwake-up.Mostappsfetchdataupdatesfromthebackendsystemafterawake-uptorefreshthecurrentUI.It’spossibleforabugtopreventtheappfromdisplayingthenewlyfetcheddatacorrectly,oritmayfreezeorcrash.IfaUIviewupdatemechanismisimplemented,youmustmakesurethismechanismworksafterthedevicewakesupandfetchesthelatestdata.AlsotestthescenarioofnothavinganInternetconnectionwhiletheappisin
standbymode.Totestthat,opentheapp,closethedevice’sInternetconnection,andputthedeviceintostandby.Afteracertainamountoftime,wakeupthedeviceandcheckthebehavioroftheapp.Itwillprobablycheckforupdates,butthereisnoInternetconnection.Inthiscase,theappmustshowapropererrormessagetotheuser,informinghimorherofthecurrentsituation.Don’tforgettocheckthatthedeviceisabletocommunicatewithdevice-
specifichardwareelementsafterwakingup.Feelfreetogetcreativewhentestingtheappforstandbyandwake-upproblems.
InstallationTestingTheinstallationprocessisthefirstimpressionamobilecustomerhasofanapp.
Iftheinstallationfailsduetoerrorsorproblems,thecustomerwillnottrytodownloadtheappagainandwillmoveontoanotherone.Toavoidsuchproblems,installationtestingmustbemadepartofthemobiletestingprocessandshouldbeperformedatleastbeforeanewversionisdueforrelease.Totestyourappforinstallationproblems,performthefollowingtasks:•Verifythattheappcanbeinstalledsuccessfullyonthelocalstorageormemorycardofthedevice.•CheckthattheinstallationworkswithdifferentInternetconnectionssuchasWi-Fiormobiledatanetworks.•ChangetheInternetconnection(Wi-Fito3G,forexample)whiletheappisinstalling.•Switchtootherappswhiletheappisinstalling.•Switchthedevice’sInternetconnectionoff,forexample,toairplanemode,whiletheappisinstalling.•Trytoinstalltheappwhenthereisnotenoughspaceleftonthelocalstorage.•Trytoinstalltheappviadatacableorsyncingfrommobile-specificsoftwareapplications.
Whenperformingthoseactions,watchoutforerrormessagesintheappaswellascrashesandfreezes.Whentheappsuccessfullyinstalls,mobiletestersshouldalsotesttheother
wayaroundbytestingtheuninstallprocess.Souninstalltheappandcheckthatitiscompletelyremovedfromthedevicewithnodataleftonthehardwareorlocalstorage.Toverifythattheappwasremovedcompletely,checkthedevice’smemoryandfoldersforleftoverdata.Anotherwaytoverifythattheappwasremovedsuccessfullyistoinstalltheappagainandtocheckthat,say,auserisnotloggedinautomaticallyorthattheappdoesn’tshowanydatafromthepreviousinstallation.Thesetestsarereallyimportantbecausesomedevicesaresharedwithinacompanyorfamilyandleftoverdatacanleadtoserioustrouble.Somemobileplatformsofferdifferentwaysofuninstallinganapp,soyou
havetotestallofthem.Whileuninstallingtheapp,watchoutforerrormessages,crashes,andfreezes.
UpdateTestingAsyouhavelearnedintheprevioussections,testinganappduringtheinstallanduninstallprocessesisreallyimportant.Besidesinstallinganduninstallingan
app,usersarealsoabletoupdatetheappfromoneversiontoanother.Duringthisupdateprocess,lotsofthingscangowrongandneedtobetestedbeforeanewappversionissubmittedtotheappstores.Testingtheupdateprocessincludesscenariossuchasthefollowing:•Logged-inusersshouldnotbeloggedoutfromtheappaftertheupdateisinstalled.•Theupdatewillnotaffectthelocaldatabase;thatis,existingdatawillnotbemodifiedordeleted.•Theappisinthesamestateasbeforetheupdatewasinstalled.•Testingtheupdateprocesswillsimulatetheupdateprocessintheappstores.
Youshouldalsotesttheupdateprocessfromafarolderappversiontothelatestonetoseewhathappenstotheapp.Whiledoingupdatetesting,keepaneyeoutforerrormessages,crashes,freezes,andperformanceissuesrightaftertheupdate.Inthefollowingtwosections,Iwilldescribehowtoperformupdatetestingon
iOSandAndroid.Nevertheless,ifyouaretestingaWindowsPhone,BlackBerry,oranyotherkindofapp,youshouldofcoursedoupdatetestingaswell.
iOSUpdateTestingTherearetwowaysofsimulatingtheupdateprocessforiOSapps.ThefirstonecanbeperformedwithiTunesbyfollowingthesesteps:
1.BuildanadhocversionoftheappthatiscurrentlyliveintheAppleAppStore.HINT:Thisversionmusthavethesamebundleidentifier(packagenameorstructureoftheappclasses)asthenewapp.
2.BesurethatnoolderversionoftheappisinstalledwithiniTunesandonthetestdevice(syncwithiTunestobesure).
3.Dragtheappfromstep1intoiTunesandsynctheversiontoyourtestdevice.
4.Launchtheappanddosomemanualtestingtomakesurethattheappisworking.
5.Buildthenewreleasecandidateversionoftheapp,dragittoiTunes,andsyncittothedevice.iTunesshouldconfirmthattheolderversionwillberemoved.
HINT:Donotdeletetheoldbuild!InthenextstepiTuneswillinstallthenewappovertheoldoneandsimulatetheupdatefromtheAppStore.
6.LaunchthenewversionoftheappandcheckthateverythingisOK.ThesecondwayoftestingtheupdateprocessforiOSistousetheApple
Configurator.16Thistoolisfareasiertouse,especiallyifyouwanttotesttheupdateprocedureonseveraliOStestdevicessuchasaniPhone4(S),iPhone5(S),oriPad.16.http://help.apple.com/configurator/mac/1.7/?lang=en
1.BuildanadhocversionoftheappthatiscurrentlyliveintheAppleAppStore.HINT:Thisversionmusthavethesamebundleidentifierasthenewapp.
2.Besurethatnoolderversionoftheappisinstalledonthetestdevice.3.Usethetooltoinstalltheappfromstep1tothedevicesforwhichyouwanttochecktheupdateprocess.
4.Launchtheappanddosomemanualtestingtomakesurethattheappisworking.
5.Buildthenewreleasecandidateversionoftheappandinstallitwiththetool.Theupdateprocesswillbesimulated.
6.LaunchthenewversionoftheappandcheckthateverythingisOK.Forfurtherinformation,pleasereadthetechnicalnoteTN228517fromApple
abouttestingupdatesoniOSdevices.17.
https://developer.apple.com/library/ios/technotes/tn2285/_index.html#//apple_ref/doc/uid/DTS40011323
AndroidUpdateTestingThesameupdatetestingcanbedonewithAndroidapps.TotesttheupdateprocessforAndroidapps,youcanusetheadbtoollocatedintheAndroidSDKfolder:
1.InstallthecurrentGooglePlaystoreversionoftheapptothetestdevice:Clickheretoviewcodeimage
./adbinstallRELEASED_APP_NAME.apk
2.Checkthatthisversionworks.3.BuildareleasecandidateoftheAndroidapp.
HINT:BesuretosignthereleasecandidatewiththePlaystorekeystore.4.Usethefollowingcommandtoinstallthenewversionoftheappandtotesttheupdateprocedure:
Clickheretoviewcodeimage
./adbinstall-rNEW_VERSION_APP.apk
Theoption-rmeanstheappwillbereinstalledandretainitsdataonthephone.
5.Thenewversionoftheappisnowinstalledandcanbetested.Asyoucansee,updatetestingisveryeasytodo.It’sreallyimportantto
performthesechecksbeforesubmittinganapptothevariousappstores.
DatabaseTestingAppsuselocaldatabases,inmostcasesaSQLite18database,topersistdataonthephone.Storingthedataorthecontentofanappinalocaldatabaseenablesmobileappstopresentthecontentwhenthedeviceisoffline.ThisisahugeadvantagecomparedtomobileWebappsthatrelyonastableInternetconnectioninordertoworkproperly.Thefactthatmobileappsusedatabasesmeansthatmobiletestersneedtotestthemandtheactionsthatwillbeexecutedonthedatabase.18.www.sqlite.org/
Localdatabasetestingcanbeperformedmanuallyorautomated.Thegoalistotestdataintegritywhileediting,deleting,ormodifyingthedata.Inordertoachievegooddatabasetesting,youneedtoknowthedatabasemodelwiththetablenames,procedures,triggers,andfunctions.Withthehelpofdatabasetools,youcanconnecttothedevicedatabasetotestandverifythedata.Thefollowingtypesoftestsshouldformpartofyourmobiledatabasetesting:•Databasevalidationtesting•Databaseintegrationtesting•Databaseperformancetesting•Procedureandfunctiontesting•Triggertesting•CRUD(Create/Read/Update/Delete)operationstestingtomakesuretheywillworkonthedatabase•TestingthatthedatabasechangesareshowncorrectlyontheUIoftheapp
•Searchandindexingfunctiontesting•Databasesecuritytesting•Testingthedatabaseformigrations
Becausedatabasesanddatabasetestingarehugetopicsinthemselves,Irecommendfurtherreadingtogathermoreinformationaboutthedifferentdatabasetechnologies.The“BooksaboutSQLite”19pagecontainslotsofusefulbooksonthistopic.19.www.sqlite.org/books.html
LocalStorageTestingLocalstoragetestinghasnothingtodowiththeappdatabase;instead,youshouldcheckhowtheappreactsindifferentstatesofthedevice’slocalstorage.Everydevicehasacertainamountofstoragecapacityformusic,images,apps,andanyotherkindofdatathatcanbestoredonthedevice.Somedeviceshaveonlyasinglecentralandpermanentlyinstalledlocalstorage,andusersarenotabletoextendthosedeviceswithadditionalstorage.However,therearemanydevicesthatofferthepossibilitytoextendthelocalstorage,forexample,withamicroSDcard.Whentestingamobileapp,youshouldtesttheapptogetherwithdifferent
statesofthelocalstoragetobesurethattheappcanhandlethemproperly.Thefollowingscenariosshouldbetested:
•Testtheappwhenthelocalstorageisfull.•Testtheappwhenthelocalstorageisfullbuttheextendedstoragestillhassomespaceleft.•Testtheappwhenthelocalstoragehassomespaceleftbuttheextendedstorageisfull.•Testtheappwhenbothstorageareasarefull.•Testtheappwhenthelocalstorageisnearlyfull.Performsomeactionswithintheappthatwillwritelotsofdatatothelocalstorageinordertofillitup.•Removethedevice’sextendedstorageandcheckthebehavioroftheapp.•Ifpossible,movetheapptotheextendedstorageandcheckthebehavior.•Testtheappwhenthelocalstorageisempty.
Whenperformingthesescenarios,watchoutforappcrashes,errormessages,appfreezes,performanceissues,UIglitches,andanyotherkindofstrange
behaviorthatcouldindicateaproblem.
SecurityTestingSecuritycanbebusinesscritical—forexample,whenattackersstealyourcustomerdata—thusmakingitaveryimportantpartofthedevelopmentandtestingprocessofyourmobileapp.Securitytestingisacomplextopicthatrequiresknowledgeinmanydifferentareas,suchasclient-servercommunication,softwarearchitecture,andsystemarchitecture.Becauseofitscomplexnatureandthespecializedskillsetrequired,securitytestingisbestdonebyexperts.Itincludesmethodssuchasmanualorautomatedpenetrationtestingwithman-in-the-middleattacks,fuzzing,scanning,andauditingthesoftware.Penetrationtestingisanapproachusedtofindsecurityweaknessesinan
applicationthatallowaccesstoitsfeaturesanddata.Thereareseveralopen-sourceandenterprisetoolsonthemarketthatcantesttheapplicationforcommonvulnerabilities.AlistofcommonsecuritytestingtoolscanbefoundontheOWASP(OpenWebApplicationSecurityProject)page.2020.www.owasp.org/index.php/Appendix_A:_Testing_Tools
Withtheaidofaproxytool,anattackercanchangeormonitorcommunicationbetweenaclientandaserver.Thisallowstheattackertoreadsensitivedatasuchasusernamesandpasswordsormanipulatethebehavioroftheclientbysendingforgeddatatoit.Thismethodisknownasaman-in-the-middleattack.Fuzzingisamethodtogenerateandsendrandomdatatoasysteminorderto
seehowthesystemreactstothedatainpreparationforapossibleattack.Thankstoscanningtools,thesystemcanbecheckedforknownvulnerabilitiesthatcanbeexploitedinordertogainaccess.Auditsaremostlikelyperformedbycertifiedsecurityproviders.Withthehelp
oftheexternalpartners,theapplicationwillbecheckedmanuallyandautomaticallyforvulnerabilitiesandpossibleattacks.Recallthatthedefaultmobileapparchitectureincludesthefollowing:•Themobileappitself•Themobiledatanetworks•Thebackendsystem
Youhavethreepossibleareasthatneedtobetestedforsecurity.Itwillbenighonimpossibletotestthedatanetworksforsecurity,butthedatanetwork
providersalreadydothistoensureasafeandsecurecommunicationenvironment.Therefore,twoareasremainforsecuritytesting:theappandthebackend.Tosimplifythings,Iwanttofocusonmobileappsecuritytesting.
CommonAppSecurityMistakesThefollowinglistcontainsthemostcommonsecurityproblemsofmobileapps:
ImportantThesepointscanbeusedduringthedevelopmentphasetocheckforcommonmistakes.However,lookingforthesecommonmistakesisnotareplacementforexpertsecuritytestinglikethatmentionedearlier.
•Cachestorage:Sensitivedatasuchaspasswordsortokensiscachedonthedevice.•Unintendedlocalstorage:Sensitivedatasuchaspasswords,tokens,orcreditcarddetailsisstoredbyaccident.•Encryption:Sensitivedatasuchaspasswordsisnotencryptedonthedevicestorage.•Client-sidevalidation:Passwordverificationisperformedonlyontheclientside.•Unencryptedcommunication:Communicationfromtheapptothebackendsystemsisnotencrypted.•Unnecessaryapppermissions:Appsusepermissionsfordevicefeaturesthattheydonotneedoruse.
Toavoidthesemistakes,youshouldkeeptheminmindandtestforthem.Toverifythecachestorageofthedevice,testyourappforinputsanddatathatarenotstoredonthedevice.Checkthedevicecachestoragefordatayoujustenteredandverifythatthedataiscachedforonlyacertainamountoftime.Toverifythecachestorage,useanappthatisabletolookatthefilesystemlevelandcheckthatthecachefolderofyourappcontainsonlythepermitteddata.Turnthedeviceoffandonagainandcheckthedeviceandappcacheasbothmustbeemptyinordertoensurethatthereisnosensitivedataleftonthedevice.Thesameappliestodatathatisstoredonthedevicebyaccident.Checkthedevicestorageandfilesfordatathatisnotallowedtobeonthe
device.Toverifythelocalstorageofthedevice,useappsordevelopertoolsthat
grantaccesstothelocalstorage.Ifyourappusesalocaldatabasetopersistthecontentordatasuchasthelogin
credentialsofyourapp,besurethatsuchdataisencryptedinthelocaldatabase.Checkthataformofencryptionisinplaceatthedatabaselevel.Ifyourappusesalogintograntaccesstothefeaturesandcontentdelivered
fromabackendsystem,besurethattheuserisnotvalidatedontheclientside.Validationshouldalwaysbeperformedonthebackendsystem.Ifthedevicegetslostandvalidationiscarriedoutonlyontheclientside,itisveryeasyforattackerstochangethevalidation,tomanipulateorstealsensitivedata.Mostappsrelyonabackendsystemfortheusertosendorreceive
informationanddata,toreadthelatestnews,tocommunicatewithfriendsinsocialnetworks,ortosende-mails.Ifthecommunicationbetweentheappandthebackendcontainssensitive,unencrypteddata,itmakessensetoencryptit,suchaswithTLS(TransportLayerSecurity).Ifanappwantstouseadevice-specificfeaturesuchasthecameraorwantsto
synchronizecontactswiththeaddressbook,specialpermissiontoaccessthoseelementsisrequired.Whendevelopingandtestinganapp,keepaneyeonthosepermissions.Useonlythepermissionsyourappreallyneeds.Otherwisetheusermaynotuseyourapp,becauseheorsheisskepticalaboutitormayfeellikeheorsheisbeingwatched.Theuseofunnecessarypermissionscanalsobeavulnerabilityfortheappandthedatastoredonthedevice.Ifyoudon’tknowthepermissionsofyourapp,talktoyourdevelopersaboutitandaskcriticalquestionsaboutthepermissionsandtheirpurpose.
SecurityRequirementsAnalysisThesecurityrequirementsanalysisshouldbepartofthefirstrequirementsanalysisphaseofeachmobileappproject.Youshouldraisethetopicasearlyaspossibletoavoidproblemsattheendoftheproject.Thefollowinglistcanhelpyouwiththesecurityrequirementsanalysis:
•Identifythepossibleuserrolesaswellastheirlimitationsandpermissionswithinthearchitecture(appandbackend).•Doestheuserrolehaveanimpactontheexistingsecurity,forexample,thebackend?•Isanexternalauditrequired?Whatshouldbepartofthisaudit?•Whatkindofsecuritytestingapproachesandtoolsarerequiredinordertoachieveagoodsecuritylevel?
•Dowehaveenoughskillsforsecuritytesting?Thislistisbynomeanscomplete.Dependingontheappanditscomplexity,thislistcanbemuchlonger.Agoodsourceofvaluableinformationaboutmobilesecurityisthe“OWASP
MobileSecurityProject”page.21Theprojectcollectsinformationaboutmobilesecuritysuchasthefollowing:21.www.owasp.org/index.php/OWASP_Mobile_Security_Project
•MobileTools(www.owasp.org/index.php/OWASP_Mobile_Security_Project#tab=Mobile_Tools•MobileSecurityTesting(www.owasp.org/index.php/OWASP_Mobile_Security_Project#tab=Mobile_Security_Testing•MobileCheatSheet(www.owasp.org/index.php/OWASP_Mobile_Security_Project#tab=Mobile_Cheat_Sheet
Thisinformationcanbeveryusefulduringboththesecurityrequirementsanalysisphaseandtheactualsecuritytestingphase.AnotherinterestingandusefulpartoftheOWASPprojectisthelistof“Top
10MobileRisks.”22Thelistincludesthefollowingtopics,withagoodexplanationofeachtopicandpossiblesolutions:22.www.owasp.org/index.php/OWASP_Mobile_Security_Project#tab=Top_10_Mobile_Risks
1.WeakServerSideControls2.InsecureDataStorage3.InsufficientTransportLayerProtection4.UnintendedDataLeakage5.PoorAuthorizationandAuthentication6.BrokenCryptography7.ClientSideInjection8.SecurityDecisionsviaUntrustedInputs9.ImproperSessionHandling10.LackofBinaryProtection
MobileSecurityTestingSummarySecuritytestingisacomplexanddifficultpartofthesoftwaredevelopmentandtestingprocess.Itrequiresspecialskillsandknowledgeoftechnologytoensurethatthesoftwareorappissecureagainstattacksandvulnerabilities.Youasa
mobiletestershouldalwayskeepsecuritytestinginmind.Talkaboutthetopicasearlyaspossibleintherequirementsanalysisphase.Ifyou’renotsurewhetheryoucantestthesecurityoftheappsufficiently,youshouldgetsomehelpfromexpertsasearlyaspossibleintheprocess.Keepthefollowingpointsinmindtoensureasecuremobileapp:•Testtheappfordifferentinputs;forexample,haveapreparedlistofpossibleattackstringsavailable.•Dopenetrationtestingontheappandthebackendsystem.•Useaproxy,afuzzer,andascannertoverifyeachpartoftheappandthebackendarchitecture.•Checktheappforcommonmistakes.•HavealookattheOWASPMobileSecurityProjectandfollowtheguidelinessetoutthere.•Checktheapp’scertifications.•Keepup-to-datewiththelatestmobiletechnologiesandsecuritynews.•Hireexperts.
ImportantThesecuritytopicscoveredinthischapterareaveryroughoverviewofpossiblethingstoconsider.Mobilesecurityisahugeandcomplextopic,sopleasescourtheInternetforfurtherinformation.
PlatformGuidelineTestingAnothertopicyouneedtobeawareofisthemobile-platform-specificguidelines.Eachapphastofollowsuchguidelinesinordertomeetthedesign,usability,andplatform-specificpatterns.Iftheappfailstofollowtheseguidelines,itmightberejectedfromtheappstore.Inaddition,followingthedesignprincipleswillmakeyourcustomershappybecausetheyknowhowtouseplatform-specificfeaturessuchasswipingfromlefttorighttoswitchviews,ortopulltorefreshandupdatethecurrentview.Itisimportantthatsuchguidelinesbeincludedinthedesignphaseofanapp.
Followingtheguidelinesfromtheverybeginningoftheprojectwillsaveyoutimeattheendbecauseyouwillencounterfewerbugsthanyouwouldbynotfollowingtheguidelines.
Whentestingamobileapp,keeptheplatform-specificguidelinesinmind.Youshouldknowthembyheartoratleastknowwheretofindthem:
•AppleiOSdesignresources(https://developer.apple.com/library/ios/design/index.html#//apple_ref/doc/uid/TP40013289•AppleiOSapprovalguidelines(https://developer.apple.com/appstore/resources/approval/guidelines.html•Androidguidelines(https://developer.android.com/design/patterns/index.html)•WindowsPhoneguidelines(http://dev.windowsphone.com/en-us/design)•BlackBerryguidelines(http://developer.blackberry.com/design/bb10/)
ImportantMakesureyou’refamiliarwiththelatestmobileplatformguidelines.Theychangewitheverynewmobileoperatingsystemversionthat’sreleased.
ConformanceTestingConformancetestingaimstoensurethatthesoftwaremeetsadefinedsetofstandards.ThesestandardsaredefinedbyindependententitiessuchastheEuropeanTelecommunicationsStandardsInstitute23(ETSI),theWorldWideWebConsortium(W3C),24andtheInstituteofElectricalandElectronicsEngineers(IEEE).2523.www.etsi.org/24.www.w3.org/25.www.ieee.org/index.html
Ifyourapphastoimplementdefinedstandards,youmustverifythatthosestandardsaremet.Itisimportantforyoutobeawareofthosestandardsandthatyouknowhowtocheckthem.However,thestandardsshouldalreadyformpartofyourrequirementsanalysisphase.Whenitcomestothedevelopmentphase,talktoyourdevelopersaboutimplementingthestandardsbyaskingquestions,performingacodereview,andwalkingthroughthedefinedstandardsdocumentation.Dependingonthemobileapptypeandpurpose,anexternalinstitutionmaybe
requiredtoverifyyourappinordertocheckthatyoucorrectlyimplementedtheirstandards.Thiswillbethecasewithmedicalappsorappsthatareusedbygovernments.Insomecases,theapphastobetestedforlegalcompliance.
ImportantKeepconformancetestinginmindandaskquestionsattheverybeginningoftheproject.
CheckingtheLogFilesWhiletestingyourmobileapp,connectyourdevicetoyourcomputerandchecktheapp’slogfileswhileusingit.Togetaccesstothelogfiles,youneedtoinstallthedevelopertoolsonyourcomputersoyoucanruntheappindebugmode.Whencheckingthelogfiles,watchoutforerrors,warnings,orexceptionsthat
occurwhileusingtheapp.Savetheinformationfromthelogfilealongwithstepsforhowtoreproducetheproblemandpasstheinformationontothedevelopers.Logfileinformationisextremelyusefultodevelopersforfindingandsolvingproblems.Besideslookingforerrorsandcrashes,youshouldlookforsensitivedatasuchastokensorpasswordsthatarevisibleinthelogs.Whentestingyourappwithoutacableconnectiontoyourcomputer,andthe
appcrashesorexhibitsstrangebehavior,leavetheappandthedeviceinthisstateandconnectitassoonaspossibletoyourcomputerandthedevelopertoolssoyoucancheckthelogfilefortheerror.Beforesubmittinganapptotheappstores,youhavetocheckthelogfilesfor
debuginformation,warnings,anderrorsagain.Debuginformationaswellaswarningsanderrorsshouldnotbepartofthereleaseversionoftheapp.Alsokeepthelogfilesofthebackendsystemsinmindastheylogdifferent
kindsofinformationsuchasrequestsandresponsesaswell.
ImportantSomebugsarevisibleonlyintheapp’slogfiles.Suchbugsmaynotbeshowntotheuserbutcouldhaveahugeimpactonthefunctionalityoftheapp.
BeCreative!Upuntilthispoint,thisbookhaspresentedasystematicapproachtomobiletestinganditsspecifics.Buteveryonewhohaseffectivelytestedsoftwareknowsthatmostofthetime,thereallynastybugsdon’toccursystematically.Bugscomeindifferentshapes,sizes,andsituations,andsometimesit’sreallyhardtofindthem.
Afteryouhavetestedyourmobileappinasystematicway(orbeforestartingwiththesystematicapproach),itisusefultothinkoutofthebox.Leanback,stepawayfromallthetestcasesandsystematicapproaches,andjusttrytobreakit!Testtheappforalimitedtimeframe—30minutes,forexample—andtrytofindsomebugsusingyourowncreativeapproaches.Becreativewithyourtesting.Trytothinkofedgecasesthataremostlikelyto
happenintherealworldwhenlotsofpeoplewillbeusingthemobileappinatotallydifferentwayfromwhatyouandyourteamexpect.Docrazythingswiththeappandthedevice.Rotateorflipthedevice,use
morethanonehandtointeractwiththeapp,andpressbuttonsasfastaspossible.Whiledoingthat,watchoutforanyunexpectedbehavior,crashes,freezes,
errormessagesthatmakenosense,andanyotherstrangethings.Isuggestthatyourecordthiskindoftestsessionbecauseitisverylikelythatanybugsthatoccurwillbehardtoreproduce.Anotherapproachcanbepairtesting,whichinvolvesonepersontestingtheappwhileanotherpersonwatchesandtakesnotesabouttheperformedtests.
Checklists,Mnemonics,andMindMapsThispartofthechapterisallaboutreminders.Yourdailytestingbusinessisstressfulandtherearesomanythingstoremember,do,andexplore.Youmaywanttousesomemodelsortoolstopreventyoufromforgettingimportantaspectsofyourwork.Threeveryniceapproachesyoucandrawuponare
•Checklists•Mnemonics•Mindmaps
ChecklistsEverynewfeaturethatwillbedevelopedneedstobetested.Softwaretestersusuallydefinetestcasesinordertosystematicallytesteverynewfeatureandavoidforgettinganything.Whenthetestcasesarefinished,asoftwaretesterusuallyprioritizesthemanualtestcasesforthetestautomationtoextendtheregressiontestsuiteandavoidmanuallytestingthenewfeatureoverandoveragain.However,noteveryfeatureortestcasecanbeautomatedbecausethetestmaybetoocomplexandwillmostlikelyendupcausinglotsofmaintenancework.Butwhatcanyoudoifsuchatestorfeatureiscriticalfortheapporthe
business?Neverexecutingitagainissimplynotanoption.Tohelpyouavoidforgettingimportantthings,youmayfinditusefultowrite
achecklistwithpartsoftheappthatneedtobemanuallytestedagainbeforegoingliveorafteraprojectmilestonehasbeenreached.Checklistscanbeverygenericandhighlevelsothattheycanbeusedin
severaldifferentprojects,ortheycanbereallylowlevelwithspecificstepstotestacertainfeature.ThemobiletestingchecklistinTable4.1isagenericonethatcoverslotsof
importantaspectsofthemobileapptestingprocess.Thislistcanbeusedforseveraldifferentmobileapps.
Table4.1GenericMobileTestingChecklist
ImportantThislistisbynomeanscomplete;itismerelyanexampleofwhatamobiletestingchecklistcouldlooklike.
Thischecklistshowssomepossiblegenericmobileteststhatcouldbeexecutedontopofyoursystematictestingapproach.Itmayalsobeusefultocreateareleasechecklistforthewholeteamsotheydon’tmissanythingimportanttotheprocess.However,checklistscanbereallyspecifictocoveraveryspecialpartoftheapp,suchasthepaymentandbillingprocesses,orthedifferentsensorsandinterfaces.
ImportantBecreativewithchecklists.Createyourowncheckliststhatarespecifictoyourappandproject.
MnemonicsSFDPOT,FCCCUTSVIDS,andISLICEDUPFUNarenottypos.They’reabbreviationsformnemonics.Mnemonicsarelearningtechniquesthataidinformationretention.Whenitcomestomobileapptesting,eachletterstandsforatestingapproachortechniqueusedforsoftwareapplications.Mnemonicsareusedtorememberimportantaspectswhiletestingsoftwareapplicationsandareveryusefulforsoftwaretestersbecausetheyareeasytorememberandincludepowerfulideasandbestpracticesonhowtotestsoftware.SFDPOT26isamnemonicfromKarenNicoleJohnsonthatsheadaptedfrom
JamesBach’smnemonicSFDPO27(betterknownas“SanFranciscoDepot”)onmobiletesting.Karenincludedthefollowingpoints:26.http://karennicolejohnson.com/2012/05/applying-the-sfdpot-heuristic-to-mobile-testing/27.www.satisfice.com/articles/sfdpo.shtml
•Structure:Testwhatit’smadeof.•CanIdownloadtheapp?•CanIdownloadanupdate?
•Function:Testwhatitdoes.•Doestheapporsiteperformthetasksorfeaturesitwasdesignedto?•Doestheapporsiteprevent,block,ornotincludefeaturesnotintendedonamobiledevice?
•Data:Testwhatitdoesitto.•Doestheappfindtime-relateddatabasedonthedevicetime?•Doestheappfindlocationsbasedonmylocation(suchasmovietheatersorhotels)?
•Platform:Testwhatitdependsupon.•Doestheappuselocationservices?•Doestheappdependonanydevicesettings?
•Operations:Testhowit’sused.•DoestheappfunctionwhenIammovingandtravelingaround?•WhathappenswhenIswitchtoWi-Fiversus3G?
•Time:Testhowit’saffectedbytime.•Whathappensifthetimezoneisswitched?•Whathappenswhenmylocationisswitched?
Foreachpart,Karenwrotesomequestionstoaskwhiletestingamobileapp.Checkoutthefulllistofquestionsinherblogpost“ApplyingtheSFDPOTHeuristictoMobileTesting.”Useandadaptthemtotestyourapp.AnothermnemonicisFCCCUTSVIDS28byMikeKelly,whonoteddown
hisideasintourstoexploreandtestapplicationsinseveralways:28.http://michaeldkelly.com/blog/2005/9/20/touring-heuristic.html
•Featuretour•Exploretheapplicationandgettoknowallthecontrolelementsandfeatures.
•Complexitytour•Findthemostcomplexpartsoftheapplication.
•Claimstour•Findalltheinformationaboutthesoftwarethattellsyouwhattheproductdoes.
•Configurationtour•Findallthewaystochangethesettingsoftheapplication.
•Usertour•Imaginehowthepossibleusersoftheapplicationwillusethesoftwareandwhattheyexpectfromit.
•Testabilitytour•Findallfeatureswithinthesoftwarethatcouldbetestedwiththehelpoftools.
•Scenariotour•Imaginepossiblescenariosastohowthesoftwarewillbeusedbyitsusers.
•Variabilitytour•Trytofindwaystochangetheapplication.
•Interoperabilitytour•Doesthesoftwareinteractwithothersoftware?
•Datatour
•Findthedataelementsoftheapplication.•Structuretour•Gatherasmuchinformationaspossibleabouttheapplication,suchastheprogramminglanguage,APIs,hardware,andsoon.
Thedescribedtoursareanexcellentwayoffindingoutpossiblesettings,features,orconfigurationsofthesoftwareapplication.Besidesthat,thetourscovertheuser’sperspectiveandareidealforexploringtheapplication.Allofthementionedtourscaneasilybeadaptedtomobileapps.AnothergreatmobiletestingmnemonicisISLICEDUPFUN29byJonathan
Kohl,whoalsoadaptedhismnemonicfromJamesBach’sSFDPOforusewithmobileapps.Hecoveredveryspecificmobileareasthathavetobetestedduringthedevelopmentphase:29.www.kohl.ca/articles/ISLICEDUPFUN.pdf
•Inputsintothedevice•Built-inkeyboard/keypad•Touchscreengesturesandtyping•Syncingwithotherdevices
•Store•Submissionspecifications•Developmentguide•Userguideforerrorhandling,locationservices,permissionsforuserprivacyitems,accessibility,andsoon
•Location•Geolocationerrors•Movement,stoppingsuddenly•Connectionissuesduetointerference
•Interactions/interruptions•Runningmultipleapplications,multitasking•Usingotherapplications,thenusingtheapplicationyouaretesting(e-mail,calendar,texting,notetaking,others)•Notificationsappearing
•Communication•Phone
•Texting•E-mails
•Ergonomics•Smallscreenscanbehardontheeyes.•Asmalldevicemeansthereisnoergonomichelpfromadeskorchair—youoftenhunchovertointeractwithit.•It’snotuncommontogetasoreback,fingers,oreyeswhenusingadeviceforanylengthoftime.
•Data•Typesofinput—seeiftheappusesspecialcharacters,differentlanguages,andsoon.•Media—seeiftheappdependsonanoutsidesourcetoplaymusic,videos,oranythingelse.•Sizeoffiles—iftheapplicationusesoutsidefiles,tryusingdifferentfiletypes.
•Usability•Noteandloganyissuesthatmakeyouuncomfortable,frustrated,angry,orupsetwhileusingtheapp.
•Platform•Android•iOS•WindowsPhone•BlackBerry
•Function•Canyouidentifyeverythingthattheapplicationdoes?•Haveyouworkedthroughalltheaspectsoftheapp?Clickedeverybutton?Filledineveryform?•Tryatouroftheproducttoidentifyeverythingitdoes.
•Userscenarios•Howisthisapplicationsupposedtobeused?•Whatproblemsdoesitsolveforusers?•Whatarethegoalsofendusersthatthisapplicationhelpsthemsolve?
•Network
•Wi-Fi•Wirelessbroadband•Deadspots
ISLICEDUPFUNisaverygoodmnemonic,andmobiletestersshouldknowitinordertoimprovetheirtestingworkandtoremembertheimportantmobileaspects.Besideshismnemonic,JonathanwroteanexcellentbookaboutmobiletestingcalledTapintoMobileApplicationTesting,30whichisavailableonLeanpubandcoverslotsofmobiletestingtopics.Thetourchapterisreallyinterestingandanexcellentsourceofknowledge.30.https://leanpub.com/testmobileapps
ThelastmnemonicaboutmobiletestingIwanttomentionisCOPFLUNGGUN,31createdbythecompanyMoolya.Thismnemoniccoversthefollowingtopicsandisworthlookingat:31.http://moolya.com/blogs/2012/04/121/Test-Mobile-applications-with-COP-who-FLUNG-GUN
•Communication•Orientation•Platform•Function•Location•Userscenarios•Notifications•Guidelines•Gesture•Updates•Network
ImportantMnemonicsaregreatapproachesyoucanusetoremindyourselfofimportantthings.Theywillhelpyouorganizeyourtestingapproach,testingstrategy,andthoughts.
MindMapsMindmapsareanothergreatwaytovisuallyorganizeinformationastheycanhelpyoutoprocessyourthoughtsandideasoncertaintopicssuchasmobile
testing.RosieSherry,thewomanbehindMinistryofTesting,32aplatformandprofessionalcommunityforsoftwaretesterswholeadthewayinsoftwaretestingcontent,training,andevents,createdandpublishedtwogreatmindmapsaboutmobiletesting.OneofhermindmapsisshowninFigures4.3through4.8.
Figure4.3Mobiletestingmindmap.CourtesyofRosieSherryandKarenNicoleJohnson.Foralargerversionofthisimage,see
informit.com/title/9780134191713
Figure4.4Mobiletestingmindmap:mobileplatforms.CourtesyofRosieSherryandKarenNicoleJohnson
Figure4.5Mobiletestingmindmap:mobileoperations.CourtesyofRosieSherryandKarenNicoleJohnson
Figure4.6Mobiletestingmindmap:mobilefunctions.CourtesyofRosieSherryandKarenNicoleJohnson
Figure4.7Mobiletestingmindmap:mobilestructure.CourtesyofRosieSherryandKarenNicoleJohnson
Figure4.8Mobiletestingmindmap:mobiledata.CourtesyofRosieSherryandKarenNicoleJohnson
32.www.ministryoftesting.com/
BothmindmapscanbedownloadedinhighresolutionfromMinistryofTesting133andMinistryofTesting2.34ThemindmapsRosiecreatedarebasedonthemnemonicfromKarenNicoleJohnson.33.www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-mindmap/34.www.ministryoftesting.com/2012/05/mobile-testing-course-pictures-and-a-mindmap/
AnothergreatinteractivemindmapprojectistheTestingMap,35whichcoverslotsofdifferentareasofsoftwaretestingandprovidessomereallyusefulideas.35.http://thetestingmap.org/
ImportantTrytocreateyourownmindmapbasedonyourappbyaddingpossibletestingtasks.Furthermore,printitoutandputitupintheoffice.Thatway,yourcolleagueswillstandtobenefitfromitaswell.
HowtoFileMobileBugsWe’venowcometothelasttopicinthischapter,“HowtoFileMobileBugs.”Ifyoufindabugwithinamobileapp,youneedtoreportitinordertogetitfixed.Filingmobilebugreportsrequiressomeadditionalinformationthatthe
developersneedinordertoreproduceandfixthebug.Butwhatisimportantwhenfilingamobilebug?Whatshouldabugreport
looklike?BeforeIanswerthosetwoquestions,Iwanttoraiseanotherone:Whyevensendabugreport?Bugreportsareveryimportantfortheproductowner,productmanager,and
thedevelopers.First,abugreporttellsthedevelopersandtheproductowneraboutissuestheywerenotawareof.Italsohelpstoidentifypossiblenewfeaturesnoonehasthoughtof,andlastbutnotleast,itprovidesusefulinformationabouthowacustomermayusethesoftware.Allofthisinformationcanbeusedtoimproveyoursoftware.Wheneveryoufindsomethingstrange,ifsomethingbehavesdifferentlyor
looksweird,don’thesitatetofileabugreport.Let’scometothequestionofwhatabugreportshouldlooklikeandwhat’s
importantwhenfilingit.Abugreportshouldcontainasmuchinformationaspossibleinorderto
identify,reproduce,andfixthebug.Thatsaid,yourreportshouldonlyincludeinformationthat’simportanttohandlingthebug,sotrytoavoidaddinganyuselessinformation.Anotherimportantpointisthatyoushoulddescribeonlyoneerrorperbugreport.Don’tcombine,group,orcreatecontainersforbugs.It’slikelythatnotallofthebugswillbefixedatthesametime,sorefrainfromcombiningorgroupingthem.Theinformationdescribedinthefollowingsectionsshouldbeincludedina
bugreport.
BugIDAbugmusthaveauniqueidentifiersuchasanumberoracombinationofcharactersandnumbers.Ifyou’reusingadefectmanagementtool,thetoolwillhandlethebugIDsforyou.Ifnot,thinkaboutauniqueIDsystemforyourproject.
•Bad:123isauniqueID,butyoumighthaveseveralprojectswhoseIDsarethesame.•Good:AppXYZ-123isgoodbecauseyou’recombininganIDwithaprojectabbreviationandanumber.
DescriptionCreateashortbutmeaningfuldescriptioninordertoprovidethedeveloperwith
aquickoverviewofwhatwentwrongwithoutgoingintodetail.Youshould,forexample,includeerrorcodesorthepartoftheapplicationwherethebugoccurred.
•Bad:“Theappcrashed,”“Whitepage,”“Sawanerror,”“Bug”•Good:“ErrorCode542ondetailmessageview”or“Time-outwhensendingasearchrequest”
StepstoReproduceThisisoneofthemostimportantpoints.Providetheexactstepstogetherwiththeinputdataonhowtoreproducethebug.Ifyouareabletoprovidethiskindofinformation,thebugwillbeveryeasytofixinmostcases.
•Bad:“Itriedtoexecuteasearch.”•Good:“Starttheappandenter‘MobileTesting’intothesearchinputfield.Pressthesearchbuttonandyou’llseetheerrorcode783onthesearchresultpageheader.”
ExpectedResultInthissectionyoushoulddescribewhatyouexpectedtohappenwhenthebugoccurred.
•Bad:“Itshouldwork”or“Ididn’texpectittocrash.”•Good:“Iexpectedtoseeasearchresultspagewithascrollablelistof20entries.”
ActualResultWhathappenedwhenthebugoccurred?Writedowntheactualresult,whatwentwrong,ortheerrorthatwasreturned.
•Bad:“Itjustwon’twork.”•Good:“Thesearchresultspagewasempty”or“Igottheerrorcode567onthesearchresultspage.”
Work-aroundIfyou’vefoundawaytocontinueusingtheappandavoidthebug,explainyoursteps.Thosestepsareimportanttoknowsincethework-aroundcouldcauseotherproblemsorindicateawayinwhichtheappshouldnotbeused.Ontheotherhand,awork-aroundcanbeveryusefulforthecustomersupportteaminordertohelpcustomerssolvethecurrentproblemuntilthebuggetsfixed.
•Bad:“Ifoundawork-around.”•Good:“Ifyouputthedeviceintolandscapemode,thesearchbuttonisenabledandtheusercansearchagain.”
ReproducibleIfyoufoundareproduciblebug,that’sfine,butdoesitoccureverytime?Ifithappenseverytime,that’sgreatasitshouldbeaneasyfixforthedeveloper.Butifthebugoccursonly20%ofthetime,itismuchhardertofindasolution.Makesureyouprovidethisinformation,asitisveryusefulforthedeveloperandwillpreventthebugfrombeingclosedwiththecomment“Can’tbereproduced.”
•Bad:“Sometimes”•Good:“Thebugoccurstwooutoftentimes.”
OperatingSystem,MobilePlatform,andMobileDeviceAnotherimportantcomponentofabugreportisinformationabouttheoperatingsystem,themobileplatform,andthemobiledevice.Writedowntheoperatingsystem,mobileplatform,anddeviceonwhichthebugoccurred.
•Bad:“OnAndroid”or“OniOS”•Good:“Android,Version4.1.2GoogleNexus4”or“iOS,Version6.1iPhone4S”
Mobile-Device-SpecificInformationMobiledeviceshavelotsofinterfacesandsensorsthatcouldhaveanimpactonyourapp.Thebatterycouldalsoaffecttheappyou’retesting.Writedownallofthisinformationinyourbugreport.
•Bad:Noinformation•Good:“GPSsensoractivated,changedtheorientationfromlandscapetoportraitmode”or“Usedthedeviceinasunnyplace”or“Batterystatewas15%”or“Batterystatewas100%.”
BrowserVersionIfyourappisamobileWebappandyoufoundanissue,it’sveryimportanttonotethebrowserversionwhereyoufoundthebugasitmayoccuronlyinacertainbrowserversion.
•Bad:“GoogleChrome”or“MozillaFirefox”
•Good:“GoogleChromeversion45.35626”or“MozillaFirefox27.6”
SoftwareBuildVersionAnotherreallyusefulpieceofinformationisthecurrentbuildversionoftheappwherethebugoccurred.Thiswillpreventthedeveloperfromwastingtimetryingtoreproduceabugthat’salreadybeenfixedinthecurrentcodebase.
•Bad:Noinformation•Good:“Appbuildversion1.2.3”
NetworkConditionandEnvironmentWhenfilingamobilebug,it’simportanttoprovidesomeinformationaboutthenetworkconditionandtheenvironmentinwhichthebugoccurred.Thiswillhelptoidentifytheproblemmoreeasilyandwillpossiblyshowsomesideeffectsnoonehasthoughtof.
•Bad:Noinformationor“Happenedonmywaytowork”•Good:“Iwasconnectedtoa3GnetworkwhileIwaswalkingthroughthecitycenter.”
LanguageIfyourappsupportsseverallanguages,providethisinformationinyourbugreport.
•Bad:Noinformation•Good:“IwasusingtheGerman-languageversionoftheapp.”
TestDataAcompletebugreportmustincludethetestdatathatwasusedtoreproducethebug.Simpletestdatacanbelogincredentialswithusernameandpassword.However,insomecasesitisnotenoughtoprovidejustausernameandpassword.Itisverylikelythatyouneedtoprovidecompletetestdatasets,forexample,asanSQLdumporatestdatascriptthatwillgeneratethenecessarydata.
•Bad:Noinformation•Good:“FindtheattachedSQLscripttoputthedatabaseinthedefinedstate”or“Enter‘MobileTesting’intothesearchinputfield.”
Severity
Everybugyoufindneedsaseveritylevel.Eitheryourdefectmanagementtoolwillofferyousomecategories,oryouhavetodefinethemwithyourteam.Itisimportanttogiveabugaseveritylevelasitwillallowtheteamtoprioritizetheirbug-fixingtimesothatcriticalandhigh-prioritybugswillbefixedfirst.Ifthisinformationisnotprovided,ittakesmuchmoretimetofindthebugsthatneedtobefixedbeforetherelease.ThedefaultseveritiesareCritical,High,Medium,andLow.
•Bad:Noinformation•Good:“Critical”or“Medium”
BugCategoryInadditiontotheseveritylevel,thebugcategoryisalsoaveryusefulpieceofinformation.Theproductownerorthedevelopercanfilterbycategorytogetanoverviewofthecurrentstatusofbugspercategory.Forexample,iftherearelotsofUXbugs,thismaybeanindicatorofapoorUIandUXoramissingdesignexpertintheteam,meaningthattheappneedssomemoredesignimprovements.
•Bad:“Noinformation”•Good:“Functionality”or“UX”or“Performance”
ScreenshotorVideoWheneveryoufindabug,trytocreatescreenshotsoravideotoprovidethedeveloperwithmoreinformation.Whenprovidingascreenshot,useanimage-editingtooltomarkthebuginthescreenshot.Avideoisalsoagreatwaytodescribeabugyou’vecomeacross.Itisalsoveryusefultogivethescreenshotorthevideoagoodnameordescription.
•Bad:“Noscreenshotsorvideosattached”or“Screenshot1.png”•Good:“01_InsertSearchTerm.png,02_SearchResultPageWithError.png”
LogFilesIfyourappcrashesorfreezes,connectthedevicetoyourcomputerandreadoutthelogfiles.Inmostcasesastacktracewillbeshownwithadescriptionoftheerror.Thiskindofinformationisextremelyusefulfordevelopers,astheyknowrightawayinwhichclassthebugortheerrorhasoccurred.
•Bad:“Noinformationprovidedwhentheappcrashed.”•Good:“Providedthefullstacktraceinthebugreport”or“Attachedthelogfiletothereport.”
TesterWhoFoundtheBugWritedownyournameorthenameofthetesterwhofoundthebug.Developersorproductownersmayhavesomequestionsaboutthereportedbug,andtheywillofcoursewanttogetintouchdirectlywiththetesterwhofoundit.Inmostcases,thisisautomaticallydonebythedefectmanagementsystemwhereeachuserhashisorherownaccount.Ifnot,makesureyouaddyoure-mailaddressand/orphonenumber.
•Bad:Noinformation•Good:“DanielKnott,[email protected]”
ThreeMorePointsAsyouhaveseen,thereisalotofinformationthatshouldbeincludedinabugreport.Therearethreeotherpointsyoushouldkeepinmindwhenwritingabugreport.ThefirstoneisDon’tgetpersonal.Whenfilingabugreport,describethe
softwaremisbehaviorratherthanthedeveloper’smindsetorthequalityofhisorherwork.Don’tuseoffensiveoremotionallychargedwordsasthosekindsofreportswillbeignoredbythedeveloperandwillendupcausingbadbloodwithintheteam.ThesecondoneisIt’snotyou.It’snotyourfaultthatthebugoccurred.Itis
thesoftwarethat’sbroken,andyouandyourcolleaguesneedtofixit.AndthethirdpointisKeepitsimple.Trytowriteyourbugreportinsucha
waythatsomeonewithnoideaabouttheprojectortheappisabletounderstandtheproblem.Ifthebugreportisthateasy,everydeveloperwithintheteamwillbeabletofixitandnontechnicalcolleaguescanunderstandtheproblemandwillvalueyourwork.
AppQualityAllianceIfyouwanttogetfurtherinformationabouthowtotestyourAndroidoriOSapp,havealookatthenonprofitgroupAppQualityAlliancesite.36ThisgroupisheadedbydifferentcoremembersandknowledgecontributorssuchasAT&T,LGE,Microsoft,Motorola,Oracle,Samsung,andSonyMobile.Themainaimofthegroupistoworkverycloselywiththeindustrytoimprovethequalityofmobileapps.36.www.appqualityalliance.org/
ThegroupdevisedtestingcriteriaforAndroidandiOSapps.Youcan
downloadbothPDFdocumentsviathefollowinglinks:•TestingcriteriaforAndroidapps(www.appqualityalliance.org/AQuA-test-criteria-for-android-apps)•TestingcriteriaforiOSapps(www.appqualityalliance.org/AQuA-test-criteria-for-iOS-apps)
Bothdocumentscontainlotsoftestcasesthatyourappshouldrunthroughbeforeyousubmitittotheappstores.Thegroupalsoprovidesinformationaboutmobileappperformancetestingas
wellasbest-practiceguidelinesfordevelopingqualitymobileapps.Youcandownloadbothdocumentsviathefollowinglinks:
•Performancetestingcriteria(www.appqualityalliance.org/aqua-performance-test-criteria)•Best-practiceguidelinesfordevelopingqualitymobileapps(www.appqualityalliance.org/AQuA-best-practice-guidelines)
Thegroup’sWebsiteisdefinitelyworthcheckingout,andmaybeyou’devenliketocontribute.
SummaryChapter4isoneofthemainchaptersofthisbookandcontainslotsoftestingideasandsolutionsthatcanbeusedinyourdailyworkinglife.Thechapterstartedwithadescriptionofthedifferencesamongemulators,simulators,andrealdevicesandwhatisimportanttoknowaboutthem.Beforethehands-onpartofthechapterIexplainedthedifferencebetweenmanualandautomatedtestingaswellastheroleoftraditionaltestinginthemobiletestingbusiness.Ifyouarefamiliarwith“traditional”softwaretestingsuchasforWebordesktopapplications,thatknowledgeisalsousefulformobileapps.LotsofthetestingapproachesforWebordesktopapplicationsarealsovalidformobileappssuchaswhiteboxandblackboxtestingtechniques.However,whenitcomestomobiletesting,therearemoreareasyouneedtotestinordertobesurethatyourappisworkingondifferentlevelsandwithdifferenthardwareandsoftwareversions.InthischapterIdescribedthefollowingmobile-relatedtopics:
•Mobileusabilitytesting•Accessibilitytesting•Batteryusagetesting•Stressandinterrupttesting•Performancetesting
•Standbytesting•Installationtesting•Updatetesting•Databasetesting•Localstoragetesting•Securitytesting•Platformguidelinetesting•Conformancetesting•Checkingthelogfiles
Ialsoprovidedchecklistsofsampletestcasesthatcanbeexecuted,suchasbeforetheappissubmittedtoanappstore.Furthermore,Ishoweddifferentmnemonicsandmindmapsthatcanbeusefulforyourmobiletestingactivities.Thechapterendedwithadetailedoverviewofhowtofileamobilebugwith
lotsofdo’sanddon’ts.
Chapter5.MobileTestAutomationandTools
Thischapterisallabouttools.Idescribethedifferenttypesandconceptsofmobiletestautomationtoolsandgiveyousomeadviceonwheretoautomateandhowtoselectamobiletestautomationtoolforyourappanddevelopmentenvironment.Ialsoexplaincontinuousintegrationandbetadistributiontools.Thischapterdoesn’tcoverhowtoinstallandconfigurethedifferentmobile
testautomationtools.Thereasonforthisissimple:Thetoolsprovideinstallationandconfigurationguidesalready,soyoucanjustvisittheirWebsitestofindalltheinformationyouneed.Detailsabouteachtoolwillbeprovidedintherelevantsectionsofthischapter.
TheFlippedTestingPyramidBeforeIstartwiththemobiletestautomationtools,Iwanttobrieflyexplainthetestautomationpyramid.AnyonewhoisinvolvedinsoftwaretestingandsoftwaretestautomationshouldknowthetestautomationpyramidintroducedbyMikeCohn.1
1.www.mountaingoatsoftware.com/
AsyoucanseeinFigure5.1,thetypicalpyramidconsistsofthreelayers.Atthebottomthereistheautomatedunittestinglayer;inthemiddle,theautomatedintegrationtestinglayer;andatthetop,thereistheautomatedend-to-end(E2E)testinglayer(includingtheuserinterfacetests).Eachlayerhasadifferentsize,indicatingthenumberofteststhatshouldbewrittenwithineachstage.Manualtestingisnotpartofthetestpyramid;henceitisshownasacloudforadditionaltestingwork.
Figure5.1Defaulttestautomationpyramid.BasedonafigurebyMikeCohn.
Butthispyramidisnotapplicabletomobileappsandmobiletestautomation.Asyoulearnedseveralchaptersago,mobiletestingrequiresatotallydifferentsetoftests—movement,sensors,differentdevicesandnetworks—fromothersoftwaresuchasdesktoporWebapplications.Lotsofmanualtestingisrequiredtobesurethatamobileappisworkingasexpectedinthedifferentusagescenarios.Mobiletestautomationtoolsarenotcurrentlyasmatureastheircounterparts
forWebanddesktopapplications,whichleadstoaflippedtestautomationpyramid.Asthetoolsbecomeincreasinglymature,thispyramidislikelytoflipbackagainbecausethedefaulttestautomationpyramidisbasedonamorestablefoundation(seeFigure5.1).Thedefaultpyramidthereforecan’tbeusedasan
indicatoroftestautomationandmanualtestinginthemobileworld.TheflippedtestingpyramidlookslikeFigure5.2.
Figure5.2Theflippedtestingpyramid
Inthisversionofthepyramid,theautomatedunittestinglayeristhesmallestone.Thisisthecasebecausenoteveryunitormethodofmobileappscanbetestedinanisolatedmanner.Insomecases,differentAPIs,layers,orsystemsmayneedtobefakedormockedinordertogetthesmallunittowork.Thisisalsothecaseforeveryothersoftwareapplication,butinsomecasesmockingorfakingothersystemsformobileappsismuchmorecomplex.Thisisnotefficientfromatechnicaloreconomicpointofview.However,it’snoexcusefornotwritingmobileunittestsatall.Thebusinesslogicofanappmustbetestedattheunitlevel.Thenextstageistheend-to-endtestautomationlayer.Withinthislayer,the
appistestedfromauserperspectivetomakesurethewholesystemisworking,fromtheapp’suserinterfacethroughtothebackendsystemviaawirelessnetwork,includingintegrationtestingwithdifferentlibrariesorAPIs.Theintegrationtestinglayeristhereforepartoftheend-to-endlayer.
Thebiggestchangeinthispyramidisthatmanualtestingispartofit.Mobiletestingrequireslotsofmanualtesting,andthiscan’tyetbereplacedbytestautomationoranyothertools.Nevertheless,mobiletestautomationisareallyimportanttopic,andevery
mobiletestershouldbeabletowriteautomatedregressionteststhatprovidefastfeedbackaboutthecurrentqualitystateofanapp.Furthermore,testautomationhelpstheteambuildareliableandrobustmobileappthatmakesthecustomershappy.
TheMobileTestPyramidTheflippedtestingpyramidhasnostablefoundation,andmobiletestingrequireslotsofmanualtesting,whichiswhyIcreatedmyownmobiletestpyramidconsistingoffourlayersincludingmanualandautomatedsteps(seeFigure5.3).Thebiggestlayerofthepyramid,manualtesting,formsthestrongfoundationforeverymobileappproject,followedbyend-to-endtesting,betatesting,andatoplayercomprisingunittesting.Thegraypartsofthepyramidindicatetheautomatedsteps,andthewhitepartsarethemanualtestingsteps.Thebetatestinglayerisnewtothepyramidbutessentialtoeverymobileappproject.Keepingthehighexpectationsofmobileusersinmindrequiresthatthislayerbepartofeverymobileprojecttogetearlyfeedbackfromyourmobilecustomers.Youcaneitheruseacrowdtestingapproachforyourbetatesting,oryoucanaskyourcolleaguestobeta-testearlyversionsofyourapptoprovideimportantfeedback.
Figure5.3Themobiletestpyramid
Ihaveusedthismobiletestpyramidinseveralprojectsandithelpedmesetupareliable,effective,andvaluabletestingprocess.
ImportantKeeptheproblemoftheflippedpyramidinmindandusethemobiletestpyramidinyourprojecttohaveagoodmixofmanualandautomatedtesting.
Inthischapter,Ifocusmoreontheend-to-endtestautomationtoolsbecausethosearethetoolsmobiletesterswillmostlikelyworkwith.However,someunittestingtoolsarealsomentioned.
DifferentTypesofTestAutomationToolsBeforeyoustartwithmobiletestautomation,itisimportantthatyouunderstandtheunderlyingarchitectureofthedifferentmobileplatforms.Youneedtoknowhowtoaccessthedifferentobjectsofapp-likebuttons,labels,lists,views,and
anyotherkindofelementinordertointeractwiththoseelementsduringthetestrun.Youalsoneedtobeabletowritecodeandbuildupreliabletestautomation
suitesthatareintegratedwithinyourapp’sdevelopmentpipeline.Whenselectingatool,youshouldknowthedifferenttypesandconceptsof
mobiletestautomationtoolsandhowthosetoolsaccessthedifferentobjects.Thisisimportanttoknowwhenselectingatoolforyourneedsandprojectbecauseeachapproachhasitsprosandcons.
ImageRecognitionToolsthatusetheimagerecognitionapproachcompareimagesinordertodrivetheuserinterfaceofanapp.Whenwritingthetestautomationscript,youtakescreenshots,forexample,ofbuttonsorlabelsthatareembeddedintoyourscript.Whenthescriptisexecuted,theimagerecognitiontoolcomparesthecurrentscreenwiththestoredbaselineimage.Ifthestoredimageisfoundonthescreen,thescriptwillexecutetheprogrammedsteps.ThosekindsoftoolsarereallyusefuliftheUIoftheappdoesn’tchangetoo
often,isdevelopedforseveralmobileplatforms,andhasexactlythesameuserinterfaceandcontrolelements.Inthatcase,imagerecognitiontoolsareagreatandfastwayofwritingtestautomationforcross-platformmobileapps.Thebiggestdisadvantageofimagerecognitiontoolsisthehighmaintenance
involvedwiththetestscripts.Thescriptswillnotworkiftheorientationofthedeviceischanged,suchasfromportraittolandscapemode,onbiggerscreens,orifthescreenresolutionchanges.Anotherdrawbackistestinganappindifferentlanguages,whichisnotpossiblesincethecapturedimagesareavailableonlyinonelanguageandwillnotworkwithotherlanguages.Herearesomeexamplesofimagerecognitiontools:•eggPlant(www.testplant.com/eggplant/testing-tools/eggplant-mobile-eggon/)•RoutineBot(www.routinebot.com/)•Sikuli(www.sikuli.org/)•TestObject(https://testobject.com/)
Coordinate-BasedRecognitionCoordinate-basedrecognitiontoolsrelyonpredefinedxandyaxiscoordinatestoaccessandinteractwiththeUIelementsoftheapp.Ifthecoordinatesofthe
elementchange,thewholescriptneedstobeadaptedtothenewvalues.Thishasamajorimpactonthemaintenanceofthescriptsinordertobuildreliabletests.Herearesomeexamplesofcoordinate-basedrecognitiontools:•MonkeyTalk(www.cloudmonkeymobile.com/monkeytalk)•PerfectoMobile(www.perfectomobile.com/)
OCR/TextRecognitionMobiletestautomationtoolsthatusetheOCRandtextrecognitionapproachobtainthetextofthecontrolelementsthatarevisibleonthescreenofthemobiledevice.Todetermineifthetextisvisibleonthescreen,OCRtechnologyisused.OCRandtextrecognitiontoolscanhandledifferentscreenresolutions,
orientations,andsizes.However,suchtoolscanverifyonlytextelementsthatarevisibleonthescreen.Ifthetextchangesorisremovedfromtheapp,theUIelementisverydifficult(orimpossible)toidentify.Youarenotabletocheckif,forexample,aUIviewisvisible,ifalistispresent,orifelementswithouttextareshownonthescreen.AnotherdrawbackofOCRrecognitiontoolsisthattheyareveryslowbecausethewholescreenneedstobescannedforthetext.HerearesomeexamplesofOCR/textrecognitiontools:•eggPlant(www.testplant.com/eggplant/testing-tools/eggplant-mobile-eggon/)•MonkeyTalk(www.cloudmonkeymobile.com/monkeytalk)•Robotium(https://code.google.com/p/robotium/)•SeeTest(http://experitest.com/)•TestObject(https://testobject.com/)
NativeObjectRecognitionToolsthatusethenativeobjectrecognitionapproachdetecttheUIobjectswithaUIelementtree.TheUIelementscanbeaccessedusingXPath(XMLPathLanguage),CSS(CascadingStyleSheet)locators,orthenativeobjectIDoftheelement.Thenativeobjectrecognitionapproachisverycommoninthedifferenttestautomationtoolscoveringnative,hybrid,andmobileWebapps.Thisapproachenablesyoutogetaccesstothenativeelementssuchasbuttons,labels,views,lists,andotherkindsofUIelements.IftheIDsorthelocatorsarewelldefinedandwritten,thetestscriptsarereliableintheeventofchangesandthereforereusableonotherdevicesaswell.ThisisahugeadvantagecomparedtoalloftheothertoolsbecausethescriptsdonotdependonchangestotheUI,
resolution,orientation,orthedeviceitself.Themajorityofthetestautomationtoolssupportthisapproachfordetectingtheelements.Herearesomeexamplesofnativeobjectrecognitiontools:•Appium(http://appium.io/)•Calabash(http://calaba.sh/)•Espresso(https://code.google.com/p/android-test-kit/)•Robotium(https://code.google.com/p/robotium/)•Selendroid(http://selendroid.io/)•Selenium(http://docs.seleniumhq.org/)•TenKodEZTestApp(www.tenkod.com/ez-testapp/)
CaptureandReplayLotsoftoolmanufacturersadvertisecaptureandreplayfunctionality.Toolsthatsupportthistechnologyareabletorecordactionssuchasclicking,scrolling,swiping,ortypingintoascript.Withthehelpofthereplayfunction,thesoftwarewillexecutetheexactsameactionsoverandoveragain.Thistechnologysoundsgreatintheory,butinrealityandday-to-daytestautomationwork,captureandreplaytoolsshouldbeusedandhandledwithcare.Butwhy?Ihaveworkedwithdifferentcaptureandreplaytools,andeverytool
presentedthesameproblems.Thecapturedscriptswerenotreliableatall.TheywereaffectedbyUI,orientation,andscreenresolutionchanges(mostofthemusecoordinate-basedorimage-basedrecognition).Thescriptswerenotreusablefromdevicetodeviceorondifferentmobileoperatingsystemversionsofthesamemobileplatform.Oftenthescriptswerenotabletostartbecausethedevicewasnotinadefined
state,meaningthatthetoolwasnotabletoreplaythescript.Oftenthescriptsmustbeeditedmanuallytobemorestableandreliable.Anotherproblemthatalwaysoccurredwasatimingissueasthetoolswereeithertoofastortooslowtointeractwiththeapplicationwhilethescriptwasreplayed.Theonlywaytosolvethisissuewastoaddwaitoperations,butthisisnotagoodapproachforwritingtestautomationscripts.Usingwaitorsleepoperationsisflawedandleadstounreliabletestresults.However,therearealsosomegoodqualities;forexample,mobiletesterswho
havenoprogrammingskillscanusethetoolstogeneratesomebasictestautomationscriptswithverylittletraining.Ontheotherhand,theyarenotabletomodifythescriptafterward.
However,don’tusecaptureandreplaytoolstobuildupahugetestautomationsuiteforyourmobileapp;otherwiseyouwillendupwithanonmaintainabletestsuite.IrecommendthatyouusethesekindsoftoolsasastartingpointtorecordbasicinteractionswithinyourappandgettheUIselectorsoftheelements.TherecordedUIselectorsorIDscanthenbeusedasafoundationforfurtherprogrammingtobuildupatestautomationsuitebasedonyourownprogramming.
ToolTypeRecommendationBeforeIrecommendatooltypeandapproach,Iwanttobrieflysummarizethemobiletestautomationtoollandscape.Mostofthetoolsarespecifictoamobileplatform,irrespectiveofwhethertheyusetheimage,coordinate,OCR,ornativeobjectrecognitionapproach.ThereareveryfewtoolsthatsupportmorethanonemobileplatformsuchasAndroid,iOS,WindowsPhone,orBlackBerry.However,notoolisabletoautomatemultiplemobileplatformswithonetestcodebase.Therearesometestautomationtoolproviderswhopromisetobeabletoautomateeverythingoneverymobileplatform,butthat’snottrue.Everymobileplatformhasspecificrequirementsforusabilityornavigationusageofanapp,meaningthatyoualsoneedtohavemultipletestcodebasesinordertoautomatethem.Inmostmobileprojects,youhavetodealwithmultiplecodebasesofdifferent
programminglanguagesandtestautomationtoolsinordertobuildupatestinganddevelopmentpipeline.Inmyopinion,amobiletestautomationtoolmustbeabletoaccessthenative
objectpropertiesthatareusedtoidentifyandtointeractwiththenativeelements.Thisisthebestandmostefficientwaytowritetestautomationscriptsformobileapps.Writingsuchtestscriptsrequiresmoreeffortandprogrammingskills,butthescriptsaremuchmorereliableandrobusttochangesandcanbeexecutedondifferentdevicesandscreenresolutions.Youshouldthereforeusenativeobjectrecognitiontoolsinyourprojectbased
ontheapptypeyouaretesting(native,hybrid,orWebapps).
ImportantRememberthatsomemobiletestautomationtoolssupportmorethanonerecognitionapproach.Combinetheapproachesinyourtestscripts,butkeeptheprosandconsinmindaswellasthefactthatnotoolisperfect.
WhatShouldBeAutomated?Whenyouhavethetaskofdeterminingwhichtestsshouldbeautomatedornot,keeponethinginmind:Itisimpossibletoautomateeveryfeatureortestcaseofamobileapp.Now,youhavetoanswerthisquestion:Whatshouldbeautomated?Creatingasmartmobiletestautomationplanrequirescarefulplanningand
designworkthatshouldbeperformedbeforeautomatingasingletestcase.First,youshoulddefineagoalforyourmobiletestautomationanddeterminewhatkindsoftestsyouwanttoautomate.Keepingthemobiletestpyramidinmindcanhelpyoutodefinetheunittestsandend-to-endusertests.Butyoushouldstillrememberthatyouneedlotsofmanualmobiletestingandthatnoteverythingcanbeautomated.However,youcanturnsomemanualtestsintoautomatedteststosavetimeandtestingeffort.Possiblegoalsforyourmobiletestautomationcouldbe:•Automatethebusiness-criticalparts.•Automateuserworkflowsandscenarios.•Automateonlycomplexappscenarios.•Automatesequencesthatneedtoberepeatedseveraltimes.•Automateonlytheacceptancecriteria.•Builduparegressiontestsuite.•Automateonlyifitiseconomicallyreasonable.
Ifsometestcasesareperformedonlyafewtimes,it’sprobablymoreeconomicaltoleavethemformanualtesting.Ifyou’reworriedaboutforgettingthem,addthemtoachecklist.Teststhatneedtoberunfrequentlyandrequirelotsofinputsandactionsshouldbeautomated.Butdon’tcreatelargeandcomplexautomatedtestscenariosasthosekindsoftestsaredifficulttomaintain,edit,anddebugandarelikelynottobestable.Keepthetestssmallandindependentofoneanother.Itisalsoimportanttokeepthecriticalpartsoftheappinmindsuchasthe
loginorpaymentprocessastheyhaveahugeimpactontheapp,yourbusiness,andthepeoplewhouseit.Ifthecriticalpartsarenotwelltestedorautomated,youmaysufferalotofdamagetoyourreputation.Anotherimportantfactorwhendecidingwhichtestsaretobeautomatedisthe
timeandthemoneyinvolved.Ifyouneedlotsoftimetoautomateamobiletestscenario,askyourselfifitisworthautomatingasalotoftimespentonone
specificscenariowillcostlotsofmoney.Andistheinvestedtimesavedlateronwhenthetestisautomatedratherthanmanual?Askyourselfthisquestionfromtimetotime;youmayfindthatit’sOKtotestmanually.Whenautomatingatestforamobileapp,keepthedifferentmobiledevicesin
mind.Onlyautomatetheteststhatcanbeexecutedoneverytargetdevice.Trytoautomatethetestsinasgeneralamanneraspossiblesoyoucanexecutethemonseveraldevices.Anddon’tforgettoautomatewithdifferentlanguagesifyourappisgoingtobeavailableinvariouscountries.UsetheUIelementIDstoidentifyanddrivethoseelementsandtoimplementrobusttests.Thereisnogenericanswertothequestion“Whatshouldbeautomated?”The
answerwillbedifferentforeverymobileappanddependontheapptype,mobileplatform,andapppurpose.Everymobiletestershouldkeepthefollowingpointsinmindwhenanswering
thisquestion:•Itisnotpossibletoautomatetestingoftheentireapporallofitsfeatures.•Defineagoalforyourtestautomation.•Defineend-to-endtests.•Starttestautomationasearlyaspossibletogetherwiththeappdevelopment.•Keepthetimeandthecoststhatareneededforthetestautomationinmind.•Keepthetestssmall,fast,simple,andindependent.•Keepthedifferentmobiledevicesinmind.•Executethetestsasearlyaspossibleandasoftenaspossible.
ImportantDon’ttrytoautomateeverything.Definetestautomationgoalsdependingonyourappandstartautomatingthosegoals.
Emulator,Simulator,orRealDevice?Thenextquestionyouneedtoansweriswhetheryoushouldautomatewithemulators,simulators,orrealdevices.InChapter4,“HowtoTestMobileApps,”Idescribedthedifferencesamongemulators,simulators,andrealdeviceswhenmobiletestingisperformedmanually.Myrecommendationtherewastousetheemulators/simulatorsforverybasictestsandtodomanualtestingonreal
devices.Butisthisalsothecasewithmobiletestautomation?BeforeIanswerthisquestion,let’shavealookattheprosandconsof
emulators/simulatorsandrealdevices.
Emulator/SimulatorProsThebiggestadvantageofemulators/simulatorsistheprice.BotharefreetouseandarepartoftheSDKfromthedifferentmobileplatforms.Besidesthat,theyarereallysimpletouseandoffervariousoptionsfordevelopersandtesters.Aftertheemulator/simulatorisinstalled,youcancreatethesimulators/emulatorsusingdifferentconfigurationssuchasoperatingsystemversionsorscreenresolutions.
Emulator/SimulatorConsIfwelookattheconsoftheemulators/simulators,therearesomemorepointsthatyoushouldbeawareof.Usingemulators/simulatorsincreasestheriskofmissingimportantbugsthatoccuronlyonrealdevices.Emulators/simulatorsarenotthesameastherealenvironment,whichisahugedisadvantage.Additionally,emulators/simulatorsofferonlya“plain”andsimplemobileoperatingsystem.Theyoffernodiversityintermsofthedifferentdevices,operatingsystems,andadaptationsoftheuserinterface.ThisisespeciallythecasewithAndroidwhereseveraldevicemanufacturerschangetheUIoftheAndroidsystemtosuittheirneeds.Fromahardwareperspective,theyalsofailtoofferarealdeviceenvironmentwithalloftherelevantsensorsandinterfaces.Thenextdisadvantageofmobileemulators/simulatorsisthatthedata
networksarenotreal.Networkspeedscanbesimulated,butthiswillnotcovertherealdatanetworkswithtrafficlossorchangeinnetworkspeedsandtechnologies.And,finally,emulators/simulatorsdonotofferthesameperformanceasrealdevicesintermsofCPU,GPU,memory,orsensors.Everymobiletestershouldkeepthefollowingpointsinmindwhenusing
emulatorsorsimulatorswithinaproject:•Itisriskynottotestinarealenvironment.•Thereisnodiversityintermsofhardwareandsoftware.•Thenetworkenvironmentissimulated.•Thereisnorealdeviceperformance.•Thereisnoaccesstodevice-specifichardwareelementssuchasthecamera,GPS,orothersensors.
RealDeviceProsTestingonrealdevicesofferslotsofadvantagescomparedtoanemulator/simulator.Thetestswillbeexecutedinarealuserenvironmentandthereforetheresultswillbeclosertoarealuserexperience,includinganydiscoveredbugs.Testingonrealdevicesalsoofferstheoptionofusingthefulldevice’shardware,software,anddevice-specificfeaturessuchassensors,CPU,GPU,andmemory.Usingtherealdevicefortestingshowstherealbehaviorintermsofperformance.Thefollowingpointssummarizetheprosofusingrealdevicesfortesting:•Realdevicetestingoffersreliabletestresults.•Realhardwareandsoftwarefeaturesareused.•Therealuserexperienceandappperformancecanbetested.•Testsaredoneinrealdatanetworks.
RealDeviceConsTestingonrealdevicesalsohassomedisadvantages.Thebiggestoneisthecostofbuyingallofthedifferentmobiledevicesfordevelopingandtestingpurposes.Ifyouwanttotestonrealdevices,youhavetobuynewdevicesnearlyeverymonthinordertoverifythatyourapprunsonallthenewfeatures(hardwareandsoftware).Inaddition,youneedsomeonewhoisresponsibleformaintainingallthosedevices.It’snotenoughjusttobuythem;youneedtocomeupwithastrategyforhowtoupdatethedevicestonewerversionsandhowtousethemwithinyourcompany.Thefollowingpointssummarizetheconsofusingrealdevicesfortesting:•Thecostofhavingtofrequentlybuynewdevicesishigh.•Maintenanceofallthedevicesistime-consuming.
WheretoAutomate?Ifwelookatalltheprosandcons,Ithinktheanswertothequestion“Wheretoautomate?”isprettysimple:ontherealdevice!Verysimpleandbasicfunctionaltestsandtestautomationcanofcoursebe
performedonanemulator/simulatortogetfastfeedbackaboutthecurrentstateofamobileapp,andthisisespeciallyusefulfordevelopers.However,ifyouasamobiletesterwanttobesurethatyourappisusingall
thedevice-specificelementssuchasthehardwareandsoftwareresourcesduring
thetestautomationsession,youneedtoexecutethetestsonrealdevices.Executingthetestautomationscriptsonrealdevicesprovidesyouwithmuchbetterresultsintermsofreliability,performance,andreal-worldbehavioroftheapp.Furthermore,youareabletoexecutethetestsonseveraldevicesatthesametimetoquicklydeterminewhetherornotanapphasaproblemonacertainmobiledevice.Agoodapproachistofindahealthymixofemulators,simulators,andreal
devicestogetthebestoutofyourtestautomation.Itisenoughif,forexample,developersdotheirtestautomationonemulatorsandsimulatorsbecausescalingandexecutingtestautomationscriptsinparallelonemulators/simulatorsismucheasierandcheaperthanonrealdevices.Ifyouwanttoscaleandbuildupanemulator/simulatormatrixforyourtestautomation,havealookattheGoogleTechTalk“BreakingtheMatrix—AndroidTestingatScale.”2
2.www.youtube.com/watch?v=uHoB0KzQGRg
Youasamobiletestershouldwritetestautomationscriptsthatwillrunonrealdevicesaswellassimulators/emulators.
ImportantWhenselectingamobiletestautomationtool,verifythatthetoolisabletoexecutethetestsonbothrealdevicesandemulators/simulators.
HowtoSelecttheRightMobileTestAutomationToolSofaryouhavelearnedaboutthedifferenttypesoftestautomationtoolsaswellaswhatshouldbeautomatedandwherethetestautomationshouldbeperformed.Nowitistimetofindtherightmobiletestautomationtoolforyourappandyourtestanddevelopmentenvironment.Whenselectingamobiletestautomationtool,youshouldkeepsomepointsin
mind.Thefirstpointisthatthereisno“onesizefitsall”testautomationtoolavailableonthemarket.Everytoolhasitsprosandcons,andnoteverytoolissuitedtoeverydevelopmentenvironmentandpipeline.ToolAcouldworkwellwithprojectAbutnotwithprojectB,meaningthattheevaluationhastoberepeatedforeveryproject.Tosavesometimeinthetoolevaluationphase,itisagoodapproachto
implementasampleappthatincludesalloftheelementsyourproductionappwillhaveinordertoseeifthetestautomationtoolisabletohandleandinteract
withthem.Ifthetoolisabletofulfillallofyourrequirementsforthesampleapp,youhaveprobablyfoundtherightmobiletestautomationtoolforyourproject.Thesecondpointyoushouldkeepinmindisthatmobiletestautomation
requiresprogrammingskillsinordertobuilduparobust,maintainable,andstabletestautomationsuite.Donotuseacaptureandreplaytooltocompensateforalackofprogrammingskills;youwillendupwitharealnightmarescenariobecauseyoucan’tfixthebrokenscripts.Ifyouhavenoprogrammingskills,trytolearnthemasyouwillneedtheminthefuture.Besidesdevelopingasampleapptoevaluatethemobiletestautomationtool,
youcanusethechecklistinthenextsectionwithsomeselectioncriteria.Thislistwillhelpyoutofindthetoolbestsuitedtoyourdevelopmentandtestingprocess.
SelectionCriteriaforaTestAutomationToolThefollowingpointsshouldbeconsideredwhenselectingamobiletestautomationtool:
1.Doesthetoolsupportdifferentmobileapptypes(native,hybrid,Webapps)?
2.Whichmobileplatformsaresupported(Android,iOS,WindowsPhone,BlackBerry)?
3.Whichrecognitiontechnologydoesthetooluse(native,image,text,coordinate)?
4.Doesthetoolchangetheappyouwanttotest(forexample,byaddingaserver,instrumentation)?
5.Isthetoolabletoexecutethetestsonrealdevicesaswellasonemulatorsandsimulators?
6.Isthereareportavailableattheendofthetestrun?7.Isthetoolabletotakescreenshotswhilethetestsareexecuted,andarethosescreenshotspartofthetestreport?
8.Canthetestsuitebeexecutedonseveraldevicesatthesametime?9.Howlongisthetestexecutiontime?Isitsufficientforyourneeds?10.DoesthetoolsupportalloftheUIandcontrolelementsofthemobile
platform?11.Istheresupportforachangeoforientationfromportraittolandscapeand
viceversa?12.Isthetoolabletoawakethedevicefromthesleeporstandbymode?13.Areallgesturessupported,suchasswipe,scroll,click,tap,orpinchto
zoom?14.Isthetoolabletosimulatenativebuttonssuchasthebackorhome
button?15.Doesthetoolusethedevice’ssoftkeyboardtoenterdata?16.Cantheappbetestedinseverallanguages?17.Doesthetoolrequiremodificationtomatchtherealdevice(jailbreak,
rooting)?18.Doesthetoolsupportaprogramminglanguagewithwhichyouareableto
writetestscripts?19.Isthetoolabletoexecutethetestsfromthecommandline?20.Canthetoolbeintegratedintoyourdevelopmentenvironment(IDE)?21.Canthetoolbeintegratedintoacontinuousintegrationsystem?22.Canthetoolbecombinedwithothertoolssuchasadefectmanagementor
testmanagementtool?23.Isthetoolabletoconnecttoatestcloudproviderinordertoexecutethe
testswithinacloud?24.Isthetoolwelldocumented?25.Isthetoolopensourceorclosedsource?26.Istherealargecommunity/supportbehindthetool?27.Sincewhenhasthetoolbeenavailableonthemarketandisitusedby
othercompaniesformobiletestautomation?28.Doesthetoolsupportcross-platformtests?Asyoucanseefromthecriterialist,therearelotsofthingstoconsiderwhen
selectingamobiletestautomationtool.Theevaluationphaseisveryimportantandshouldnotbeunderestimated.Ifyouchoosethewrongtool(duetotimeconstraints,forexample)beforetheprojectstarts,itisverylikelythatyouwillstrugglewiththetoolduringtheproject.Point18inthelist,“Doesthetoolsupportaprogramminglanguagewith
whichyouareabletowritetestscripts?”isveryimportant.Youshouldtrytofindatoolthatsupportstheprogramminglanguagewithwhichyouareabletowritecode;thiswillreduceyourlearningcurvebecauseyouneedtobecome
familiarwithjustthetoolandnottheprogramminglanguage.You’llalsosavelotsoftimeandmoneyonadditionaltraining.
ImportantDevelopasampleapporuseachecklistwithyourcriteriatofindthetoolthatbestfitsyourtestanddevelopmentprocess.
CurrentStateofToolsThispartofthechaptershouldgiveyouanoverviewofpossiblemobiletestautomationtools.Iselectedmainlyopen-sourceend-to-endtestingtoolsfortheiOSandAndroidmobileplatformsandtoolsthatIhaveusedinmobileproj-ects.Furthermore,Ichosetoolsthatarewellknownformobiletestautomationandusedbymostcompanies.Iwillgiveyousomerecommendationsandusefulinformationaboutthetoolsandhowtousethem.Iwillnotexplainordescribetheinstallationorconfigurationprocessofthetoolsbecausethisinformationcan(orwill)beoutdatedsoon.Allofthementionedtoolsrequireprogrammingskillsinorderforyoutobeabletoworkwiththemefficiently.And,asalways,thelistoftoolsmentionedhereisbynomeanscomplete.
AndroidToolsMostoftheAndroidtestautomationtoolsarebasedontheAndroidInstrumentationFramework3fromGoogle.TogetstartedwithAndroidtestautomation,youneedtounderstandtheviewhierarchyofanAndroidapp.Youalsoneedtoknowwhatkindsofcomponentsandelementstheappusesaswellashowalltheseelementsarearrangedonthescreenandwhattheyrepresentinthelogicalstructureoftheapp.
3.http://developer.android.com/tools/testing/testing_android.html
GoogleprovidesaveryusefultoolcalledUIAutomatorViewer4toinspecttheapp’sviewandlayouthierarchy.Thistoolletsyouviewtheproperties—thenameorIDofeachUIcomponentthatisdisplayedonthescreen.Youneedthiskindofinformation(nameorIDofanelement)towriteyourmobiletestscripts.
4.http://developer.android.com/tools/testing/testing_ui.html
YoucanfindtheUIAutomatorViewerintheAndroidSDKlocationonyourcomputer,forexample:Clickheretoviewcodeimage
/android/sdk/tools/uiautomatorviewer.sh
ThistoolisparticularlyimportantwhentheAndroidappcodeisnotavailableandyoujusthavethecompiled.apkfiletowriteyourmobiletestautomation.However,someoftheAndroidtoolsmentionedoffertheirownUIautomatorviewerthatcanbeusedtoinspecttheelements.
RobotiumRobotium5isthedefactostandardopen-sourcetoolforAndroidtestautomationandwasoneofthefirstAndroidtestautomationtoolsonthemarket.RobotiumisablackboxtoolthatprovidesfullsupportfornativeandhybridAndroidapps.RobotiumisanextensionoftheaforementionedAndroidInstrumentationFrameworkandprovidesaverysimpleandbasicAPIforwritingUItests.Itsuppliestheso-calledsoloobjecttocallmethodssuchasclickOnTextorenterText.CheckoutthecodeinListing5.1forsomepossibletestactions.
5.https://code.google.com/p/robotium/
Listing5.1SampleCodefromRobotium
Clickheretoviewcodeimage
/*Robotiumwillclickonthetext"Welcome"*/solo.clickOnText("Welcome");/*RobotiumwillenterthestringMySecretPasswordintotheinputfieldwiththeID2*/solo.enterText(2,"MySecretPassword");/*Robotiumwillclickonthebuttonwiththelabel"Login"*/solo.clickOnButton("Login");/*Robotiumwillsimulateaclickonthenativebackbutton*/solo.goBack();
Robotiumrequiresonlyminimalknowledgeoftheappyouwanttotestandprovidesexcellentreadability.Byjustreadingthetestmethodsyoualreadyknowwhatishappeningandwhatwillbetestedontheviewofthedevice.TheUIAutomatorViewersuppliesalloftheviewinformationyouneedtowriteyourtests.RobotiumtestsarewrittenintheJavaprogramminglanguageandcanbeexecutedonarealdeviceoronemulators.Testscanbeexecutedononlyasingledeviceatatime,andRobotiumisabletotestonlytheappthatisbeingtested,meaningthatthereisnowayoftestingoutsidethisapplication.AttheendofthetestrunaJUnitreportisgenerated.Thewrittentestscanbeexecutedeitherfromthecommandline,fromanIDE,
orfromacontinuousintegrationserverusingMaven,Gradle,orAnt.
IfyouwanttoknowmoreaboutRobotiumandhowtogetstarted,checkouttheRobotiumwikipage:
•RobotiumGettingStarted(https://code.google.com/p/robotium/wiki/Getting_Started)
AlsocheckouttheRobotiumprojectpages:•LatestRobotiumversionandsamples(https://code.google.com/p/robotium/wiki/Downloads)
RobotiumExtensionsIn2014,thedeveloperofRobotium,RenasReda,foundedthecompanyRobot-ium6toprovidetheso-calledRobotiumRecorder,whichallowsdevelopersandtesterstorecordRobotiumtestsinsteadofwritingthecodemanually.
6.http://robotium.com/
AnotherniceextensionforRobotiumistheExtSolo7projectfromthecompanyBitbar.ExtSoloaddssomeveryusefultestmethodstoRobotiumsuchasthefollowing:
7.http://docs.testdroid.com/_pages/extsolo.html
•changeDeviceLanguage(java.util.Localelocale):switchesthecurrentlanguageofthedeviceduringthetestexecution•setGPSMockLocation(doublelatitude,doublelongitude,doublealtitude):setsthedevice’scurrentGPSlocation•turnWifi(booleanenabled):turnsWi-Fioffandon,toseehowtheapphandlestheconnectionloss
FullAPIdocumentationisavailablehere:•ExtSoloAPIdocumentation(http://docs.testdroid.com/_static/extSolodocs/com/bitbar/recorder/extensions/ExtSolo.html
SpoonSpoon8isanotherpowerfulAndroidtestautomationframework,developedbythecompanySquare.9SpoonisalsoanextensionoftheexistingAndroidInstrumentationFramework.ThemaindifferencefromRobotiumisSpoon’sabilitytosimultaneouslyexecutethesametestsonmultipledevicesoremulators.Spooncanexecutethetestsoneverytarget(deviceoremulator)thatisvisibletoadb(AndroidDebugBridge)andisconnectedtothetestserver.
8.http://square.github.io/spoon/
9.https://squareup.com
ThetestsarealsowritteninJavaandcanbeexecutedfromthecommandline,fromanIDE,orfromacontinuousintegrationserver.Spoon’sstructureandtestcodearealsoverysimple,asyoucanseeinthecodeinListing5.2.
Listing5.2SampleCodefromSpoon
Clickheretoviewcodeimage
Spoon.screenshot(activity,"Login_Screen");assertThat(password).hasNoError();instrumentation.runOnMainSync(newRunnable(){@Overridepublicvoidrun(){password.setText("MySecretPassword");}});
Afterallofthesetestshavebeenexecutedoneverydevice,SpoongeneratesastaticHTMLreportwithdetailedinformationabouteachdeviceandtest(seeFigure5.4).Thereportcomparesthetestresultsonthedifferentdevicesinaniceoverview.Ifscreenshotsaretakenduringthetestrun,SpoongeneratesananimatedGIFimagefromthemsoyoucanseetheexecutedteststepsagaininthetestreport.
Figure5.4Spoonsamplereport
Spoonalsorequiresverylittleknowledgeofthecodefortheappyouwanttotest.SpooncanbecombinedwithotherAndroidtestautomationtools,soyoucan,forexample,usethescreenshotfunctionfromSpoonandtestmethodsfromRobotiumorEspresso.ThelatestversionofSpoonandanexampletestappcanbefoundonGitHub:•Spoonsamplesandlatestversion(https://github.com/square/spoon)
SelendroidSelendroid10isatestautomationtoolfornativeAndroid,hybrid,ormobileWebapps.ThenameSelendroidcomesfromthewordsSeleniumandAndroid.SelendroidisfullycompatiblewiththeJSONWireProtocol,11andthetestsarewrittenusingtheSelenium2ClientAPI.1210.http://selendroid.io/11.https://code.google.com/p/selenium/wiki/JsonWireProtocol
12.http://docs.seleniumhq.org/docs/03_webdriver.jsp
IfyouarefamiliarwithwritingautomatedtestswithSelenium2forbrowser-basedapplications,itisveryeasytowritetestautomationcodewithSelendroidforAndroidapps.SelendroidisabletosimultaneouslyexecuteandinteractwithmultipleAndroiddevices(realdevicesoremulators).Listing5.3showsacodeexample.
Listing5.3SampleCodefromSelendroid
Clickheretoviewcodeimage
WebElementloginButton=driver().findElement(By.id("startLogin"));WebElementpasswordInput=driver().findElement(By.id("password"));passwordInput.sendKeys("MySecretPassword");loginButton.click();
TheUIelementsofthemobileappcanbefoundbydifferentlocatortypes,forexample,byID,name,linktext,class,tagname,orXPath.ToinspecttheUIcomponentsoftheappundertest,SelendroidprovidesaveryusefultoolcalledSelendroidInspector,13whichprovidesahierarchyviewerthatletsyouseetheUIcomponentproperties.Itisabletocreateascreenshotwiththeviewlocator,recordtheclickactionsonthemobileapp,displaytheHTMLsourceofaWebview,andprovideaveryusefulXPathhelpertoidentifytheWebelements.13.http://selendroid.io/inspector.html
Inordertosupportdifferentgestures,SelendroidusestheAdvancedUserInteractionsAPI.14Thewrittentestscanbeexecutedfromthecommandline,fromanIDE,orfromacontinuousintegrationserver.14.https://code.google.com/p/selenium/wiki/AdvancedUserInteractions
SelendroidcanbefullyintegratedasanodeintoaSeleniumGrid15forscalingandparalleltesting.And,finally,theappundertestwillnotbemodifiedforautomationpurposes.15.https://code.google.com/p/selenium/wiki/Grid2
AdditionalusefulinformationonSelendroidcanbefoundontheWebsite:•Selendroidgettingstarted(http://selendroid.io/setup.html)•ScalingSelendroid(http://selendroid.io/scale.html)•LatestSelendroidversionandsamples(https://github.com/selendroid/selendroid)
CalabashforAndroidCalabash16isacross-platformmobiletestautomationframeworkfornativeandhybridAndroidandiOSapps.ThetoolmakesitpossibletowriteautomatedUIacceptancetestsinCucumber.17WiththehelpofCucumber,youcanexpressthebehavioroftheappyou’retestingusinganaturallanguage.Thisapproachiscalledbehavior-drivendevelopment(BDD),anditcanbeveryhelpfulwhenbusinessexpertsornontechnicalcolleaguesareinvolvedintheacceptancecriteriaprocess.16.http://calaba.sh/17.http://cukes.info/
CucumberusesGherkin18asthedomain-specificlanguage(DSL)toannotatethebehavioroftheapplication.18.https://github.com/cucumber/cucumber/wiki/Gherkin
PleaserefertothecodeinListings5.4and5.5forexamples.Listing5.4showsaCucumber(Gherkin)scenariousingrealtexttodescribethebehavioroftheapplication.
Listing5.4SampleGherkinCode
Clickheretoviewcodeimage
Feature:AsauserIwanttologinScenario:LoginusingvalidcredentialsGivenIamontheloginscreenWhenIenter"Username"intotheuserfieldAndIenter"PWD"intothepasswordfieldAndIclicktheloginbuttonThenImustseemyuseraccount
Listing5.5showstheRubycodeneededtomaptherealtextintocommandsthatthecomputerneedstounderstandinordertocommunicateandinteractwiththeapp.
Listing5.5SampleStepDefinitionCodeforOneGherkinStep
Clickheretoviewcodeimage
When(/^Ienter"(.*?)"intotheuserfield$/)do|username|fill_in("IDUserName",:with=>"username")end
Asyoucansee,actualtestautomationisperformedwithRubyandwithintheso-calledstepdefinitions.Gherkinisresponsiblefordescribingthebehavioroftheapplication,Rubyisneededfortheactualcoding,andCucumberistheframeworkthatexecuteseverythingtogetherontherealdevicesoremulators.TheCalabashtestscanbeexecutedfromthecommandline,fromanIDE,orfromacontinuousintegrationserver.Calabashsupportsascreenshotfunction,isabletouselocalizationwithinthe
app,andsupportsdifferentgestures.AdditionalusefulinformationaboutCalabash,Gherkin,andCucumbercanbe
foundontheGitHubprojectpage:•CalabashforAndroid(https://github.com/calabash/calabash-android)•PredefinedstepdefinitionsforAndroid(https://github.com/calabash/calabash-android/blob/master/ruby-gem/lib/calabash-android/canned_steps.md)
AppiumAppium19isanopen-source,cross-platformtestautomationtoolfornative,hybrid,andmobileWebapps.AppiumsupportsthemobileplatformsAndroid,iOS,andFirefoxOS.LikeSelendroid,AppiumusestheWebDriverJSONWireProtocoltodriveandtotesttheUIofthemobileapps.Youcanuseseveralprogramminglanguagestowriteyourtestautomation.Currently,Appiumsupportsthefollowinglanguages:19.http://appium.io/
•C#•Clojure•Java•JavaScript•Objective-C•Perl•PHP•Python•Ruby
Thewrittentestscanbeexecutedbyemulators/simulatorsorrealdevices.ThecodeinListing5.6showssomeoftheteststeps.
Listing5.6SampleAppiumCode
Clickheretoviewcodeimage
WebElementloginText=driver.findElement(By.name("TextLogin"));assertEquals("TextLogin",loginText.getText());WebElementloginTextView=driver.findElementByClassName("android.widget.TextView");assertEquals("TextLogin",loginTextView.getText());WebElementbutton=driver.findElement(By.name("Login"));button.click();
OneofthemainadvantagesofAppiumisthatthetoolnotonlyisabletocommunicatewiththeappbeingtestedbutisalsoabletostartanotherappfromtheappundertest,suchasthecameraapporthecontactsapp.Besidesthat,theappundertestwillnotbemodifiedwhenusingAppiuminordertoautomateit.MoreinformationaboutAppiumcanbefoundontheGitHubpageaswellas
onthemanufacturer’spage:•AppiumonGitHub(https://github.com/appium/appium)•Appiumintroduction(http://appium.io/introduction.html)•AppiumAPIreferencedocumentation(http://appium.io/slate/en/master)
EspressoEspresso20istheAndroidtestkitprovidedbyGoogle.EspressoisbasedonanimprovedInstrumentationTestRunnercalledGoogleInstrumentationTestRunner21tomakeAndroidtestautomationmorereliableandfaster.20.https://code.google.com/p/android-test-kit/wiki/Espresso21.https://code.google.com/p/android-test-kit/wiki/GoogleInstrumentationTestRunner
Espressoprovidesasmallandeasy-to-learnAPItointeractwiththeUIelementsofanativeAndroidapp.Espressoismainlyaimedatdeveloperswhohaveaccesstothecodebaseinordertowritefastandreliabletests.However,ifyouareabletowriteJavacodeandhaveaccesstothecodebaseoftheappyouwanttotest,Espressoisanicelittletoolforwritingthetestautomation.Asanexample,pleaserefertothecodeinListing5.7.
Listing5.7SampleCodefromEspresso
Clickheretoviewcodeimage
onView(withId(R.id.login)).perform(click());onView(withId(R.id.logout)).check(doesNotExist());onView(withId(R.id.input)).perform(typeText("Hello"));
Espressoisabletoexecutetestseitherfromthecommandline,fromanIDE,orfromacontinuousintegrationserveronrealdevicesoremulatorsbutnotinparallel.However,testexecutionismuchfastercomparedtothatofanyotherAndroidtestautomationtools.AdditionalusefulinformationaboutEspressocanbefoundontheGoogle
projectpage:•Espressostartguide(https://code.google.com/p/android-test-kit/wiki/EspressoStartGuide)•Espressosamples(https://code.google.com/p/android-test-kit/wiki/EspressoSamples)
MoreAndroidTestingToolsAsIsaidatthebeginningofthischapter,thelistofAndroidtestautomationtoolsmentionedinthisbookisbynomeanscomplete.Therearesomanyopen-andclosed-sourcetoolsavailableonthemarket,andmoretoolsareboundtofollow.ThefollowinglistcontainsthenamesofsomeotherAndroidtestautomationtoolsthatyoushouldtry.Thislistcontainssomeclosed-sourceenterprisetoolsaswellassomeunittestingtools.
•eggPlant(www.testplant.com/eggplant/)•Experitest(http://experitest.com/)•JamoSolutions(www.jamosolutions.com/)•Keynote(www.keynote.com/solutions/testing/mobile-testing)•MonkeyTalk(www.cloudmonkeymobile.com/monkeytalk)•PerfectoMobile(www.perfectomobile.com/)•Ranorex(www.ranorex.com/)•Robolectric(http://robolectric.org/)•Siesta(https://market.sencha.com/extensions/siesta)•SilkMobile(www.borland.com/products/silkmobile/)•SOASTA(www.soasta.com/products/soasta-platform/)•TenKodEZTestApp(www.tenkod.com/ez-testapp/)•TestObject(https://testobject.com/)
•UIAutomator(http://developer.android.com/tools/help/uiautomator/index.html)
AndroidToolRecommendationRecommendingamobiletestautomationtoolisnoteasy.Therearesomanyfactorsthatneedtobeconsideredwhenchoosingamobiletestautomationtool,andthosefactorsaredifferentforeachappandproject.FrommypointofviewandjudgingfromtheappsIhaveworkedwith(socialmediaandbookingapps),IrecommendtakingacloserlookatRobotium,Spoon,Appium,andSelendroid.Allofthetoolsaregreattoworkwith.Theyofferfullsupportfornative,
hybrid,andWeb-basedapps.Besidesthat,allofthetoolscomewithgooddocumentation,codesamples,andagreatcommunityifyouwanttoaskquestions.Andlastbutnotleast,writingtestcodewiththetoolsisveryeasyandlotsoffun.WhenchoosingRobotiumasatestautomationtool,Ihighlyrecommend
combiningitwithSpoon.Spoon’stestreportingfeatureisexcellent,andtheoptiontorunyourtestsonseveraldevicesatthesametimeisunbeatable.Robotiumisawell-developedAndroidtestautomationtoolwithahugecommunityandlotsofsupportbehindit.AppiumandSelendroidarealsotoolsyoushouldkeepinmind.Bothoffera
greatwaytodevelopyourautomatedtestsinseveralprogramminglanguages.BothtoolshavegreatoptionsforscalingyourtestingprocessinacloudoraSeleniumGrid.
ImportantKeeponethinginmind:Nomatterwhichtoolyouusefortestautomation,usetheresourceIDsoftheUIcomponentsifpossibleasdoingsospeedsuptestautomationandmakesitmorereliable.
iOSToolsLet’sgofurtherwithsomeiOStestingtools.WhatIalreadymentionedfortheAndroidtoolsalsoappliestoiOStools:
•Theselectedtoollistisnotcomplete.•Ihaveincludedend-to-endtestautomationtools.•Allmentionedtoolsrequirecodingskills.•BeforeyoustartwithiOStestautomation,makesureyou’refamiliarwith
theiOSUIstructureofiOSapps.
UIAutomationUIAutomation22istheiOStestingtoolthatispartofInstruments23providedbyApple.WiththehelpofUIAutomation,youareabletoeitherrecordthetestsorwritethemmanuallyusingJavaScript.IfyouarefamiliarwithiOSapps,youknowthatiOSappsuseso-calledaccessibilitylabelstodescribetheUIelementsandmakethemaccessible,suchasforscreenreaders.MostiOStestingtoolsandUIAutomationusetheseaccessibilitylabelsinordertocommunicateandinteractwiththeappbeingtested.Ifyourapphasnodefinedaccessibilitylabels,youarenotabletowritetestautomationforit.22.
https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html23.
https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html
UIAutomationisabletosimulaterealuserinteractionssuchastap,swipe,scroll,pinch,ortype,eitheronarealdeviceorontheiOSsimulator.ThecodeinListing5.8showssometestactionsthatcanbeperformed.
Listing5.8SampleCodefromUIAutomation
Clickheretoviewcodeimage
app.keyboard().typeString("Sometext");rootTable.cells()["ListEntry7"].tap();alert.buttons()["Continue"].tap();
UIAutomationisabletochangethedeviceorientationfromportraittolandscapemodeandbackagain.Itisalsoabletohandledifferentalertsthatmayoccurduringthetestrunonthemobiledevice.Theautomatedtestscanbeexecutedfromthecommandline,withintheIDE,andfromacontinuousintegrationserver.MoreinformationaboutUIAutomationcanbefoundonApple’sdeveloper
pages:•UIAutomationJavaScriptreference(https://developer.apple.com/library/ios/documentation/DeveloperTools/Reference/UIAutomationRef/_index.html
CalabashforiOSCalabash24isacross-platformmobiletestautomationframeworkfornativeand
hybridAndroidandiOSapps(ifyouhavealreadyreadaboutitintheAndroidsection,youcanskipahead).ThetoolenablesautomatedUIacceptancetestswritteninCucumber.25WiththehelpofCucumber,youcanexpressthebehavioroftheappyou’retestingusinganaturallanguage.Thisapproachiscalledbehavior-drivendevelopment(BDD),anditcanbeveryhelpfulwhenbusinessexpertsornontechnicalcolleaguesareinvolvedintheacceptancecriteriaprocess.24.http://calaba.sh/25.http://cukes.info/
ImportantIalreadydescribedthefeaturesofCalabashintheAndroidtoolssection,sopleasereferbackthere.AlmostexactlythesameprocessisinvolvedwhenwritingthefeatureandstepdefinitionfilesforAndroidandiOS.
TogetmoreinformationaboutCalabashforiOS,checkouttheCalabashiOSprojectsite:
•CalabashforiOS(https://github.com/calabash/calabash-ios)•GettingstartedwithCalabashforiOS(https://github.com/calabash/calabash-ios/wiki/01-Getting-started-guide)•PredefinedstepsforCalabashiOS(https://github.com/calabash/calabash-ios/wiki/02-Predefined-steps)
ios-driverios-driver26isabletoautomatenativeiOS,hybrid,andmobileWebappsusingtheSeleniumWebDriverAPI.ItusesthesameapproachasSelendroidbutforiOSapps.ItimplementstheJSONWireProtocolinordertocommunicateandtestiOSappsusingInstruments.Thetoolisabletoexecutethetestseitheronarealdeviceoronasimulator.LikeAppiumandSelendroid,ios-driveroffersyouadifferentsetofprogramminglanguageswithwhichyoucanwriteyourtestscripts.Youcanchoosefromthefollowing:26.http://ios-driver.github.io/ios-driver/
•C#•Clojure•Java
•JavaScript•Objective-C•Perl•PHP•Python•Ruby
ThecodeinListing5.9showssomeofthepossibletestcommandsforanativeiOSappwritteninJava.
Listing5.9SampleCodefromios-driver
Clickheretoviewcodeimage
Bybutton=By.id("Login");WebElementloginButton=driver.findElement(button);Assert.assertEquals(loginButton.getAttribute("name"),"Login");loginButton.click();
InordertoidentifytheUIelementsoftheapp,ios-driverprovidesaUIinspector27similartoSelendroidthatidentifiesandviewsthepropertiesoftheUIelements.ios-driverisabletohandlelocalizedappsanddoesn’trequireanychangestotheappundertest.Thetestscanbeexecutedfromthecommandlineorfromacontinuousintegrationserver.Furthermore,thetoolcanbeusedasanodewithinaSeleniumGridtoscaleandparallelizethetesting.27.http://ios-driver.github.io/ios-driver/?page=inspector
Moreinformationaboutios-drivercanbefoundonthemanufacturer’sWebsite,aswellasontheGitHubprojectpage:
•ios-drivergettingstarted(http://ios-driver.github.io/ios-driver/?page=setup)•Sourcecodeandsamples(https://github.com/ios-driver/ios-driver)
KeepItFunctionalKeepItFunctional28(KIF)isanopen-sourceiOStestingtooldevelopedbythecompanySquare.29KIFisabletoautomatenativeiOSappsusingtheaccessibilitylabelsprovidedbytheapp.Thistoolusestheso-calledtesterobjecttobeabletosimulateuserinputssuchastouch,swipe,scroll,andtyping.Objective-CisusedtowriteautomatedtestscriptsforKIF,andKIFisableto
executethetestsonarealdeviceoriOSsimulator.28.https://github.com/kif-framework/KIF29.http://corner.squareup.com/2011/07/ios-integration-testing.html
HavealookatthesamplecodeofKeepItFunctionalshowninListing5.10.
Listing5.10SampleCodefromKeepItFunctional
Clickheretoviewcodeimage
[testerenterText:@"userone"intoViewWithAccessibilityLabel:@"UserName"];[testerenterText:@"Mypassword"intoViewWithAccessibilityLabel:@"LoginPassword"];[testertapViewWithAccessibilityLabel:@"Login"];
KIFcanbefullyintegratedwithinXcodetostartanddebugthetestautomationscripts.Furthermore,theautomatedtestscanbeexecutedfromthecommandlineorfromacontinuousintegrationserversuchasBots.3030.https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-
continuous_integration/ConfigureBots/ConfigureBots.html
There’sonethingyouneedtokeepinmindwhenautomatingtestswithKIF:itusesundocumentedAppleAPIs.Thisisnotaproblemwhentestinganapp,butit’scrucialthatyourtestscriptsnotbepartoftheproductioncode.Iftheyare,ApplewillrejectyourappduetotheuseofundocumentedAPIs.IfyoufollowKIF’sinstallationinstructions,thisshouldnotbeanissue.
AppiumAppium31isanopen-source,cross-platformtestautomationtoolfornative,hybrid,andmobileWebapps(ifyouhavealreadyreadaboutitintheAndroidsection,youcanskipahead).AppiumsupportsthemobileplatformsAndroid,iOS,andFirefoxOS.LikeSelendroid,AppiumusestheWebDriverJSONWireProtocoltodriveandtesttheUIofthemobileapps.31.http://appium.io/
ImportantIalreadydescribedthefeaturesofAppiumintheAndroidtoolssection,sopleasereferbackthere.
MoreiOSTestingTools
AsIdidforAndroid,I’dliketoprovideyouwithalistofadditionaliOStestingtools.Thefollowinglistcontainsunittestingandend-to-endopen-andclosed-sourcetestingtools;itisbynomeanscomplete:
•Experitest(http://experitest.com/)•Frank(www.testingwithfrank.com/)•GHUnit(https://github.com/gh-unit/gh-unit)•JamoSolutions(www.jamosolutions.com/)•Keynote(www.keynote.com/solutions/testing/mobile-testing)•Kiwi(https://github.com/kiwi-bdd/Kiwi)•MonkeyTalk(www.cloudmonkeymobile.com/monkeytalk)•OCMock(http://ocmock.org/)•PerfectoMobile(www.perfectomobile.com/)•Ranorex(www.ranorex.com/)•SilkMobile(www.borland.com/products/silkmobile/)•SOASTA(www.soasta.com/products/soasta-platform/)•Specta(https://github.com/specta/specta)•Subliminal(https://github.com/inkling/Subliminal)•XCTest(https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/testing_2_testing_basics/testing_2_testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW3)•Zucchini(www.zucchiniframework.org/)
iOSToolRecommendationRecommendinganiOStestautomationtoolisalsonotaneasytask.JustasforAndroid,therearesomanyfactorstoconsiderwhenchoosinganiOStestautomationtool.Irecommendthatyoutakeacloserlookatios-driver,Appium,andKeepItFunctional.Allofthetoolsprovidereallygoodandpowerfulfeaturesinordertobuild
reliableandrobusttestautomationscriptsforiOSapps.IfyoujustwanttoautomateanativeiOSapp,KIFwouldbeagoodchoiceasyoucansetupandwritereliableandrobustautomatedtestsveryquickly.AnotheradvantageofKIFisthatthetestscriptsarewrittenwithObjective-C,thesamelanguagewithwhichtheappwillbewritten.IfyoustrugglewithObjective-C,youcansimplyaskyourdevelopersforsupportorhavethemwritethetestautomation.
IfyouwanttoautomateahybridiOSorWebapp,youshoulduseeitherios-driverorAppiumasbothoffergreatsupportforvariousprogramminglanguagesaswellastheoptiontousetheminacloudorSeleniumGridenvironment.Thisprovidespowerfulscalingandparalleltestexecutiononseveraldifferentdevicesandoperatingsystems.Allthreetoolscomewithgooddocumentationandverygoodcodesamples,
areeasytouse,andhaveahugecommunitybehindthemthatisonhandtohelpifyourunintoanyproblems.
MobileTestAutomationToolsSummaryAsyouhaveseen,therearemanydifferentmobiletestautomationframeworksavailableonthemarket.Eachtoolhasitsownstyleofwritingtestscriptsandsupportsdifferentfeaturesets,differentmobileplatforms,anddifferentmobileapptypes.Everytoolcurrentlyavailableonthemarkethasitsprosandcons.Youshouldreallykeepinmindthatnotoolisperfect,beitanopen-orclosed-sourcetool.Beforeselectingamobiletestautomationtool,scanthemarketforpossibletoolsandsolutionstohelpyoumaketherightdecision.Useasampleapporachecklisttoevaluatethevarioustools.Andlastbutnotleast,it’simportanttostartsimplewithamobiletest
automationtool.Don’ttryandfindTHEoneandonlytestautomationsolutionforyourmobileapp.Maybeyouneedtousemorethanonetoolortocombinetoolsinordertobuildupatestautomationsuitethatcoversyourneedsandrequirements.Itisbettertohaveonlyacertainamountoftestautomationinplacethatcovers,forexample,thecriticalpartsofyourappinsteadofeverypart.Whenchoosingatool,askyourselfthequestion“Whatshouldbeautomated?”
ContinuousIntegrationSystemContinuousintegration(CI)isnothingnew,andthisdevelopmentpracticeofintegratingandtestingthecodefromacentrallysharedcoderepositoryseveraltimesadayhasbeeninuseforseveralyearsnow.Everycheck-inisthenverifiedbyadifferentsetofautomatedbuildstepstoensurethatthelatestcodechangeswillnotbreakthesoftwareandintegrationwithothermodules.ACIservershouldbeavailableineveryproject,nomatterifthesoftwareisa
desktop,Web,ormobileapplication,asitwillhelptheteamreducetheriskofbrokensoftware,givefastfeedbacktoeveryoneinvolvedintheproject,andintegratesmallersoftwarepartsintoothersasearlyaspossiblewithinthe
process.Nowadays,thereareplentyofopen-andclosed-sourceCIsystemsavailable
onthemarket.IfyouhaveaCIsysteminyourteam,integratetheautomatedmobiletestsintoit.NearlyeverymobiletestautomationtoolcanbeintegratedintoaCIsystem.Ifthisisnotpossiblewiththetoolyou’reusing,you’llhavetofindawaytointegrateitsuchaswithexternalbuildscriptsthatwillrunoutsidetheCIenvironmenttofulfillthetask.Thisisveryimportantfortheprojectasawholesothatacompletebuildpipelineincludingallbuildandtestscriptscanbeestablished.Whenthetestautomationtoolhasbeenintegrated,defineabuildandtest
strategywithyourteam.Talktoyourdevelopersanddefinewhichtestsshouldbeexecutedaftereverycommitorduringthenight.IfyourautomatedtestsstarttoturnfastfeedbackfromyourCIsysteminto
slowfeedback,splitthemintoseparatetestsuites.Forexample,youcandefineasmoketestsuitecontainingteststhatcheckwhetherthemainpartsoftheapplicationarestillworking.Thistestsuiterunsforonlyacoupleofsecondsorminutesandshouldbeexecutedaftereverycommit.Anothertestsuitecanbearegressiontestsuitethatruns,say,fourtimesadaytochecktheappinmoredetail.Andanothersuitecanbeafulltestsuitethatrunseverytesteverynighttomakesurethecodechangesfromthepreviousdayhavenotaffectedtheexistingpartsoftheapp.AnotherimportantpointwhenaddingamobiletestautomationtooltoaCI
systemistestreporting.TheCIsystemmustbeabletodisplaydifferentkindsoftestreportformatsinordertoprovidethewholeteamwithvisualfeedback.Thereportingcomponentofthesystemshouldbeeasytoreadandunderstand.OncetheCIsystemandallofthemobiletestinganddevelopmenttoolshave
beenintegrated,defineacompletebuildandtestpipelineforyourmobileapplication.Thebuildpipelineshouldbeabletostartautomaticallywithoutanyuserinputs,forexample,byeitherlisteningtoacentralcoderepositoryortriggeringthebuildsatacertaintimeduringthenight.Furthermore,thebuildstepsshouldtriggerotherbuildstepsinordertounit-
test,end-to-end-test,buildanapplicationondifferentstagingsystems,buildalphaorbetaversionsoftheapp,oruploadtheapplicationtoabetadistributionserver.Hereisanexampleofapossible,verysimplemobilebuildpipeline:1.Performstaticcodeanalysis,suchaswithPMD,FindBugs,Lint,orCheckstyle.
2.Performunittests.3.Performend-to-endUItests.4.Buildamobileappversionondifferentstagingsystems.5.Buildabetaversionofthemobileapp.6.Uploadthebetaversiontoabetadistributionsystem.7.Signandbuildareleasecandidateoftheapp(theonlybuildstepthatshouldbetriggeredmanually).
Buildsteps1and2shouldbeexecutedonthedeveloper’scomputerbeforeheorshecommitsthecodetothecentralrepository.IfyouhaveaCIsystemforyourmobileappinplace,don’tforgettoplugreal
devicesintothatserverinordertoexecuteallofthetestautomationontherealdevice.ThefollowingaresomeoftheavailableCIsystems:•Bamboo(www.atlassian.com/de/software/bamboo)•Bots(https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/ConfigureBots/ConfigureBots.html)•Buildbot(http://buildbot.net/)•CruiseControl(http://cruisecontrol.sourceforge.net/)•Janky(https://github.com/github/janky)•Jenkins(http://jenkins-ci.org/)•TeamCity(www.jetbrains.com/teamcity/)•TravisCI(https://travis-ci.org/)
ImportantHaveaCIsysteminplaceandintegrateyourmobiletestautomationtooltogetfastfeedbackaboutthequalityoftheappaftereverycodechange.
BetaDistributionToolsAsyouhavelearnedfromthepreviouschapters,mobileusershaveexpectationsintermsoftheusability,performance,andfeaturesofmobileapps.Youandyourteamthereforehavetobesurethatyourmobileappprovidesagreatuserexperienceandisfast,reliable,andfuntouse.Tomeetalloftheseexpectations,
youandyourteamhaveachallengingjobandneedtotesttheappwithotherpeopletogetfeedbackasearlyaspossibleinthedevelopmentprocess.Togetthisfeedbackfromotheruserssuchascolleaguesorusersfromyour
targetcustomergroup,youneedatooltodistributebetaversionsofyourapp.Withthehelpofthistool,youcangivepotentialusersaccesstoabetaversionofthenextreleasecandidate.Betadistributiontoolsincludeseveralusefulfeaturessuchasover-the-airapp
distribution,crashreporting,bugreporting,anddirectin-appfeedback.Sometoolsprovideso-calledcheckpointswithintheappwhereyoucanplacequestionsfortheuseraboutthefeatureheorshejustused.Anothernicefeatureisso-calledsessions,whichcanbeincludedtotrackhowthebetatesterusestheapporafeature,thushelpingyoutoidentifyunexpectedappusage.Thetoolsalsoprovidedataandstatisticsaboutthemobileoperatingsystemversions,devicehardware,andinterfacelanguage.Alloftheinformationprovidedbyabetadistributiontoolisreallyimportant
toknowbeforeyourappisusedbythemajorityofyourtargetcustomergroup.Youcandrawonthisinformationtorefineanddevelopyourappintherightdirection,thusmakingitfarmorereliable,stable,andofcoursefuntouse.Whenusingabetadistributiontool,itisveryimportanttoinformthepotential
betatestersaboutallofthesefeaturesandthatinformationisgatheredaboutthedeviceandtheuser.Asastartingpoint,useabetadistributiontoolwithinyourcompanybyasking
yourcolleaguestotesttheappandprovidefeedback.Noteverymobileappcanbedistributedasabetaversiontotheoutsideworldduetonetworkrestrictions,companyguidelines,orpertinentlaw.Hereisalistofbetadistributiontools:•Appaloosa(www.appaloosa-store.com/)•AppBlade(https://appblade.com/)•ApplauseSDK(www.applause.com/mobile-sdk)•BetabyCrashlytics(http://try.crashlytics.com/beta/)•BirdFlight(www.birdflightapp.com/)•GooglePlaynativeAppBetaTesting(https://play.google.com/apps/publish)•HockeyApp(http://hockeyapp.net/)•HockeyKit(http://hockeykit.net/)
•TestFlight(https://developer.apple.com/testflight/index.html)GoogleandApplealsoprovideawaytodistributeabetaappversiontoa
wideruserbase.WithinGooglePlay32youareabletoaddbetatesterswiththeirGmailaddresseswhocanthendownloadthebetaversionfromtheGooglePlaystore.Alternatively,youcandefineastagedrolloutwhereanewversionofyourappisavailableonlytoacertainnumberofusers,forexample,10%ofthecurrentuserbase.Iftheappworksasexpected,youcanincreasetheapprollouteithertoyourentireuserbaseorbyanotherincrement.32.https://support.google.com/googleplay/android-developer/answer/3131213?hl=en
OntheApple33side,youarealsoabletobuildabetaversionofyourappanddistributeittoregisteredbetatesters.ThebetatestershavetoberegisteredwiththeiruniquedeviceID(UDID)usinganadhocprovisioningprofile.However,youareabletoregisteronly100testdeviceswithinonemembershipyear.YoucangetaroundthistestdevicerestrictionifyourcompanyispartofApple’senterpriseprogram.33.
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/AppDistributionGuide/TestingYouriOSApp/TestingYouriOSApp.html
ImportantWhereverpossible,useabetadistributiontooltogatherearlyfeedbackfrombetatestersinordertobuildabettermobileapp.
SummaryThefifthchapterofthisbookconcentratedonmobiletestautomation.AtthebeginningofthechapterIexplainedtheproblemwiththetraditionaltestautomationpyramidandmobileapps.Iintroducedandexplainedtheflippedtestingpyramidandshowedanewpyramid—themobiletestpyramid.Thispyramidcontainsautomated,aswellasmanual,testingtofitalltherequirementsofthecurrentstateformobileapps.InthenextsectionofthischapterIdescribedthedifferentapproachesand
typesofmobiletestautomationtools.Thesearetoolsthatuseimagerecognition,coordinate-basedrecognition,textrecognition,ornativeobjectrecognition.Iexplainedeveryapproach,withprosandcons.Furthermore,Iassignedtheavailabletoolsonthemarkettothedifferentapproachestoprovideyouwithanoverview.InanothersectionIexplainedwhyitisabadideatojustusecaptureand
replaytoolsforyourtestautomation.Thosetoolsdoprovideagoodstarting
pointtogetsometestautomationupandrunninginthefirstplace,butinthelongruntheywillcauselotsofmaintenancetroubleandthetestsaren’treliableatall.Inthesection“WhatShouldBeAutomated?”Iexplainedwhichpartsofyour
appneedtestautomationandwhichdonot.Forexample,itisagoodapproachtoautomatethebusiness-criticalpartsoftheapp.Ontheotherhand,thosepartsthatarelikelytochangeofteninthenearfuturearefineformanualtestingbecausethetestautomationwillnotruninastablefashioniftheyareincluded.Tohelpyouselecttherighttestautomationtoolforyourmobileapp,I
includedalistwithselectioncriteriatofindthetoolthatfitsbestintoyourdevelopmentandtestenvironment.Thebiggestpartofthischaptercoveredthecurrentstateofmobiletest
automationtoolsfortheiOSandAndroidplatforms.Iexplainedthedifferenttoolswithcodesamplesaswellasprosandcons.Idescribedthefollowingtools:
•Robotium•Spoon•Selendroid•CalabashforAndroidandiOS•Appium•Espresso•UIAutomation•ios-driver•KeepItFunctional
Theclosingsectionsofthischaptercoveredthetopicsofcontinuousintegrationandbetadistributionofmobileapps.IoutlinedasamplemobileCIbuildpipelinethatcaneasilybeadaptedtoyourenvironment.IaddedalistofCItoolsthatcanbeusedformobileapps.Inthebetadistributionsection,Iexplainedthepurposeofdistributingabetaversionofyourapptocolleaguesorbetatesterstogetearlyfeedbackandbugreports.
Chapter6.AdditionalMobileTestingMethods
Sofaryou’velearnedaboutmobiletechnologiesandhowtomanuallytestmobileappsindifferentscenariosaswellaswhileoutandabout.You’velearnedaboutmobiletestautomationandtheconceptsbehindthedifferentmobiletestautomationtools.You’renowawareofseveraltoolsforthedifferentmobileplatformsandknowhowtoselecttherighttoolforyourtestingprocess.Toextendyourknowledgeandyourtoolbox,inthischapterIintroducesome
otherpossiblemobiletestingapproaches:crowdandcloudtesting.Bothoftheseapproachescanbebeneficialinyourdailyworkwithinamobileteam.
CrowdTestingAcompanyhasthreepossiblewaystoestablishsoftwaretestingwithinanorganization.Testingcanbedonewiththeaidofanin-housequalityassurancedepartment,viaoutsourcing(nearshoring/offshoring),orusingacrowdtestingapproach.In-housetestingandoutsourcingarenothingnew,andbothareconsideredestablishedapproachesamongvariousorganizationsandindustries.Butthisisnotthecasewithcrowdtesting.Thetermcrowdsourcingwas
introducedbyJeeHowe1in2006andisacombinationofthewordscrowdandoutsourcing.Inthesoftwaretestingbusinessthewordcrowdsourcinghasbecomecrowdtesting.
1.www.crowdsourcing.com/
Withthehelpofacommunityofexternalsoftwaretesters,severalcrowdtestingprovidersofferanewwaytoperformsoftwaretesting.Theexternalsoftwaretesterscomefromdiversebackgrounds,bothgeographicallyandintermsoftheirleveloftechnicalknowledge.Dependingonthecrowdtestingprovider,thecrowdcanrangefromafewpeopletoseveralthousandtestersworldwide,anditmayincludesoftwaretestingexpertsandpeopleofanyage,gender,profession,andeducationalbackground.Furthermore,thecrowdtestershavelotsofdifferentdeviceswithdiversehardwareandsoftwarecombinationsandaccesstoanumberofdifferentdatanetworks.Crowdtestingistantamounttotestinginthewild.Withtheaidofcrowdtesters,anappwillbetestedunderasetofrealistic
scenariosthatcan’tbecreatedbyanin-housetestingteam.Themobileappwillbetestedunderreal-worldconditions—withdifferentdatanetworks,hardware,andsoftwareaswellasdifferentusers.Externaltestersprovideafreshsetof
eyesforyourmobileappandwilldoubtlessreportlotsofbugs,deviations,aswellasperformance,usability,andfunctionalissues.Crowdtestingprovidersofferaplatformwherecrowdtesterscanregisterand
createaprofilestatingtheirdevices,skills,anddemographicbackground.Theclientcanaddtheappundertest,thepreconditions,samplescenarios,instructions,knownbugs,anddetailedtestplans.Theclientisalsoabletodefinethedemographicbackground,targetcustomergroup,skillset,anddevicesonwhichthecrowdshouldtestthemobileapp.Somecrowdprovidershaveaprojectmanagementframeworkinplace,
includinggovernanceandlegalstructuresforthetestingphase.Inaddition,testprovidersassignaprojectmanagerwhoisresponsibleforthetestcycle.Theprojectmanagerisalsothepersonwhofilters,rates,andcategorizesthebugsreportedbythecrowdandsummarizesthetestingcyclefortheclient.Somecrowdtestingprovidershaveanassessmentcenterortrialprojectin
placewherepossiblecrowdtestersmustparticipateinordertoverifytheirtestingabilitiesbeforebeingacceptedintothetestingcommunity.Mostcrowdtestingproviderschargeafeefortheirservices,butcrowdtesting
isrelativelycheapbecauseyoupayonlyforthepackagethatyouhaveagreeduponwiththecrowdtestingprovider.Therearedifferentkindsofpackagesavailable,suchassimplebugreporting,exploratorytesting,and/orexecutingdefinedtestcases.Figure6.1showsthetypicalcrowdtestingprocessfromthefirstbriefingwith
thecrowdprovidertothefinalpresentation.Duringthetestcycle,theclientisalwaysabletoseetheliveprogressofthecrowdtesters.
Figure6.1Typicalcrowdtestingprocess
1.Thefirststepistheinitialbriefingbetweenthecrowdproviderandtheclient.
2.Thecrowdproviderselectsthecrowdbasedontheclient’srequirements.Duringthetestcycletheprovidersupportsthecrowd.
3.Thecrowdteststhesoftware.4.Thecrowdtestersfilereportsdependingontheaimofthetestcycle,forexample,bugs,feedback,oranyotherkindofproblems.
5.Thecrowdproviderensuresthatthequalityofthereportsisgoodenough.Heorshewillfollowuponbugsifmoreinformationisneeded.
6.Attheendofthetestcyclethecrowdproviderwritesthefinaltestreport.7.Thereportisthenpresentedtotheclient.However,therearesomechallengesyouneedtobeawareofifyouwantto
useacrowdtestingapproachinyourproject.Itcantakealongtimetoprepareandorganizeacrowdtestingcycle.You
needtodefinetheexactgoalforthetestcycleandcollectandpreparealloftheinformationneededforthecrowdinordertogetvaluableresults.Youneedto
briefthecrowdtestingprovider,andattheendofthetestcycleyou’llneedsomemoretimetoreviewallofthebugandtestreportsyoureceive.However,thesestepsarealsonecessaryifyouwanttobriefanewin-housetesteronanewproject.Thereportedbugsmaybeofverylowqualitybecauseofthecrowd’slackof
knowledge.Thecrowdtestingproviderwillofcoursefilterandcategorizethebugsandpaytestersonlyforthoseofrealvalue,butit’salsolikelythatthebugreportsaren’tdetailedorpreciseenoughforyourneeds.Itmaybeverydifficultforcrowdtesterstoaccessthedevelopmentandtest
systems.Dataprivacyandsecurityissuesmaypreventyoufromgrantingexternalaccesstointernalstagingsystems,forcingthecrowdtesterstousetheproductionenvironmentwhich,inturn,hastoworkandinteractwithabetaversionofyourapp.Maybeyouneedtocreateanisolatedsectionwithintheproductionenvironmentthatcanhandlethebetarequestsandbeusedforthecrowdtestingcycle.Attheendofthetestcycleyouneedtobesurethatthecrowdtestersarenolongerabletoaccesstheproductionenvironmentandthattheappwillbeofnousetothem.AnotherchallengecomesintheformoflegalhurdlesandNDAs
(nondisclosureagreements).Ifyourappisconfidential,forexample,crowdtestingmaysimplynotbeanoption.Theprosofcrowdtestingincludethefollowing:•Thecrowdincludesdifferenttestersfromaroundtheworld,withdifferentdemographicbackgroundsandskillsets.•Lotsofdifferentmobiledeviceswithdifferenthardwareandsoftwarecombinationscanbeusedfortesting.•Themobileappistestedinreal-worldconditionswithrealusers.•Thecrowdprovidesafreshsetofeyesforyourapplication.•Lotsofissueswillbereported.•Crowdtestingprovidersfilterandcategorizethebugs.
Theconsofcrowdtestingincludethefollowing:•Crowdtestersarenotgenerallytestingexperts.•Youdon’treallyknowwhomyou’redealingwith.•Bugreportsmaybeoflowquality.•Accesstostagingsystemscanbeverydifficultduetolegalhurdles,dataprivacy,andsecurityconcerns.
•Crowdtestingcantakealongtimetoprepare.•Communicationwiththecrowdcanbedifficult.•Reproducingbugscanbedifficult.•Thereisariskthatthemobileappwillcontinuetobeusedbythecrowdafterthetestcyclehasended.
ImportantSomecrowdtestingprovidershaveamechanismtoautomaticallyuninstalltheappundertestfromtesters’mobiledevices.
Thefollowinglistincludessomecrowdtestingproviders(thislistisbynomeanscomplete):
•99tests(http://99tests.com/)•Applause(www.applause.com/)•crowdsourcedtesting(https://crowdsourcedtesting.com)•GlobalAppTesting(http://globalapptesting.com/)•Mob4Hire(www.mob4hire.com/)•passbrains(www.passbrains.com/)•Testbirds(www.testbirds.de/)•testCloud(www.testcloud.io/)•TestPlus(www.testplus.at/)
ImportantThecrowdtestingapproachisagoodextensionofyourin-housetestingteam.However,crowdtestingwon’tandshouldn’treplaceyourin-housetestingactivities.
PrivateCrowdTestingIfyou’renotabletouseapubliccrowdtestingproviderbecauseoflegalrestrictions,NDAs,ordataprivacyanddatasecurityconcerns,orbecauseyou’renotabletograntaccesstoyourdevelopmentenvironment,youcanuseaprivatecrowdtestingapproach.Youcanstartandbuildupaninternalcrowdtestingsessionwithyour
colleagues.Dependingonthesizeofyourcompany,you’llbesurroundedbylots
ofpeoplefromdifferentbackgroundsworkinginvariousdepartments,soyoucancollectfeedbackfromdevelopers,designers,productmanagers,projectmanagers,management,sales,andmarketingcolleagues.Withtheinputfromyourcolleagues,you’llbeabletosimulaterealusersindifferentusagescenariostogetaninitialimpressionofyourapp.Oneofthebiggestadvantagesyou’llnoticewhenestablishingaprivatecrowd
testingapproachisthatyou’llneedlesstimetoprepareandorganizeitthanwithapubliccrowdtestingapproachbecauseyourcolleaguesarealreadyfamiliarwiththecorporateenvironment,theproduct,anditsfeatures.Employeescanalsogainaccesstothedevelopmentorstagingenvironment,whichallowsyoutosidestepanylegalrestrictionsandNDAs.Furthermore,itisveryeasytocommunicatewithyourcolleaguesduringthe
testcycleandtoobservethemwhiletheyareusingtheapp.Thiswillprovideyouandyourteamwithextremelyvaluableinsightsintohowusersactuallyinteractwithyourappintermsofusabilityandfunctionality.Attheendofatestcycleyoucaninterviewyourcolleaguesabouttheappanditsnewfeaturestogathermoreinformationabouttheiropinionsoftheappandanyproblemstheyencounteredduringtesting.Theinternalcrowdtestingsessioncanbekeptshortandcanberepeated
severaltimesduringyourapp’sdevelopmentphase.Thisgivesyoumoreflexibilitywhenitcomestoreactingtopossibleusability,functional,orperformanceproblems.Onewaytokeepyourcolleaguesdedicatedandmotivatedistointroduceyour
internaltestingsessionasacompetition.Trycreatingcategoriesforthecompetitionandawardprizesforeachcategory.Possiblecategoriescouldincludethefollowing:
•Bestusabilitybug•Bestfunctionalbug•Bestperformancebug•Bestsecuritybug•Bestfeedbackprovided•Bestbugreport•Bestoveralltestengagement
Prizescouldincludethefollowing:•Companymugs
•Funnystickers•T-shirtswithfunnyslogans•Vouchers
Prizesdon’thavetobeexpensive;theyjustneedtomotivatepeopletofindandreportasmanybugsaspossible.Furthermore,competitionwillgiveyourcolleaguesanincentivetotakepartinupcomingtestcycles.
ImportantTrytocreateaprivatecrowdtestingsessionwithinyourcompanyandseehowyourcolleaguesperformastesters.You’llbesurprisedatthebugsandthereportedresults.
MobileCloudTestingManufacturersofmobilecloudtestingsolutionsprovideawiderangeofcurrentmobiledeviceswithdifferenthardwareandsoftwarecombinationsinthecloud.Cloudtestingprovidersusethecharacteristicsofcloudcomputingtoprovidethisservicetomobilecompanies,teams,andmobiletesters.Suchcharacteristicsincludethefollowing:
•Thecloudusesadynamic,shared,andvirtualITinfrastructure.•Thecloudprovideson-demandself-service.•Thecloudisscalablebasedontheload.•Thecloudispricedaccordingtoconsumption.•Thecloudisavailableacrossdifferentnetworkconnections.
MobilecloudtestingsolutionsareaccessibleviatheWebandprovidedifferenttypesoftestingthatcanbeperformedwithinthecloudonrealoremulated/simulateddevices.Thedifferenttestingtypesavailableare:
•Functionaltesting•Performancetesting•Loadtesting•Mobiledevicetesting•Cross-browsertesting
AsisthecasewithOpenDeviceLabs,themobilecloudprovidesyouandyourteamwitheasyaccesstoacomprehensiverangeofcurrentmobiledevicesusingallofthedifferentmobileplatforms,nomatterwhereyouareintheworld.
Ifyouwanttotestyourmobileappinthecloud,allyouneedtodoisallocatethephysicaldevice,uploadandinstallyourapp,andstartyourmanualorautomatedtesting.Providersofferdifferentkindsofadditionalservicessuchasreportingfeatures,screenshots,andvideosofyourtestsessionoranAPItoexecuteyourtestautomationscriptsonseveraldevicesinparallel.Thefactthatmobiletestcloudsaredistributedindifferentgeographical
regionsallovertheworldmakesiteasytosimulatedifferentnetworktechnologiesandscenariosfrompotentialmobileusers.Onemajoradvantageofamobiletestcloudisthatyoudon’tneedtobuynew
phonesforyourdevelopmentortestingeffortsasthiswillbedonebythecloudprovider.Furthermore,youdon’tneedtomaintainallthedifferentdevices,whichmayhaveapositiveimpactonyourprojectcosts.However,therearealsosomelimitationsifyouwanttotestyourmobileapp
withinamobilecloud.Forexample,ifyourappusesaBluetoothconnectiontoconnecttootherphysicalelementssuchasspeakers,itisnotpossibletotestwithinacloud.Itisalsonotpossibletouseallthesensorsandinterfacessuchastheproximitysensor,brightnesssensor,accelerationsensor,orgyroscopesensorbecausetheclouddevicesaremountedandconnectedtoaserverwithinadatacenter.Ontopofthat,youcan’ttestyourmobileappforinterruptionsornotificationsfromotherappsorthedeviceitself.Anotherdrawbackoftestingyourappmanuallywithinamobilecloudisthat
you’reinteractingwithyourappthroughthecomputermouse.Youhavenophysicalcontactwiththedeviceorappwithyourhandorfingers,makingitverydifficulttogetafeelfortheusabilityandresponseoftheapp.Furthermore,multitouchgesturescan’tbeperformedonthetouchinterface.And,lastbutnotleast,securityandprivacyissuesshouldn’tbe
underestimatedwhenusingamobiletestcloud.Youneedtobesurethattheproviderwillcompletelyremoveyourappanditsdatafromthetestdevicesafterthetestsessionhasbeencompleted;otherwisethenextcloudtestingusermightbeabletoseeanduseyourapp.Beforechoosingamobilecloudtestingprovider,checktheprovidedfeatures
andcomparethemwiththoseofothervendors.Youshouldalsoweightheprosandconsofmobilecloudtestingtoseeifthisapproachfitsyourprojectanddevelopmentenvironment.Theprosofmobilecloudtestingincludethefollowing:•Easyaccesstothephysicaldevices•Easyaccesstoemulatorsandsimulators
•Fastandeasysetupofthemobiledevices•Accessibilityfromanywhereintheworld•Lowercostsasthere’snoneedtobuynewdevices•Nodevicemaintenancecosts•Differenttesttypes•Simulatednetworkprovidersfromallovertheworld•Goodreportingfeaturessuchasreports,screenshots,andvideos
Theconsofmobilecloudtestingincludethefollowing:•Youhavelesscontroloverthemobiledevices.•Networkissuescanaffecttheavailabilityandthefunctionalityofthemobiletestcloud.•Securityandprivacyissues:Othercompaniesusethesamedevices,soyouneedtobesuretheappwillbedeletedinfullafterthesessionhasbeencompleted.•Performanceproblems:TestingtheappviatheInternetcanhaveanimpactontheappexecutiontimeandtestresults.•Firewallsetup:Youneedtochangesomeofthefirewallsettingsinordertogainaccesstothedevelopmentandtestenvironments.•Ifthecloudhasanysystemproblemsoroutages,yourapptestenvironmentmayalsoexperiencepoorperformanceoroutages.•Itisdifficulttotrackdownintermittentproblemsbecauseyouhavenoaccesstothesystem.•Noteverythingcanbetestedwithinacloud,forexample,sensors,interfaces,interrupts,andnotifications.•Youhavenophysicalcontactwiththedevice.
Thefollowingaresomemobilecloudtestingproviders(thislistisbynomeanscomplete):
•AppThwack(https://appthwack.com/)•Appurify(http://appurify.com/)•CloudMonkey(www.cloudmonkeymobile.com/)•Experitest(http://experitest.com/cloud/)•Keynote(www.keynote.com/)•Neotys(www.neotys.com/product/neotys-cloud-platform.html)
•PerfectoMobile(www.perfectomobile.com/)•Ranorex(www.ranorex.com/)•SauceLabs(https://saucelabs.com/)•TestChameleon(www.testchameleon.com/)•Testdroid(http://testdroid.com/)•Testmunk(www.testmunk.com/)•TestObject(https://testobject.com/)•XamarinTestCloud(http://xamarin.com/test-cloud)
ImportantMobilecloudtestingisagoodextensiontoyourin-housetestingworkbutcomeswithsometestinglimitationsthatneedtobetakenintoconsideration.
PrivateCloudIfthedrawbacksoutweightheadvantagesofapubliccloud,butyou’restillinterestedinusingacloudtestingapproach,considerusingaprivatemobiletestingcloud.Almosteveryproviderintheprecedinglistcanprovideaprivatemobiletestcloud.Aprivatemobilecloudcanbeofferedasahostedorlocallyinstalledsolution.
Thehostedsolutionisthemorecommononebecauseiteliminatesthelogisticsandcostsincurredbybuyingnewphonesandmaintainingdevices,suchasinstallingupdatesandperformingconfigurations.Privatecloudprovidersofferasecureareawithinthedatacenterthatgrantsexclusiveaccesstothephysicaldevices.Ontopofthat,theyoffervarioussecurityoptionstomeetacompany’ssecuritypoliciesandrequirements.Thelocallyinstalledsolutionisalsoknownasaprivatemobiletestlab.Ifyou
decidetousethissolution,themobilecloudproviderwillprovideyouwithamobiletestingrackincludingdevicemanagementsoftwaretomaintainandallocatetestdeviceswithinyourcompanytodevelopersortesters.Youcanalsoextendtherackwithnewdevicesonyourown.Thisdevicerackwillbelocatedbehindyourfirewall,anddevicesare
connectedtothedevelopmentenvironment,whichalleviatesanyproblemswiththespeedandconnectionproblemsofapubliccloud.Securityandprivacyconcernsarealsonolongeranissuewiththissolution.Bywayofexample,refer
tothemobiletestlabfromMobileLabs.22.http://mobilelabsinc.com/products/deviceconnect/
Alocallyinstalledsolutionalsocomeswiththeproblemthatyou’reresponsibleforbuyingandmaintainingnewdevicestobeincludedinthedevicerack.Furthermore,aprivatemobiletestingcloudcanbereallyexpensiveasyouhaveexclusiveaccesstothetestdevices,thecloudvendorprovidesyouwithexclusivesupport,andbothyouandyourcompanyneedtotrainyourcolleaguestobeabletoworkwiththeprivatecloudsoftwareandsystem.Theprosofaprivatemobiletestcloudincludethefollowing:•Easyaccesstothephysicaldevices•Fastandeasysetupofthemobiledevices•Accessibilityfromanywhereintheworld•Nodevicemaintenancecosts(hostedsolutiononly)•Differenttesttypes•Simulationofvariousnetworkprovidersfromallovertheworld(hostedsolutiononly)•Exclusiveaccesstothetestdevices•Nosecurityconcerns
Theconsofaprivatemobiletestcloudincludethefollowing:•Itisfarmoreexpensivethanapublicmobiletestcloud.•Networkissuescanstillaffecttheavailabilityandthefunctionalityofthemobiletestcloud.•Firewallsetup:Youneedtochangesomefirewallsettingstogainaccesstothedevelopmentandtestenvironments(hostedsolutiononly).•Ifthecloudhasanysystemproblemsoroutages,yourappwillperformpoorlyandmayalsoexperienceoutages(hostedsolutiononly).•Itisdifficulttotrackdownintermittentproblemsbecauseyouhavenoaccesstothesystem(hostedsolutiononly).•Noteverythingcanbetestedwithinacloud,forexample,sensorsandinterfaces.•Additionaltrainingisrequiredtoworkwiththecloudprovidersoftwareandsystem.
Important
Becauseofthemanualtestinglimitations,youshouldconsiderusingmobiletestcloudsformobiletestautomationpurposesonlywiththeaimofhandlingautomatedtestingandfragmentationacrossseveraldevices.Manualtestingshouldstillbeperformedonrealdevicesinreal-worldenvironmentsandwhileonthemove.
Cloud-BasedTestAutomationInChapter5,“MobileTestAutomationandTools,”Iexplainedthedifferentconceptsofmobiletestautomationtools.IalsodescribedsomemobiletestautomationtoolsfortheiOSandAndroidplatforms.Whenchoosingamobiletestautomationtool,checktoseeifthetoolisabletoexecutethetestscriptswithinamobiletestcloud.SomeofthemobiletestcloudprovidersofferanAPIforvariousmobiletestingtoolssoyoucanexecuteyourscriptswiththeirservices.ThisAPIcanhelpyouscaleyourtestingeffortsandtestdifferentdevicesinparallel.SomeprovidersofferthepossibilityofwritingtestautomationscriptsontheWebwithinthecloudtestingsoftware.Theadvantagesofamobiletestcloudalsoapplytotestautomationwithinthe
cloud.Amobiletestcloudcanhelpyoucreateanautomatedon-demandtestenvironmentforyourmobileappandmobileplatform.However,usingacloud-basedtestautomationapproachalsohassome
additionaldrawbacks.Testexecutiononclouddevicesisslowerthanwithalocaltestautomationsolution,whichispartlyduetothecommunicationbetweenthecloudprovidernetworkandyourcompanynetworkwhenrequestingandsendinglotsofdata.Thislatencycanhaveanimpactonthetestresultsandbehaviorofyourapp.Testingtheperformancewithtestautomationscriptsonamobiletestcloudisthereforenotanidealsolution.Debuggingthetestautomationscriptsontheclouddevicesisanotherissueasscriptdebuggingispossiblebutnotyetgoodenoughtoworkefficiently.Ifyou’reconsideringusingacloud-basedtestsolution,evaluateseveral
providerstoseeiftheyofferthefeaturesyouneedforyourmobileapp.Acloud-basedtestautomationapproachcanbeausefuladditiontoyourin-housetestautomationandcanmakeyourtestingworkmoreefficient.
SummaryInthesixthchapteryouhavelearnedaboutcrowdandcloudtestingservices.Bothcanbeadditionstoyourin-housemobiletestingactivitiesbutshouldneverbeusedastheonlymobiletestingsolutionforyourapp.
InthecrowdtestingsectionIexplainedthetypicalprocessofacrowdtestingcycle.Youneedtokeepinmindthatthisprocesswilltakequitesometimeforpreparationaswellasinthebuganalysisphase.Theseeffortsshouldnotbeunderestimated.Furthermore,Idescribedthedifferencesbetweenaprivateandapubliccrowdtestsession.Bothapproachesdifferalotinthedetails;however,thosedetailsareimportantinordertoavoidinfrastructure,dataprotection,andsecurityconcerns.Inthecloudtestingsection,Iexplainedthefeaturesofacloudtesting
provider.Inaddition,Idescribedwhichtesttypescanbeperformedwithinacloud.IntheprosandconssectionIoutlinedpossibleproblemswithpublicandprivatecloudsaswellastheadvantages.
Chapter7.MobileTestandLaunchStrategies
Sofaryou’velearnedalotaboutmobiletestingandthedifferenttesttechniquesandapproaches.Thischaptercoversmobiletestandlaunchstrategiesandwhatyouneedtoincludeinyourstrategies.Boththetestandlaunchstrategiesareveryimportanttoeveryproject,andyoushouldn’tunderestimatejusthowusefulitistokeepawrittenrecordofthem.InthefollowingsectionsI’llprovideyouwithsomeexamplesofmobiletestandlaunchstrategies.You’llalsofindsomequestionsthatwillhelpyoutowriteyourownstrategies.
MobileTestStrategyIngeneral,ateststrategyisadocumentthatdescribesthetestingapproachandworkinvolvedinthesoftwaredevelopmentcycle.Thisstrategycanbeusedtolettheprojectmanager,productmanager,developers,designers,andanyoneelseinvolvedinthesoftwaredevelopmentcycleknowaboutthekeyissuesofthesoftwaretestingprocess.Ateststrategyincludesthetestingobjective,testlevelsandtechniques,
resourcesrequiredtotestthesystemundertest,andthetestingenvironment.Italsodescribestheproductrisksandhowtomitigatethemforstakeholdersandcustomers.Finally,italsoincludesadefinitionoftestentryandexitcriteria.Thedefinedteststrategywillhelp,remind,andguideyousoyoudon’tforget
theimportantcomponentsandfeaturesoftheapp.Youreallyshouldtakethetimetowritedownthestepsandresourcesneededtotestthemobileapp.Furthermore,theteststrategydocumentsyourworkandendeavorswithintheproject.Writingateststrategyoncedoesn’tmeanthatit’sthensetinstoneandyoucan’tmakeanysubsequentchanges.Onthecontrary,it’simportantthatyoutalkaboutthestrategywithyourteamfromtimetotimesoyoucanadaptittoanyproductchangesorothermorerecentcircumstances.However,thereisnosinglemobileteststrategythatcanbeusedbyevery
teamorforeverymobileappasalmosteveryapphasdifferentrequirements,goals,andtargetgroupsandrunsondifferentmobileplatforms,whichmakesitimpossibletoreusetheentirestrategyineveryproject.Buttherewill,ofcourse,besomeitemsthatcanandshouldbereused.Thefollowingsectionsofthischaptershouldgiveyouanideaofhowtoshape
amobileteststrategy.Youcanusethemasastartingpointandguideforputtingtogetheryourownmobileteststrategy.
ImportantDraftingamobileteststrategydoesn’tnecessarilyinvolvewritingendlessdocumentationasyouandanyothertesterssimplywon’thavethetimeand/ortheresourcestorunthroughitall,andflexibilityisthenameofthegameinthemobiletestingbusiness.Amobileteststrategyshouldservetoguideyouandanyoneelseinvolvedintheprojectsoyoucankeeptabsontheimportantpartsofthetestingprocess.
DefineRequirementsThefirstthingyouandyourteamshoulddoisdefineyourapp’srequirementsandfeaturesattheverybeginningoftheproject.Writethemalldownanddescribethefeaturesandpossibleusescenariostogetabetterfeelingfortheappanditspotentialusers.Roughdescriptionsoftherequirementsandfeaturesareabsolutelyfineatthispointastheywillbespecifiedinmoredetailduringthedevelopmentprocess.Awrittenrecordoftheserequirementswillmakeitmucheasierforyoutoderiveyourmobiletestingstrategy.Hereisalistofsomepossiblerequirementsandfeatures:•Providearegistrationform.•Providealoginformtoaccesstheapp’scontent.•Providealogoutoption.•Implementasearchfunctionwithintheapp.•Theusershouldbeabletocreateauserprofile.•Theusershouldbeabletosharecontentwithotherusers.•Theusershouldbeabletosharecontentonsocialnetworks.•Theusershouldbeabletotakepictures.•Theappshouldbeavailableindifferentlanguages(English,German,French,andsoon).
Furthermore,youshouldknowwho’sgoingtouseyourapp.AsdescribedinChapter3,“ChallengesinMobileTesting,”youreallyneedtoknowyourtargetgroupandtheirexpectations.Gatherasmuchinformationasyoucantogainimportantinsightsintoyourcustomers’usescenarios.Here’saquickrecapofpossibleinformationaboutyourtargetgroup(thefull
listcanbefoundinChapter3):•Gender
•Monthlyincome•Educationalbackground•Location•Otherappstheyuse•Smartphonehabits•Devicestheyuse
ImportantIfyoudon’tknowanythingaboutyourtargetgroup,checkmobile-platform-specificnumbersofoperatingsystemsandhardwarespecifications.Furthermore,analyzeandgatherinformationaboutappsthataresimilartoyours.Thisisagoodstartingpointtohelpyougatherpotentialuserinformation.
Basedontherequirementsandfeaturesofyourappandknowledgeofyourtargetgroup,youcanaskspecificquestionstocollectinformationforyourtestingworkandscope:
•Isitimportanttofindcriticalbugsquickly?•Shouldtheappbetestedonlyincommonuserscenarios?•Onwhichmobileplatformsshouldtestingbeperformed?•Whatareourcustomers’carriernetworks?•Arethereanyareaswithintheappthatarelikelytochangeonaregularbasis?•Istherelease(submission)dateoftheappalreadyknown?•Istherearoadmapforfuturereleases?
Don’thesitatetoaskthesekindsofquestionsasthey’reimportant,andtheanswerswillhelpyoutodefineyournexttestingstepsandpriorities.Don’tworryifyouforgettoaskaquestionbeforeyoustartwritingtheteststrategy—it’sbettertoaskaquestionwheneveritcomesupthannotatall.Inthenextstepyoushouldcollectinformationaboutthedevelopment
environmentwithinthecompany.Itisimportanttoknowwhichtoolsareusedtosetupadevelopmentandtestpipeline.Whichcontinuousintegrationserverisbeingused?Whichtoolsareusedtobuildtheapp?Andwhichbackendtechnologiesareusedtoprocesstherequestsfromthemobileapp?Youalsoneedtoknowaboutthearchitectureoftheproductionenvironment.
Togoaboutcollectingallofthisinformation,youshouldascertaintheanswerstothefollowingquestions:
•Whatkindofsoftwaredevelopmenttoolsarealreadyavailableandinusewithinthecompany?•Isthereacommonbuildpipelinethatmustbeused?•Whichcontinuousintegrationserverisusedfortheproject?•Whichtoolsareusedtobuildthemobileapp?•Whichtechnologiesareusedtoprocessthemobilerequests?•Whatskills,suchasprogramminglanguages,areavailablewithinthemobileteam?•Howmanymobiledevicesareavailablewithinthecompanyfortesting?•Whattoolsandtechnologiesareusedintheproductionenvironment?•Howmanypeopledoweexpecttousethemobileapp?
Theanswerstothesequestionswillhelpyouchoose,forexample,thetestautomationtoolbasedonthetechnicalknowledgewithinyourteam.Theywillhelpyoutodefinethetestlevelsandtechniques,andfurthermoretheywillprovideafirstoverviewofalltechnologiesinvolvedinthemobileproject.Knowledgeofthedevelopment,test,andproductionenvironmentisveryimportantwhenitcomestocoordinatingtestingwithintheproject.
ImportantCollectingrequirementsandfeaturesisimportantasyouneedsuchusefulinformationforyourmobileteststrategy.Thiskindofinformationisagoodstartingpointforplanningyourtestingactivitiesandwillhelpyoutodefineatestingscope.
TestingScopeOnceyou’vedefinedtherequirements,youcanspecifythescopeforyourteststrategy.It’snotpossibletotestanapponeverypossiblehardwareandsoftwarecombination.Youshouldthereforereducethescopeofyourtestingeffortsandinitiallyconcentrateontheimportantpartsofthemobileapp.Therearefourwaystoreduceyourtestingscope:•Single-devicescope•Multidevicescope
•Maximumdevicescope•Use-casescope
Single-DeviceScopeThesingle-devicescopefocusesononemobiledeviceduringtesting.Thisapproachcanorwillbeusedifonlyonedeviceistobesupportedbytheapporifthereisverylittletimeavailablefortheproject.Intheeventoftimepressure,you’llprobablychooseonlythemostpopulardeviceusedbyyourtargetedcustomers.ThisdevicewillbeusedfortestingwithjustonemobilecarriernetworkandpossiblyaWi-Ficonnection.Anotherapproachcouldbetochooseadevicefromthedevicegroupthathasolderhardwarespecificationsandcouldthereforecausemoreproblemsforthedevelopersintermsofsupport.You’relikelytofindmorebugsandproblems,suchasperformanceorotherissues,withthisdevicethanwiththelatestdevice.Usingonlythesingle-devicescopecanbedangerousforthemobileapp,for
thesuccessoftheproject,orevenforthewholecompany.Itisverylikelythatyou’llmissimportantbugsthatoccuronotherdevicesandthatyourcustomerswillsubmitbadratingstotheappstores.Iftheappsupportsonlyonedevice—forexample,ifitisaninternalenterpriseapp—thisapproachcanbeused.Ontheotherhand,it’sbettertotestononlyonedevicethantoskiptheentiretestingprocess.
MultideviceScopeAsthenamesuggests,themultidevicescopefocuseseitheronseveraldevicesfromonemobileplatformoronmultiplemobileplatforms.Selecttheplatformsandtestdevicesbasedonyourtargetgroup,andthengroupthedevicesasdescribedinChapter3inthesection“MobileDeviceGroups.”Ifyoudon’thaveanyinformationaboutyourtargetgroup,usetheInternettosearchforplatform-specificnumbersandstatistics,whichwillhelpyouselectthemobileplatformanddevicestoconcentrateon.AreallyniceWebpageprovidedbyGoogleis“OurMobilePlanet”1where
youcangetinformationbasedonthecountry,age,gender,userbehavior,andbehaviorduringthecurrentyear.
1.http://think.withgoogle.com/mobileplanet/en/
MaximumDeviceScopeThemaximumdevicescopefocusesonasmanymobileplatformsanddevicesaspossible.Thisapproachcanbeusedformass-marketappsintendedtoreachas
manycustomersaspossibleallovertheworldwithnorestrictionsintermsofplatform,device,carriernetwork,ortargetgroup.Testinganappforthemassmarketisverydifficultbecausetherewillalmostalwaysbeahardwareandsoftwarecombinationthatdoesn’tworkwellwithyourapp,andit’snearlyimpossibletofindthiscombination.Inordertohandleandreducethisrisk,youneedtofindawaytotestonasmanydevicesaspossible.Thisapproachrequireslotsofresearchtogatherinformationandstatistics
aboutcurrentdeviceusage,mobileplatforms,andoperatingsystemversions.Itrequiresinformationaboutthedifferentcarriernetworksandconnectionspeedsfromaroundtheworld,andsoforth.Onceyou’vecollectedtherequiredinformation,considerusingacombination
ofin-housetesting,cloudtesting,andcrowdtestingtohandlethemass-marketsituation.Bearinmindthattestingjustwithin-houseresourcesanddeviceswillbeeithertoolimitedinscopeortooexpensive.
Use-CaseScopeBesideschoosingahardwaretestingscopetodownsizeyourtestingwork,youcanalsochooseause-casescopetolimittheworkload.Withthisapproachyoucanconcentrateoncertainpartsorfeaturesofthemobileappandleaveoutlessimportantones,suchashelptextsoredgecases.Nearlyeveryprojectisunderextremetimepressure,thusmakingitimportanttodefinewhichpartsoftheappmustbeincludedinthetestingscopeandwhichcanbeleftout.Writedownbothpartsinyourteststrategyanddescribetheusecasesthathavetobetestedandwhy.Ifthereisenoughtime,thelessimportantpartsshouldbetestedaswell.Youmaywantorneedtoincludethefollowingtestscopeinformationinyour
teststrategy:•Whichapproachshouldbeusedintheproject?•Isitpossibletocombineapproachesforcertainpartsoftheapp?•Whywasthisapproachselected?•Whataretherequiredandmainusecasesoftheapp?
DefineTestLevelsandTestTechniquesOnceyou’vedefinedtherequirementsandscope,youneedtothinkaboutthedifferenttestlevelsandtesttechniquesyouwanttouseinyourproject.KeepinmindthequalityassurancemeasuresoverviewfromChapter4,“HowtoTestMobileApps”(thesection“TraditionalTesting”),whendefiningthem.Thisoverviewwillhelpyoutoderivethetestlevelsandtechniquesforyourmobile
app.
TestLevelsAsshowninChapter5,“MobileTestAutomationandTools,”thefocusoftestlevelsfromnon-mobilesoftwaredevelopmenttendstoshiftthroughoutthemobilesoftwaredevelopmentprocess.The“mobiletestpyramid”showsthattheunittestinglevelisthesmallestpartcomparedtoend-to-endtesting,betatesting,andmanualtesting.Inmostsoftwaredevelopmentprojects,thedevelopersareresponsiblefor
writingunittests.Thisisalsothecasewithmobileappprojects.Mobiletestersareresponsibleforwritingtheend-to-endtestautomationincludingintegrationtesting.However,everyoneontheteamshouldberesponsiblefortheapp’squality—theyshouldallsupportthemobiletesterinhisorherwork.Asmentionedinseveralchaptersinthisbook,manualtestingisavery
importantpartofamobiledevelopmentprojectandformsanessentialpartofthetestlevelsinamobileproject.However,thereareotherlevelsthatareimportantformanualtesting:acceptance,alpha,andbetatesting.Useraccep-tancetestscanbeperformedtotestthemobileappagainsttheuserrequirementstoverifythattheappcoversthemall.Thisstepisusuallycarriedoutbyatester,productmanager,orthecustomer.Inmobiledevelopmentprojects,alphaandbetatestingareimportanttest
levelsthatshouldformpartofyourteststrategy.Wheneverafeatureisimplementedwithinyourapp,youshouldtestitwithpotentialcustomerstogatherearlyfeedbackaboutit.Iftestingwithpotentialcustomersisnotpossiblewithinalphatests,youcouldtrytestingthefeaturewithyourworkcolleaguestogetfeedbackfromoutsideofthedevelopmentteam.Oncethemobileapphasreachedadefinedmaturity—forexample,allofthe
featureshavebeenimplementedoronlytwobugswerefoundinthelastweek—youshouldconsiderusingbetadistributiontoolsoracrowd-basedtestingapproachtoperformbetatestswithpotentialcustomers.Thecriteriathatdeterminethatatestingphaseisfinishedandthatthenextoneisreadyforexecutionalsoneedtobedocumentedintheteststrategy.Typicalsoftwaretestlevelsthatshouldbeusedinamobiledevelopmentproj-
ectarelistedhere:•Automatedtesting•Unittesting•End-to-endtesting(includingintegrationtesting)
•Manualtesting•Acceptancetesting•Alphatesting•Betatesting
•RegressiontestingOnceyou’vedefinedthetestlevels,youshouldalsoconsiderdefininghow
intensivelyeachlevelshouldbetestedfromafunctionalandnonfunctionalpointofview.Pleasekeepinmindthatnotalltesttypeswillbeincludedateverytestlevel.Furthermore,itcanbehelpfultodefinesomemetricsthatmeasurethecurrent
stateoftheapplication.Possiblemetricsmayincludethefollowing:•Eachfeaturemusthaveatleastoneunittest.•Eachfeaturemusthaveatleastoneend-to-endtest.•Thereshouldbenowarningsorerrorsinthestaticanalysischeck.
Functionaltestingshouldincludethefollowing:•Identifythefunctionalityofthemobileapp.•Testthedifferentpartsagainstthefunctionalrequirements.•Defineandexecutethetestcases.•Createinputdatabasedonthespecifications.•Comparetheactualandtheexpectedoutputs.
Nonfunctionaltestingshouldincludethefollowingpoints:•Loadtesting•Performancetesting•Usabilitytesting•Securitytesting•Portabilitytesting•Accessibilitytesting•Internationalization/localizationtesting
Thefollowingtest-levelinformationcouldbeusedinyourteststrategy:•Whichtestlevelswillbeusedintheprojectandwhy?•Whichpartsaretobeusedforfunctionalandnonfunctionaltesting?•Defineanddescribetheautomatedtestlevels.Whichpartsneedtobeunit
testedandwhichpartswillbetestedusinganend-to-endtestautomationtool?•Defineanddescribewhenthesoftwarehasreachedacertainmaturityandcanbeusedforalphaandbetatesting.•Defineanddescribemetricsrelevanttotheproject.
TestTechniquesYoucandrawuponqualityassurancemeasures(Chapter4)todefineyourtesttechniquesandmethods.Considerusingstaticanddynamicapproachestotestyourmobileappfromdifferentpointsofview.AsdescribedinChapter4,Irecommendusingstaticcodeanalysistoolsin
yourstatictestingapproachtotestyourmobileapp’scodeforanybugsorproblems.Keepinmindthattheappcodeisnotexecutedduringstatictesting.Furthermore,alloftheprojectdocumentationshouldbereviewedforcompleteness.Withthedynamictestingapproachyoushouldusewhiteandblackboxtesting
techniquestotestyourapp.Whiteboxtestingshouldbedonebythedevelopersandcoversthefollowing:
•Statementcoverage•Pathcoverage•Branchcoverage•Decisioncoverage•Controlflowtesting•Dataflowtesting
Blackboxtestingshouldbedonebythesoftwaretestersincluding:•Equivalenceclasses•Boundaryvalueanalysis•Decisiontables•Statetransitions•Cause-effectgraph
However,thedevelopersshouldalsotestforboundaryvaluesandstatetransitionsattheunittestinglevelinordertobesurethateachunitcorrectlyhandlesthosesituations.Yourteststrategyshouldalsoincludeawrittenrecordofwhichtesttechnique
istobehandledbywhom.Besidestheaforementionedtechniques,youshouldconsiderusingexploratory
andrisk-basedtestingtoorganizeanddownsizethetestingworkwithinyourmobileteam.
ImportantDefinetestlevelsforyourmobileappbasedonitsfeaturesandrequirements.Qualityassurancemeasureswillhelpyoudefineyourtestmethodsandtechniques.
Thefollowinginformationabouttesttechniquescouldformpartofyourteststrategy:
•Whichtesttechniquewillbeusedwithyourprojectandwhy?•Defineanddescribetheorderofthetesttechniques;forexample,staticcodeanalysisanddocumentreviewaretobefollowedbywhiteboxtesting,blackboxtesting,andthenexploratorytestsessionsinvolvingthewholeteam.•Whichteammembersaretoapplythedifferenttesttechniques?•Defineanddescribethemanualtestingprocess,suchasacceptancetesting,exploratorytesting,alphaandbetatesting.•Defineanddescribethetestexitcriteriaforwhiteandblackboxtesting,forexample,80%branchcoveragewithwhiteboxtesting.
TestDataNearlyeveryappprocesses,creates,andsendsdatafromthemobileappviadatanetworkstodifferentbackendsystems.Theprocessedandtransferreddatadiffersfrommobileapptomobileappandhasdifferingrequirementsandcomplexities.Onecomponentofyourmobileteststrategyshouldbetherequiredtestdata.Itisimportanttodefinethetestdataasrealisticallyaspossiblebasedonthefeaturesandrequirementsofyourapp.Thefollowingthreepointsareanexampleofdifferenttestdatatypes:•Configurationdata:Thisdataincludesconfigurationsforthemobileapporforthebackendsystem.Itcould,forexample,includedecisionrulesforrulesenginesand/orsettingsfordatabasesandfirewalls.•Stabledata:Thisusuallyinvolvesdatawithalowrateofchangeandalongduration.Atypicalexampleofthisiscustomerinformationsuchas
usernameandpassword,orproductinformation.•Temporarydata:Thiskindofdataislikelytochangefrequently,meaningthatitcanbeusedonlyonceorwillbecreatedwhiletheappisrunning,forexample,paymentdetailsorvouchers.
Oncethetestdatarequirementsareclear,findawaytosavethetestdatasoyoucanre-createandresetthedatainadefinedstatewheneveryouneedit.Onepossiblesolutionforthisisadatabasewherethestoredinformationcanbeusedduringthedevelopmentandtestingprocess.Anotheradvantageofthisisthatyoucanalsousethedatabaseformanualandautomatedtesting.Ontheotherhand,youcanusetestdatamanagementtoolstoorganizedatawithinyourproject.Onceyou’vesetuptheresetandre-creationprocess,youshouldstartcreating
thedataassoonaspossibleasthiswillhelpyouandyourcolleaguesduringtheappdevelopmentprocess.Dependingontheapp,youmayneedlotsoftestdata.Ifthisisthecase,it
mightbeagoodideatouseageneratortocreatethedataautomatically.Ifadatageneratorisused,itisimportantthatthefunctionalityandnecessaryparametersbedocumented.Iftestdataisavailablewithinyourproject,don’tforgettoadaptittonew
featuresandchanges.Itisverylikelythatovertimefeatureswillbeimprovedandthetestdatarequirementswillchange.Yourteststrategyshouldthereforealsooutlineaprocessforupdatingthetestdata,includingresponsibilitiesandtriggersfortheupdateprocess.Itisalsorecommendedthatyoudefineastrategyforhowoutdatedtestdatashouldbearchivedinordertoreproduceincomingbugsinoldfeaturesorversionsofyourapp.Thefollowingtestdatainformationcouldformpartofyourteststrategy:•Howisthetestdatagenerated?•Whereisthetestdatastored?•Howisthedocumentationhandled,forexample,testdatatogetherwithtestresults?•Howoftenwillthetestdatabeupdated?•Whoisresponsibleforthetestdata?
SelectTargetDevicesandTestEnvironmentNowthatyou’vedescribedthefeatures,requirements,testlevel,andtechniqueaswellasthetestdata,youneedtothinkaboutthetestenvironmentandtest
targetdevicesforyourstrategy.AsyoulearnedinChapter3,groupingyourtestdevicesormobileWebbrowsersisagoodapproachtodeterminewhichdevicesshouldbeusedforthemobileapp.Creatingsuchmobiledevicegroupsrequiresinformationaboutyourtargetcustomergroupandtheirusagescenarios.Don’tforgetthetestingscopesdescribedinthischapterwhendoingso.Oncethedevicegroupsareinplace,you’llneedtoselectdevicesfromthose
groupstohavethemavailablewithinyourteam.Itisrecommendedthatyouhaveatleastonedevicefromeachgroupavailablefortesting.However,Irecommendatleastfivedevicesfromeachgroupinordertohaveabroadermixofhardwareandsoftwarecombinationswithdifferentformfactorsanddisplays.Youshouldalsokeeparecordofwhyyouchosethosetestdevices.Nowthatyouknowwhichdevicesareneededfortesting,youneedtobuyor
rentthem.Buyingallofthedevicesyouneedcanbeexpensiveandperhapsisnotanoptionduetotheproject’sbudget.Agoodapproachtosavesomemoneyisonlineauctionswhereyoucanbuyuseddevices.Inmostcasesthedevicesareingoodenoughshapeforyourtestingwork.Ifbuyingisnotanoptionatall,youcanrentthedevices.Asmentionedin
Chapter3,thereareseveralmobiledevicelabprovidersonthemarketthatwilllendyouthedevicesyouneedforasetperiod.However,checktherentalpricesandcomparethemwiththedeviceprice.Ifyouwanttorentthedevicesforaprolongedperiod,rentingwillprobablybemoreexpensivethanbuyingthedeviceinthefirstplace.AnotheralternativeisOpenDeviceLabs2wherethedevicescanbeborrowed
forfree.ChecktheOpenDeviceLabsmaptofindoneinyourarea.Ifyouwantafreedevice,youcanalsoaskaroundatyourcompanyorevenseeifsomeoneinyourfamilyhasthedeviceyouneedthatyoucanborrowforawhile.
2.http://opendevicelab.com/
Onceyou’vespecifiedyourdevicestrategy,youneedtothinkaboutthetestenvironment:thebackendsystems.Youneedtoknowthearchitectureofthebackendsystemssuchasdatabases,paymentsystems,APIs,andanyotherkindofsysteminvolvedinthemobileproject.Ifyouhavesystemsinformation,youneedtobesurethatit’salsoavailablewithinthetestenvironmentsoyoucantestasthoughyou’reintheproductionenvironment.Ifyourteststrategycontainsin-the-wildoron-the-movetests,there’sanadditionalrequirement:thetestenvironmentmustbeaccessiblefromoutsidethecompanynetwork.Thefollowinginformationaboutthetargetdevicesandthetestenvironment
couldformpartofyourteststrategy:
•Whichdeviceswillbeusedfortesting?•Whyarethosedevicesusedfortesting?•Arethedevicesavailablewithinthecompanyordoyouneedtoacquirethem?•Whatareyourreasonsforchoosingthosetestdevices?•Whataretherequirementsforthetestdevicesandthetestenvironment?•Isthereanupdatepolicyforthemobiledevices?•Whenwillnewdevicesbeintegratedintothedevelopmentandtestingprocess?•Whataretheusagescenariosofthesystem?•Whatdoesthebackendsystemconsistof?•Isthetestenvironmentsimilartotheliveenvironment?•Canthetestenvironmentbeusedfortestingpurposesfromoutsidethecompanynetwork?
Manualandin-the-WildTestingAsyouhavelearnedsofar,mobiletestingrequireslotsofmanualtestingandtestinginreal-lifeenvironments.ThinkoftheexamplewiththeskiandsnowboardappfromChapter1,“What’sSpecialaboutMobileTesting?,”wheretestingonamountainisrequiredtoseeifthemobileappactuallyworksunderrealconditions.Manualtestinginthewildisessentialforyourmobileappandrequireslotsof
planningbeforehandtoavoiduselesstestscenarioswhiletestingonthemove.Youshouldthereforetrytoidentifycommonreal-worldusagescenariosforyourmobileappanditsfeatures.Writethescenariosdowninyourteststrategyandrankthembasedonpriorityandimportancetoyourproject.Furthermore,itisrecommendedthatyoudefinetestroutes,suchasbybus,
train,car,plane,orwhilewalking.Withintheseroutes,describepossiblescenariosthatshouldbetested.Theseroutesallowyoutosimulaterealmobileuserswhiletheycommutetoworkorwhilethey’retravelingaround.Lastbutnotleast,youshoulddefinedatanetworkscenariosbasedonyour
targetgroup.Aftergatheringinformationaboutyourtargetgroup,youwillknowinwhichregionstheyliveandwhatkindofdatanetworksandspeedsareavailable,forexample,4G,3G,orEDGE.Basedonthatinformation,youcanlimitapptestingtothosenetworkspeeds,butdon’tforgettotestondifferent
networkproviders.Hereareafewexampleusagescenarios:•Testyourappbasedonitsfeatures,suchasoutsideinsunnyplacesorinsideanoffice.•Testtoseeiftheappcanbeusedindifferentweatherconditions.•Usemultipleappssuchase-mail,chat,andnewswhileyourappisrunninginthebackground.Checktoseeiftheappisinfluencedbyotherapps.
Herearesomeexamplesofroutescenarios:•Usetheappwhilecommutingtoworkbytrain,bus,orcar.•Usetheappwhilerunningandcheckthebehaviorofthesensors.•UsetheappwhilewalkingthroughacityorthecountrysideandcheckthesensorsandtheGPSorcompass.
Herearesomeexamplesofdatanetworkscenarios:•Testhowtheappworksinfastdatanetworkslike4Gor3G.•Testhowtheapphandlesthedatanetworkchangefrom4Gto3Goreven2G.•Testhowtheapphandlespacketlossoracompletelossofnetwork.
In-the-wildtestingrequireslotsofmovementandisachallengingtaskthatneedstobehandledduringtheapp’sdevelopmentprocess.Ifyoudon’thavethetimeoroptiontotestyourappinreal-worldscenarios,thinkaboutusingcrowdtestingintandemwithyourin-housetestingwork.However,remembertheprosandconsofcrowdtestingfromChapter6,“AdditionalMobileTestingMethods,”astheycanhaveanimpactonyourprojectplanning,coordination,timing,andbudget.Don’twritecomplextestcasesandscenarioswithexactstepsforin-the-wild
testing.Whileyou’reonthemove,you’reunlikelytohavealaptopwithyoutocheckthetestcases.Thisisinefficientbecauseitwilldestroythereal-lifetestingscenarios,behavior,andusersimulation.You’llprobablyhaveabagwithyoucontaininglotsofdevicesforyourin-the-wildtesting.Whenplanningsuchscenarios,keepthemshortandinformativesothattheycanbeusedwhileonthemove.Irecommendthatyouprintoutthescenariosorwritethemdownonapieceofpapersoyoucanjustreadthemandkeeptheminmind.Thefollowinginformationaboutin-the-wildtestingcouldformpartofyour
teststrategy:
•Defineanddescribetheusagescenariosofyourcustomers.•Defineanddescribethetestscenariosinthewild.•Defineanddescribethedifferentdatanetworksthatneedtobeusedfortesting.
MobileChecklistsandToursYoucanaddmobilechecklistsandmobiletourstoyourmobileteststrategy.AsdescribedinChapter4,mobilechecklistscanbeveryimportantforyourmobileappastheyhelpyoukeeparecordofthingsthatcan’tbeautomatedorarelikelytochangeonafrequentbasis.Ifyouknowtherequirementsandfeaturesofyourapp,you’llprobablyalsoknowthepartsoftheappthatneedtobetestedrepeatedly.Inthatcaseit’sgoodtoaddthosefeaturestoachecklistaspartofyourteststrategy.Basedonthefeaturesofyourapp,it’susefultodefinemobiletestingtoursto
concentrateyourtestingeffortsonspecialpartsofyourapp.CemKaner3describesatouras“...anexplorationofaproductthatisorganizedaroundatheme.”
3.http://kaner.com/?p=96
Usingtoursinyourmobiletestingworkhelpsyoutoexploreandunderstandhowthemobileappworks.Italsohelpsyoucomeupwithnewtestideaswhileyou’retestingtheapp.PleasereferbacktoChapter4foradescriptionofsometoursbackedupwithmnemonics.Herearesomeexamplesoftestingtours:•Featuretour:Exploreandtestallofthepossiblefeatureswithintheapp.•Configurationtour:Exploreandtesteverypartoftheappthatcanbeconfigured.•Gesturetour:Useeverypossiblegestureoneveryscreentoseehowtheapphandlesthedifferentinputs.•Orientationtour:Changetheorientationofeveryscreenfromportraittolandscapeandviceversatoseeifanyproblemsarise.
Iusethefollowingmobilemnemonicsinmyprojects:•FCCCUTSVIDS(http://michaeldkelly.com/blog/2005/9/20/touring-heuristic.html)fromMichaelKelly•ISLICEDUPFUN(www.kohl.ca/articles/ISLICEDUPFUN.pdf)fromJonathanKohl
Thefollowinginformationaboutmobilechecklistsandtourscouldformpartofyourteststrategy:
•Whichchecklistswillbeusedintheprojectandwhy?•Describetheusedchecklistsandtours.•Describewhenthechecklistsandtoursshouldbeusedandbywhom.
TestAutomationTestautomationcanalsoformpartofamobileteststrategy.Dependingonthemobileappanditslifecycle,youmaynotneedtoautomateit.Ifthat’sthecase,it’simportantthatyoudocumentanddescribethereasonswhytestautomationisnotnecessary.However,ifyourapprequirestestautomation,youshouldstarttothinkabout
automationandthetoolsyouwanttouseassoonaspossible.ThinkbacktothedifferentmobiletestautomationconceptsinChapter5andtheirprosandcons.Choosethetoolthatbestfitsyourcurrentprojectsituationandwhereyouhavealreadygainedsomeexperiencewithoneofthetoolsorprogrammingskills,orwhereyouhaveamobiletestingsetuporenvironmentinplace.Thiswillsaveyoulotsoftimeandmoney.Describethemobiletestautomationtoolinyourteststrategyandwhyyou
choseitforthisproject.Basedonthedescribedfeaturesandrequirements,youcandefinethepartsoftheappthatshouldbeautomatedandwhichdonotneedtobe.Duringthenextstepyoushoulddescribethedevicesonwhichtheautomated
testhastorunandinwhatenvironmentthetestsshouldbeexecuted.Yourmobileappprojectshouldfindtherightbalancebetweenrealdevices
andvirtualdevicesifyou’renotabletotesteverythingonrealdevices.Amixtureofrealandvirtualdevicescanalsobemorecost-effective.Onceyou’vedefinedthedevicesandtestautomationenvironment,I
recommendthatyoualsodefinetestsuitestogroupthedifferentautomatedtestcasesbasedoncommonfeatures,areas,andrequirementswithinyourapp.Withthehelpofthetestsuitesyoucandecidewhichtestsshouldbeexecutedwhenandhowoften.Forexample,youcandefineasmoketestsuitecontainingseveralautomatedtestsfromeachpartoftheapptomakesureacommittothecentralcoderepositorydidn’tbreakanything.Thistestsuiteshouldbesmallandruneverytimethecodechangesinordertoreceivefastfeedbackandtoswiftlyinformthedevelopersofanyproblems.
Herearesomeexamplesoftestsuites:•Smoketestsuitecontainingafewtestscriptstocheckthebasicfunctionsoftheappandtoprovidefastfeedback•Mediumtestsuiteoratestsuitecontainingonlyspecificfunctionality•Fullregressiontestsuitecontainingalltestscriptstobeexecutedonceadayorduringthenight•Testsuitescontainingonlyuserscenariossuchastheregistrationorcheckoutprocess
Testsuitescanbeagreatwaytocreateabalancebetweenfastfeedbackandbroadtestcoverage.Whenthetestsuitesareinplace,definewhentheyshouldbeexecuted;for
example,thesmoketestsuitemustberunaftereverycommitbythedeveloper.Themediumtestsuitecouldbeexecutedeverytwohours.Thefullregressiontestsuiteshouldbeexecutedonceeverynight.Youshouldalsodefinewherethetestautomationshouldbeexecuted.One
suchsolutioncanbewheretheunittestsareexecutedonthedeveloper’slocalenvironmentbeforechangesarepushedtothecentralrepository.Theunittestsandend-to-endtestscanbeexecutedontheCIserver.Thefollowinginformationabouttestautomationcouldformpartofyourtest
strategy:•IsaCIserverusedforthebuildprocess?•WhichCIserverisusedandwhy?•Whichtestenvironmentisusedtoexecutethetestautomation?•Whichtestautomationtoolsareusedintheproject?•Arethetestsexecutedonrealdevicesandvirtualdevices?•WhichdevicesmustbeconnectedtotheCIserverandwhy?•Isacloudproviderusedforthetestautomation?•Defineanddescribethetestsuitesandgroups.•Defineanddescribethedifferentbuildtriggersandexecutiontimes.•Defineanddescribewherethedifferenttestautomationsuiteswillbeexecuted,forexample,everythingonalocaldevelopermachineoronlyontheCIserver.
ProductRisks
Everyprojectisexposedtodifferentkindsofrisk.Itisimportanttoworkoutbothprojectandfeatureriskssoyoucanthendefinepossiblesolutionsforthem.Considerboththelikelihoodofriskoccurrenceandriskimpact.Iftheproductrisksareclear,youcanstartimplementingariskanalysisapproachthatcanbeusedbythewholeteamwhendefining,implementing,andtestingnewfeatures.Thefollowinginformationabouttheproductriskscouldformpartofyourtest
strategy:•Whichpartsarecriticaltothebusiness?•Whatisthelikelihoodoftheproductrisksoccurring?•Howshouldthebusiness-criticalpartsbetested?•Whatisthepotentialimpactifacriticalproblemoccurs?•Howisthefeatureriskanalysisperformed?•Isthereadisasterplaninplace?
ImportantCreatingamobileteststrategyisnoteasyasitneedstocoverlotsofmobiletestinginformation.Yourstrategymayalsoneedtobemodifiedduringthedevelopmentprocessbecauseofchangedproductfeaturesorpriorities.
MobileLaunchStrategyAsimportantasthemobileteststrategyis,it’salsoreallyimportanttowritedownamobilelaunchstrategy.Launchinganappisnoteasy,andlotsofproblemscanoccurduringandafterithasbeenlaunched.Thispartofthechapterwilldescribetheimportantpre-andpost-launchactivitiesforamobileapp.
Pre-Launch—ChecktheReleaseMaterialAsmentionedinChapter4,youshouldthinkaboutputtingtogetherareleasechecklisttomakesurethatallthereleaseinformationisavailableandinplace.Youshouldalsoperformtheupdateandinstallationtestbeforesubmittingtheapptotheappstore.Thetestsyouperformbeforecommittingtheapptotheappstoreshouldn’tbe
limitedtotheappitself.Yourlaunchstrategyshouldoutlineifandhowthebackendservicesaretestedbeforeanewrelease.
Askyourselfthequestion“DoesthenewversionoftheapprequiresomenewbackendservicesorAPIcalls?”Ifso,arethoseservicesorAPIcallsalreadyliveinthebackendsystem?Ifnot,it’sverylikelythatyourappwillberejectedbytheappstorevendor.Ifbackendservicesareavailable,checkthenewandexistingfeaturesoftheappintheproductionenvironmentonefinaltime.Whentheappisreadyforlaunch,checkthereleasenotesandfeature
descriptioninthenewappstoreinformationmaterial.Readthroughthetextsandcomparethemwiththenewandexistingfeatures.Inthereleasenotesit’simportanttobespecificaboutthenewfeatures;describethemwellandexplainthemtotheusers.Itisalsoimportanttoprovidethereleasenotesandappdescriptionineverysupportedlanguage.Don’tforgettocheckthescreenshotsoftheapp.Theyshouldbeinthesame
languageasthenotes,bethesamesizeasthepreviousscreenshots,andhavethesamestatusbariconsshowingthetime,battery,andnetworkstate.Figure7.1andFigure7.2provideexamplesofwhatnottodo:thestatusbarcontainsdifferenticonsforthesameapp.
Figure7.1Appstorescreenshot
Figure7.2Appstorescreenshotofthesameappwithdifferentstatusbarinformationandsizes
Theappstorescreenshotsneedtohavethesamestatusbarandthesamelook-and-feelinordertoportrayaprofessionalimage.Ifthenewfeaturesaredescribedbymeansofavideo,watchthevideoagain
andlistentothefeaturedescriptionandinformationitprovides.Askifthereismoremarketingmaterialavailablethatneedstobechecked.Lastbutnotleast,thereisonereallyimportantpoint:don’treleaseyourappto
yourcustomersonaFridayeveningorjustbeforeyouleavetheoffice.Inmostoftheappstoresittakessometime(uptoseveralhours)beforeyourappislistedandavailablefordownload.Iftheofficeisempty,noonecanreacttoanycriticalbugsorreleaseproblemsthatmayoccurrightaftertherelease.Someappstoresdon’thaveanappsubmissionprocessandcanthereforereact
quicklytoproblemsbyreleasingahot-fixversionofyourapprightafterthelast
release.Whenpossible,releaseyourappeitherinthemorningorontheweekendsothatyouhavesometimetoreacttocriticalissues.Thefollowinginformationaboutthereleasematerialcouldformpartofyour
launchstrategy:•Defineanduseareleasechecklist.•Performtheupdateandinstallationtestagain.•ChecktomakesureallofthebackendservicesandAPIsareavailableontheproductionenvironment.•Definewhichnewandoldfeaturesshouldbecheckedagainontheliveenvironment.•Defineanddescribehowtheappstorematerial,includingreleasenotes,screenshots,andvideos,shouldbechecked.•Defineanddescribewhentheappshouldbesubmittedtotheappstore.
Post-Release—WhatHappensafteranAppLaunch?Thepost-releasephasestartsonceyou’vereleasedyourmobileapp.Duringthisphaseyouandyourteamshouldpayattentiontoseveralpointstogetfeedbackfromyourcustomersandtohandleanyquestionsorproblemstheysendyou.Thefirstthingyoushoulddorightafterreleasingyourappistodownloadit
andinstallitfromtheappstoretomakesureit’sworkingasexpected.IftheversionisOK,youshouldarchiveitonafileservertobeabletoinstallitagainlater,forexample,toreproduceproblemsorbugsthatmaybereported.Thisstepcanalsobehandledbyyourbuildpipelinewithinthecontinuousintegrationsystem.However,thereareafewmoreissuesthatyou,yourteam,andyourcompany
needtohandle.Thefollowingsectionsdescribetheactivitiesthatcanbeperformedafterreleasingyourappinordertogetfurtherinformationaboutyourcustomersandanypotentialproblems.Itisimportanttodefineandwritedownsuchitemsinyourapplaunchstrategy.
CommunitySupportProvidingcommunityorcustomersupportforareleasedproductisessential,especiallyifyou’reofferingapaidapp.Wheneverusershaveaproblem,theyshouldbeabletocontactsomeoneatyourcompanyorthemobileteamtogetanswerstotheirquestionsorproblems.Ifnoonelooksaftertheusersorcommunities,theywillwritebadreviewsorsimplystopusingyourapp.
Youshouldthereforemakesureyoumaneverypossiblesocialmediachannelandcheckforcustomerfeedback,questions,orproblemsthatneedtobelookedinto.Itisimportanttorespondtosuchqueriestogivecustomersthefeelingthatsomeoneislisteningtothem.Itisalsopossibletoselectsomeusersandaskthemquestionsaboutthenewversionoftheapp.Thegatheredfeedbackcanthenbeusedtoimprovetheappinfuturereleases.
ImportantIfyourcompanyhasacustomersupportdepartment,Irecommendthatyouspendabitoftimewiththemtogetafeelingforcustomerneedsandproblems.
ReviewsThenextchannelyouhavetomonitorafterreleasingyourappistheappstorereviews.Readthemcarefullytogetsomefeedbackandbugreportsfromyourusers.However,Ihighlyrecommendthatyouhandlethosereviewswithcare.Therearelotsofpeopleouttherewholiketowritenegativefeedbackaboutappsthatisnottrue.Wheneverausercomplainsaboutyourapp,trytoreproducetheprobleminordertobeabletocreateabugreportaimedatfixingtheissueinasubsequentrelease.Ifyou’renotabletoreproducetheissue,trytoreplytothereviewertoaskspecificquestionsinordertogetmoreinformation.However,notallappstoresprovideareplyfunctionwithintheirreview
sectionsothatyoucantalktoyourcustomers.Iftheappstoredoesn’tprovidesuchafeature,youcanwriteyourownreview,statingthatyouarethedeveloper,tester,orproductmanageroftheappandwouldlikefurtherinformationorareofferingsomesortofsolutiontotheproblem.However,besureyou’reallowedtowritereviewswithintheappstoretoreplytopreviousreviews.Pleaserefertotheappstorereviewguidelinestomakesureyou’renotviolatinganyrules.Ifyou’reabletoreplytoreviews,usethisfunctionalitytointeractwithyourcustomersandtolearnfromthem.Ifyougetlotsofnegativefeedbackduetoamisunderstandingwithintheapp
oritsfeatures,youcanalsoprovidesomesortoftroubleshootingguideortutorialwithintheappdescription.However,ifyougetlotsofnegativefeedbackaboutyourappbecausethecustomersdidn’tunderstandthefeature,youneedtorethinkthewholeapporfeature,performadditionalusabilitytesting,andprovideanupdateassoonaspossible.
CrashReportsAnothervaluablesourceofinformationisthecrashreportsofyourapp.IfyouimplementedtoolslikeHockeyApp,4crashlytics,5orTestFlight,6youshouldcheckthemafterthereleasetoseeifthereareanyproblemswithyourapp.Noteveryproblemthatoccursduringruntimewillendupbeinganappcrash.Ifyourapphasgoodexceptionhandlinginplace,errorswillnotbeshowntotheuserbutwillbecapturedbythecrashreportingtools.Thiskindofinformationisveryimportanttohelpyouimprovetheappinfurtherreleases.
4.http://hockeyapp.net/features/5.http://try.crashlytics.com/6.www.testflightapp.com
ThetoolsprovideaWebinterfacewherethetopappcrashesareranked,grouped,andcategorized.Theyshowthetotalnumberofcrashesandthenumberofusersaffectedbytheseproblems.Nearlyeverycrashreportingtoolprovidesnicegraphsshowing,forexample,appdistributionandappcrashesovertime.Someofthecrashreportingtoolsprovideanoptiontosendfeedbackfromwithintheapptothecrashreportingbackend.Furthermore,someofthetoolsofferthird-partyintegrationforabugtrackingsystem.Ifyouhaven’timplementedacrashreportingtool,someappstore
manufacturersprovidebasiccrashreportingfunctionalitythatcanbeusedasastartingpoint.
ImportantImplementacrashreportingtoolasitwillhelpyouandyourteamtogetmoreinsightintotheproblemsandcrasheswithinyourapp.
TrackingandStatisticsTogatherinformationaboutyourcustomersandtheirusageofyourapp,youshouldimplementsomesortoftrackingmechanismtocollectimportantdata.Thiskindofinformationwillbeaggregatedbytrackingtoolstogeneratestatisticsaboutyourappandfeatureusage.Ifyourappusesatrackingmechanism,checkthestatisticsgeneratedpost-release.Dependingonthetrackingimplementation,youcangetinformationsuchas:•Mobileoperatingsystemversion•Mobiledevicemanufacturer•Devicemodel
•Displaysize•Mobilebrowserversion•Numberofpageviews•Howoftenacertainfeaturewasused•Howoftentheregistrationprocesswasaborted
Withthehelpofthosestatisticsandnumbers,trytounderstandyourusers’behaviorsoyoucantweaktheappanditsfeatures.Thefollowinglistisasamplingofsomemobiletrackingtools:
•adjust(www.adjust.com/)•appsfire(http://appsfire.com/)•AppsFlyer(www.appsflyer.com/)•Clicktale(www.clicktale.com/)•iMobiTrax(www.imobitrax.com/)•MobileAppTracking(www.mobileapptracking.com/)
Asyoucansee,it’snoteasytocreatemobiletestandlaunchstrategiesasbothrequireandcontainlotsofinformationabouttestingtopicsandpre-andpost-launchactivities.Pleasekeepinmindthatsuchstrategiesarenotsetinstone.Youshouldreworkandadaptbothstrategieswheneverchangesoccurintermsoftheproduct,risks,oranyotherpriorities.Mobiletestandlaunchstrategydocumentsareanongoingprocess,andeveryteammembershouldberesponsibleforupdatingandextendingthem.
SummaryThemaintopicofChapter7wasthemobiletestandlaunchstrategies.Everymobileteamneedstestandlaunchstrategiesinordertorememberimportanttasksbeforeandafterthereleaseoftheapp.Whenestablishingamobileteststrategy,thefollowingtopicsshouldbe
coveredanddefined:•Requirementsdefinition•Testingscope•Testlevels•Testtechniques•Testdata•Targetdevicesandenvironment
•TestautomationWiththequestionsprovidedinthispartofthechapteryouarenowabletoset
upyourownmobileteststrategyforyourapporforyourcompany.Theotherpartofthischaptercoveredthemobilelaunchstrategy.The
importanceofthereleasematerial,includingfeaturedescriptions,screenshots,andanyothermarketingmaterial,wasdescribed.Withthehelpofthequestionsprovided,youshouldfinditveryeasytocheckwhethereverythingisavailablefortherelease.Afterthereleaseoftheappitisimportanttohavecommunitysupportin
place,whereuserscanaskquestionswhentheyhaveanykindofproblemwithyourapp.Furthermore,theimportanceofcrashreports,tracking,anduserstatisticswasestablished.
Chapter8.ImportantSkillsforMobileTesters
Softwaretestersandmobiletestersinparticulararefacingmoreandmorerequirementsasmobileappsarebecomingincreasinglycomplexandthetimetomarketisgettingshorter.Mobiletestersneedtobeabletoefficientlytestcomplexmobileapplicationswithinaveryshortperiodoftimetodeliverexcellentproductstocustomers.Besidestestingknowledge,mobiletestersmusthaveseveralotherimportantskillstomanagethecomplexityofthesystemsandthehugenumberofdifferentscenarios.Thischapterisallaboutsoftwaretestingskillsandhowtoimproveyour
mobiletestingskillstobecomeabettermobiletester.
SkillSetofaMobileTesterBesidesknowledgeandskillsregardingsoftwaretestingmethods,approaches,mobileapps,anddevices,testersmusthaveasolidbasicsetofsoftskillsiftheywanttobesuccessfulinthemobiledevelopmentbusiness.Inthefollowingsectionsofthischapteryou’llreadaboutvariousskillseverysoftwaretestershouldhave.
CommunicationTheabilitytocommunicateisoneofthemostimportantskillsasoftwaretestermusthave.Softwaretestersmustbeabletodescribetheirtestingworktodifferentkindsofpeopleatdifferentlevelswithinthecompany.Theyneedtobeabletotalktodevelopers,designers,productmanagers,projectmanagers,othersoftwaretesters,andmanagementaswellastocustomers.Talkingtoothersoftwaretestersanddevelopersrequirestechnicalskillsanddetailedknowledgeofthedifferentpartsofthesoftware.Reportingbugsandshowingotherstheirmistakescanquicklyleadtonegativeemotions,whichiswhyit’simportanttoreportbugsinaclearandconcisewaywithoutbringingemotionsintotheconversation.Talkingtoproductmanagers,projectmanagers,designers,ormanagement
requirestheabilitytodescribeproblemsandbugstonontechniciansatahigherlevelinawaythatisclearandunderstandable.Inadditiontoverbalcommunication,softwaretestersneedstrongwritten
communicationskillsastheyhavetobeabletodescribeproblemsandbugsinsuchawaythateverypotentialstakeholdercanunderstandthem.
Listeningisalsoanessentialpartofcommunication,andsoftwaretestersmustbeabletolistencarefullywhenotherpeoplearetalkinganddescribingtheirthoughtsorproblems.Itisimportantthatyoudon’tinterruptotherpeoplewhenthey’retalking.Ifyouhaveanyquestions,makeanoteofthemandaskthemoncethepersonhasfinished.Tohelpyouimproveyourcommunicationskills,Irecommendthatyouread,
read,andread—notjustbooksbutblogs,newspapers,andanyotherkindofmaterialthatwillhelpyouimproveandexpandyourvocabulary,especiallyifyou’renotcommunicatinginyournativelanguage.IalsorecommendthatyouwatchmoviesorTVseriesinotherlanguagesto
bolsteryourvocabulary.Ifyouhavemoretimeavailableandwanttospendsomemoney,youcanattendlanguageandcommunicationclasses,whichareagreatwaytoimproveyourlanguageandcommunicationskills.Lastbutnotleast,Irecommendthatyougivetalksatusergroups,
conferences,orwithinyourcompany.Suchexperiencewillhaveamajorimpactonyourcommunicationskillsbecauseyougetfeedbackfromyouraudiencerightaway.Poorcommunicationgenerallyleadstodisagreementandmisunderstandings,
whichcanbeavoidedbyfollowingsomesimplerules:•Listencarefully.•Don’tinterruptotherpeoplewhilethey’respeaking.•Don’tspeaktooloudly.•Don’tspeaktooquickly.•Speakclearlyandprecisely.•Makeeyecontactwithyouraudience.•Don’tgetpersonalwhencommunicatingwithotherpeople.•Beabletocommunicateondifferentlevels,rangingfromtechnicaltonontechnicalaudiences.•Improveyourvocabularybyreadingbooks,blogs,andnewspapers.
ImportantSoftwaretestersneedtobediplomats,technicians,andpoliticiansallrolledintooneastheyhavetobeabletotalkandlistentodifferentstakeholderswithinthecompany.
CuriosityIt’shumannaturetobecurious,andsoftwaretestersneedtobecurioustoexplore,discover,andlearnnewthingsaboutthesoftwarethey’retestingandtheproductdomain.Acurioussoftwaretesterexploresthesoftwaretogetasmuchinformationaspossibleoutofthesystemtoidentifypotentialproblemsandraiseinterestingquestionsabouttheproduct.It’simportanttogobeyondtheusualsoftwaretestingapproachesandmethodstodiscovernewthings.Tobeabletodiscovernewthings,it’simportanttobeopentonew
technologiesandwillingtotrynewapproachesandmethods.Acurioussoftwaretesterdoesn’trelyonstatementsfromotherpeople;heorshequestionsthemtogainmoreinformation.Ifyou’dliketotrainyourcuriosity,Irecommendthatyoudownloadarandom
mobileapporsoftwareapplicationandstartexploringitsfeatures.Trynewapproachesandmethodswhileyou’reexploringthesoftware.Trytobreakthesystemandstartquestioningthefeatures.Makenoteofeverythingthatfeelswrongorevokessomesortofstrangereactionduringyourexplorationsoyoucanraisequestionsorpointoutanypossibleproblemswiththesoftware.
ImportantBecurious;exploreanddiscovereverypartofthesoftwaretoraiseproblemsorquestions.Don’trelyonstatementsfromotherpeople;questionthem.
CriticalThinkingAnotherreallyimportantskilleverysoftwaretestermusthaveiscriticalthinking.Withthehelpofcriticalthinkinggoodsoftwaretestersareabletoseethelargercontextofthesoftwareanditsfeatures.Theyarealsoabletobreakdownthesoftwareortherequirementsthroughanalysisandreflection.Thisisveryimportanttogainadeepunderstandingoftheproductandtofocusontherighttestingwork.ThefollowingquotefromMichaelBoltondescribescriticalthinkinginanice
way:“Criticalthinkingisthinkingaboutthinkingwiththeaimofnotgettingfooled.”1
1.www.developsense.com/presentations/2010-04-QuestioningBestPracticeMyths.pdf
It’simportanttoquestionyourownthinking,testingmethods,andapproachesaswellasyourowndecisionsandthesoftwarethatneedstobetested.Ask
yourselfthefollowingquestions:•Whatistheproblemofthisfeature/software?•Isitreallyaproblem?•Whyhaveyoutestedthisfeaturethatway?•Haveyouthoughtaboutthis?•Areyousureaboutthis?
Averygoodthree-wordcriticalthinkingheuristicfromJamesBach2isHuh?Really?So?Eachwordsuggestsaninvestigationpatternthatindicatesassumptions,sloppyinferences,andmisunderstandings:
2.www.satisfice.com/
•Huh?•Doyouunderstandwhatothersaretalkingabout?•Isitconfusing?•Isitvague?
•Really?•Isitfactuallytrue?•Whatevidencedowehaveforit?
•So?•Whydoesthismatter?•Towhomdoesitmatter?•Howmuchdoesitmatter?
Usethiscriticalthinkingheuristicinyourprojectandstartquestioningyourownworkandthemobileapp.Forfurtherinformationaboutcriticalthinking,takealookattheslidesfromMichaelBolton’scourse,CriticalThinkingforTesters.3
3.www.developsense.com/presentations/2012-11-EuroSTAR-CriticalThinkingForTesters.pdf
TenacityReportingbugsorraisingissuescanbeexhaustinganddifficult.Noteveryissuefoundbyasoftwaretesterwillbefixed.Theissuemaynotbeimportantenoughforotherteammembers,orperhapsthereisnotenoughtimetofixit.Itispartofthesoftwaretester’sjobtobetenaciousandfighttogetbugsresolved.Ifthesoftwaretesterthinksabugmaybecriticalforthecustomersorthesystem,heorsheneedstoinitiateadiscussiontodescribeandexplainwhyitneedstobefixed
inthenextrelease.Thekeywordshereare“BugAdvocacy.”TheAssociationforSoftwareTestingprovidesatrainingcourseonthisimportanttopic.4IfyouwanttogetafirstimpressionofBugAdvocacy,takealookattheslidesfromCemKaner.5
4.www.associationforsoftwaretesting.org/training/courses/bug-advocacy/5.www.kaner.com/pdfs/BugAdvocacy.pdf
Highstresslevelsarecommonbeforeareleaseandoftencausedevelopersorprojectmanagerstoneglecttheagreed-uponsoftwarequalitystandards.Insuchsituationssoftwaretestersmustbetenaciousandexplainorraisebugsoverandoveragainuntilthesoftwarequalitystandardshavebeenmet.Butbecarefulwiththisasyoumayendupbeingconsideredanuisance.Hereit’simportanttorelyonyourstrongcommunicationskills.Softwaretestershavetobetenaciouswhiletestingsoftwaresuchasmobile
apps.Dependingonthekindofappbeingtested,suchasagame,itisverylikelythatcertaingamelevelshavetobetestedoverandoveragaininordertobesurethateachlevelworksasexpected.Thiscanalsobeveryexhaustingbutrequirestenacityorsometestautomation.
ImportantBetenaciousduringtestingandduringpossiblediscussionsaboutbugsanderrorswithintheapplication.
ConstantLearnerThecurrentmobileworldandtechnologyarechangingrapidly.Tomaintainpacewiththisenvironment,softwaretestersand,inparticular,mobiletestersmustbeabletoadaptandlearnnewthingsreallyquickly.Softwaretestersneedtotakenoteofchangestakingplacearoundtheminordertoadaptandlearnnewapproaches,methods,andtechnologies.Tokeeppaceandlearnnewtechniquesandtools,softwaretesterscanread
blogsorbooksandattendconferencesandtrainingcourses.Ontheotherhand,it’simportantthatsoftwaretestersbeabletolearnduringtheirdailyjob,whiletestingsoftware,andwhileusingtoolssuchastestautomationtools.Wheneveranewtool,technique,ortechnologyentersthemarket,everysoftwaretestershouldbemotivatedtogatherinformationaboutthesenewitemsandtolearnaboutthem.
Important
Learningandimprovingpersonalskillsshouldbealifelonghabit.
CreativityAnotherimportantskillasoftwaretestershouldhaveiscreativity.Itisimportanttobeabletogeneratecreativeideastotestsoftwareinverydifferentwayssoastofindmorebugsandprovidetheteamwithusefulinformation.Thecreativityprocessstartswithdesigningthetestcasesandtestdata.Softwaretestersneedtothinkindifferentwaystofindeveryconceivableuseforapieceofsoftware.Whenthedefaulttestingapproachiscompleteandthere’ssomeprojecttime
leftfortestingactivities,Irecommendthatyoutestthesoftwareagainfromacompletelydifferentpointofview;forexample,walkthroughthebugsagaintogeneratenewtestingideas,ortalktocolleaguesorbetatesterstogetnewideasforyourtestingwork.Trytobecreativewithyourdatainputs,whenusingthenavigation,oranythingelsethatcomestomind.You’llbesurprisedabouttheresultsfromthatkindoftestingandwillnodoubtcomeacrosssomemorebugs.
ImportantMobiletestersinparticularhavetobecreativeinordertousemobiledevicesindifferentwaysbypayingattentiontoalltheinterfaces,sensors,andlocations.
CustomerFocusEverysoftwaretestershouldhaveastrongcustomerfocus.It’simportantthatsoftwaretesterstrytothinklikeacustomerinordertodeterminewhetherthesoftwarebeingtestedmeetscustomerneeds.Testersthereforeneedtohavelotsofpassionanddeterminationandbeabletoidentifystronglywithcustomers.Astrongcustomerfocusrequiresyoutobeaproductandfieldexpertwithin
yourteam.Youalsoneedtohaveanoverviewofreleasedfeaturesandfunctionalityandbeabletokeepaneyeoutforfuturereleases.Itisveryimportanttobeawareofcustomerbehaviorinordertoknowwhichfeaturesandfunctionalitytheyuse.Ifpossible,softwaretestersshouldtalktocustomerstodeterminetheirneedsandproblems.Thiscanbeachallengingjob,sosoftwaretestershavetobepatient.Whensoftwaretestershaveastrongcustomerfocus,theycancontributetheir
knowledgetoeveryphaseofthesoftwaredevelopmentprocess,whichinturnhelpstobuildbetterproducts.Tohelpimprovecustomerfocus,Irecommend
spendingacoupleofweeksworkingwithyourcustomersupportdepartmenttogetabetterfeelingforcustomerneeds.
ProgrammingandTechnicalSkillsThefactthatsoftwareproductsandmobileappsarebecomingincreasinglycomplexleadstothechallengethatmobiletestersalsoneedtohavesolidprogrammingskillsastheyhelpsoftwaretesterstounderstandthesystemundertest,tocommunicatewithdevelopersatcodelevel,toreviewcodefromdevelopersorothersoftwaretesters,andtowritetestautomationcodewhichisnowbecomingessentialineveryproject.Mobiletesterswithnoprogrammingskillsneedtotrainthemselvesbyreading
abookaboutprogramminglanguagesorpatterns,byfollowingaprogrammingtutorialontheInternet,orbyattendingaprogrammingcourse.It’salsopossibletoaskadeveloperifheorshecantrainthemobiletesterwithinaprojectorcompany.Thankstoprogrammingskills,mobiletestersareabletowritetestautomation
codefromunittoend-to-endlevel.Theyareabletoattendcodereviewstoasktechnicalquestionsandarelikelyabletowriteshellscriptsinordertoautomateeitherabuildpipelineoranyothertaskthattheteamneedstoperform.Besidescodingskills,everymobiletestermustbeabletounderstandtechnical
systemarchitecturesinordertobeabletoaskcriticalquestionsaboutthearchitectureandtoknowhowtotesteverypartofit.
ImportantEverymobiletesterneedsprogrammingskillsinordertobeabletowritetestautomationcodeandtobeabletoattendcodereviewsandtechnicaldiscussions.
HowtoImproveYourMobileTestingSkillsAsmentionedinseveralpartsofthisbook,themobileworldischangingrapidly,soyou’llneedtohoneyourskillseverydaytokeeppacewiththemobiletestingworld.Youhavetoconstantlylearnnewthingstogeneratenewtestingideas,tocollaboratewithdevelopersduringtheirworkwithprogrammingskills,andyou’llalsoneedtounderstandcustomerneeds.Inordertoimproveyourmobiletestingskillsitisimportanttohaveatleast
onemobiledeviceavailable.Inmostcasesthisisyourprivateandpersonal
device.Ifpossible,Irecommendthatyouhaveseveraldevicesavailableathomewithdifferentmobileplatformssoyoucanlearneverythingaboutthoseplatforms.Youdon’tneedtokeepbuyingthelatestdevices;youcansettleforusedphonesorevenolderversionstolearnabouteachplatform.Ifyou’renotinapositiontobuylotsofdevices,keepinmindtheOpenDeviceLabs,whereyou’reabletoborrowdifferentdevicesforfree.
LearnfromOtherAppsAveryeasywaytoimproveyourmobiletestingskillsistolearnfromotherapps.Irecommendinstallingandusingasmanyappsaspossiblefromdifferentcategorieswithindifferentappstorestoseehowtheyworkandbehave.Checkhowotherappshaveimplementedtheirnavigationandupdatemechanismandhowtheyusemobile-specificfeaturessuchasthecameraorothersensors.However,besidesusingthem,it’simportanttochecktheupdatetextsofthose
apps.Irecommendthatyouunchecktheautomaticupdatefunctionalityofallofyourappssoyoucaninstallnewversionsmanually.Beforepressingtheupdatebuttoninthedifferentmobileappstores,readtheupdatetextsandappdescriptionscarefully.Therearelotsofcompaniesordeveloperswhoarereallyprecisewhendescribingwhatthenewversionoftheappisallabout.Theydescribewhichbugsarefixedwiththenewversionandwhichnewfeatureshavebeenadded.Ifthereisabugdescriptioninanapp’supdatetext,trytoreproducethebugso
youcanseeitwithyourowneyes.Thiscanbealotoffun,butyoumayfindittakesawhiletoprovokethebug.Butthisisinitselfagreatwaytolearnlotsofnewthings.Youwillprobablygetnewtestingideas,comeacrossnewwaystouseanapp
andnewapproachestoprovokingabug,andlearnthingsthatyoumayneverhavethoughtaboutbefore.ThefollowingsectionswillprovidesomeexamplesinvolvingdifferentkindsofbugsanddescriptionsfrommobileappsIcheckfromtimetotime.
ImportantTheappscreenshotsinthe“CrashesonSpecificDevices”sectionareanonymized.AlloftheexamplesarebasedontheGooglePlaystore.However,thesamesortofbugandfeaturedescriptioncanbefoundineveryothermobileappstore.
CrashesonSpecificDevicesTheappinFigure8.1crashedwhenusingAndroidversion4.3andonsomex86devices.Ifyouwanttoreproducethiscrash,you’llneedtogetadevicewithAndroid4.3.Ifsuchadeviceisavailable,findthesectionorappviewthat’scrashing.ThismayprovetoberatherdifficultastherearelotsofdeviceswithAndroid4.3availableandeverydevicecanbehavedifferently.Ifyoumanagetofindthecrash,trytounderstandwhyithappened.Maybeit’sduetoapoorInternetconnectionorjustabadimplementation.
Figure8.1AppcrashesonAndroid4.3andx86devices
Nevertheless,asshowninFigure8.2,someappshaveproblemsonlywithacertainversionontheAndroidplatform,sotestingonseveralmobileoperatingsystemsisveryimportant.
Figure8.2Appcrashesonsomedevices
KeyboardsAsImentionedinChapter3,“ChallengesinMobileTesting,”usersareabletoreplacesystemappssuchasthekeyboardappwithathird-partysolution.ThiscanleadtovariousproblemsasshowninFigure8.3,whichcontainsabugreport.Toreproducethisissue,youneedtoinstallthethird-partykeyboardandstarthuntingthebug.
Figure8.3Appproblemsduetoalternativekeyboards
WidgetsSomemobileplatformssupporttheuseofwidgets.Whenprovidingawidget,besurethatitwon’tfreeze,crash,orconsumetoomuchbatterypowerasshowninFigure8.4.
Figure8.4Widgetconsumestoomuchbatterypowerandfreezes.
PerformanceAsIhavementionedinseveralpartsofthisbook,theloadingtimeandperformanceofanappareessentialtoasuccessfulapp.TheappdescriptioninFigure8.5providessomeinformationaboutpossibleperformanceissuesonthestatisticspage.Toreproducetheperformanceissueofthisappyouneedtogettwoverysimilardevices.Onthefirstdeviceyoushouldkeeptheoldversionoftheappwhileinstallingtheupdateontheotherdevice.Thenyoucancomparetheperformanceofthedescribedsectionaswellastheloadingtimestoseeifthereisanimprovement(seeFigure8.6).
Figure8.5Performanceissuesinsomesectionsoftheapp
Figure8.6Loadingperformanceoftheapp
LoginandPaymentIfyourappprovidesaloginfeatureormobilepaymentprocess,itiscriticalthatthosefeatureswork.Ifyouruserscan’tloginorbuyanything,youwilllosemoneyandharmyourreputation.Featurescriticaltoyourappmustworktothemaximumpossibleextent,soyouneedtobesurethatthey’rewelltestedandcoveredbytestautomation.TakealookatthescreenshotinFigure8.7,whereyoucanseethatthemobileappproviderhadanissuewithitssubscriptionmodel.
Figure8.7Checkoutproblemswithpremiumsubscription
PermissionsAsImentionedinChapter4,“HowtoTestMobileApps,”it’sveryimportantthatyouuseonlythemobileapppermissionsyoureallyneedforyourfeaturestowork.Ifyouusepermissionsthatusersdon’tunderstandorthataren’trequired(seeFigure8.8),youwillprobablygetlotsofbadreviewsinthestores,anditmayevennegativelyimpactyourapp’ssecurity.Besuretocheckthepermissionsagainbeforereleasingyourapp.
Figure8.8Usingpermissionsthataren’trequired
MobileDeviceHardwareUsageIfyourappusesmobile-device-specifichardwarefeaturessuchasthecamera,besurethatthefeatureworksonasmanydevicesaspossible.AsyoucanseeinFigure8.9,theappproviderhadanissuewiththecamerapreviewonsomedevices.Testingforhardwaresupportondifferentdevicesisagoodtaskforcrowdtesters.
Figure8.9Camerapreviewnotworkingproperly
Sincecomingupwiththeideaofcheckingappupdatetexts,IhavecontinueddoingthiseverytimebeforeIupdatetoanewerversionofanapp.Thisissometimestime-consumingandfrustratingwhenI’munabletoreproducethedescribedbug,butithelpedmealotinimprovingmymobiletestingskills.JustbyreadingthroughtheappdescriptionsIlearnedsomanynewwaysofgeneratingmobiletestingideasaswellaslotsofnewwaystoprovokebugsduringmydailytestingwork.
ObserveAnotherwaytoimproveyourmobileskillsisbyobservingotherpeople.Watchotherpeoplewhilethey’reusingtheirmobiledevices.Trytoobserveotherpeoplewhenamongthepublic,forexample,onatrain,inthesupermarket,oranywherethey’reusinganapp.Itisveryinterestingtoseehowotherpeopleuseappsintotallydifferentways,andyoucanlearnallsortsofthingsfromyourobservationsthatyoucanreusewhiletestingyourappandplanningnewfeatureswithyourteam.Itwillalsohelpyoutogeneratenewtestingideasandbearotherusagebehaviorsinmind.
ImportantDon’tbetooobviousanddon’tstartstalkingpeoplewhileobservingthem.
DuringmyobservationsIhavenoticedthatmanypeopledon’tknowthata
navigationdrawercanbeopenedbyswipingfromtheleftofthescreentotheright.Theytouchthenavigationiconinthetopleft-handcornertoopenit.Irealizedthatnoteveryoneisfamiliarwitheveryfeaturethatamobiledeviceorapphastooffer.Youshouldthereforetakethetimetoobserveotherpeopleinthewildorintestlabstoseehowtheyusetheirappsandthenapplythisknowledgeinyourmobiledevelopmentprocess.AsastartingpointIrecommendthatyouobserveeitheryourcolleaguesor
yourfamily.Thiswayyou’lllearnalot,andtheyprobablywon’tmindyouobservingthem.
TakePartinCompetitionsandTestCyclesIfyou’reinterestedincompetingwithothersoftwaretestersfromaroundtheworldandlearningfromtheirtestingexperience,Irecommendthatyoutakepartinatestingcompetition.Therearelotsofcompetitionsthroughouttheyearwheresoftwaretesterscanregistereitherasateamoralonetotestapieceofsoftwareindifferentcategories.Thegreatthingaboutcompetitionsisthatyoucanshareyourknowledgeandlearnfromothersoftwaretesters.It’salsofuntocompetewithothersoftwaretesterstoseehowgoodyourtestingskillsarecomparedtotheirs.Igenerallyattendtestingcompetitionsinordertolearn.Idon’tcareaboutmy
finalcompetitionranking;Ijustwanttolearnandimprovemytestingskills.Ireallyliketoseeothertestersduringtheirworkandpickupnewtestingideas.Hereareacoupleoftestingcompetitions:•SoftwareTestingWorldCup(www.softwaretestingworldcup.com/)•Testathon(http://testathon.co/)
AnothernicewaytoimproveyourtestingskillsandcontributetothetestingcommunityisWeekendTesting.6WeekendTestingisaplatformwheresoftwaretesterscancollaborateandlearnfromthetestingcommunity.Asthenamesuggests,WeekendTestingtakesplaceontheweekendandisallabouttestingsoftwarefromdifferentperspectivesandsharingyourtestingworkwithothersoftwaretesters.HavealookattheWeekendTestingWebsitetocheckouttheupcomingtestingdates.
6.http://weekendtesting.com/
AsmentionedinChapter6,“AdditionalMobileTestingMethods,”crowdtestingisasoftwaretestingapproachwhereyouaddsoftwaretestersfromallovertheworldtoyourin-housetestingefforts.However,crowdtestingcanserveanotherpurposeaswell:Registeringasacrowdtesterandattendingtesting
cyclesisagreatwaytolearnasyougettoseeothermobileapps,theproblemstheappproviderfaces,andthebugsthatcomeup.Furthermore,itisveryinterestingtoseeifyourbugreportsaregoodenoughtobeacceptedbythecrowdtestingproviderandmobileappprovider.
ImportantTakepartintestingcompetitions,shareyourknowledgewithothersoftwaretesters,andregisterwithcrowdtestingplatformstoseehowothermobileappswork.Remember:Whileitmaybeinterestingtoseeifyourbugreportsareacceptedbytheprovider,yourfocusshouldbeonimprovingyourmobiletestingskills.
TheMobileCommunityandtheMobileWorldAsmentionedintheprevioussection,learningfromothersoftwaretestersandmobiletestersisagreatwaytoimproveyourownmobiletestingskills.Ithereforerecommendthatyoubecomeanactivepartofthemobilecommunity,forexample,byregisteringonsoftwaretestingplatformslikeSoftwareTestingClub.7
7.www.softwaretestingclub.com/
Everysoftwaretestingplatformalsohasamobilesectionwhereyoucanexchangenewsandviewswithothermobiletesters.It’salsoveryusefultojoinmobiletestinggroupsonvarioussocialmediaplatforms.Therearealwayslotsofgreatmobiledevelopersandtesterswhowanttointeractonacertaintopicorwhohaveaproblem.Helpingsomeonewithaproblemisagreatwaytocontributetothemobilecommunity.Anddon’tbeshyaboutaskingquestionsevenifyouthinktheymightbesilly.There’snosuchthingasastupidquestion!Ifyoudon’thaveaTwitteraccountyet,Ihighlyrecommendthatyoucreate
one.EverysoftwaretestingandmobiletestingexpertusesTwittertowriteaboutnewtestingideas,blogposts,andotherimportantinformationthatisboundtohelpyouduringyourdailyworkinglife.Mostmobiletestingexpertsalsoblog,soyoushouldsubscribetotheirupdatestogetthelatestinformationfromthem.InthenextsectionIwilllistafewblogsandbooksthatmaybeofinteresttoyou.Testingeventsandconferencesareanotherwayofexchangingwiththe
mobilecommunity.Lotsofsoftwaretestingconferencesarehostedallovertheworldwheresoftwaretestingandmobiletestingexpertsmeetandsharetheirknowledgeintheformoftalksorworkshops.Ifyouhavetheopportunitytodo
so,Ihighlyrecommendthatyouattendsomeconferencesandmeetothermobiletestersinpersontotalkaboutarangeoftopics.Youcanalsochecktoseeifthereareanytestingusergroupsinyourarea;thesearegenerallyfreetoattendandagreatopportunitytomeetothersoftwaretestersnearby.Ifyouworkinthemobiletestingbusiness,youcouldstartbloggingtokeepa
recordofyourexperienceandcontributetothemobilecommunity.Youcanhelpothermobiletesterswithyouracquiredknowledgeorblogaboutyourexperienceswhilebecomingamobiletester.Besideslearningfromothermobiletesters,it’sveryimportantthatyoustay
up-to-datewithregardtothelatesttechnologiesandfeaturesofthemobileoperatingsystemsanddevices.Youneedtoknowwhenthedifferentmobiledevicemanufacturersreleasenewdevicesandnewversionsoftheiroperatingsystems.Foracompleteoverviewofsuchfeatures,Irecommendthatyouwatchthekeynotevideospostedbythemajormanufacturers.Ialsorecommendthatyouuseasmanyappsaspossiblefromarangeof
differentcategoriessoyoualwayshaveanoverviewofpossiblenewfeaturesaswellasnewwaystoimplementanduseanapp.
ValuableSourcesThissectionofthechapterprovidesyouwithsomeinterestingsoftwaretestingcommunities,books,magazines,andblogsthatyoucanusetoimproveyourknowledge.
ImportantTheselistsarebynomeanscomplete.
ConferencesThefollowingconferencesareworthgoingto:
•AgileTestingDays(www.agiletestingdays.com/)•BelgiumTestingDays(http://btdconf.com/)•DutchTestingDay(www.testdag.nl/)•EuroSTAR(www.eurostarconferences.com/)•GoogleTestAutomationConference(https://developers.google.com/google-test-automation-conference/)•Iqnite(www.iqnite-conferences.com/index.aspx)
•Let’sTest(http://lets-test.com/)•MobileAppEurope(http://mobileappeurope.com/)•Øredev(http://oredev.org/)•STAREAST(http://stareast.techwell.com/)•STARWEST(http://starwest.techwell.com/)•TestBash(www.ministryoftesting.com/training-events/testbash/)•TestExpo(http://testexpo.co.uk/)
CommunitiesThefollowingsoftwaretestingcommunitiesareworthlookinginto:
•AssociationforSoftwareTesting(www.associationforsoftwaretesting.org/)•MobileQAZone(www.mobileqazone.com/)•SoftwareTestingClub(www.softwaretestingclub.com)•TestingCircus(www.testingcircus.com/)•uTestcommunity(www.utest.com)
BooksThefollowingbooksareworthreading.Notallofthemareaboutmobiletesting,buttheyareanexcellentsourceofknowledgeforsoftwaretesters.
•AgileTesting(http://lisacrispin.com/agile-testing-book-is-now-out/)byLisaCrispinandJanetGregory•BeautifulTesting(www.amazon.com/gp/product/0596159811?tag=sw-testing-books-20)editedbyTimRileyandAdamGoucher•ExploreIt!(http://pragprog.com/book/ehxta/explore-it)byElisabethHendrickson•HowGoogleTestsSoftware(http://books.google.de/books?id=VrAx1ATf-RoC)byJamesA.Whittaker,JasonArbon,andJeffCarollo•LessonsLearnedinSoftwareTesting(www.amazon.com/gp/product/0471081124?tag=sw-testing-books-20)byCemKaner,JamesBach,andBretPettichord•SpecificationbyExample(http://specificationbyexample.com/)byGojkoAdzic•TapintoMobileApplicationTesting(https://leanpub.com/testmobileapps)byJonathanKohl
MagazinesThefollowingmagazinesprovidegreatcontentfromexpertsinvariousindustries.Thesemagazinesfocusonspecificsoftwareandmobiletestingtopics.
•AgileRecord(www.agilerecord.com/)•ProfessionalTester(www.professionaltester.com/magazine/)•Tea-timewithTesters(www.teatimewithtesters.com/)•TestingCircus(www.testingcircus.com/)
BlogsThefollowinglistcontainsgreatblogsfromsoftwaretestingexperts:
•GojkoAdzic,http://gojko.net/•JamesBach,www.satisfice.com/blog/•MichaelBolton,www.developsense.com/blog/•LisaCrispin,http://lisacrispin.com/•MartinFowler,http://martinfowler.com/•MarkusGärtner,http://blog.shino.de/•ShmuelGershon,http://testing.gershon.info/•AndyGlover,http://cartoontester.blogspot.co.uk/•AdamGoucher,http://adam.goucher.ca/•ElisabethHendrickson,http://testobsessed.com/•JimHolmes,http://frazzleddad.blogspot.com/•LenaHouser,http://trancecyberiantester.blogspot.com/•EricJacobson,www.testthisblog.com/•StephenJanaway,http://stephenjanaway.co.uk/stephenjanaway/blog/•ViktorJohansson,http://therollingtester.com/•JonathanKohl,www.kohl.ca/blog/•RobLambert,http://thesocialtester.co.uk/•AlanPage,http://angryweasel.com/blog/•HuibSchoots,www.huibschoots.nl/wordpress/•RosieSherry,www.rosiesherry.com/
Thefollowingblogshavemultiplecontributors:•http://blog.inthewildtesting.com/
•http://blog.utest.com/(uTestemployees)•http://googletesting.blogspot.de/(Googleemployees)•www.ministryoftesting.com/testing-feeds/(agreattestingfeedcollection)•http://mobileapptesting.com/•http://webapptesting.com/
Andthenthere’smyblog:•www.adventuresinqa.com
SummaryChapter8wasallaboutskillsandtherequiredskillsetofamobiletester.Hiringmobiletestersisnoteasy,becausetheyareveryrare.Ifyouhavefoundsomeonewhocouldfityourcompanyandposition,besureheorshehasthefollowingsoftskillsinordertobesuccessfulintheroleofmobiletester:
•Communication•Curiosity•Criticalthinking•Tenacity•Constantlearner•Creativity•Customerfocus•Programmingandtechnicalskills
Ifyouareamobiletesterandwanttoimproveyourtestingskills,thischapterprovidedsomesuggestions.Withlotsofexamples,thesection“LearnfromOtherApps”showedyouhowtoimproveyourtestingskillsbytryingtoreproducebugsinexistingapps.Inaddition,itisimportanttobeactiveinthemobiletestingcommunity,tolearnfromothermobiletestersandtoshareyourknowledge.Attheendofthechapteranoverviewofimportantconferences,books,blogs,andmagazineswasprovided.
Chapter9.What’sNext?AndFinalThoughts
Welcometothefinalchapterofthisbook!Thischapterdealswiththequestion“What’snext?”What’sthenextbigthingmobiletestersaregoingtohavetodealwith?What
kindsofnewtechnologyarealreadyoutthere,areontheirway,orcouldarriveinthefuture?Arethereanynewtestingchallengesthatweneedtohandle?Arethereanynewtestingtoolsandtestenvironmentsonthehorizon?Toanswerthosequestions,thefollowingsectionsofthischapterdescribe
somenewtechnologiesthatarealreadyonthemarketorduetoarriveinthenearfuture.Tokeeppaceinthefast-growingworldoftechnology,it’simportantforyoutoknowwhat’sinthepipeline.Thefollowingsectionscanbeusedforfurtherinvestigationsandresearchif
needed.Nooneisabletopredictthefuture,butI’mconvincedthatthefollowingtechnologieswillbecomemoreandmoreimportantformobiletestersinthenextcoupleofyears.
InternetofThingsTheInternetofThings(IoT)referstotheinterconnectionofuniquelyidentifiableembeddedcomputingdeviceswithintheexistingInternetinfrastructuretoofferdifferentkindsofservices.ThingsintheIoTcanincludeawidevarietyofdevicessuchashumanmedicalimplants,biochipsforanimals,andcarswithbuilt-insensorsthatcommunicatewithoneanothertoexchangeinformationaboutthecurrenttrafficsituationorprovidedriverswithcertaininformationabouttheircars.TherearealsodevicessuchaswashingorcoffeemachinesthatcanconnecttotheInternetsoyoucanmonitorthemremotely.EverythingthatcanbeassignedanIPaddressandhastheabilitytoprovideandtransferdataoveranetworkisathingintheIoT.AccordingtoastudyconductedbyGartner,1therewillbeupto26billion
devicesontheInternetofThingsby2020.Andthatfiguredoesn’tincludecomputers,tablets,orsmartphones,whichwillreachthe7.3billionmarkby2020.Ifyoucomparethosenumbers,IoTdeviceswilldwarfalltheexistingsmartdevices.ThiswillofcoursegiverisetoawholenewindustrywithpeopletryingtoconnecteverythingtotheInternet.
1.www.gartner.com/newsroom/id/2636073
HerearesomepossibleusageareasandscenariosforIoT:
•Environmentalmonitoring:Sensorscanbeusedtomonitorwaterquality,soilconditions,ortheatmosphere.•Infrastructuremanagement:Bridges,railwaytracks,andwindfarmscanbemonitored.•Energymanagement:Industrialmanufacturerscanoptimizeenergylevelsinrealtime.•Medicalandhealthcaresystems:People’shealthcanbemonitoredremotely.•Buildingandhomeautomation:Alarmandheatingsystemscanbemonitoredandmanaged.•Transportsystems:Carscancommunicatewithoneanother,suchastoavoidtraffic.
InordertostandardizetheIoT,aconsortium2ofcompanieshasbeenformedtopushalongtheIoTanddevelopdefaultcommunicationstrategies,interfaces,andprotocols.ThefollowingtwosectionsprovidesomeexamplesofcurrentIoTdevicesandscenariosfromdifferentmanufacturers.
2.www.openinterconnect.org/
ConnectedHomeMobiledevicemanufacturersGoogleandApplearecurrentlybuildingtheirfirstIoTservicesandproductswhichwillformpartoftheIoTfamilyalongsideseveralothercompanies.In2014,GoogleboughtNestLabs,3whichbuildsintelligentthermostatsandsmokealarmsforsmarthomes.ThethermostatsandsmokealarmsareconnectedtoaWi-Finetworkandcanbeaccessedfromanywhereintheworldusingacomputer,tablet,orsmartphone.
3.https://nest.com/
Googleiscurrentlydevelopingmobileappsfordifferentmobileplatformsthatgrantaccesstoconnecteddevicesaroundthehome.Theproductisintelligentasitcanlearnfromuserhabitstocontroltheheatingsystembasedonthetimeofdayandthecurrentmonthanddeterminewhetherornottheuserisathome.Userscanalsodefinevariousscenariostocontroltheirentirehomeheatingsystembasedontheirneeds.BecausethedevicesareconnectedtotheInternet,it’seasytoauto-updatethemwithnewsoftwareversionsincludingbugfixesandfeatures.ApplehasintroducedHomeKit4withiOS8,whichprovidesaframeworkto
communicateandcontrolconnecteddevicesintheuser’shome.Userswillbe
abletoautomateandcontrolIoTdeviceswiththeirvoicebyusingSiri.HomeKitwillbeabletocontroldevicessuchasthermostats,lighting,doors,andotherelementsthatcanconnecttotheInternet.AppleprovidesthedevelopmentframeworkforHomeKitandiscurrentlylookingforindustrypartnerswhowanttoimplementApple’sHomeKitaccessoryprotocoltobecontrolledbytheHomeKitapp.
4.https://developer.apple.com/homekit/
Asyoucansee,connectedhomesarealreadyavailableonthemarketandenableentirelynewwaysofinteractingwithdevicesandpartsofourdailylife.Thereareplentyofpotentialnewtestscenarios,testenvironments,andtestingchallengesthataretotallydifferentfromtraditionaltestingormobiletesting.
ImportantBesidesGoogleandApple,therearealotofothercompaniesthatareinvestinginandalreadyhavesolutionsinplaceforconnectedhomes.IoptedtowriteaboutGoogleandApplebecausetheyalsoprovideAPIsfordevelopersthatallowthemtobuildmobileapplicationsaroundtheconnectedhometechnology.
ConnectedCarConnectedcarsarethenextIoTexample.Again,GoogleandApplearealreadyontheirwaytointegratingtheAndroidandiOSmobileoperatingsystemsintocarstomakethemevenmoreintelligent.GooglehasintroducedAndroidAuto,5andApplereleasedCarPlay.6BothGoogleandApplewillprovidealightweightversionoftheirmobileoperatingsystemstoprovideuserswithfeaturestheycanusewhiledriving,suchasnavigation,music,contacts,phone,andmessages.
5.www.android.com/auto/6.www.apple.com/ios/carplay/
Besidestheirownmobileapps,bothGoogleandAppleoffertheoptiontouseinstalledthird-partyappsincars.Therearealreadylotsofcarmanufacturersthatsupportbothsystemsandletbuyerschoosethesystemtheyprefer.However,usingmobileappswithcardisplaysgivesrisetosomenew
challengesforanyoneinvolvedinthesoftwaredevelopmentprocess.Forexample,theprovidedappsandfeaturesshouldn’tdistractthedriverandneedtoofferaverysimpleuserinterfacewithlessinformationcomparedtomobileappsorWebapplications.
Thefollowingfourpointsmustbeconsideredwhendevelopingandtestingmobileappsorotherapplicationsforconnectedcars:
•Simpleinterface:Carapplicationsandinterfacesshouldnotdistractthedriver.TheUIelementsmustbeeasytousewhilethedriverisatthewheel.Traditionalinputmethodsneedtobereconsideredandshouldalsoincludevoicecontrol.•Avoiduselessfeatures:Thefeaturesetofanapprequiredwhiledrivingisprobablyalotsmallerthanforanappusedonasmartphone.Theappshouldthereforeofferfewerfeaturesonthecardisplaytopreventthedriverfrombeingoverwhelmedorfrustratedwhileatthewheel.•Third-partyappsneedguidelines:CarmanufacturersneedtoprovideanAPIforthird-partydeveloperssotheirservicescanbeintegrated.However,thisposesthechallengeofcreatingverystrictguidelinesaboutwhatisandisn’tpossible.•Testinginthecar:Developingappsforcarsischallengingenoughinitself,buttestingappsforcarsisevenmorecomplex.Itissimplynotenoughtotesttheappinalabsituationbecausecarsaregenerallyonthemove,havedifferentmanufactureryearsandmodels,andhavelotsofinterfaceswithothersystems.Electronicinterferenceinacarcanhaveahugeimpactonyourappandthewholesystem.Theprovidedappmustbesafeforthedrivertousewhileatthewheel.Lastbutnotleast,theappmustbewelltestedtoavoidanycriticalbugs.
AgoodexampleofaconnectedcarandIoTisthecarmanufacturerTesla,7whichbuildscarsthatarecompletelyconnectedtotheInternetandcanbepartiallycontrolledwiththeaidofamobileapp.Thecarreceivesautomaticupdatesthatimprovefeatures,fixbugs,andevensolveproblemswithdifferentpartsoftheengine.WiredmagazinepublishedaninterestingarticleaboutTeslaasanexampleofIoT.8
7.www.teslamotors.com/8.www.wired.com/2014/02/teslas-air-fix-best-example-yet-internet-things/
Asyoucansee,thetwoexamples—connectedhomeandcar—representnewchallengesfortheentiresoftwaredevelopmentprocess.Fromatestingpointofviewinparticular,thesenewtechnologiesrequiredifferenttestingmethodsaswellasnewtestingenvironments,newtestingdevices,andcompletelynewtestingscenarios.
Wearables
Wearabletechnologyisarapidlygrowingfieldthat’sexpectedtogrowexponentiallyoverthenextfewyears.Therearelotsofnewandinnovativeformfactorsfordevicesthatcanbewornondifferentpartsofthebody.Thesenewformfactorsposenewchallengesforcompaniesastheylooktofindsmartwaystomaketheirproductfunctional,usable,andlovablefortheircustomers.Thesameappliestodevelopersandsoftwaretesterswhoneedtorethinktheirworkandthewaytheydevelopandtestsuchproducts.Wearablesgenerallyinvolvesmartwatches,smartglasses,andfitnesswristbands.
SmartWatchesandFitnessWristbandsSmartwatchesandfitnesswristbandsareextensionstomobiledevicesthatsendandreceiveinformationsuchasmessages,news,incomingcalls,andhealthstatustoandfrommobiledevices.Togetthelatestinformationfromanapp,usersnolongerneedtotaketheirdeviceoutoftheirpocket.Mostdevicescanbecontrolledbytheuser’svoiceorasmalltouchscreen.However,asmartwatchorfitnesswristbandisessentiallyuselesswithoutamobiledevicetointeractwith.Theusabilityanddesignofsmartwatchesandfitnesswristbandsneedtobe
thoroughlytestedandchecked.Designingsoftwareforreallysmallscreensisnoteasy,whichiswhydesignersandUXexpertsneedtorethinktheirconceptsinordertobuildniceproductsthatwearablesuserswillloveanduse.JonathanKohlwroteanexcellentarticleabouthislessonslearnedwhendesigningproductsforsmartwatchesandwearables.9
9.www.kohl.ca/2014/lessons-learned-when-designing-products-for-smartwatches-wearables/
Ifyouhavetheopportunitytotestwearables,especiallysmartwatchesorfitnesswristbands,youshouldkeepaneyeonthelook-and-feelofthedevicetogetherwiththesoftwareundertest.Thisalsoincludestestingthedesignandusability.Whendoingso,youshouldaskyourselfthefollowingquestionstogeneratevaluablefeedbackabouttheproduct:
•Isthedevicenicetowear?•Doestheappmakesenseonthewearabledevice?•Arethefeatureseasytouseandhelpful?•Docertainpartsofthedevicegetinthewaywhileyou’reonthemoveandwhileyou’reusingthesoftware?•Howcantheuserinteractwiththesmartwatchorfitnesswristband?
Thelook,feel,design,andusabilityarethemainsuccessfactorswhenitcomestowearabletechnology.Ifawearabledevicedoesn’tfeelgood,userswill
notbuyorwearit.Fromatechnologicalpointofview,testingsmartwatcheshassomeadditional
challengescomparedtomobileappsanddevices.Thefactthatsmartwatchesareextensionstomobiledevicesrequirestestingofthewearabledevicetogetherwiththesoftwaretoseehowbothcommunicatewithmobiledevicesinordertoreceiveandtransferdata.Thisscenarioisn’tsomethingyoucanautomate.I’msureyou’realreadyawareofmobiledevicefragmentation,butthisiscompoundedbythefactthatsmartwatchesandfitnesswristbandsneedtoworkcorrectlywithinadifferentsetofuniquedailyuserscenarios,andallthosescenariosrequireextensivein-the-wildtesting.Suchtestinginareal-lifeenvironmentwillplayanessentialpartinthesuccessofsoftwareforsmartwatches.GoogleintroducedAndroidWear10in2014tokickoffthewearabledevice
era.AppleintroducedtheAppleWatch11inSeptember2014andstartedsellingitinearly2015.IfyousearchtheInternetforsmartwatchesandfitnesswristbands,you’llcomeacrossvariousdevicemanufacturersandallthedifferentdevicesonthemarket.10.www.android.com/wear/11.www.apple.com/watch/
Forinformationaboutbuildingsoftwareproductsforwearabledevices,checkoutthePebbledeveloper,12theGooglewearable,13andtheAppleWatch14featurepages.12.https://developer.getpebble.com/13.https://developer.android.com/training/building-wearables.html14.www.apple.com/watch/features/
SmartGlassesGoogleGlassisanotherwearablethatwasintroducedbyGoogle.GoogleGlass15includesalmostthesamehardwareasamobiledeviceextendedwithanopticalhead-mounteddisplay(OHMD)tohavethecontentandinformationdirectlyinfrontofyoureyes.Theglassesareequippedwithlotsofsensorsandacameratointeractwithyoursurroundings.Youcancontrolthiswearabledevicewithyourvoiceorbyusingthetouchpadonthesideoftheframe.15.www.google.com/glass/start/
TheexplorereditionofGoogleGlasshasbeenonsaleinsomecountriessince2014,butlotsofcountriesandcompanieshaveexpressedprivacyconcernssinceitsintroductionin2012becausethedeviceisabletorecordpeopleinpublic
withouttheirpermission.Furthermore,thereareconcernsabouttheproductintermsofcorporatesecretsandsafetyconsiderationswhileusingitindifferentscenarios,suchaswhiledrivingacarorridingamotorbike.However,Googleprovidedanewwayofusingmobiletechnologiesandset
newstandardsandinnovationsintheworldofwearabledevices,eventhoughtheproductisn’treadyforthemassmarketyet.GoogleGlassisagreatexampleofthedirectiontechnologywilltakeoverthecomingyears.Ifyouhavetheopportunitytodevelopandtestsoftwareforsmartglasses,
don’tforgettorunthroughthelistofquestionsapplicabletosmartwatches.You’llalsoneedtorethinkyourtestingapproachforthisdevice.
HealthAppsAnotherinterestingandgrowingmarketismobilehealthapps.“ThenumberofmHealthappspublishedonthetwoleadingplatforms,iOSandAndroid,havemorethandoubledinonly2.5yearstoreachmorethan100,000apps(Q12014)”and“...willreach$26billioninrevenuesby2017,...”asquotedinthecurrentmHealthAppDeveloperEconomics16report.ThishugeincreaseshowsthatmHealthappswillbeontheriseinthenearfuture.16.http://mhealtheconomics.com/mhealth-developer-economics-report/
ThetopfourmHealthappsare:1.Fitnessapps(30%)2.Medicalreferenceapps(16%)3.Well-beingapps(15%)4.Nutritionapps(8%)
TheremainderofthemHealthappsaredistributedamongdifferentcategoriessuchasmedicalconditionmanagement,diagnostics,compliance,reminders,alerts,andmonitoring.Healthappsinvolvetheuseofmobiledevicesorwearablestomonitorthe
humanbodyforcurrentbloodpressure,pulse,heartrate,sleeppatterns,calorieconsumption,orcurrentspeedwhilerunning.ThehugenumberofmHealthappsonthetwoleadingplatforms,iOSandAndroid,hasconvincedAppleandGoogletoinvestinanddevelopmHealthAPIsandappsfortheirmobileplatforms.AppleintroducedHealth17foritscustomersandHealthKit18foritsdevelopersatthesametimeasitrolledoutiOS8.GoogleintroducedGoogleFit19in2014.ThefactthatbothAppleandGoogleareenteringthehealthmarketwillleadtoseveralnewdevicesandappsbeingrolledoutinthenearfuture.
17.www.apple.com/ios/whats-new/health/18.https://developer.apple.com/healthkit/19.https://developers.google.com/fit/
Asthenumbersshow,mostoftheseappshelpcustomerstotracktheirfitnessordietaryhabits,buthealthappsalsoposehighriskstocustomers.Appsthatmanageinsulindosesforpatientswithdiabetescouldhavedisastrousconsequencesifabugoccurs.Thisriskraisesanimportantquestion:Canwetrusthealthcareapps?MedicaldevicesaregenerallyregulatedbytheUnitedStatesFoodandDrug
Administration(FDA),butthisisnotthecasewitheverymobilehealthapp.ExpertsfromtheNewEnglandJournalofMedicine20saythattheFDAdoesn’thaveenoughresourcestoregulateallthehealthappsavailableinthedifferentmobileappstores.AnotherchallengethatisnearlyimpossiblefortheFDAandappproviderstohandleisallthemobileoperatingsystemupdatesprovidedbythedifferentvendors.Eachmobileplatformreceivesmorethanoneortwoupdatesayear,andeachoperatingsystemupdatemustbecompliantwiththeFDAregulations.20.www.nejm.org/doi/full/10.1056/NEJMhle1403384
Sotheanswertothepreviousquestionisno.Wecan’ttrusthealthappsiftheyaren’tregulatedbyanyinstitutionbecausewecan’tbesurethatthedelivereddataiscorrectandfreeofmistakes.Ifyouhavetheopportunitytotestmobilehealthandfitnessapps,pleasebear
thefollowingpointsinmind:•GetinformationfromtheFDAandothermedicalinstitutionswithregardtoregulationsandhealthcareworkflows.•Theprovideddatamustbecorrectinordertoprotecthumanlife.•Datasecurityisaveryimportantaspectduetotheprivacyofaperson’sstateofhealth.•Mobilehealthappsmusthaveexcellentusabilityinordertocoverthetargetgroup’sneeds.•Geolocationdatamustbecorrectforfitnesstrackers.
Besidesthat,allofthemobiletestingknowledgeyouhaveacquiredinthisbookalsoappliestohealthandfitnessapps.Ifyou’dliketofindoutmoreaboutmHealth,visitthemHealthNews21or
mobileHealthEconomics22WebsitesorhavealookatthemHealthAppDeveloperEconomics2014study.23
21.www.mhealthnews.com/22.http://mhealtheconomics.com/23.http://mhealtheconomics.com/mhealth-developer-economics-report/
FinalThoughtsThisisthelastsectionofmybookaboutmobiletesting,andI’dliketoprovideyouwithsomefinalthoughts.Duringthecourseofthisbookyou’velearnedagreatdealaboutmobiledevices,mobileapps,mobileusers,andthetoolsthatareimportantwhenitcomestomobiletesting.Ialsohopeyou’velearnedfrommyideasandexperiencesduringmytimeasamobiletester.Thisbookisdesignedtohelpyouinyourdailylifeasamobiletester,mobile
developer,orproductmanagerbygivingyoutheimpetustogeneratenewtestingideasandtryoutnewmobiletestingapproaches.Itshouldalsoserveasabasisfordevelopingyourowntestingideasandapproaches,whilealsohelpingyoutoextendyourknowledgelevel.Asyouhaveseen,themobiledevelopmentandtestingbusinessisrapidly
changingwithlotsofnewtechnologyenteringthemarketeverydayandplentymoreinthepipeline.Thisiswhyit’simportantthatyoustayup-to-date,strivetolearnconstantly,andadaptyourskillstotheever-changingworldoftechnology.
FiveKeySuccessFactorsToroundthingsoff,I’dliketoprovideyouwithmyfivekeysuccessfactorsforbecomingasuccessfulmobiletester.
SuccessFactor1:HaveHighExpectationsMobileusershavehighexpectations,andyoushouldalsohaveveryhighexpectationswhenitcomestomobileappsandtheirusability,performance,andfeatureset.Bearinmindthatmobilecustomerswilluninstallyourappveryquicklyifthey’renothappywithit,andthey’llprobablysubmitabadreviewtotheappstore.It’sthereforeimportantthatyoualwayskeepyourcustomersinmind,encouragegoodusabilityandperformance,andmakesurethatalltheimportantbugsgetfixed.KeeptheKIFSUprincipleinmindandlistentoyourcustomers’needs.
SuccessFactor2:BeanExpertonMobileDevicesAsuccessfulmobiletesterneedstobeanexpertonmobiledevices.Itisessentialtoknowallthedifferenthardwareandsoftwarefeaturesofmobiledevicesfromthevariousplatforms.Thisknowledgewillhelpyoutokeeplotsofdifferenttest
scenariosinmindduringyourdailybusiness.Ifyou’reabletodoso,buymobiledevicesusingthedifferentplatformssoyoucanstayup-to-date.Ifbuyingisnotanoption,trytorentthemfromamobiledevicelab.Youshouldalsosubscribetovarioustechnologyblogsandnewspagesin
ordertogetthelatestnewsaboutmobileoperatingsystemsandmobiledevices.Irecommendthatyouwatchthekeynotevideosofthemajormobilemanufacturerstofindoutthelatestinformationaboutthedifferentplatforms.
SuccessFactor3:BeontheMoveOneofthemostimportantpointsyoushouldbearinmindwhentestingamobileappistobeonthemovewhileyou’retesting.Yourcustomersusetheirmobileappsinmanydifferentscenarios,locations,anddatanetworks.Therefore,it’sessentialthatyoutestyourappinseveraldatanetworkswithdifferentnetworkspeedstoreplicatereal-lifescenarios.Whiletestingyourappinthewild,you’lldoubtlesscomeacrosslotsofdifferentproblemsthatwouldprobablynevershowupintheoffice.Whentestinganapponthemove,therewillbelotsofinterferencesthatcouldhaveanimpactonyourappwhenusingthevarioussensorsandinterfacesamobiledevicehasonoffer.Sogrababag,fillitwithmobiledevices,andstarttestinginthewildright
now!
SuccessFactor4:BuildUpYourProgrammingSkillsMobiletestersneedtobeabletowritetestautomationcode.Ifyoudon’thaveanyprogrammingskillsrightnow,doyourbesttogetintoprogrammingsoyoucanwritereliableandrobusttestautomationscriptsforyourmobileapp.Programmingskillswillalsohelpyousupportthedeveloperswiththeirregressiontests,andyou’llbeabletocommunicateanddiscusstheapp’scodewiththedevelopers.Ifyourprogrammingskillsneedbrushingup,now’sthetimetoreadsomeprogramminglanguagebooksorrunthroughsomeonlinetutorials.
SuccessFactor5:BeaConstantLearnerThefinalsuccessfactorinbecomingabettermobiletesteristobeaconstantlearner.Thisdoesn’tjustapplytomobiletesters;itshouldbethecaseforanyoneinvolvedintheITbusiness.Thetechnologiesusedtobuildcomplexsystemsincludingmobileappsarechangingconstantly.Furthermore,newwaysofusingandcommunicatingwithnewtechnologiesareontheirway,andit’simportantforyoutofindoutaboutthemassoonaspossible.
Besideslearningnewtechnologies,youshouldalsoworkonimprovingyourtestingskills.Greatwaystodothisincludereadinglotsofblogsandbooks,attendingconferences,andtakingpartincompetitionstolearnfromothermobiletestersandshareexperienceswiththem.Thiswillhelpyoutoimproveyourtestingideas,approaches,andskills.Don’tshyawayfromtryingoutnewthings—makemistakesandlearnfrom
them.
SummaryThelastchapterofthisbookcoveredthetopic“What’snext?”Whataretheupcomingtechnologytrendssoftwaretestershavetodealwith?Idescribedfivepossibletechnologytrendsthatarealreadyonthemarketorontheirway.Thefivetechnologiesare
•InternetofThings•Connectedhomes•Connectedcars•Wearables•Healthapps
Inthe“FinalThoughts”sectionIoutlinedfivekeysuccessfactorstobecomeasuccessfulmobiletester.Thosesuccessfactorsare
•Havehighexpectations.•Beanexpertonmobiledevices.•Beonthemove.•Buildupyourprogrammingskills.•Beaconstantlearner.
That’sit;thankyouverymuchforreadingmybook.Ihopeyoulearnedalotofnewthingsandgotnewideasforyourdailylifeasamobiletester.Happymobiletesting!
Index
0Gnetworks,121Gnetworks,122Gnetworks,12–133Gnetworks,134Gnetworks,135Gmobilestandard,1399tests,145
Accelerationsensor,testing,38AccessibilityguidelinesAndroid,62iOS,62WC3initiative,62WindowsPhone,63
AccessibilitytestingAndroid,62auditoryimpairment,61cognitiveimpairment,61–62iOS,62overview,60physicalimpairment,61visualimpairment,60–61WC3initiative,62WindowsPhone,63
Actualresult,inbugreports,98AdvancedUserInteractionsAPI,125Advertisements,selling,24AgileRecord,197AgileTesting,196AgileTestingDays,195AmazonAppstoreGuidelines,26Ambientlightsensor,testing,37
AMPS(AdvancedMobilePhoneService),12AMTS(AdvancedMobileTelephoneService),12Androiddesignguide,18guidelines,81
Androidappstestautomation.SeeTestautomationtools,Android.testingcriteria,103
AndroidAssetPackagingTool,66AndroidAuto,201AndroidEmulator,51Androidoperatingsystemaccessibilityguidelines,62accessibilitytesting,62AppQualityAlliancesite,103pseudo-randomtestevents,66
Androidsmartphonebatterystatus,displaying,18docksection,18generatingpseudo-randomtestevents,66interrupttesting,66–67introductionof,14vs.iOS,17–19JouleUnitenergyprofilingtool,65Monkeytool,66networkstrength,displaying,18notifications,displaying,18pinningapps,18statusbar,18stresstesting,66–67time,displaying,18userinterface,18
AndroidWebbrowservs.otherbrowsers,48Appdevelopmentdevelopmentguidelines,2–3
KIFSU(KeepItFastSimpleUsable)principle,3KISS(KeepItSimple,Stupid)principle,3rejection,reasonsfor,18testing.SeeMobiletesting.
AppQualityAlliancesite,103AppstoresAppleAppStore,25BlackBerryWorld,26GooglePlayStore,25listof,25MicrosoftStore,26numberofavailableapps,25rejectingapps,reasonsfor,18reviewandpublishingguidelines,26Samsung,25uploadingappsto,26forWindowsPhone,26
Appaloosa,138AppBlade,138AppiumforAndroid,127–128foriOS,134nativeobjectrecognition,111onlineresources,111,134
Applause,138,145AppleAppStore,25AppleAppStoreGuidelines,26AppleWatch,204“ApplyingtheSFDPOTHeuristictoMobileTesting,”88Appsdeveloping.SeeAppdevelopment.health,205–207mHealth,205–207numberavailable,2,25pinning,18
preinstalled,testing,44reasonsfordeleting,2system,testing,44testing.SeeMobiletesting.uploadingtoappstores,26WindowsPhone,downloading,26
Apps,Androidtestautomation.SeeTestautomationtools,Android.testingcriteria,103
Apps,iOStestautomation.SeeTestautomationtools,iOS.testingcriteria,103
Apps,typesofhybrid,20–21native,19–20Web,22–23
appsfire,177AppsFlyer,177AppThwack,36,149Appurify,149AssociationforSoftwareTesting,196Auditoryimpairment,accessibilitytesting,61Automatedtesting.SeeTestautomation.
Bach,James,88,90Bamboo,137Batteryconsumptiontools,64–66Batterylife,savingwithproximitysensor,37–38Batterystatus,displaying,18Batteryusage,testingbatteryconsumptiontools,64–66Charlestool,63Fiddlertool,64fullychargedbattery,63–64Instrumentsenergyusagefunction,65
JouleUnitenergyprofilingtool,65lowbattery,64overheating,64usagestatistics,64–66
BeautifulTesting,196BelgiumTestingDays,195BetabyCrashlytics,138Betadistributiontools,138–139BirdFlight,138BlackBoxSoftwareTesting,55Blackboxtesting,54–55,164BlackBerryguidelines,81relativemarketshare,14,16
BlackBerryWebbrowservs.otherbrowsers,48BlackBerryWorld,26BlackBerryWorldGuidelines,26Blindness.SeeVisualimpairment.Blink,47Blogs,196–197BooksaboutSQLite,75Booksandpublications.SeealsoOnlineresources.AgileRecord,197AgileTesting,196BeautifulTesting,196ExploreIt!,196HowGoogleTestsSoftware,196TheInmatesAreRunningtheAsylum,30LessonsLearnedinSoftwareTesting,196MobileUsability,57ProfessionalTester,197SpecificationbyExample,196TapintoMobileApplicationTesting,92,196Tea-timewithTesters,197TestingCircus,197
Bots,137Boundaryvaluetesting,55Browserversion,inbugreports,99BugAdvocacy,183Bugcategory,inbugreports,101BugID,inbugreports,97Bugreportsbepolite,102keepitsimple,102mindmaps,examples,94–96overview,93,96–97remaincalm,102
Bugreports,contentsactualresult,98browserversion,99bugcategory,101bugID,97description,97expectedresult,98language,100logfiles,101–102mobiledevice,99mobileplatform,99mobile-device-specificinformation,99networkconditionandenvironment,100operatingsystem,99reproducible,99screenshotorvideo,101severity,101softwarebuildversion,100stepstoreproduce,98testdata,100testerwhofoundthebug,102work-around,98
Bugs,findingandprotectingfrom,53–55
Buildbot,137Businessmodelschoosing,24–25freemium,23–24in-apppurchases,24paid,24sellingadvertisements,24transaction,24
CalabashforAndroid,125–126foriOS,131–132nativeobjectrecognition,111
Camerasmobiletesting,43–44resolutionandimagesize,43–44testing,43–44
Capacitivetouchscreens,41Captureandreplaytools,111–112CarPlay,201CDMA(CodeDivisionMultipleAccess),12Cellphonesgenerationsof,12–13heads,12historyof,4–5transceivers,12
Charlesbatterytool,63Checklists.SeealsoGuidelines;Mindmaps;Mnemonics.genericmobiletesting,85–87mobileteststrategy,169–170overview,84
Checkstyletool,54ChromeWebbrowservs.otherbrowsers,48CIsystems,137Clicktale,177
Cloudtesting,147–152CloudMonkey,36,149Cognitiveimpairment,accessibilitytesting,61–62Cohn,Mike,105,106Colordeficiencies.SeeVisualimpairment.Color-blindness.SeeVisualimpairment.Communication,testerskill,179–181Communities,196Communitysupport,174–175Conferences,195–196Configurationdata,mobileteststrategy,165Connectedcars,201–203Connectedhomes,200–201Constantlearning,testerskill,183–184Cooper,Alan,30Coordinate-basedrecognitiontools,110COPFLUNGGUN(Communication,Orientation,Platform,Function,Location,Userscenarios,Notifications,Guidelines,Gesture,Updates,Network),92Crashreports,176Crashesonspecificdevices,testerskillsimprovement,187–192Creativity,testerskill,184Criticalthinking,testerskill,181–182Crowdtesting,141–147Crowdtestingproviders,145crowdsourcedtesting,145Crowdsourcing,141CruiseControl,137CSS3,22CucumberforAndroid,125–126Curiosity,testerskill,181Customerfocus,testerskill,184–185Customers.SeealsoUsers.gettinginformationabout,29–32invitingtousabilitytestingsessions,31–32representingwithpersonas,30–32
satisfying,checklistfor,32
Databases,testing,75Dateformats,testing,46Deafness.SeeAuditoryimpairment.Decisiontables,testing,55Deletingapps,reasonsfor,2Description,inbugreports,97Developingapps.SeeAppdevelopment.Devicemovement,detecting,38Deviceorientation,detecting,38Docksection,AndroidandiOSdevices,18DolphinWebbrowservs.otherbrowsers,48Doubletapgesture,42Downloadingresources.SeeOnlineresources.Draggesture,42Dumbphones,5DutchTestingDays,195Dynamictesting,53–55
EDGE(EnhancedDataratesforGlobalEvolution),13eggPlantforAndroid,129imagerecognition,110OCR/textrecognition,111
EmulatorsAndroidEmulator,51handlingfragmentation,35mobiledevicetestclouds,35prosandcons,115–116vs.realdevices,51–52vs.simulators,51–52
EnergyUsageInstrument,65–66EspressoforAndroid,128
nativeobjectrecognition,111ETSI(EuropeanTelecommunicationsStandardsInstitute)standards,82EuroSTAR,195Expectedresult,inbugreports,98ExperitestforAndroid,129foriOS,134mobilecloudtesting,149
ExploreIt!,196ExtSolo,122–123
FCCCUTSVIDS(Featuretour,Complexitytour,Claimstour,Configurationtour,Usertour,Testabilitytour,Scenariotour,Variabilitytour,Interoperabilitytour,Datatour,Structuretour),89–90,170Featurephones,5Fiddlerbatterytool,64FindBugstool,54FirefoxMobilebrowservs.otherbrowsers,48Fitnesswristbands,203–2045Gmobilestandard,134Gnetworks,13Fragmentation,definition,5Fragmentation,handlingemulators,35groupingtestdevices,33–34mobilecloudtesting,35–36mobiledevicelabs,34–36prioritizingmobiledevicegroups,33–34privacy,35rentingtestdevices,34–36security,35simulators,35
Frank,134Freemiumbusinessmodel,23–24Fullychargedbattery,testing,63–64
Functionaltesting,56,162
Gatheringcustomerinformation,30Gecko,47GeneralPacketRadioService(GPRS),13Generationsofmobiledevices,11–12Geographicalregions,adaptingto.SeeL10n(localization).GesturesupportforAndroid,125Gesturesdetecting,40testing,41–42
GherkinforAndroid,126GHUnit,134GlobalAppTesting,145Google,OurMobilePlanet,34GoogleBestPractices,57GoogleGlass,204–205GoogleInstrumentationTestRunner,128GooglePlaynativeAppBetaTesting,138GooglePlayStore,25GooglePlaystoreGuidelines,26GoogleTestAutomationConference,195Googlewearables,204GPRS(GeneralPacketRadioService),13GPScoordinates,detecting,39–40GPSlocation,settingforAndroid,122Groupingtestdevices,33–34GSM(GlobalSystemforMobileCommunications),12Guidelines.SeealsoChecklists;Mindmaps;Mnemonics.AmazonAppstoreGuidelines,26Android,81appdevelopment,2–3appstores,reviewandpublishing,26AppleAppStoreGuidelines,26AppleiOSapproval,81
BlackBerry,81BlackBerryWorldGuidelines,26forcorporatelook-and-feel,59GooglePlaystoreGuidelines,26platform-specificmobiletesting,81–82testautomation,114–115testingconnectedcars,202WindowsPhone,81
Guidelines,accessibilitytestingforAndroid,62foriOS,62WC3initiative,62forWindowsPhone,63
Gyroscopesensor,testing,38
Hardwareusagetesting,192Heads,cellphones,12Healthapps,205–207Hearingimpairment.SeeAuditoryimpairment.Help.SeeBooksandpublications;Onlineresources.HockeyApp,138HockeyKit,139HomeKit,201HowGoogleTestsSoftware,196Howe,Jee,141HTCDreamsmartphone,14HTML5,22Humiditysensor,testing,39Hybridapps,20–21
ISLICEDUPFUN(Inputs,Store,Location,Interactions/interruptions,Communication,Ergonomics,Data,Usability,Platform,Function,Userscenarios,Network),90–92,170I18n(internationalization),45–46IEEE(InstituteofElectricalandElectronicsEngineers),82Ignite,195
Imagerecognitiontools,109iMobiTrax,177In-apppurchases,24TheInmatesAreRunningtheAsylum,30Inputs,Store,Location,Interactions/interruptions,Communication,Ergonomics,Data,Usability,Platform,Function,Userscenarios,Network(ISLICEDUPFUN),90–92,170Installationtesting,71–72Instruments,energyusagefunction,65Internationalization(I18n),45–46Internetconnections,testing,4InternetExplorerMobileWebbrowservs.otherbrowsers,48InternetofThings(IoT).SeeIoT(InternetofThings).Interrupttesting,66–68In-the-wildtesting,167–169iOSaccessibilitytesting,62approvalguidelines,81designguide,18designresources,81generatingpseudo-randomtestevents,67simulators,52
iOSappstestautomation.SeeTestautomation,iOS.testingcriteria,103
iOSsmartphonevs.Android,17–19batterystatus,displaying,18docksection,18generatingpseudo-randomtestevents,67Instrumentsenergyusagefunction,65interrupttesting,67–68loggingenergyusage,onlineresource,66networkstrength,displaying,18notifications,displaying,18
pinningapps,18statusbar,18stresstesting,67–68time,displaying,18UIAutoMonkeytool,67–68userinterface,18
ios-driver,132–133IoT(InternetofThings)AndroidAuto,201AppleCarPlay,201AppleHomeKit,201connectedcar,201–203connectedhome,200–201mobiletesting,199–203NestLabs,200overview,199–200
ISTQBclass,55
JamoSolutionsAndroid,129iOS,134
Janky,137Javastaticanalysistools,54JavaScript,22Jenkins,137Johnson,KarenNicolemindmaps,93–96mnemonics,88
JouleUnit,energyprofilingtool,65JSONWireProtocolcompatibilitywithSelendroid,124andios-driver,132–133
JUnittool,54
Kaner,Cem,183
Kelly,Mike,89Keyboardinput,testing,44Keyboards,testing,187–188KeynoteforAndroid,129foriOS,134mobilecloudtesting,149
KeynoteMobileTesting,36KIF(KeepItFunctional),133KIFSU(KeepItFastSimpleUsable)principle,3KISS(KeepItSimple,Stupid)principle,3,102Kiwi,134Kohl,Jonathan,90,92
L10n(localization),45–46Languageinbugreports,100switching,forAndroid,122
Languagesadaptingto.SeeI18n(internationalization);L10n(localization).fallback,46
Launchstrategy.SeeMobilelaunchstrategy.LessonsLearnedinSoftwareTesting,196Let’sTest,195Linttool,54Localstoragetesting,76Localization(L10n),45–46Locationsensor,testing,39–40Logfilesinbugreports,101–102checking,82–83
LoggingenergyusageiniOS,66Logintask,testing,190Longtouchgesture,41Lowbattery,testing,64
LTE(LongTermEvolution),13
Magazines.SeeBooksandpublications.Magneticfields,detecting,38–39Magneticsensor,testing,38–39Manualtesting,mobileteststrategy,162,167–169Maximumdevicescope,160Metrics,mobileteststrategy,162mHealthapps,testing,205–207Microphones,testing,42–43MicrosoftStore,26Mindmaps,93,94–96MinistryofTesting,93Mnemonics.SeealsoChecklists;Guidelines;Mindmaps.COPFLUNGGUN(Communication,Orientation,Platform,Function,Location,Userscenarios,Notifications,Guidelines,Gesture,Updates,Network),92FCCCUTSVIDS(Featuretour,Complexitytour,Claimstour,Configurationtour,Usertour,Testabilitytour,Scenariotour,Variabilitytour,Interoperabilitytour,Datatour,Structuretour),89–90ISLICEDUPFUN(Inputs,Store,Location,Interactions/interruptions,Communication,Ergonomics,Data,Usability,Platform,Function,Userscenarios,Network),90–92mobileteststrategy,170overview,87SFDPOT(Structure,Function,Data,Platform,Operations,Time),88
Mob4Hire,145Mobile,definition,11MobileAppEurope,195MobileCheatSheet,79Mobilecloudtestinghandlingfragmentation,35–36mobiledevicetestclouds,35–36
Mobilecloudtestingproviders,149–150Mobilecommunity,testerskillsimprovement,194–195Mobiledevice,inbugreports,99
Mobiledevicelabs,34–36Mobiledevicetestcloudsemulators,35groupingtestdevices,33–34mobilecloudtesting,35–36mobiledevicelabs,34–36prioritizingmobiledevicegroups,33–34privacy,35rentingtestdevices,34–36security,35simulators,35
Mobiledevices.SeealsoSmartphones;specificbrands.acquiringfortesting,6–8cellphones,4–5dumbphones,5featurephones,5fragmentation,5rentingfortesting,7selectingfortesting,6
MobileFrameworksComparisonChart,21MobileLabs,36Mobilelaunchstrategycheckreleasematerial,172–174communitysupport,174–175crashreports,176post-release,174pre-launch,172–174reviews,175statistics,176–177timingoftherelease,173tracking,176–177
Mobilenetworks,historyofAMPS(AdvancedMobilePhoneService),12AMTS(AdvancedMobileTelephoneService),12CDMA(CodeDivisionMultipleAccess),12
EDGE(EnhancedDataratesforGlobalEvolution),13generations,11–12GPRS(GeneralPacketRadioService),13GSM(GlobalSystemforMobileCommunications),12LTE(LongTermEvolution),13MobileRadioTelephoneSystem,12MTD(MobileTelephonySystemD),12MTS(MobileTelephoneService),12NMT(NordicMobileTelephone),12OLT(OffentligLandmobilTelefoni),12overview,11–14speed,11–13UMTS(UniversalMobileTelecommunicationsSystem),13WiMAX(WorldwideInteroperabilityforMicrowaveAccess),13
MobilePersonas,30Mobileplatform,inbugreports,99MobileQAZone,196MobileRadioTelephoneSystem,12Mobilereleasecycles,6–8MobileSecurityTesting,79MobileTelephoneService(MTS),12MobileTelephonySystemD(MTD),12Mobileteststrategyautomatedtesting,162blackboxtesting,164configurationdata,165definingrequirements,156–158FCCCUTSVIDS,170functionaltesting,162ISLICEDUPFUN,170in-the-wildtesting,167–169manualtesting,162,167–169mnemonics,170mobilechecklists,169–170mobiletours,169–170
nonfunctionaltesting,162–163overview,155–156possiblemetrics,162productrisks,172regressiontesting,162stabledata,165targetdevices,selecting,166–167targetgroupinformation,gathering,157–158temporarydata,165testautomation,170–171testdata,164–166testenvironment,selecting,166–167testlevels,161–163testtechniques,163–164usagescenarios,168whiteboxtesting,163
Mobileteststrategy,scopeoftestingmaximumdevicescope,160multidevicescope,159–160single-devicescope,159use-casescope,160–161
Mobiletesting.SeealsoQualityassurancemeasures;specifictests.acquiringdevicesfor,6–8vs.automatedtesting,52–53batteries.SeeBatteryusage,testing.becreative,83boundaryvaluetesting,55cameras,43–44checkinglogfiles,82–83CI(continuousintegration),136–137cloudtesting,147–152connectedcar,201–203forconnectedcars,202connectedhome,200–201crowdtesting,141–147
databases,75decisiontables,55documentingresults.SeeBugreports.ETSI(EuropeanTelecommunicationsStandardsInstitute)standards,82functionaltesting,56generatingpseudo-randomtesteventsforiOSsmartphone,67genericchecklistfor,85–87gestures,41–42hardwareusage,192IEEE(InstituteofElectricalandElectronicsEngineers),82installationtesting,71–72interrupttesting,66–68IoT(InternetofThings),199–203keyboardinput,44keyboards,187–188localstoragetesting,76logintask,190mHealthapps,205–207microphones,42–43mindmaps,93,94–96onmultipleplatforms,187–192vs.othertechnologies,1–2,8paymenttask,190forpeoplewithdisabilities.SeeAccessibilitytesting.performance.SeePerformancetesting.permissions,190–191platform-specificguidelines,81–82preinstalledapps,44pseudo-randomtesteventsforAndroid,66qualityofInternetconnections,4inreallifeenvironments,3–4remotelocations,4rentingdevicesfor,7security.SeeSecuritytesting.selectingdevicesfor,6
smartglasses,204–205soundinputs,42–43standardsconformance,82standbytesting,71stresstesting,66–68systemapps,44TestingMap,93touchscreen,41–42uninstalltesting,72userinterface.SeeUsabilitytesting.voicecommands,42–43W3C(WorldWideWebConsortium),82wearables,203–204weatherconditions,4widgets,187–189
Mobiletesting,mnemonics.SeealsoChecklists;Guidelines;Mindmaps.COPFLUNGGUN(Communication,Orientation,Platform,Function,Location,Userscenarios,Notifications,Guidelines,Gesture,Updates,Network),92FCCCUTSVIDS(Featuretour,Complexitytour,Claimstour,Configurationtour,Usertour,Testabilitytour,Scenariotour,Variabilitytour,Interoperabilitytour,Datatour,Structuretour),89–90ISLICEDUPFUN(Inputs,Store,Location,Interactions/interruptions,Communication,Ergonomics,Data,Usability,Platform,Function,Userscenarios,Network),90–92overview,87SFDPOT(Structure,Function,Data,Platform,Operations,Time),88
Mobiletesting,sensorsandinterfacesaccelerationsensor,38ambientlightsensor,37changesindevicemovement,detecting,38deviceorientation,detecting,38gestures,detecting,40GPScoordinates,detecting,39–40gyroscopesensor,38
humiditysensor,39locationsensor,39–40magneticfields,detecting,38–39magneticsensor,38–39possibletests,40–41pressuresensor,39proximitysensor,37–38screenbrightness,37temperaturesensor,39touchlesssensor,40
Mobiletesting,updatetestingadbtool,74–75forAndroidapps,74–75foriOSapps,73–74overview,72typicalscenarios,73
MobileTools,79Mobiletours,169–170Mobiletrackingtools,177MobileUsability,57MobileWebbrowserscomparisonof,48layoutengines,47
MobileAppTracking,177Mobile-device-specificinformation,inbugreports,99Mobilityanddatanetworks,3–4Monkeytool,66MonkeyTalkforAndroid,129coordinate-basedrecognition,110foriOS,134OCR/textrecognition,111
Moolya,92MTD(MobileTelephonySystemD),12MTS(MobileTelephoneService),12
Multidevicescope,159–160Multitouchgesture,42
Nativeapps,description,19–20NativeObjectRecognition,111Neotys,149NestLabs,200Networkconditionandenvironment,inbugreports,100Networkstrength,displaying,1899tests,145NMT(NordicMobileTelephone),12Nonfunctionaltesting,162–163Notifications,displaying,18
OCMock,134OCR/textrecognition,110–111OLT(OffentligLandmobilTelefoni),121Gnetworks,12Onlineresources.SeealsoBooksandpublications.5Gmobilestandard,1399tests,145adjust,177AmazonAppstoreGuidelines,26Androiddesignguide,18AndroidEmulator,51Androidguidelines,81AppQualityAlliancesite,103apprejection,reasonsfor,18appstorereviewandpublishingguidelines,26Appaloosa,138AppBlade,138Appium,134Applause,138,145AppleAppStore,25AppleAppStoreGuidelines,26
AppleiOSapprovalguidelines,81AppleiOSdesignresources,81AppleWatch,204“ApplyingtheSFDPOTHeuristictoMobileTesting,”88appsfire,177AppsFlyer,177AppThwack,36,149Appurify,149Bamboo,137batteryusage,65–66BetabyCrashlytics,138betadistributiontools,138–139BirdFlight,138BlackBerryguidelines,81BlackBerryWorld,26BlackBerryWorldGuidelines,26Blink,47BooksaboutSQLite,75Bots,137Buildbot,137CDMA(CodeDivisionMultipleAccess),12CIsystems,137Clicktale,177CloudMonkey,36,149crowdtestingproviders,145crowdsourcedtesting,145CruiseControl,137CSS3,22EDGE(EnhancedDataratesforGlobalEvolution),13eggPlant,129EnergyUsageInstrument,65–66Espresso,128Experitest,129,134,149Frank,134gatheringcustomerinformation,30
Gecko,47GHUnit,134GlobalAppTesting,145GoogleBestPractices,57GoogleInstrumentationTestRunner,128GooglePlaynativeAppBetaTesting,138GooglePlayStore,25GooglePlaystoreGuidelines,26Googlewearables,204GPRS(GeneralPacketRadioService),13GSM(GlobalSystemforMobileCommunications),12HockeyApp,138HockeyKit,139HTML5,22iMobiTrax,177iOSdesignguide,18iOSsimulators,52ios-driver,133JamoSolutions,129,134Janky,137JavaScript,22Jenkins,137Keynote,129,134,149KeynoteMobileTesting,36Kiwi,134loggingenergyusageiniOS,66MicrosoftStore,26Mob4Hire,145mobilecloudtestingproviders,149–150mobiledevicetestcloud,36MobileFrameworksComparisonChart,21MobileLabs,36MobilePersonas,30mobiletrackingtools,177mobileusability,57
mobileusabilitydesignprinciples,57mobileuserinterfacedesignprinciples,57MobileAppTracking,177MonkeyTalk,129,134Neotys,149OCMock,134OpenDeviceLab,36operatingsystemmarketshares,16OurMobilePlanet,34passbrains,145Pebble,204PerfectoMobile,36,129,134,150PhoneGapframework,20platform-specificguidelines,81–82Presto,47Ranorex,129,134,150Roboelectric,129Robotiumprojectpages,122Robotiumwikipage,122SauceLabs,36,150Siesta,129SilkMobile,129,134SOASTA,129,134Specta,134Subliminal,134TeamCity,137TenKodEZTestApp,129TestBirds,145TestChameleon,150testCloud,145Testdroid,150TestFlight,139testingcriteriaforAndroidandiOS,103testingtools,54Testmunk,36,150
TestObject,36,129,150TestPlus,145TouchTest,36TravisCI,137Trident,47UIAutomator,129UMTS(UniversalMobileTelecommunicationsSystem),13usabilityheuristics,57usabilityprinciplesandtechniques,57usabilitytesting,57wearables,204Webbrowserlayoutengines,47WebKit,47widgets,18WiMAX(WorldwideInteroperabilityforMicrowaveAccess),13WindowsPhoneguidelines,81XamarinTestCloud,36,150XCTest,134Zucchini,134
Onlineresources,fortestersAgileTestingDays,195AssociationforSoftwareTesting,196BelgiumTestingDays,195blogs,196–197booksandpublications,196–197communities,196conferences,195–196DutchTestingDays,195EuroSTAR,195GoogleTestAutomationConference,195Ignite,195Let’sTest,195MobileAppEurope,195MobileQAZone,196Øredev,195
SoftwareTestingClub,196STAREAST,196STARWEST,196TestBash,196TestExpo,196TestingCircus,196uTestcommunity,196
OpenDeviceLab,36OperaMobileWebbrowservs.otherbrowsers,48Operatingsystemsthebigtwo,16–19.SeealsoAndroid;iOS.inbugreports,99relativemarketshares,16
Øredev,195OurMobilePlanet,34Overheating,batteries,64OWASP(OpenWebApplicationSecurityProject),77OWASPMobileSecurityPageMobileCheatSheet,79MobileSecurityTesting,79MobileTools,79Top10MobileRisks,80
Paidbusinessmodel,24passbrains,145Paymenttask,testing,190Pebble,204PerfectoMobileforAndroid,129coordinate-basedrecognition,110foriOS,134mobilecloudtestingproviders,150mobiledevicetestcloud,36
Performancetestingcriticalareas,69
mobileapps,70–71overview,68–69reproducingissues,189–190
Permissions,testing,190–191Personas,30–32PhoneGapframework,20Physicalimpairment,accessibilitytesting,61Pinchopen/closegestures,42Pinningapps,18Platform-specificguidelinesfortesting,81–82PMDtool,54Post-release,mobilelaunchstrategy,174Preinstalledapps,testing,44Pre-launch,mobilelaunchstrategy,172–174Pressuresensor,testing,39Presto,47Prioritizingmobiledevicegroups,33–34Privacy,35Processfocus,53–55Productfocus,53–55Productrisks,mobileteststrategy,172ProfessionalTester,197Programminglanguage,choosing,foriOStests,132–133Programmingskills,testerskill,185Proximitysensor,testing,37–38Pyramid.SeeTestautomationpyramid.
Qualityassurancemeasures.SeealsoMobiletesting.blackboxtesting,54–55dynamictesting,53–55findingbugs,53–55Javastaticanalysistools,54processfocus,53–55processfocusphase,54productfocus,53–55
protectingfrombugs,53–55statictesting,53–55whiteboxtesting,54
RanorexforAndroid,129foriOS,134mobilecloudtestingproviders,150
Realdevices,prosandconsoftesting,116–117Reda,Renas,122Regressiontesting,mobileteststrategy,162Rejectingapps,reasonsfor,18Releasecycles.SeeMobilereleasecycles.Releasematerial,checking,172–174Rentingdevicesfortesting,7Rentingtestdevices,34–36Reproducible,inbugreports,99Requirementsdefinition,mobileteststrategy,156–158ResearchinMotion,14Resistivetouchscreens,41Reviewandpublishingguidelines,appstores,26Reviews,mobilelaunchstrategy,175Roboelectric,129RobotiumforAndroid,121–122nativeobjectrecognition,111OCR/textrecognition,111recordingtestscripts,122–123
RobotiumExtensions,122–123Robotiumprojectpages,122RobotiumRecorder,122–123Robotiumwikipage,122Rotategesture,42Rotatingtouchscreens,42RoutineBot,110
SafariWebbrowservs.otherbrowsers,48Samsungappstore,25SauceLabs,36,150Scopeoftestingmaximumdevicescope,160multidevicescope,159–160single-devicescope,159use-casescope,160–161
Screenbrightness,testing,37Screens.SeeTouchscreens.Screenshotorvideo,inbugreports,101Securityhandlingfragmentation,35mobiledevicetestclouds,35
Securitytestingcachestorage,78client-sidevalidation,78commonappsecuritymistakes,77–79encryption,78fuzzing,77overview,76–77penetrationtesting,77securityrequirementsanalysis,79–80summary,80–81testingtools,77Top10MobileRisks,80unencryptedcommunications,78unintendedlocalstorage,78unnecessaryapppermissions,78
SeeTest,111SelendroidforAndroid,124–125nativeobjectrecognition,111
Selenium,111Sellingadvertisements,24
Sensors,testing.SeeMobiletesting,sensorsandinterfaces.Severityoferror,inbugreports,101SFDPOT(Structure,Function,Data,Platform,Operations,Time),88Sherry,Rosie,93–96Siesta,129Sikuli,110SilkMobileforAndroid,129foriOS,134
Simulatorsvs.emulators,51–52handlingfragmentation,35iOS,52mobiledevicetestclouds,35prosandcons,115–116vs.realdevices,51–52
Single-devicescope,159Smartglasses,204–205Smartwatches,203–204Smartphones.SeealsoMobiledevices;specificbrands.commonfeatures,5externalcomponents,15–16hardwarecomponents,15historyof,14–16marketpenetration,5post-2007,14–16
SOASTAforAndroid,129foriOS,134
Softwarebuildversion,inbugreports,100SoftwareTestingClub,196Soundinputs,testing,42–43SpecificationbyExample,196Specta,134Speedofmobilenetworks,historyof,11–13
Spoon,123–124Square,123,133Stabledata,mobileteststrategy,165Standardsconformance,testing,82Standbytesting,71STAREAST,196STARWEST,196Staticanalysistools,54Statictesting,53–55Statistics,mobilelaunchstrategy,176–177Statusbar,18Stepstoreproduce,inbugreports,98Strategyforproductlaunch.SeeMobilelaunchstrategy.fortesting.SeeMobileteststrategy.
Stresstesting,66–68Structure,Function,Data,Platform,Operations,Time(SFDPOT),88Subliminal,134Swipegesture,41Systemapps,testing,44
Tapgesture,41TapintoMobileApplicationTesting,92,196Targetdeviceselection,mobileteststrategy,166–167Targetgroupinformationgathering,mobileteststrategy,157–158TeamCity,137Tea-timewithTesters,197Technicalskills,testerskill,185Temperaturesensor,testing,39Temporarydata,mobileteststrategy,165Tenacity,testerskill,183TenKodEZTestAppforAndroid,129nativeobjectrecognition,111
Testautomation
cloud-based,115–116,152emulators,prosandcons,115–116goalsfor,113–114guidelinesfor,114–115vs.manualtesting,52–53mobileteststrategy,162,170–171vs.mobiletesting,52–53realdevices,prosandcons,116–117selectingtasksfor,113–115simulators,prosandcons,115–116
Testautomationpyramiddefaultconfiguration,105–106flippedconfiguration,106–108mobiletestconfiguration,108–109
TestautomationtoolsAppium,111betadistribution,138–139Calabash,111captureandreplay,111–112coordinate-basedrecognition,110eggPlant,110,111Espresso,111imagerecognition,109MonkeyTalk,110,111NativeObjectRecognition,111OCR/textrecognition,110–111overview,109PerfectoMobile,110recommendations,112–113Robotium,111RoutineBot,110SeeTest,111selectioncriteria,118–120Selendroid,111Selenium,111
Sikuli,110TenKodEZTestApp,111TestObject,110,111
Testautomationtools,AndroidAdvancedUserInteractionsAPI,125Appium,127–128automatingUIacceptancetests,125CalabashforAndroid,125–126Cucumber,125–126eggPlant,129Espresso,128Experitest,129ExtSolo,122–123gesturesupport,125Gherkin,126GoogleInstrumentationTestRunner,128JamoSolutions,129Keynote,129MonkeyTalk,129overview,120–121PerfectoMobile,129Ranorex,129recommendations,129–130recordingtestscripts,122–123Roboelectric,129Robotium,121–122RobotiumExtensions,122–123RobotiumRecorder,122–123Selendroid,124–125settingGPSlocation,122Siesta,129SilkMobile,129SOASTA,129Spoon,123–124switchingcurrentlanguage,122
TenKodEZTestApp,129TestObject,129togglingWi-Fionandoff,122UIAutomator,129UIAutomatorViewer,121
Testautomationtools,iOSAppium,134CalabashforiOS,131–132choosingaprogramminglanguage,132–133Experitest,134Frank,134GHUnit,134ios-driver,132–133JamoSolutions,134Keynote,134KIF(KeepItFunctional),133Kiwi,134MonkeyTalk,134OCMock,134PerfectoMobile,134Ranorex,134SilkMobile,134SOASTA,134Specta,134Subliminal,134UIAutomation,129–130XCTest,134Zucchini,134
Testdatainbugreports,100mobileteststrategy,164–166
Testenvironmentselection,mobileteststrategy,166–167Testlevels,mobileteststrategy,161–163Testscripts,recording,122–123Teststrategy.SeeMobileteststrategy.
Testtechniques,mobileteststrategy,163–164TestBash,196TestBirds,145TestChameleon,150testCloud,145Testdroid,150Testersblogs,196–197booksandpublications,196–197communities,196conferences,195–196namedinbugreports,102successfactors,207–209
Testers,requiredskillscommunication,179–181constantlearning,183–184creativity,184criticalthinking,181–182curiosity,181customerfocus,184–185programmingskills,185technicalskills,185tenacity,183
Testers,skillsimprovementcrashesonspecificdevices,187–192engagewiththemobilecommunity,194–195learnfromotherapps,186observeotherpeople,192–193participateintestingcompetitions,193–194Twitter,194
TestExpo,196TestFlight,139Testingapps.SeeMobiletesting.TestingCircus(community),196TestingCircus(magazine),197
Testingcompetitions,testerskillsimprovement,193–194TestingcriteriaforAndroidandiOS,103TestingMap,93Testingtoolsadjust,177appsfire,177AppsFlyer,177forautomation.SeeTestautomationtools.Checkstyle,54Clicktale,177FindBugs,54iMobiTrax,177JUnit,54Lint,54formobiletracking,177MobileAppTracking,177PMD,54staticanalysis,54TestNG,54whiteboxtesting,54XCTest,54
Testmunk,36,150TestNGtool,54TestObjectforAndroid,129imagerecognition,110mobilecloudtestingproviders,150mobiledevicetestcloud,36OCR/textrecognition,111
TestPlus,1453Gnetworks,13Time,displaying,18Timingoftherelease,173T-MobileG1smartphone,14Tools.SeeOnlineresources;Testingtools;specifictools.
Top10MobileRisks,80Touchgestures,41–42Touchlesssensor,testing,40Touchscreenscapacitive,41effectsofweatherconditions,42mobiletesting,41–42resistive,41rotating,42testing,41–42touchgestures,41–42typesof,41
TouchTest,36Tracking,mobilelaunchstrategy,176–177Trackingtools,177Transactionbusinessmodel,24Transceivers,cellphones,12TravisCI,137Trident,47Twitter,testerskillsimprovement,1942Gnetworks,12–13
UIacceptancetests,automating,125UIAutomation,129–130UIAutomator,129UIAutomatorViewer,121UMTS(UniversalMobileTelecommunicationsSystem),13Uninstalltesting,72Updatetestingadbtool,74–75forAndroidapps,74–75foriOSapps,73–74overview,72typicalscenarios,73
Uploadingappstoappstores,26
Usabilityheuristics,57TheInmatesAreRunningtheAsylum,30principlesandtechniques,57
Usabilitytestingallowmistakes,58consistency,59corporateidentityguidelinesforlook-and-feel,59details,checking,58documentyourobservations,59–60invitingcustomersto,31–32lessismore,57navigation,checking,58overview,56self-explanatoryapps,57usabilityanddesignpatterns,58verifytext,59workflows,checking,58
Usabilitytestingsessions,31–32Usagescenarios,mobileteststrategy,168Usagestatistics,batterylife,64–66Use-casescope,160–161UserinterfaceAndroidvs.iOS,18components,viewingpropertiesof,121designprinciplesformobiledevices,57testing.SeeMobiletesting,sensorsandinterfaces.
Users,expectations.SeealsoCustomers.overview,2–3priorityappcharacteristics,2reasonsfordeletingapps,2
uTestcommunity,196
Visualimpairment,accessibilitytesting,60–61Voicecommands,testing,42–43
W3C(WorldWideWebConsortium)accessibilityguidelinesinitiative,62conformancetesting,82
Wearablesfitnesswristbands,203–204GoogleGlass,204–205mobiletesting,203–204overview,203smartglasses,204–205smartwatches,203–204
Weatherconditionseffectsontouchscreens,42testing,4
Webapps,description,22–23Webbrowserlayoutengines,47Webbrowsers.SeeMobileWebbrowsers.WebKit,47WeekendTesting,193–194Whiteboxtestingmobileteststrategy,163toolsfor,54
WidgetsiOSvs.Android,18testing,187–189
Wi-Fi,togglingonandoff,122WiMAX(WorldwideInteroperabilityforMicrowaveAccess),13WindowsPhoneaccessibilityguidelines,63accessibilitytesting,63appstore,26relativemarketshare,16
WindowsPhoneapps,downloading,26Work-around,inbugreports,98WorldwideInteroperabilityforMicrowaveAccess(WiMAX),13
XamarinTestCloud,36,150XCTest,134XCTesttool,54
0Gnetworks,12Zucchini,134
CodeSnippets