16 June 2017

The consumer mobile experienceMeasuring consumer experience of using mobile services

Contents

Section Page1 Introduction 42 Data service availability 93 Data performance 114 Voiceperformance 15

Annex 1 Technical and research methodology2 Statisticalmethodology

3

Executivesummary

Thisreportsummarisestheinitialresultsfrom Ofcom’s new mobile research, which is designed to measure the consumer experience of using mobile services. It providesinformationondataserviceavailability, and the performance of mobile voice and data services.

Thepurposeofthisfirstreportistopresent a high-level picture of the consumer experience of using mobile services, and it does not include data comparing the performance of mobile network operators. We will look to extend the scope of future reports to include comparisons of consumers’ experience by mobilenetworkoperatorandlocation.

We welcome feedback on this report at mobileresearchapp@ofcom.org.uk.

Keyfindings:

Dataserviceavailability• More than nine in ten mobile data

downloads are successful for both 4G(95.6%)and3G-only(92.4%)users.

• Almostsevenintenusers(69%)arehappy with their overall service, with 4Gusersmoresatisfiedthan3G-onlyusers(71%vs60%).

• Whenusingapps,4GconsumersareconnectedtoWi-Fi69%ofthetime.When4Gusersareconnectedtoacellularnetwork,65%oftimeisspentona4Gnetwork,30%on3Gand5%on2G.

Dataperformance• Connectionspeedswhenusing

YouTube and Chrome are faster over 4Gthan3G,withWi-Fiprovidinghigher average speeds than both mobile technologies.

• 4Gnetworksaremoreresponsivethan3G(48msvs64msresponsetime).Wi-Fiisevenmoreresponsiveat27ms.

Voiceperformance• Onceinitiated,lessthan1%ofall

calls are dropped due to loss of service.

• Nineinten(90%)panellistssaytheyare happy with the performance of their network when making a call.

This research is part of a wider programme of work by Ofcom to research andprovideinformationaboutmobilequality of service, which also includes our SmartphoneCities research and our broadband and mobile checker app.

Introduction

Section1

5

Introduction

Last year we piloted a new methodology for measuring the consumer experience of using mobile voice and data services, recruitingapanelofUKconsumerswhoinstalled an Ofcom-branded research app on their Android smartphone. This new research complements our Smartphone Citieswork,whichhasbeenrunningsince2014andcomparestheperformanceofmobilenetworkoperators(MNOs)inselectedUKcities.

AbouttheOfcomMobileResearchApp

The app, which is provided by our technical partner P3, measures consumers’ experience of using mobile services as the panellists use their smartphones. It does this by running a programme of ‘passive’ tests which are designed to measure network availability and performance, while minimising the impact on the user experience(forexample,byminimisingdatauseandbatterydrain).Additionalsatisfactionmeasurementsarecapturedvia pop-up style surveys, which allow us to compare consumer experience and perception.

More information about our testing methodology can be found in Annex 1: Technical methodology.

Theappgivesitsuserssummaryinformationabouttheirconnectionandusagehabits(mostmetricsarebasedonthepastsevendays):

• the current network technology the phone is using and the signal strength;

• theproportionoftimespentonthedifferentnetworktypes(Wi-Fi,2G,3Gand4G);

• theproportionofdataconnectionteststhat have been successful on Wi-Fi and mobile networks;

• the maximum download speed recorded over each wireless network type;

• thedroppedcallratio;• networklatency(responsetime)

recorded over each wireless network type.

The app can be downloadedfromtheGooglePlay Store.

6

Theappcollectsinformationonanumberofmeasures,including:

• Data service availability and accessibility

• Data throughput and responsiveness (latency)

• Voicecallperformance• Usageinformation

As the data are collected from a panel ofconsumersacrosstheUK,theresultsprovidedbelowareagoodreflectionoftheconsumer experience. This experience is affectedbyanumberoffactorsinadditionto network performance, including handset settingsand,potentially,thetariffthattheconsumerison.Assuch,thefindingsarelikelytobedifferentfromthosepublishedinSmartphoneCitiesandsimilarreports,which focus on measuring the performance of the mobile network.

The data used in this report were collected between27September2016and23December2016.Intotal,6,632peopledownloadedtheapp;ofthose,4,288metour requirements in terms of spending at least a week on the panel by the end of thefieldwork,andhavingvalidinformationregardingtheir:

• mobile provider;• mobile service technology;• home postcode;• age; and• gender.

The map shows the geographic spread of ouroverallpanel(i.e.wherethetestswereattempted),althoughforthepurposesofthis report we used sub-panels of the data toensurethatouranalysisisrepresentativeoftheUKAndroidmobilepopulation.

More information about our sub-panels and how we ensured that the data were representative can be found in Annex 2: Statistical methodology.

7

Overallsatisfaction

4Gusersreportedhigherlevelsofsatisfactionthan3G-onlyusers

Overall,69%ofourpanellistssaidtheywere‘very’or‘fairly’satisfiedwiththeiroverallmobile service.

Thosewithatariffandhandsetwhichenabledthemtouse4Gservicesweremoresatisfiedthan3G-onlyusers(71%vs60%).

People in urban areas were more likely than those in rural areas to say they were satisfiedwiththeirservice(72%vs56%).Thiswastruebothfor4Gusers(73%vs57%)andfor3G-onlyusers(63%vs49%),andmayreflectthebetternetworkcoveragein urban areas.

While the performance of a service is a key driver of consumer experience, it is important to note that many other factors alsoaffecttheconsumerexperienceandthereforelevelsofsatisfaction.Theseinclude price, handset type, quality of customer service, contract terms and the activitiesthatthephoneisusedfor.

In our panel, people said that web browsing (90%),voicecalls(75%)anddownloadinglargefiles(52%)wereamongthemostimportantactivitiesthattheyusedtheirphones for.

Satisfactionwithoverallmobileservice

Activitiesimportanttomobileusers

90%75%

52% 46% 44%27%%

of p

anel

lists

8

Networkshare

4GconsumersusedappsoverWi-Fimorethantwo-thirdsofthetime

Our mobile research app records the type of network that users connect to when activelyusingappsandtransferringdata,and provides an overall picture of the type of network mobile users connect to most often.

OurfindingsshowthatWi-Fiisafundamental part of the experience of consumers using mobile phones. More thantwo-thirdsofthetime,4Gconsumersconnected to Wi-Fi rather than mobile networks when using apps.

ThismaybeduetobetterexperienceoverWi-Fi or users trying to minimise their mobile data use, enabling them to save their data allowances for when they are outside the home or other Wi-Fi areas.

When transferring data over a mobile data network,35%ofthetime4Gcustomerswereconnectedto3Gor2Gnetworks.Thisreflectsthefactthat4Gnetworkscurrentlyhavelowercoveragethan3Gor2Gnetworks,andevenwhere4Gcoverageisavailable,connectionsmaysometimesbemadetoa3Gor2Gnetworkaspartofmobile providers’ capacity management.

Typeofnetworkconnectedtowhiledataistransferredduringappsession

9

What is data service availability?

Every15minutes,theresearchapprunsabackgroundtestwhichattemptstodownloadasmallfileandlogswhetherthiscanbecompletedsuccessfully.Thismetricdefinesthepercentageofcaseswhentheusercanbothconnectto the network and download data, however, it is not a direct measure of coverage.

More information on how this metric is derived can be found in Annex 1: Technical methodology.

Dataserviceavailability

Section2

10

Dataserviceavailability

Morethannineintenmobiledatadownloadsaresuccessful

Inmostcases(95.6%ofoccasions),4Guserscouldaccessamobilenetwork(either2G,3Gor4G)andsuccessfullydownloaddata.Thiswaslowerfor3G-onlyusers(92.4%).

This analysis includes only those tests that were run when the phone’s screen was on, i.e. when the panellist was trying to use it. It doesnottakeintoaccountconnectiontestsrunwhilethephonewasnotinuse(‘screenoff’).

Analysingtheresultsbytimeofday,weseeastrongpositivecorrelationbetweenthenumberofconnectiontestsperhourand the percentage of tests that fail. This indicates that it is more likely that data downloadwillnotbesuccessfulattimeswhen more people are using their phones and mobile networks are busy.

Percentageofcasespeoplecouldaccessamobilenetworkandsuccessfully

downloaddata

Correlationbetweenthenumberofconnectiontestsperhour

andthepercentageoffailedtests

0

20,000

40,000

60,000

80,000

100,000

120,000

0%

1%

2%

3%

4%

5%

6%

7%

1 3 5 7 9 11 13 15 17 19 21 23

Num

ber o

f tes

ts

% o

f fai

led

test

s

Time of day% of failed tests Number of tests

11

Download speed

Downloadspeedisthespeedatwhichinformationistransferredfromtheinternettoadevice.Ineffect,itdetermineshowquicklyafilecanbedownloaded to a smartphone. The unit of measurement for download speeds ismegabitspersecond(Mbit/s).

Response time

Responsetime(referredtotechnicallyaslatency)isthedelaybetweenaconsumermakingarequesttotheirmobilenetworkforinformationandthenetworkprovidingthisinformationtothedevice.Aconnectionwithlowlatency will ‘feel’ more responsive.

More information on how this metric is derived can be found in Annex 1: Technical methodology.

Dataperformance

Section3

12

People use their smartphones for a variety of purposes, and what the phone is used foraffectstheconsumerexperience:someapplicationsrequirehigherspeedsinorderto deliver a good consumer experience, whileforothersresponsetimemayhaveabigger impact.

WhenusingappsliketheGooglePlayStore,thephonetriestodownloadafileasquicklyas possible and therefore needs speeds significantlyhigherthanmostotherapps.VideostreamingservicessuchasYouTubealso need higher speeds in order to deliver a goodexperience(over2Mbit/sforstandard-definitionvideostreamingandover5Mbit/sforhigh-definition);lowspeedsmaylead

tovideobufferingandfailuretostreamhigh-definitionvideos.Incomparison,webbrowsing and video calling require speeds ofover1Mbit/sforagoodexperience(theindicativespeedsarebasedonthose used to determine the status of service for the Ofcom’s broadband and mobile checker app).

Lookingatvariationthroughouttheday,wesee that the highest levels of mobile use are between10amand8pm,andthatduringthesetimestherecordedaveragedatathroughput is lowest. This indicates that networkcongestionresultsinlowerspeedsatbusytimes.

Dataperformance

Correlationbetweenthenumberofpeopleusingtheirphonesandtheaveragespeed,bytimeofday

0

0.5

1

1.5

2

2.5

3

3.5

0

500

1000

1500

2000

2500

3000

0 2 4 6 8 10 12 14 16 18 20 22

Mbi

t/s

Num

ber o

f use

rs

Time of day

Number of users Average throughput

13

YouTube

The average speeds recorded when using YouTube vary by network technology.

Onaverage,thoseusingYouTubeover4Gcouldreceivespeedsof3.8Mbit/s,whiletheaveragespeedover3Gwaslower,at2.7Mbit/s.PeoplewatchingYouTubeoverWi-Fi had the highest average speed, at 6.6Mbit/s.

TheaveragedurationofaYouTubesessionwaslongeroverWi-Fi(aroundfive-and-a-halfminutes)comparedto4G(justoverfourminutes)and3G(aroundthreeminutes).

Thismayreflectabetteruserexperienceover higher speeds. But it is also likely to be duetoconsumerslimitingtheirmobiledatause to avoid exceeding their mobile data allowance.

GoogleChrome

TheGoogleChromewebbrowserrequiresmuch lower speeds than YouTube in order toprovideasatisfactoryuserexperience.

AttheUKlevel,4Gusersonaveragereceived2.0Mbit/sspeeds,whilewhenon3G,theaveragespeedwaslower,at1.4Mbit/s.Onaverage,therewaslittledifferencebetweenspeedson4GandWi-Fi(2.2Mbit/s)whenusingChrome.

ConsumerstendedtospendmoretimeusingChromewhenonWi-Fi(justovertwominutes)comparedto4Gand3G(atslightlyabove and slightly below one-and-a-half minutesrespectively).

It should also be noted that people may spendmoretimeusingappswhenathomeand likely using Wi-Fi than when they are out and probably using mobile data, thus explaining higher app use on Wi-Fi.

YouTubeandGoogleChromeperformance

1.4

2.0 2.2

1.41.6

2.2

3G 4G Wi-Fi

Speed (Mbit/s) Duration (min)

2.73.8

6.6

3.14.2

5.5

3G 4G Wi-Fi

Speed (Mbit/s) Duration (min)

14

Downloadspeedsareparticularlyimportantfor video streaming and downloading larger files,suchasfilmsorapps,howeveritisnotthe only measure that determines consumer experience.

Foractivitiesthatrequireinformationtobedeliveredwithaslittledelayaspossible,aquickresponsetime(i.e.lowlatency)maybemoreimportant.Aconnectionwithlowlatency will also feel more responsive for simple tasks like web browsing and certain applicationsperformfarbetterwithlowerlatency,includingvideocalling,VoIP(voiceoverinternetprotocol)andonlinegaming.

Ourresearchindicatesthat4Gnetworksaremoreresponsivethan3Gones;onaverage,responsetime(latency)on3Gwas64mscomparedto48mson4G.ResponsetimeonWi-Fiwasevenlowerat27ms.

Mostactivitiesrequirearesponsetimeoflessthan100mstoprovideagoodexperience, although some online gaming appsrequirearesponsetimeoflessthan50ms.

Responsetime

3Gresponsetime 4Gresponsetime Wi-Firesponsetime

64ms 48ms 27ms

15

Measuring voice performance

Asvoicecallperformanceisstillamajordriverofcustomersatisfaction,Ofcom’s mobile research app measures whether people can successfully maintainvoicecallsbydetectingcaseswherethecallisinterruptedduetoaloss of service.

Duetolimitationsinpassivetestingmethodology,wearenotabletolookintocallsetupsuccess,sotheresultsdonottakeintoaccounttimeswhenpeoplewere not able to make a call.

More information on how this metric is derived can be found in Annex 1: Technical methodology.

Voiceperformance

Section4

16

Voiceperformance

Voicecallperformanceremainsimportantformanypeople

Three in four respondents in our panel said that the ability to make voice calls on a daily basis was ‘extremely’ or ‘very’ important to them,whileonly5%claimedthatvoicecallswere not at all important.

Thereweredifferencesbasedonage;alowerproportion(56%)of18-24sinourpanel said calls were important to them, comparedto81%ofover-45s.

Callsdroppedduetolossofservice

Ourresearchshowsthatonceinitiated,lessthan1%ofallcallsweredroppedduetolossof service during the research period.

Satisfactionwithvoicecallperformanceishigh

AtaUKlevel,90%ofpanellistssaidthattheywerehappy(eitherpositiveorneutral)with the performance of their network when making a phone call.

However, those in urban areas were more likely than those in rural areas to say that theyweresatisfied(90%vs86%).

Morethanhalfofpanellistssaidthat makingvoicecallswasextremelyimportant

Agebreakdownofpanellistswhosaidvoicecallswereextremelyorveryimportant

Percentageofpanelistswhoclaimtheyarehappywithvoicecallperformance

17

Nextsteps

• Wewillpublishadditionalanalysisofusagetrendinformationcapturedby the research app in the UK Communications Market Report in August2017.

• We aim to increase the size of our panel in the coming months. If you would like to take part in the research, please download the app from the GooglePlayStore.

• ThesecondphaseoffieldworkwillruninQ42017.

• We will look to extend the scope of future reports to include comparisons of the experiences of consumers by mobilenetworkoperatorandlocation.