Barnsley User Experience Testing - AWTG Limitedawtg.co.uk/app/uploads/2014/10/Barnsley_User_Experience_v1.3.pdfBarnsley User Experience Testing ... •an Android master ... AWTG has

Commercial in confidence

Barnsley User Experience Testing

3G/WiFi

Page 2 Commercial in confidence

Overview and Background

Test Equipment

Test Overview

Test Methodology

Test Results

• User Connectivity

• Walk Tests

• Outdoor Static

• Indoor Static

Result Summary

Outline


Overview and Background

This document describes the user experience testing conducted in Barnsley Town Centre by AWTG on

17th and 18th October 2013. The aim of the exercise was to measure the performance of MNO’s 2G/3G

network within WiFi network’s boundary on user smart handsets and evaluating handover from 2G/3G

to WiFi and vice versa. It should be noted that the MNO did not have 4G coverage within the

assessment area at the time of testing. Table below shows overview of the tests:

Test

Connectivity

Smartphone 1 (S1) Smartphone 2 (S2) Smartphone 3 (S3)

WiFi Only 3G Only Roaming

2G/3G/WiFi

User Connectivity Test X X

Indoor Static Test

Outdoor Static Test

Outdoor Walk Test


Test Equipment: SwissQual QualiPoc Freerider

QualiPoc Freerider II consists of: • an Android master tablet • up to 6 Android slave Smartphones (3 Smartphones used for

the QoE measurement tests mentioned in this document) • GPS device.

AWTG has selected the SwissQual QualiPoc Freerider tool to carry out the tests in the area to capture user experience. MNO SIM cards installed in Samsung Galaxy SIII handsets have been used for these tests.


Test Overview

User Connectivity Tests

• Handover from 3G To WiFi

• Handover from WiFi To 3G

• Measuring latency and web browsing throughput

Synchronized 3G and WiFi tests in indoor and outdoor Static locations

• FTP UL/DL on 3G and WiFi

• Web browsing measurement on 3G and WiFi

• Latency measurement (RTT) on 3G and WiFi

• Online streaming measurement (Youtube) on 3G and WiFi

Outdoor Walk Test

• Running a test package (consisting FTP UL/DL, Web Browsing, Latency and Youtube Streaming) every 3 minutes while

engineer walks through the root in and out of the WiFi coverage.

• The route is selected in a way that network performance is measured inside and outside the WiFi coverage area. Please

refer to slide8 to view the test route.


Test Methodology - User Connectivity

Start and end points of the route

Walk Route

WiFi coverage boundary

The aim of this test is to measure the following:

• The effect of handover from 3G to WiFi when user enters WiFi zone

• The effect of handover from WiFi to 3G when user leaves WiFi zone

The following User Connectivity Test procedure have been followed by AWTG engineers:

• Enable WiFi on a smartphone configured with 3G Sim (while on the WiFi network edge with no WiFi coverage) and start Web Browsing.

• Move from a point in 3G coverage to another location within WiFi coverage.

• Latency (ping) tests were running while conducting the above steps.


Test Methodology - Outdoor Walk Test

3G-WiFi walk tests have been conducted on a route within the WiFi network coverage area. One engineer walked

through the route carrying SwissQual Freerider tool which consisted of three Smartphones running the following

test package every 3 minutes.

• Latency/Ping (maximum test duration of 10 sec)

• Web Browsing – www.bbc.co.uk– (maximum test duration of 30sec)

• FTP DL – 100MB file – (maximum test duration of 30sec)

• FTP UL – 100MB file – (maximum test duration of 30sec)

• YouTube Streaming – 30 second video- (maximum test duration of 1min)

Latency Web Browsing FTP DL FTP UL YouTube Streaming

10s 30 s 30s 30s 60s

180s 20s

wait

Test Cycle


Test Methodology - Outdoor Walk Test Route

3G-WiFi walk tests have been conducted on a route within

the WiFi network coverage area. One engineer walked

through the route carrying SwissQual Freerider tool which

consisted of three Smartphones running the test package

every three minutes.

The walk test route which engineers followed is shown on

the map. The route is selected in a way that 3G network

performance is measured inside and outside the WiFi

coverage area.

Walk Route

WiFi AP

WiFi Network Boundary


Test Methodology - Static Tests

3G-WiFi measurement static tests have been conducted in three fixed outdoor and two fixed indoor locations.

The tests ran on all three Smartphones and the following user experience measurements have been collected:

• Latency/Ping (maximum test duration of 10 sec)

• Web Browsing – www.bbc.co.uk– (maximum test duration of 30sec)

• FTP DL – 100MB file – (maximum test duration of 30sec)

• FTP UL – 100MB file – (maximum test duration of 30sec)

• YouTube Streaming – 30 second video- (maximum test duration of 1min)

10 cycles have been conducted at each location in order to achieve a certain confidence level for the investigated KPIs.


Test Results - User Connectivity S3 - Roaming 3G/WiFi

This section shows the handover between 3G and WiFi. The heat maps below show the user was switched to WiFi when entering the WiFi boundary which resulted in lower latency and improved web browsing performance. Users within the WiFi boundary experienced the following improvements:

• Lower RTT in WiFi Boundary • Higher HTTP Throughput. • Lower Time to Load HTTP Webpage

WiFi Boundary WiFi Boundary WiFi Boundary

Technology Heat-map Latency Heat-map HTTP Throughput Heat-map

Technology WiFi

3G


Test Results - User Connectivity S3 - Roaming 3G/WiFi

Route started outside WiFi area. Test engineer moved inside the WiFi boundary and walked back towards the start point.

Route started outside WiFi area. Test engineer moved inside the WiFi boundary and walked back towards the start point.

WiFi AP WiFi Network Boundary

Start point End Point

UE connected to WiFi from 3G when entering the WiFi coverage.

latency improved significantly when the UE connected to WiFi.


Test Results - Walk Tests 3G RSCP

WiFi Network Boundary

Average RSCP = -78.2 dBm

Minimum RSCP = -117.7 dBm

Maximum RSCP = -45 dBm

Standard Deviation = 11.25


Test Results - Walk Tests WiFi Signal Strength

WiFi AP WiFi Network Boundary

Average RSSI = -69.19 dBm

Minimum RSSI = -98 dBm

Maximum RSSI = -47 dBm

Standard Deviation = 8.46


Test Results - Walk Tests Technology Heat-map

This slide shows the technology which the roaming handsets were

connected during the walk test. The heat-map shows the user was

on WiFi majority of the time in the WiFi boundary.

• 86% of the time connected to WiFi

• 14% of the time connected to 3G

Roaming 3G/WiFi

Technology

WiFi

3G

WiFi Boundary

WiFi AP


Walk Test – Youtube Time to Load Comparison

WiFi 3G 3G/WiFi

S2 (3G only) Time to Start 30 sec Video

Average (sec) 7.03

Minimum (sec) 1.89

Maximum (sec) 26.45

Standard Deviation 6.01

S3 (Roaming ) Time to Start 30 sec Video

Average (sec) 3.14

Minimum (sec) 0.71

Maximum (sec) 16.90


S1 (WiFi only ) Time to Start 30 sec Video

Average (sec) 2.73

Minimum (sec) 0.75

Maximum (sec) 9.81



Walk Test - HTTP Time to Load the Webpage Comparison

S2 (3G only) Time to Load Webpage (URL: www.bbc.co.uk)

Average (sec) 7.77

Minimum (sec) 2.71

Maximum (sec) 16.89


S3 (Roaming ) Time to Load Webpage (URL: www.bbc.co.uk)

Average (sec) 5.51

Minimum (sec) 1.84

Maximum (sec) 16.98


S1 (WiFi only ) Time to Load Webpage (URL: www.bbc.co.uk)

Average (sec) 5.94

Minimum (sec) 1.38

Maximum (sec) 18.20


WiFi 3G 3G/WiFi


Walk Test - HTTP Throughput Comparison

S2 (3G only) HTTP Throughput (URL: www.bbc.co.uk)

Average (Kbps) 178.61

Minimum (Kbps) 74.06

Maximum (Kbps) 455.44


S3 (Roaming ) HTTP Throughput (URL: www.bbc.co.uk)





S1 (WiFi only ) HTTP Throughput (URL: www.bbc.co.uk)





WiFi 3G 3G/WiFi


Walk Test - FTP Download Comparison

WiFi 3G 3G/WiFi

S2 (3G only) FTP DL Throughput

Average (Mbps) 0.65

Minimum (Mbps) 0.0004

Maximum (Mbps) 2.05


S3 (Roaming ) FTP DL Throughput

Average (Mbps) 2.22

Minimum (Mbps) 0.01

Maximum (Mbps) 12.27


S1 (WiFi only ) FTP DL Throughput

Average (Mbps) 3.22

Minimum (Mbps) 0.09




Walk Test - FTP Upload Comparison

S2 (3G only) FTP UL Throughput

Average (Mbps) 0.20

Minimum (Mbps) 0.07

Maximum (Mbps) 1.2


S3 (Roaming ) FTP UL Throughput

Average (Mbps) 1.87

Minimum (Mbps) 0.01



S1 (WiFi only ) FTP UL Throughput

Average (Mbps) 2.80

Minimum (Mbps) 0.11



WiFi 3G 3G/WiFi


Walk Test - FTP DL/UL Distribution Comparison


Test Results – Outdoor Static In Front of Barnsley Train Station

Test Location WiFi AP

Date Time Longitude Latitude

18/10/2013 13:30 -1.47857 53.55469

Tests were conducted outside Barnsley Train Station, 10 metres from WiFi AP connected to fibre. Tests were conducted during busy hours on Friday .


Test Results – Outdoor Static In Front of Morrisons



18/10/2013 14:30 -1.48002 53.54979

Tests were conducted outside Morrisons, 12 metres from a Root WiFi AP.


Test Results – Outdoor Static In the Corner of Regent Street & Church Street



18/10/2013 15:20 -1.48179 53.55427

Tests were conducted In the Corner of Regent Street & Church Street , 4 metres from WiFi AP 1 hop away from Root AP.


Test Results – Indoor Static Blah Bar


18/10/2013 17:45 -1.48176 53.55341

Tests were conducted in Blah Bar, 35 metres from WiFi AP, 1 hop away from fibre root.

• Tests were conducted near window • Window type: double glazed


Indoor Tests were conducted 5m from double glazed window


Test Results – Indoor Static Train Station



18/10/2013 18:15 -1.47813 53.55458

Tests were conducted in Subway inside Barnsley Train Station, 35 metres from WiFi root AP connected to fibre.

• Tests were conducted near window

• Window type: double glazed

Indoor Tests were conducted 4m from window


The following measurements on WiFi and 3G have been

conducted in two indoor and three fixed outdoor locations.

• FTP Download

• FTP upload

• Web Performance (www.bbc.co.uk)

• HTTP Throughput

• Time to Load the webpage

• Latency (Ping from handsets to FTP Server)

• Video Streaming (YouTube streaming)

Summary Results Indoor and Outdoor Static Tests

The Average static test results show better WiFi performance for FTP-DL/UL, Latency, YouTube Streaming and HTTP

web browsing.

FTP Tests

Operation Technology Avg Throughput [Mbps]

Outdoor Avg Throughput [Mbps]

Indoor

FTP (DL) WiFi 9.40 2.29

3G 1.08 0.71

FTP (UL) WiFi 7.76 2.02

3G 0.48 0.14

Web Browsing

Host Technology Avg Throughput [Kbps] Avg Time to Load (sec)

Outdoor Indoor Outdoor Indoor

http://www.bbc.co.uk WiFi 476.40 336.43 2.63 3.72

3G 259.91 155.84 6.35 10.75

YouTube Streaming

Technology Avg Time to Start Play (s)

Outdoor Avg Time to Start Play (s)

Indoor

WiFi 1.54 4.01

3G 11.79 13.53

Latency

Ping Host Technology Avg RTT [ms] Outdoor Avg RTT [ms] Indoor

FTP Server (217.138.0.114) WiFi 29.2 89.3

3G 229.0 274.5


Summary Results Indoor and Outdoor Static Tests

The following Pie Charts show the average of the tests conducted in three indoor and two fixed outdoor locations on WiFi and 3G networks.


Measurements have been conducted on three handsets:

• Smartphone1 : locked to WiFi

• Smartphone2 : locked to 3G

• Smartphone3 : Roaming 3G/WiFi (86% on WiFi & 14% on 3G)

Please note that in the majority of the tests (86% of test

duration), 2 handsets were conducting simultaneous tests on

WiFi while one handset was on 3G. The average results of the

walk test per handset are shown in the tables.

Summary Results Outdoor Walk Tests

The average walk test results show better WiFi Performance

for FTP DL/UL, Latency, HTTP Web browsing and YouTube

video streaming. Nevertheless, the handset roaming

between 3G and WiFi had improved performance compared

with the handset locked to 3G.

FTP Tests

Operation Technology Average Throughput (Mbps)

FTP (DL)

WiFi (S1) 3.22

3G (S2) 0.65

3G/WiFi (S3) 2.22

FTP (UL)

WiFi (S1) 2.8

3G (S2) 0.2

3G/WiFi (S3) 1.87

YouTube Streaming

Technology Average Time to Start Play (s)

WiFi (S1) 2.73

3G (S2) 7.03

3G/WiFi (S3) 3.14

HTTP

Host Technology Average Throughput

(Kbps) Average Time to Load (s)

http://www.bbc.co.uk

WiFi (S1) 281.81 5.94

3G (S2) 178.6 7.77

3G/WiFi (S3) 315.5 5.51

Latency

Technology Avg RTT (ms)

WiFi (S1) 47.37

3G (S2) 281.94

3G/WiFi (S3) 101

Commercial in confidence Page 29

Summary Results Outdoor Walk Tests

The following Pie Charts show the average results of the tests conducted on three handsets while walking through the rout.

Commercial in confidence Page 30

Registered in UK No. 5793772HQ in London UK

Office: +44 (0) 208 799 0368

Fax: +44 (0) 208 728 9610

www.awtg.co.uk

Thank You

Documents

Barnsley User Experience Testing - AWTG Limitedawtg.co.uk/app/uploads/2014/10/Barnsley_User_Experience_v1.3.pdfBarnsley User Experience Testing ... •an Android master ... AWTG has