Upload
lynhi
View
221
Download
3
Embed Size (px)
Citation preview
Commercial in confidence
Barnsley User Experience Testing
3G/WiFi
Page 2 Commercial in confidence
Overview and Background
Test Equipment
Test Overview
Test Methodology
Test Results
• User Connectivity
• Walk Tests
• Outdoor Static
• Indoor Static
Result Summary
Outline
Page 3 Commercial in confidence
Overview and Background
This document describes the user experience testing conducted in Barnsley Town Centre by AWTG on
17th and 18th October 2013. The aim of the exercise was to measure the performance of MNO’s 2G/3G
network within WiFi network’s boundary on user smart handsets and evaluating handover from 2G/3G
to WiFi and vice versa. It should be noted that the MNO did not have 4G coverage within the
assessment area at the time of testing. Table below shows overview of the tests:
Test
Connectivity
Smartphone 1 (S1) Smartphone 2 (S2) Smartphone 3 (S3)
WiFi Only 3G Only Roaming
2G/3G/WiFi
User Connectivity Test X X
Indoor Static Test
Outdoor Static Test
Outdoor Walk Test
Page 4 Commercial in confidence
Test Equipment: SwissQual QualiPoc Freerider
QualiPoc Freerider II consists of: • an Android master tablet • up to 6 Android slave Smartphones (3 Smartphones used for
the QoE measurement tests mentioned in this document) • GPS device.
AWTG has selected the SwissQual QualiPoc Freerider tool to carry out the tests in the area to capture user experience. MNO SIM cards installed in Samsung Galaxy SIII handsets have been used for these tests.
Page 5 Commercial in confidence
Test Overview
User Connectivity Tests
• Handover from 3G To WiFi
• Handover from WiFi To 3G
• Measuring latency and web browsing throughput
Synchronized 3G and WiFi tests in indoor and outdoor Static locations
• FTP UL/DL on 3G and WiFi
• Web browsing measurement on 3G and WiFi
• Latency measurement (RTT) on 3G and WiFi
• Online streaming measurement (Youtube) on 3G and WiFi
Outdoor Walk Test
• Running a test package (consisting FTP UL/DL, Web Browsing, Latency and Youtube Streaming) every 3 minutes while
engineer walks through the root in and out of the WiFi coverage.
• The route is selected in a way that network performance is measured inside and outside the WiFi coverage area. Please
refer to slide8 to view the test route.
Page 6 Commercial in confidence
Test Methodology - User Connectivity
Start and end points of the route
Walk Route
WiFi coverage boundary
The aim of this test is to measure the following:
• The effect of handover from 3G to WiFi when user enters WiFi zone
• The effect of handover from WiFi to 3G when user leaves WiFi zone
The following User Connectivity Test procedure have been followed by AWTG engineers:
• Enable WiFi on a smartphone configured with 3G Sim (while on the WiFi network edge with no WiFi coverage) and start Web Browsing.
• Move from a point in 3G coverage to another location within WiFi coverage.
• Latency (ping) tests were running while conducting the above steps.
Page 7 Commercial in confidence
Test Methodology - Outdoor Walk Test
3G-WiFi walk tests have been conducted on a route within the WiFi network coverage area. One engineer walked
through the route carrying SwissQual Freerider tool which consisted of three Smartphones running the following
test package every 3 minutes.
• Latency/Ping (maximum test duration of 10 sec)
• Web Browsing – www.bbc.co.uk– (maximum test duration of 30sec)
• FTP DL – 100MB file – (maximum test duration of 30sec)
• FTP UL – 100MB file – (maximum test duration of 30sec)
• YouTube Streaming – 30 second video- (maximum test duration of 1min)
Latency Web Browsing FTP DL FTP UL YouTube Streaming
10s 30 s 30s 30s 60s
180s 20s
wait
Test Cycle
Page 8 Commercial in confidence
Test Methodology - Outdoor Walk Test Route
3G-WiFi walk tests have been conducted on a route within
the WiFi network coverage area. One engineer walked
through the route carrying SwissQual Freerider tool which
consisted of three Smartphones running the test package
every three minutes.
The walk test route which engineers followed is shown on
the map. The route is selected in a way that 3G network
performance is measured inside and outside the WiFi
coverage area.
Walk Route
WiFi AP
WiFi Network Boundary
Page 9 Commercial in confidence
Test Methodology - Static Tests
3G-WiFi measurement static tests have been conducted in three fixed outdoor and two fixed indoor locations.
The tests ran on all three Smartphones and the following user experience measurements have been collected:
• Latency/Ping (maximum test duration of 10 sec)
• Web Browsing – www.bbc.co.uk– (maximum test duration of 30sec)
• FTP DL – 100MB file – (maximum test duration of 30sec)
• FTP UL – 100MB file – (maximum test duration of 30sec)
• YouTube Streaming – 30 second video- (maximum test duration of 1min)
10 cycles have been conducted at each location in order to achieve a certain confidence level for the investigated KPIs.
Page 10 Commercial in confidence
Test Results - User Connectivity S3 - Roaming 3G/WiFi
This section shows the handover between 3G and WiFi. The heat maps below show the user was switched to WiFi when entering the WiFi boundary which resulted in lower latency and improved web browsing performance. Users within the WiFi boundary experienced the following improvements:
• Lower RTT in WiFi Boundary • Higher HTTP Throughput. • Lower Time to Load HTTP Webpage
WiFi Boundary WiFi Boundary WiFi Boundary
Technology Heat-map Latency Heat-map HTTP Throughput Heat-map
Technology WiFi
3G
Page 11 Commercial in confidence
Test Results - User Connectivity S3 - Roaming 3G/WiFi
Route started outside WiFi area. Test engineer moved inside the WiFi boundary and walked back towards the start point.
Route started outside WiFi area. Test engineer moved inside the WiFi boundary and walked back towards the start point.
WiFi AP WiFi Network Boundary
Start point End Point
UE connected to WiFi from 3G when entering the WiFi coverage.
latency improved significantly when the UE connected to WiFi.
Page 12 Commercial in confidence
Test Results - Walk Tests 3G RSCP
WiFi Network Boundary
Average RSCP = -78.2 dBm
Minimum RSCP = -117.7 dBm
Maximum RSCP = -45 dBm
Standard Deviation = 11.25
Page 13 Commercial in confidence
Test Results - Walk Tests WiFi Signal Strength
WiFi AP WiFi Network Boundary
Average RSSI = -69.19 dBm
Minimum RSSI = -98 dBm
Maximum RSSI = -47 dBm
Standard Deviation = 8.46
Page 14 Commercial in confidence
Test Results - Walk Tests Technology Heat-map
This slide shows the technology which the roaming handsets were
connected during the walk test. The heat-map shows the user was
on WiFi majority of the time in the WiFi boundary.
• 86% of the time connected to WiFi
• 14% of the time connected to 3G
Roaming 3G/WiFi
Technology
WiFi
3G
WiFi Boundary
WiFi AP
Page 15 Commercial in confidence
Walk Test – Youtube Time to Load Comparison
WiFi 3G 3G/WiFi
S2 (3G only) Time to Start 30 sec Video
Average (sec) 7.03
Minimum (sec) 1.89
Maximum (sec) 26.45
Standard Deviation 6.01
S3 (Roaming ) Time to Start 30 sec Video
Average (sec) 3.14
Minimum (sec) 0.71
Maximum (sec) 16.90
Standard Deviation 3.22
S1 (WiFi only ) Time to Start 30 sec Video
Average (sec) 2.73
Minimum (sec) 0.75
Maximum (sec) 9.81
Standard Deviation 2.31
Page 16 Commercial in confidence
Walk Test - HTTP Time to Load the Webpage Comparison
S2 (3G only) Time to Load Webpage (URL: www.bbc.co.uk)
Average (sec) 7.77
Minimum (sec) 2.71
Maximum (sec) 16.89
Standard Deviation 2.82
S3 (Roaming ) Time to Load Webpage (URL: www.bbc.co.uk)
Average (sec) 5.51
Minimum (sec) 1.84
Maximum (sec) 16.98
Standard Deviation 3.36
S1 (WiFi only ) Time to Load Webpage (URL: www.bbc.co.uk)
Average (sec) 5.94
Minimum (sec) 1.38
Maximum (sec) 18.20
Standard Deviation 4.41
WiFi 3G 3G/WiFi
Page 17 Commercial in confidence
Walk Test - HTTP Throughput Comparison
S2 (3G only) HTTP Throughput (URL: www.bbc.co.uk)
Average (Kbps) 178.61
Minimum (Kbps) 74.06
Maximum (Kbps) 455.44
Standard Deviation 59.79
S3 (Roaming ) HTTP Throughput (URL: www.bbc.co.uk)
Average (Kbps) 315.49
Minimum (Kbps) 74.37
Maximum (Kbps) 688.41
Standard Deviation 171.77
S1 (WiFi only ) HTTP Throughput (URL: www.bbc.co.uk)
Average (Kbps) 281.81
Minimum (Kbps) 69.01
Maximum (Kbps) 902.80
Standard Deviation 196.60
WiFi 3G 3G/WiFi
Page 18 Commercial in confidence
Walk Test - FTP Download Comparison
WiFi 3G 3G/WiFi
S2 (3G only) FTP DL Throughput
Average (Mbps) 0.65
Minimum (Mbps) 0.0004
Maximum (Mbps) 2.05
Standard Deviation 0.47
S3 (Roaming ) FTP DL Throughput
Average (Mbps) 2.22
Minimum (Mbps) 0.01
Maximum (Mbps) 12.27
Standard Deviation 2.43
S1 (WiFi only ) FTP DL Throughput
Average (Mbps) 3.22
Minimum (Mbps) 0.09
Maximum (Mbps) 12.64
Standard Deviation 3.03
Page 19 Commercial in confidence
Walk Test - FTP Upload Comparison
S2 (3G only) FTP UL Throughput
Average (Mbps) 0.20
Minimum (Mbps) 0.07
Maximum (Mbps) 1.2
Standard Deviation 0.17
S3 (Roaming ) FTP UL Throughput
Average (Mbps) 1.87
Minimum (Mbps) 0.01
Maximum (Mbps) 12.98
Standard Deviation 2.77
S1 (WiFi only ) FTP UL Throughput
Average (Mbps) 2.80
Minimum (Mbps) 0.11
Maximum (Mbps) 23.69
Standard Deviation 3.85
WiFi 3G 3G/WiFi
Page 20 Commercial in confidence
Walk Test - FTP DL/UL Distribution Comparison
Page 21 Commercial in confidence
Test Results – Outdoor Static In Front of Barnsley Train Station
Test Location WiFi AP
Date Time Longitude Latitude
18/10/2013 13:30 -1.47857 53.55469
Tests were conducted outside Barnsley Train Station, 10 metres from WiFi AP connected to fibre. Tests were conducted during busy hours on Friday .
Page 22 Commercial in confidence
Test Results – Outdoor Static In Front of Morrisons
Test Location WiFi AP
Date Time Longitude Latitude
18/10/2013 14:30 -1.48002 53.54979
Tests were conducted outside Morrisons, 12 metres from a Root WiFi AP.
Page 23 Commercial in confidence
Test Results – Outdoor Static In the Corner of Regent Street & Church Street
Test Location WiFi AP
Date Time Longitude Latitude
18/10/2013 15:20 -1.48179 53.55427
Tests were conducted In the Corner of Regent Street & Church Street , 4 metres from WiFi AP 1 hop away from Root AP.
Page 24 Commercial in confidence
Test Results – Indoor Static Blah Bar
Date Time Longitude Latitude
18/10/2013 17:45 -1.48176 53.55341
Tests were conducted in Blah Bar, 35 metres from WiFi AP, 1 hop away from fibre root.
• Tests were conducted near window • Window type: double glazed
Test Location WiFi AP
Indoor Tests were conducted 5m from double glazed window
Page 25 Commercial in confidence
Test Results – Indoor Static Train Station
Test Location WiFi AP
Date Time Longitude Latitude
18/10/2013 18:15 -1.47813 53.55458
Tests were conducted in Subway inside Barnsley Train Station, 35 metres from WiFi root AP connected to fibre.
• Tests were conducted near window
• Window type: double glazed
Indoor Tests were conducted 4m from window
Page 26 Commercial in confidence
The following measurements on WiFi and 3G have been
conducted in two indoor and three fixed outdoor locations.
• FTP Download
• FTP upload
• Web Performance (www.bbc.co.uk)
• HTTP Throughput
• Time to Load the webpage
• Latency (Ping from handsets to FTP Server)
• Video Streaming (YouTube streaming)
Summary Results Indoor and Outdoor Static Tests
The Average static test results show better WiFi performance for FTP-DL/UL, Latency, YouTube Streaming and HTTP
web browsing.
FTP Tests
Operation Technology Avg Throughput [Mbps]
Outdoor Avg Throughput [Mbps]
Indoor
FTP (DL) WiFi 9.40 2.29
3G 1.08 0.71
FTP (UL) WiFi 7.76 2.02
3G 0.48 0.14
Web Browsing
Host Technology Avg Throughput [Kbps] Avg Time to Load (sec)
Outdoor Indoor Outdoor Indoor
http://www.bbc.co.uk WiFi 476.40 336.43 2.63 3.72
3G 259.91 155.84 6.35 10.75
YouTube Streaming
Technology Avg Time to Start Play (s)
Outdoor Avg Time to Start Play (s)
Indoor
WiFi 1.54 4.01
3G 11.79 13.53
Latency
Ping Host Technology Avg RTT [ms] Outdoor Avg RTT [ms] Indoor
FTP Server (217.138.0.114) WiFi 29.2 89.3
3G 229.0 274.5
Page 27 Commercial in confidence
Summary Results Indoor and Outdoor Static Tests
The following Pie Charts show the average of the tests conducted in three indoor and two fixed outdoor locations on WiFi and 3G networks.
Page 28 Commercial in confidence
Measurements have been conducted on three handsets:
• Smartphone1 : locked to WiFi
• Smartphone2 : locked to 3G
• Smartphone3 : Roaming 3G/WiFi (86% on WiFi & 14% on 3G)
Please note that in the majority of the tests (86% of test
duration), 2 handsets were conducting simultaneous tests on
WiFi while one handset was on 3G. The average results of the
walk test per handset are shown in the tables.
Summary Results Outdoor Walk Tests
The average walk test results show better WiFi Performance
for FTP DL/UL, Latency, HTTP Web browsing and YouTube
video streaming. Nevertheless, the handset roaming
between 3G and WiFi had improved performance compared
with the handset locked to 3G.
FTP Tests
Operation Technology Average Throughput (Mbps)
FTP (DL)
WiFi (S1) 3.22
3G (S2) 0.65
3G/WiFi (S3) 2.22
FTP (UL)
WiFi (S1) 2.8
3G (S2) 0.2
3G/WiFi (S3) 1.87
YouTube Streaming
Technology Average Time to Start Play (s)
WiFi (S1) 2.73
3G (S2) 7.03
3G/WiFi (S3) 3.14
HTTP
Host Technology Average Throughput
(Kbps) Average Time to Load (s)
http://www.bbc.co.uk
WiFi (S1) 281.81 5.94
3G (S2) 178.6 7.77
3G/WiFi (S3) 315.5 5.51
Latency
Technology Avg RTT (ms)
WiFi (S1) 47.37
3G (S2) 281.94
3G/WiFi (S3) 101
Commercial in confidence Page 29
Summary Results Outdoor Walk Tests
The following Pie Charts show the average results of the tests conducted on three handsets while walking through the rout.
Commercial in confidence Page 30
Registered in UK No. 5793772HQ in London UK
Office: +44 (0) 208 799 0368
Fax: +44 (0) 208 728 9610
www.awtg.co.uk
Thank You