Upload
shirjeel31
View
221
Download
0
Embed Size (px)
Citation preview
8/19/2019 Ascom Unattended Trial Report Dubai
1/22
UADT Report
December 2013
Page 1
Unattended drive test trial reportME GD2.0 NPO Lever
Trial country – UAETrial network – du
Trial dates – 25th, 26th & 27th Nov 2013
Trial conducted in collaboration between Nasir Mahmood – NPO expert Sean Ettehadieh – NPO expert Usman Sharif – NPO expert Ashish Tandon – ME GD transition lead Faisal Ghazaleh – ASCOM Technical expert
Malick Noor –
ASCOM AM
Report prepared by Nasir Mahmood – NPO expert
Report reviewed by Sean Ettehadieh – NPO expert Usman Sharif – NPO expert Ashish Tandon – ME GD transition lead
Company confidentialNot to be distributed further without consent of the author & / or reviewers
8/19/2019 Ascom Unattended Trial Report Dubai
2/22
Page 2
TABLE of Contents
1 Purpose................................................................................ 3
2 Executive Summary ............................................................. 4
3 Trial scope & setup .............................................................. 5
3.1 Scope of trial .................................................................................................. 5 3.2 Drive test sites ............................................................................................... 5 3.3 Equipment ..................................................................................................... 5 3.4 Setup ............................................................................................................. 5 3.5 Drive test route planning ................................................................................ 6
4 Test results .......................................................................... 7
4.1 Compliancy score rating matrix ...................................................................... 7 4.2 2G test results ............................................................................................... 8 4.2.1 Major observations ................................................................................... 9 4.2.2 Short term & permanent solutions for overcoming shortcomings .............. 9 4.2.3 Snapshot of test ....................................................................................... 9 4.3 3G test results ..............................................................................................10 4.3.1 Major observations ..................................................................................11 4.3.2 Short term & permanent solutions for overcoming shortcomings .............11 4.3.3 Snapshot of test ......................................................................................11
4.4 LTE test results .............................................................................................12 4.4.1 Major observations ..................................................................................13 4.4.2 Short term & permanent solutions for overcoming shortcomings .............13 4.4.3 Snapshot of test ......................................................................................13 4.5 Speech Quality test results ...........................................................................14 4.6 Tool automation test results ..........................................................................14 4.7 Trial results from Saudi trial ..........................................................................15 4.7.1 Major observations ..................................................................................16 4.7.2 Short term & permanent solutions for overcoming shortcomings .............16
5 TEMS Automated Road-map ............................................. 17
6 Frequently asked questions ............................................... 18
7 Glossary of Technical Terms .............................................. 21
8/19/2019 Ascom Unattended Trial Report Dubai
3/22
Page 3
1 Purpose
This document serves to present an impartial view of trial results that have been conducted extensively (on
an existing site and on a new site) in UAE on the du network using ASCOM TEMS automated solution and
to showcase the proof of concept for automation in drive testing i.e. drive testing without a technician in the
car.
The report also aims to provide the local delivery teams all the relevant information in order to be fully
familiar with some shortcomings in automation of drive testing and potential short term workarounds and
long term solutions envisaged.
An overall technical score based on compliancy per tested scenario is also presented in the document only
for the purpose of benchmarking with the NEMO solution.
8/19/2019 Ascom Unattended Trial Report Dubai
4/22
Page 4
2 Executive Summary
On the basis of the trial conducted involving region NPO experts & observations recorded; it can be
confirmed with reasonable confidence that TEMS automated solution is currently capable of fulfilling upto
76% of the project requirements for pre launch (new sites) DT scope
&
85% of the project requirements for post launch (routine & benchmarking) DT scope (refer
Saudi report in section 4.7)
It is important to note that there are current shortcomings of the tool as outlined below:
TD-LTE and CSFB
Instant live monitoring and Locking on target cell
MMS
However, the above shortcomings do not pose a serious challenge in respect of rolling out the solution
widely due to the proposed workarounds (see explanations in the subsequent sections) and permanent
solution already envisaged during Q1 and Q2 of 2014 (see relevant 5 on ASCOM roadmap)
Advantages
High efficiency gains in overall drive test process
o In the course of the trial, one of the clearly visible advantages of the solution was the quick
turnaround time starting from drive testing & ending with generating a drive test report.
During the drive tests, the raw data was received within 10mins (server based in Sweden) of
the start and a report was prepared within minutes thereafter using standardized templates
in TEMS discovery
New features & value add to customer
o TEMS Automated provides additional features related to “Speech Quality Assessment”
using PESQ / POLQA which can be used to detect echo delay, attenuation volume levels,
silent calls etc. TA will be a value add to our customers as it can be used for managing
special events like Hajj & Ramadan, World Cup, Expo etc by getting live feeds about
network health and KPIs as perceived by subscribers.
On the basis of the above observations, on a scale of 0 to 5, TEMS automated secured an overall technical
score of 3.81 for pre launch scope and 4.27 for post launch scope.
Scoring methodology & matrix is presented in the subsequent sections of this trial report.
8/19/2019 Ascom Unattended Trial Report Dubai
5/22
Page 5
3 Trial scope & setup
3.1 Scope of trial
Drive test on a newly commissioned site & an existing site to simulate tests related to pre-
optimization, site rehomings, expansions & swap.
3.2 Drive test sites
Network – du
o DXB 1996 – existing site – 3G & LTE
o DXB 2349 – new site on air – 2G
3.3 Equipment
Hardware: RTU-Version 5 with three active probes and one idle probe
Software: 9.2.0
Voice Card: f3607
Data Card: MC7710
SIM Card: 3 SIMs (2G, 3G, LTE enabled) with 5GB, 5GB & 25GB data plan & 300voice min
3.4 Setup
B1 – ASCOM performed the drive tests with driver only
B2 – DT data was loaded automatically in the backend servers in Sweden
B3 – ASCOM engg controlled the DT remotely from the NSN Dubai office
B4 – DT log files were pulled via FTP & ASCOM engg showcased reporting & analysis
B5 – DT reports & analysis capabilities assessed by NSN NPO experts for technical compliancy
8/19/2019 Ascom Unattended Trial Report Dubai
6/22
Page 6
3.5 Drive test route planning
Drive test route for both the sites DXB 1996 & DXB 2349 were clearly marked on a printout and
handed over to the driver Instructions were given to the driver to start driving from a particular location marked on the printout
on the pre defined route
Upon reaching the site, fleet manager (ASCOM engg) and the car driver maintained contact on
phone whenever it was needed
Additionally, the driver was a given an android based smart phone (GPS and Mobile data enabled)
with pre-load app called “LIFE360” and the location of the driver was monitored LIVE from the NSN
office using the same smart phone app installed an another android based device.
Figure 1: Drive test route for site DXB 2349 Figure 2: Drive test route for site DXB 1996
Figure 3: Life 360 app for monitoring driver location
8/19/2019 Ascom Unattended Trial Report Dubai
7/22
Page 7
4 Test results
This section summarizes the drive test results per technology domain for which the trial was conducted.
Each test has been rated and given a compliance score as per methodology explained below.
The overall technical score awarded to the TA as mentioned in the executive summary chapter was an
average of the score awarded per feature test.
For each test availability of a short term solution (temporary workaround) and a permanent solution
(solution as per ASCOM roadmap) has also been highlighted.
4.1 Compliancy score rating matrix
Score Abbreviation Methodology
0 Non compliant (NC) Non compliant
1 Partially compliant (PC) Available short term solution = No
Expected permanent solution on ASCOM roadmap = No
2 Partially compliant (PC) Available short term solution = Yes
Expected permanent solution on ASCOM roadmap = No
3 Partially compliant (PC) Available short term solution = No
Expected permanent solution on ASCOM roadmap = Yes
4* Partially compliant (PC) Available short term solution = Yes
Expected permanent solution on ASCOM roadmap = Yes
Will Stay as PC even with permanent solution based on manual
efforts needed to perform the test
4 Partially compliant (PC) Available short term solution = Yes
Expected permanent solution on ASCOM roadmap = Yes
5 Compliant Fully compliant
8/19/2019 Ascom Unattended Trial Report Dubai
8/22
UADT Report
December 2013
Page 8
4.2 2G test results
2G trial was conducted on site DXB2439 for validation of various testing scenarios related to pre-optimization of a newly commissioned 2G site. These tests
are also valid for site rehomings, TRX expansions and swaps. Results of this trial are summarized in table below:
Test
scenario
Scenario detail Compliance Short term solution
available
Permanent solution available Score
1 Idle mode coverage test for Rx level & footprint PC Yes Yes (2Q 2014) 4
2 Dedicated Mode Long Call & Short for checking Rx-Lev, Rx-Qual and
KPIs
PC Yes Yes (2Q 2014) 4
3 TRX Testing to find out faulty TRXs PC Yes Yes (2Q 2014) 4*
4 SMS Test PC Yes Yes (2Q 2014) 4
5 MMS Test NC No No 0
6 Sector Swap (Cross Feeder) Test PC Yes Yes (2Q 2014) 4*
7 Hand-over Tests PC Yes Yes (2Q 2014) 4
8 Data Testing (Ping, Throughput & Data KPIs) PC Yes Yes (2Q 2014) 4
9 Data Testing (DL Static Testing) PC Yes Yes (2Q 2014) 4
8/19/2019 Ascom Unattended Trial Report Dubai
9/22
UADT Report
December 2013
Page 9
4.2.1 Major observations
1. TA Solution is capable of capturing information related to idle mode, dedicated mode (short & long
Calls), SMS, handover and data tests.
2. TA lacks capability of camping on a cell and visbility about current location, serving cell and RF
conditions which increase the probability of deviation from targeted area of test-drive & pose a
challenge for certain tests which require precise information about prevailing radio conditions.
3. TA is not capable of performing TRX and sector swap tests as per routines used by project teams due
to lack of ability to lock on targeted Cell. However, these tests could be performed with TA solution
using indirect ways and correlation of information captured during other tests.
4. TA Solution is not capable of performing MMS tests.
4.2.2 Short term & permanent solutions for overcoming shortcomings
Short-Term Solution: The challenges described above could be managed by using a TPR with
pre-fed drive-map and team-viewer application to provide the required visibility and control to central
fleet technician. This workaround is envisaged to be secured via commercial negotiations with
ASCOM
Permanent Solution: ASCOM plans to embed a feature (see roadmap section) in the TEMS
automated that would be capable of providing the basic on line cell information to the fleet manager
remotely before (& during) drive test.
4.2.3 Snapshot of test
RxLev
RxQual Foot-Print Data
8/19/2019 Ascom Unattended Trial Report Dubai
10/22
UADT Report
December 2013
Page 10
4.3 3G test results
3G trial was conducted on an operational site DXB1996 for validation of various testing scenarios related to pre-optimization of a newly commissioned 3G site.
These tests are also valid for site rehomings and swaps. Results of this trial are summarized in table below:
Test
scenario
Scenario detail Compliance Short term solution
available
Permanent solution available Score
10 Idle Mode for CPICH RSCP, EcNo & Serving SC PC Yes Yes (2Q 2014) 4
11 Active Mode for Voice Call CPICH RSCP, EcNo & Serving SC PC Yes Yes (2Q 2014) 4
12 Hand-over Tests PC Yes Yes (2Q 2014) 4
13 Sector Swap(Cross Feeder) Test PC Yes Yes (2Q 2014) 4*
14 SMS Test PC Yes Yes (2Q 2014) 4
15 MMS Test NC No No 0
16 Data UL/DL Test for Rel 99 PC Yes Yes (2Q 2014) 4
17 Data UL/DL Test for HSPA PC Yes Yes (2Q 2014) 4
18 Data Testing (DL Static Testing) PC Yes Yes (2Q 2014) 4
19 2G to 3G re-selection PC Yes (2Q 2014) 4*
20 3G to 2G HO (IRAT) PC Yes (2Q 2014) 4
8/19/2019 Ascom Unattended Trial Report Dubai
11/22
UADT Report
December 2013
Page 11
4.3.1 Major observations
1. TA Solution is capable of capturing information related to idle mode, active mode (short & long Calls),
SMS, handover and data tests.
2. TA lacks capability of camping on a cell and visbility about about current location, serving cell and RF
conditions which increases the probability of deviation from targeted area of test-drive & poses a
challenge for certain tests like static data DL/UL which require precise information about prevailing radio
conditions.
3. TA is not capable of performing sector swap test as per routines used by project teams due to lack of
ability to lock on targeted SC. However, this test could be performed with TA solution using indirect
ways and correlation of information captured during other tests.
4. TA Solution is not capable of performing MMS tests.
4.3.2 Short term & permanent solutions for overcoming shortcomings
Short-Term Solution: The challenges described above could be managed by using a TPR with
pre-fed drive-map and team-viewer application to provide the required visibility and control to central
fleet technician. This workaround is envisaged to be secured via commercial negotiations with
ASCOM
Permanent Solution: ASCOM plans to embed a feature (see roadmap section) in the TEMS
automated that would be capable of providing the basic on line cell information to the fleet manager
remotely before (& during) drive test.
4.3.3 Snapshot of test
CPICH RSCP CPICH EcNo Best Server SC
8/19/2019 Ascom Unattended Trial Report Dubai
12/22
UADT Report
December 2013
Page 12
4.4 LTE test results
LTE trial was conducted on an operational site DXB2439 for validation of various testing scenarios related to pre-optimization of a newly commissioned LTE
site. These tests are also valid for site rehomings and swaps. Results of this trial are summarized in table below:
Test
scenario
Scenario detail Compliance Short term solution
available
Permanent solution available Score
21 Idle Mode PCI, RSRP, RSRQ, SINR PC Yes Yes (2Q 2014) 4
22 Connected Mode PCI, RSRP, RSRQ, SINR, CQI, RI & DL Throughput PC Yes Yes (2Q 2014) 4
23 Hand-over Tests PC Yes Yes (2Q 2014) 4
24 Sector Swap Tests PC Yes Yes (2Q 2014) 4*
25 SMS Test PC Yes Yes (2Q 2014) 4
26 MMS Test NC No No 0
27 Ping Test PC Yes Yes (2Q 2014) 4
28 DL & UL throughput in static mode PC Yes Yes (2Q 2014) 4
29 CSFB (Circuit Switched Fall Back) PC No Yes (1Q 2014) 3
30 3G to LTE re-selection PC Yes Yes (2Q 2014) 4*
8/19/2019 Ascom Unattended Trial Report Dubai
13/22
UADT Report
December 2013
Page 13
4.4.1 Major observations
1. TA Solution is capable of capturing information related to idle mode, connected mode, SMS, handover
and data tests.
2. TA lacks capability of camping on a cell and visbility about current location, serving cell and RF
conditions which increases the probability of deviation from targeted area of test-drive & poses a
challenge for certain tests like which require precise information about prevailing radio conditions.
3. TA is not capable of performing sector swap tests as per routines used by project teams due to lack of
abaility to lock on targetted PCI. However these tests could be performed with TA using indirect ways
and co-relation of information captured during other tests.
4. TA Solution is not capable of performing CSFB and MMS tests.
4.4.2 Short term & permanent solutions for overcoming shortcomings
Short-Term Solution: The challenges described above could be managed by using a TPR with
pre-fed drive-map and team-viewer application to provide the required visibility and control to central
fleet technician. This workaround is envisaged to be secured via commercial negotiations with
ASCOM
Permanent Solution: ASCOM plans to embed a feature (see roadmap section) in the TEMS
automated that would be capable of providing the basic on line cell information to the fleet manager
remotely before (& during) drive test. CSFB is also part of official road-map from ASCOM (see road-
map section).
4.4.3 Snapshot of test
RxLev
RxQual SINR Data
8/19/2019 Ascom Unattended Trial Report Dubai
14/22
Page 14
4.5 Speech Quality test results
The functionality related to Speech Quality Testing is not available in traditional drive test tools. It was
tested during trial and results of evaluation are summarized in table below:
Test
scenario
Scenario detail Compliance Short term
solution available
Permanent solution
available
Score
31 Additional – PESQ/POLQA
Quality of Voice Testing on
2G/3G
PC Yes Yes (2Q 2014) 4
4.6 Tool automation test results
In addition to the technical tests, TA was also evaluated for its capabilities on automation of drive test.
Results of this evaluation are summarized in table below:
Feature
scenario
Scenario detail Compliance Score
32 Transfer of log files while the drive test is ongoing C 5
33 Viewing and processing of log files while the drive test is ongoing C 5
34 Automated processing and DT report generation by TEMS discovery C 5
35 Defining automated routines via Fleet Manager for New Sites On-Air C 5
36 Preparing and downloading scripts to TA while the drive test is ongoing C 5
37 Ability to run multiple technology drive tests within one drive C 5
8/19/2019 Ascom Unattended Trial Report Dubai
15/22
UADT Report
December 2013
Page 15
4.7 Trial results from Saudi trial
In addition to the trial that was conducted extensively in UAE, a trial was also conducted in Saudi in P9 2013, focusing primarily on routine optimization and
benchmarking related drive tests. Results of this evaluation are summarized in table below:
Test
scenarioTechnology Operator Test Cases Compliance
Permanent
Solution
Score
1 2G, 3G, LTE
STC,
Mobily,
Zain
STC-2G Idle Mobily-2G IdleZain-2G
Idle
STC-LTE
Idle
PC
Due to TD-LTE
Yes
(1Q 2014)
3
2 2G, 3G, LTE
STC,
Mobily,
Zain
STC-3G
Short Call
Mobily-3G Short
Call
STC-3G
FTP DL/UL
Zain-3G
FTP DL/UL
PC
Due to TD-LTE
Yes
(1Q 2014)
3
3 2G, 3G, LTE
STC,
Mobily,
Zain
STC-Dual
Mode Long
Call
Mobily-Dual
Mode Long Call
STC-LTE
FTP DL/UL
Zain-LTE
FTP DL/UL
PC
Due to TD-LTE
Yes(1Q 2014)
3
4 2G,LTESTC,
Mobily,
Zain
STC-2G
Short Call
STC-2G Long
Call
STC-LTE
Ping
Mobily-LTE
Ping
PC
Due to TD-LTE
Yes(1Q 2014)
3
The evaluation score related to speech quality tests and tool automation features listed in above sections are also valid for post launch testing for routine
optimization and benchmarking activities.
8/19/2019 Ascom Unattended Trial Report Dubai
16/22
UADT Report
December 2013
Page 16
4.7.1 Major observations
1. TA Solution is capable of peforming all tests related to routine optimization and benchmarking actvities
for 2G, 3G, and FDD-LTE.
2. TA Solution is not capable of conducting any tests on TD-LTE.
3. TA Solution is not capable of performing CSFB tests.
4.7.2 Short term & permanent solutions for overcoming shortcomings
The major shortcomings reported during trial are already covered in official road-map provided by ASCOM.
The support for TD-LTE and CSFB will be available during 1Q of 2014.
8/19/2019 Ascom Unattended Trial Report Dubai
17/22
UADT Report
December 2013
Page 17
5 TEMS Automated Road-map
Ascom has shared an official road-map confirming support for major shortcomings including TD-LTE,
CSFB, cell locking and real time monitoring of probes.
It is expected that these features will get embedded in TEMS automated during Q1 and Q2 of 2014 which
will drive the TEMS automated solution to a higher compliancy and ease of use in our NPO projects – pre
and post launch.
Details of the roadmap from ASCOM are attached below.
Road Map TEMS Automatic.pdf
8/19/2019 Ascom Unattended Trial Report Dubai
18/22
Page 18
6 Frequently asked questions
1. Is it possible to simulate ‘in-car’ and ‘outdoor’ test scenarios using TEMS Automated?
Answer : Yes, it is possible to use TEMS Automated in both modes. TEMS Automated comes with ‘indoor’
and ‘outdoor’ antennas which can be used for ‘in-car’ and ‘outdoor’ tests based on project needs.
2. How are the log files transmitted over air during drive-test?
Answer: There are multiple options provisioned in TEMS Automated for transmission of log files during
drive test. Fleet technician can select any technology (i.e., GPRS, EDGE, 3G, HSPA or LTE) on one of the
chipsets of TEMS Automated and trigger automatic transmission of log files. It is possible to change the
technology as well as chipset during course of drive test. It is also possible to use a chipset in multi-RAB
mode in case all four chipsets are being used for testing.
3. Is there any interruption to the measurement collection procedure during transmission of log files?
Answer : No, there is no interruption to any measurement collection procedure. TEMS Automated uses a
technique of splitting log files in smaller files (~500 KB) depending upon type of test in progress (i.e., short
call, long Call etc). These files are transmitted over air at regular intervals using technology chosen by fleet
technician during course of drive test.
4. Will there be any impact on data and voice tests of targeted site(s) since TEMS Automated uses same
serving cell for uploading log-files during drive test?
Answer : Yes, there could be a direct or indirect impact of continuous uploading of log files using same
serving cell. It is strongly recommended to use alternative technology or alternative operator for
transmission of log files to avoid any impact on quality of information being collected from the targeted
site(s).
5. Is it possible to connect an external scanner with TEMS Automated?
Answer : Yes, TEMS Automated is provisioned with external ports for connection of scanners (like PCTEL)
for conducting all kinds of tests related to scanning.
6. Are the chipsets provisioned in TEMS Automated capable of conducting voice and data tests on 2G,
3G, HSPA and LTE?
Answer : Yes, all chipsets are capable of 2G, 3G, HSPA and FDD-LTE. However, the capability related to
data testing on HSPA and LTE is limited to 7.2Mbps. So, each TEMS Automated unit is equipped with one
8/19/2019 Ascom Unattended Trial Report Dubai
19/22
Page 19
or more ‘category 3’ data-chipsets capable of data testing up to 100 Mbps. However it is expected that
ASCOM will introduce a new chipset MC7304 in Q1 2014 which will be capable of voice and data testing on
2G, 3G & LTE upto 100Mbps. Please also see the roadmap of ASCOM for further details on this aspect.
7. Is TEMS Automated capable of conducting tests for TD LTE?
Answer : No, TEMS Automated is not capable of perfuming any tests for TD-LTE. However, this
functionality is envisaged in TEMS Automated by Q1 2014.
8. Is it possible to retrieve log files from TEMS Automated in case of interruption in connectivity with back-
end server?
Answer : Yes, it is possible to retrieve log files from TEMS Automated by connecting it directly to a laptop.
9. Is it possible to enhance the configuration of TEMS Automated beyond 4 chipsets for benchmarking of
multiple operators for multiple technologies?
Answer : Yes, it is possible to connect multiple TEMS Automated units using ‘daisy-chain’ technique to
enhance TA configuration from four to eight and max upto twelve based on project needs.
10. Does TEMS Automated require special power arrangements inside the car?
Answer : No, there are no special requirements for TEMS Automated. A 12V socket in the car is sufficient
to keep TEMS Automated powered-up during drive test.
11. Does TEMS Automated require periodic calibration like other testing devices?
Answer : No, there is no need for periodic calibration of TEMS Automated. Ascom will provide after-sales
maintenance support in case any abnormal behavior is observed during measurement collection.
12. What are the technical specifications of data-chipset ‘MC7710’ used in TEMS Automated?
Answer : The technical specifications of MC7710 are exactly same as a category-3 device. It is capable of
testing up to 100Mbps.
13. Is it possible to reconfigure and change testing scenarios based on observations recorded during the
drive-test?
Answer : Yes, it is extremely easy to change test scenarios while drive-test is in progress. The entire
process takes few minutes and test routines are changed on-the-run.
8/19/2019 Ascom Unattended Trial Report Dubai
20/22
Page 20
14. Do we need a special server with fixed-line connectivity for performing ‘speech quality test’ using
PESQ or POLQA techniques?
Answer : No, there is no need for any special server or fixed line connectivity. TEMS Automated performs
‘speech quality tests’ using two voice chipsets of the same unit.
8/19/2019 Ascom Unattended Trial Report Dubai
21/22
Page 21
7 Glossary of Technical Terms
1. TEMS Test Equipment Mobile Station
2. TA TEMS Automated
3. TPR TEMS Pocket Remote
4. PESQ Perceptual Evaluation of Speech Quality
5. POLQA Perceptual Objective Listening Quality Assessment
6. PCI Physical Cell IDs
7. CSFB Circuit Switched Fall Back
8. FTP File Transfer Protocol
9. SMS Short Messaging Service
10. MMS Multimedia Messaging Service
8/19/2019 Ascom Unattended Trial Report Dubai
22/22
Nokia Solutions and NetworksP.O. Box 1FI-02022Finland
Visiting address:Karaportti 3, ESPOO, FinlandSwitchboard +358 71 400 4000
Product code C401-00649-B-201008-2-EN
©2013 Nokia Solutions and Networks. All rights reserved.
PublicNSN is a trademark of Nokia Solutions and Networks. Nokia is a registeredtrademark of Nokia Corporation. Other product names mentioned in thisdocument may be trademarks of their respective owners, and they arementioned for identification purposes only.