Voice quality assessment

SW/HW migration and feature activation - Non-regression test plan -Anca BecaNE/ Voice Quality Expert

Alcatel-Lucent Romania

Table of Contents

31Scope

42Needed equipment

42.1Prerequisites

42.2General overview on Qvoice

52.3Speech sample classification thresholds

63Proposed tests

94Expected reported results

1 ScopeThe scope of this document is to define the non-regression test cases, needed equipment and time schedule for voice quality assessment at the introduction of new software release, new optional features or hardware enhancements that might impact the voice quality.

Other tests, that might be required to asses the improvement in voice quality, do not make the scope of this document.The tests shall perform MS to MS and MS to ISDN calls for EFR, FR, HR, AMR-FR and AMR-HR. If one or more of this codecs is not available in the network, the correspondent tests will be removed.For a fair comparison on field of the codec performance before and after the changes in the network (ex: feature activation), the tests have to be done as much as possible in the same radio conditions and at the same network load. For this, static tests are recommended to be performed. The static tests have to be performed both in good and bad radio conditions to evaluate the best and worst case scenario. As the voice quality in the BSS is affected not only by the radio conditions but also by handovers it is recommended to perform as well mobility tests. For the mobility tests, the same route should be drove, in the same conditions and in the same direction, for the measurements before and after feature activation.2 Needed equipment2.1 PrerequisitesIn order to carry on with the measurements the following points should be taken into consideration

The QVoice tool (from ASCOM) should be used, equipped with at least 2 test mobiles with forcing functions (speech version, band). QVM (for mobile to mobile calls) and QVS (for mobile to ISDN calls) are required

QVP for post-processing can be remotely

The frequency band has to be the same during the tests.

Available ISDN line at the MSC is required

Significant number of samples is necessary for reliable statistical distributions (1500 samples/mobility test case and 200 samples/static test case). The network should allow enabling EFR, HR, AMR FR, AMR HR The AMR parameters should be the ALU optimized:

Codec subset: 12.2, 10.2, 7.4 and 4.75

Thresholds: T1=4dB, T2=7dB, T3=10dB, Hyst=2dB

For the stationary tests, at least two points with good radio conditions and two with bad radio conditions have to be considered. For this, 2 different cells with normal load have to be chosen. The parameter settings in the cells chosen for these tests have to be the ones used in the most part of the network. The route for the mobility tests has to be defined (same route for before/after tests) including good and bad radio conditions, with reduced number of handovers (avoid ping-pong). The route has to de drove in the same direction with max. 50km/h. Drive test car needed. NPO access needed, for the monitoring of the network KPIs in the tested area, during the tests, to see any abnormal behavior of the network that might impact the results of the tests. The reference tests and the tests after feature activation or sw/hw migration have to be performed in the same traffic conditions, same time of the day (load conditions) and exactly on the same route (for the mobility tests).

2.2 General overview on Qvoice

The QVoice consists of two modules (QVM & QVS). The QVM (QVoice Mobile unit) is the mobile measurement tool installed in the car and QVS (QVoice Server unit) is the stationary one existing on the ISDN side.The principle of operation is the following: the "QVM" (or "QVS" according to the direction, UL/DL, we want to measure) sends an audio pattern in a chosen language (English, French, German or Cantonese). The "QVS" (or "QVM") receives the pattern, compares it to the reference and calculates a note, under the PESQ metric. 5 seconds speech samples in different languages are transmitted in both directions.

After the measurements are terminated, log files (*.QMD) from both modules are imported in the QVoice Presentation station in order to perform analysis and produces reports and maps.

For MS-MS measurements, the QVM is both the transmitter and the receiver of the audio pattern, as the ISDN/PSTN network is not involved in the call. The call is establish between two mobiles located at the same QVM.

Figure 1: Different modules of QVoice tool for MS-ISDN measurementThe Principle of all end-to-end Voice Quality tools is the following:

1. A reference speech sample IN.

2. A degraded speech sample OUT.

3. The comparison between them giving a score.

Figure 2: Principles of Voice quality tools2.3 Speech sample classification thresholds

In the following table the ranges of the PESQ and corresponding classification are presented.

Speech QualityPESQSpeech Sample

FromTo

Excellent53.5

Good3.52.7

Fair2.72.1

Poor2.11.5

Bad1.51

Table 1: Speech sample classification 3 Proposed testsThe measurement test cases should be the ones in the following table. Each of the tests proposed bellow should be made before and after the sw/hw migration or feature activation. Also, the results of the tests have to be sent by the measurement team after each day, so that we can find in time some problems that might appear and correct them.Time

Task

1 daySetting up equipment (QVM + QVS), testing equipment, validating proposed drive route, administrative issues

1 day

(200 samples ~ 20 min)

(~ 4h with 4 mobiles)Static measurements in two selected cells in good radio conditions (RxLev >= -60dBm, RxQual 0):

Mobile to ISDN: FR ISDN (200 samples/cells) EFR ISDN (200 samples/cell)

HR ISDN (200 samples/cell)

AMR-FR ISDN (200 samples/cell)

AMR-HR ISDN (200 samples/cell)Mobile to mobile: FR FR (200 samples/cell)

EFR EFR (200 samples/cell) HR HR (200 samples/cell)

AMR-FR AMR-FR (200 samples/cell)

AMR-HR AMR-HR (200 samples/cell)

1 day

(~ 4h with 4 mobiles)Static measurements in two selected cells in bad radio conditions (-90