Benchmarked Key Performance Indicators

Benchmarked Key Performance Indicators

Crisphine J. Ogongo,Engineer/Compliance

Communications Commission of Kenya

ITU-T Workshop on Delivering Good Quality Telecommunication Service

in a Safe Environment in Africa (Nairobi, Kenya, 26th July 2010)

Nairobi - Kenya 26 - 27 July 2010

Content

ITU/ETSI StandardsQoS Regulation And Comparison With Int’l StdsConclusionQuestions


ITU-T/ETSI STANDARDS


Definition

Quality of service (QoS): The collective effect of service performances, which determine the degree of satisfaction of a user of the service (ITU-T E.800)


What QoS Really Is!Network Performance Parameters translated to;

system design, configuration, operation and maintenance

Quality of service parameter influenced by; Statistics e.g «Call Block Rate»

User/customer requirementsIndividual experience e.g «inaccessibility»-

User opinion/requirements feed back in to the network planning process to alter planned performance and/or practical operational standards


ITU-T QOS Parameters with Targets

Call Set Up Time/Post Dialing DelayCall Release DelayAnswer Signal DelayEnd to End BlockingHandover Success Rate/Unsuccessful HandoverSpeech Quality Multimedia QoS


Call Set up timeCall type F-M M-F M-M Definition

Authentication/ciphering

0.0 2.5 2.5

Time interval between the end of dialing by the user and the reception by him of the appropriate tone or recorded announcement, or the abandon of the call without tone.

Paging/alerting 4.0 0.0 4.0

Routing number transfer 2.0 0.0 2.0

Post-selection delay

Local connection 3.0 3.0 3.0

Toll connection 5.0 5.0 5.0

International connection 8.0 8.0 8.0

TOTAL PLMN PSTN

F-M M-F M-M Normal Load High Load

Mean 95% Mean 95%

Local connection 9 5.5 11.5 3 s 6 s 4.5 s 9 s

Toll connection 11 7.5 13.5 5s 8 s 7.5 s 12 s

International connection 14 10.5 16.5 8 s 11 s 12 s 16.5 s


Answer Signal Delay

PLMN Connection Definition

Call type F-M M-F M-M Time interval between the establishment of a connection between calling and called users, and the detection of an answersignal at the originating exchange.

Local connection 1.0 1.0 1.25

Toll connection 1.75 1.75 2.0

International connection 2.75 2.75 3.0

PSTN

Normal Load High Load

Mean 95% Mean 95%

Local connection 0.75 sec 1.5 sec 1.0 sec 2.0 sec

Toll connection 1.5 sec 3.0 sec 2.0 sec 4.0 sec

International connection 2.0 sec 5.0 sec 3.3 sec 6.5 sec


Call Release Delay

Call release delay (secs)

Connection Type

Call type F-M, M-F, M-M PSTN

Calling, or called, party clears

1.0 sec 0.4 – 1sec

Call release delay is defined as the time interval from the instant the first bit of the DISCONNECT message is passed by the user terminal which terminated the call to the access signaling system, until the last bit of the RELEASE message is received by the same terminal (indicating that the terminals can initiate/receive a new call).

Definition


End to End Blocking

1990 Improvement

PLMNOn radio Channel 5-10% 1%

PLMN to Fixed 1% 0.5%

Normal Load High Load

PSTNLocal 2% 3%

Toll Connection 3% 4.5%

Int’l Connection 5% 7.5%

Definition: - The probability that any call attempt will be unsuccessful due to a lack of network resources


Unsuccessful Handovers

Call type F-M, M-F, M-M

Probability of unsuccessful land cellular handover

0.5%

Is the probability that a handover attempt fails because of lack of radio resources in the target cell, or because of a lack of free resources for establishing the new network connection. The failure condition is based either on a specified time interval since the handover request was first issued or on a threshold on signal strength


Speech Quality

PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs PESQ compares an original signal X(t) with a degraded signal Y(t) that is the result of passing X(t) through a communications system. The output of PESQ is a prediction of the perceived quality that would be given to Y(t) by subjects in a subjective listening test(6).Provides Speech Quality raw values -0.5 to 4.5


Speech Quality

Subjects in

Listening Test

Original Signal X(t)Perceived

Speech quality

PESQ: - Perceptual evaluation of speech quality ITU-T P.862 An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs


Speech Quality cont’dITU-T P.862.1 - Mapping function for transforming P.862 raw result scores to MOS

The mapping ensures a domain rescaling from –0.5 ... 4.5 to 1.02 ... 4.56

P.862.1_F1

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

– 1 0 1 2 3 4 5P.862

Map

ped

P.86

2


Speech Quality Cont’d

Speech quality depends on Transmission Rating Factor RThe R‑value is a measure of a quality perception to be expected by the average user when communicating via the connection under consideration

5060708090100T1211030-99

linear quality scale

high quality

mediumquality

low quality

Area notrecommended

OverallRating "R"


Speech Quality Cont’d

R-value rangeEquivalent MOS Values

Speech transmission quality category

User satisfaction

90 R < 100 5 Excellent Very satisfied80 R < 90 4 Good Satisfied70 R < 80 3 Fair Some users dissatisfied60 R < 70 2 Bad Many users dissatisfied50 R < 60 1 Poor Nearly all users dissatisfiedConversion of R‑values into MOS

MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶

Transmission Rating Factor conversion to MOS equivalent


MOS User Satisfaction Levels

R-Value

MOS Value

Comment Country

R ≤ 0 1 Nearly all users dissatisfied

R = 55 2.835 Many users dissatisfied Kenya

R = 60 3.1 Some users dissatisfied Nigeria/Kenya in 3yrs

R = 70 3.597 Some users dissatisfied

R = 80 4.024 Satisfied

R ≥ 100

4.5 Very satisfied Satisfied

MOS = MOS = 1+0.035R + R(R-60)(100-R)7.10⁻⁶

Typical values used in


Data/Multimedia Parameters Critical to Users

Delay variationsOccurs at the transport layer in packetized data systems due to the inherent variable arrival times of individual packets

Solved through buffering

Delaythe time taken to establish a particular service from the initial user request and the time to receive specific information once the service is established.

Packet LossOf packets or bits during transmissionIncludes coding degradation


Multimedia (Audio and Video) QoS Parameters and targets

KPI

1-way delay

Delay Variation

Information Loss (Packet Loss Ratio-PLR)

Audio Conversational Voice <150 ms < 1 ms < 3%

Audio Voice Messaging < 1 s playback

< 1 ms < 3%

High Quality Streaming Audio < 10 s << 1 ms < 1%

Videophone (2-way) < 150 ms < 1%

Videophone (1-way) < 10 s < 1%


Multimedia QoS Parameters

KPI Symmetry T-Values Delay

Web-browsing– HTML

1-way ~10 KB P< 2 s /pageA < 4 s /page

No delay variation or information Loss

KeyP-preferredA-AcceptableT-typical

Bulk data transfer/retrieval

1-way 10 KB-10 MB P < 15 sA < 60 s

Transaction services – high priority e.g. e‑commerce

2-way < 10 KB P < 2 s A< 4 s

Command/control 2-way ~ 1 KB < 250 ms

Still image 1-way < 100 KB P < 15 sA < 60 s

Interactive games 2-way < 1 KB < 200 ms

Telnet 2-way < 1 KB < 200 ms

E-mail (server access) 1-way < 10 KB P< 2 s A< 4 s

E-mail (server to server transfer)

1-way < 10 KB Can be severalminutes


QOS REGULATION IN AFRICA AND COMPARISON WITH INTERNATIONAL

STANDARDS

Case studiesKenyaNigeriaUgandaTanzania


Status of Country QoS Regulation

Nigeria Tanzania Uganda Kenya

QoS regulated √ √ √ √

QoS Parameters and Targets √ √ √ √

Regulations x √ x x

Guidelines √ x √ X

Licenses x x x √

Objective parameters √ √ √ √

Subjective Parameters √ √ √ √

PSTN Network x √ √ √

Cellular Mobile Network √ √ √ √

Internet x √ √ √

Data (Leased Lines) x √ √ x


Objective KPIs and Targets Benchmarks

KPIITU Targets

Kenyan Targets

Nigeria Tanzania Uganda

Completed Call (%) - 90 90 >99 99

Speech Quality (MOS)%age with Good SQ

4.0 ≤ MOS ≥ 4.5

95%>2.7 & 3.1 in 3 yrs

98 >2.0 MOS

>95 95

Call Drop Rate (%) - 2 2 <3 2

Call Block Rate (%) 1 10 - - 2

Call Set Up Time (s) varied 13.5 < 10 -

Call Set Up Success Rate (%)

90 90 - -

Handover Success Rate (%) 99.5% 85 90 - -

Call release delay 1s - - 2s -

Multimedia (IP) QoS Varied ITU - - -


Subjective QoS parametersAccount complaint rateAccount complaint resolution timeDisconnection complaint rateDisconnection complaint resolution time Miscellaneous complaint rate Miscellaneous complaint resolution time Fault report rate Fault repair timeBilling Accuracy Service supply time Call centre answer success ratio Call centre answer timeComplaint resolution


Conclusion

ITU-T recommendations have covered many QoS issues but very few targets have been specifiedWhere ITU-T targets are specified, they should be held as minimum thresholds not negotiable downwardsMany regulators monitor QoS parametersTargets adopted by regulators vary widely


Questions

How many regulators monitor QoS performance of providers?What are the KPI’s monitored?How comprehensive are KPIs in addressing customers requirements?How do the targets compare with international standards or best practices?Is there need for African Regulators to benchmark and adopt similar and adequate parameters and targets?


Thank You


Documents

Benchmarked Key Performance Indicators