Paper The New Metric and the New

Software Tool for Determining QoS

in the Short Messages Service

in Mobile NetworksMarcus Rompf1 and Tadeus Uhl2

1 Flensburg University of Applied Sciences, Flensburg, Germany2 Maritime University of Szczecin, Szczecin, Poland

Abstract—With traveling professions and nomadic lifestyles

on the increase there is a growing need for comprehensive mo-

bile networking within the Next Generation Network, and the

security, reliability and technical independency of the Short

Message Service (SMS) is becoming more and more indis-

pensable for business processes. Since competitiveness goes

hand-in-hand with this business capability, and methods to

determine the quality of this service are becoming equally in-

dispensable. This paper presents possible methods to eval-

uate the “Completion Rate of SMS” and “End-to-End De-

livery Time” and combinations of them (new metric in this

paper). A software tool, QoSCalc(SMS), has been developed

in collaboration with the authors in order to verify whether

service providers are abiding by the Service Level Agree-

ments. The tool has been tested in a real measurement envi-

ronment and the results are presented graphically here and

interpreted.

Keywords—communication network, ETSI TS 102-250-2, QoS

measurement environment, QoS measurement techniques, queu-

ing model, SMS service.

1. Introduction

3G networks are becoming more and more widespread

every day. Their services also include the Short Message

Service (SMS). Although their popularity has experienced

a slight decrease in private use, the trend in business use

is upward. Constant availability has become a prerequisite

of both traveling professions and mobile lifestyles. Board-

ing passes via SMS, and SMS TAN (Transaction Authen-

tication Number) for online banking have become com-

mon usages and are making SMS increasingly attractive

for competitive businesses for which security, reliability

and technical independency are crucial factors. “Service

Level Agreements” are signed to guarantee a specific end-

to-end quality of service, which can, however, seldom be

verified conclusively by the average service subscriber.

In November 2009 the European Parliament and Euro-

pean Council adopted Directive 2009/136/EC [1] amending

Directive 2002/22/EC [2] on universal service and users’

rights relating to electronic communications networks and

services, Directive 2002/58/EC [3] concerning the pro-

cessing of personal data and the protection of privacy in

the electronic communications sector and Regulation (EC)

No. 2006/2004 [4] on cooperation between national au-

thorities responsible for the enforcement of consumer pro-

tection laws. Within this directive, providers of electronic

communications services that allow calls should not only

ensure that their customers are adequately informed about

the limitations of the services but also about the routing

of emergency calls. Furthermore, information about ser-

vices, which are not provided over a switched telephone

network, should also cover the level of reliability of the

access and provide caller location information in compari-

son with that of a service that is provided over a switched

telephone network, taking into account current technology

and quality standards and any quality of service parameters

specified under Directive 2002/22/EC (Universal Service

Directive) [2].

Especially with regarding to Quality of Service (QoS),

Member States of EU shall ensure that national regulatory

authorities empowered to require service providers to pub-

lish comparable, adequate and up-to-date information for

end-users on the quality of their services, when requested.

These national regulatory authorities may specify, amongst

other things, the QoS parameters to be measured and the

contents, form and manner in which the information is to

be published. In addition, the national regulatory author-

ities should be able to set minimum QoS requirements to

prevent degradation, traffic-slowing or traffic-hindering. As

a result, it can be said that the European Parliament and the

European Council are not only generally aware of QoS and

all that it entails, but are also ensuring that the Member

States apply their directives.

In 2011, the ETSI technical committee Speech and multi-

media Transmission Quality (STQ) produced two multi-part

technical specifications to cover “QoS aspects for popular

services in mobile networks” [5] and “User related QoS

parameter definitions and measurements” [6]. The second

part of [5] in particular discusses the topic “Definition of

Quality of Service parameters and their computation”, and

contains an abstract definition, which gives a generic de-

scription of the parameter, an abstract equation and the

corresponding user and technical trigger points. The sec-

3

Marcus Rompf and Tadeus Uhl

Fig. 1. SMS traffic growth in 1992–2013 and 2015–2017 forecast. (See color pictures online at www.nit.eu/publications/journal-jtit)

ond part of [6] is devoted to the topics “Voice telephony,

Group 3 fax, modem data services, and SMS”.

Research on the current telecommunications market has re-

vealed that SMS still plays an important role even in the age

of over-the-top (OTT) Messaging, which also has become

very popular. In 2011 the world press declared SMS for

“dead” and heralded OTT Messaging as its replacement.

SMS continues to offer a few benefits such as its flexible

platform and reliability. Figure 1 shows the growth of SMS

traffic [7].

It is obvious that SMS was still popular with about 7.6 tril-

lion messages in 2013. Although it has now entered the

decline phase, it is worth taking a closer look at SMS

nonetheless.

The examination will start with a presentation of the QoS

parameters for SMS as they stand in the ETSI Recommen-

dation TS 102-250-2 [6]. Then, a new metric for SMS

will be defined and illustrated. The paper will then turn

to an introduction of the new tool QoSCalc(SMS), begin-

ning with its general architecture and functional principle.

After that, there is a description of how the new tool was

put to the test in a series of practical measurements. The

paper closes with a conclusion and an outlook.

2. QoS Parameters for SMS

This chapter introduces the main QoS parameters recom-

mended in ETSI TS 102-250-2 [6]:

SMS Service Non-Accessibility. The service non-accessi-

bility denotes the probability that the end user cannot access

the SMS when requested while it is offered by display of

the network indicator on the user equipment.

SMS Service Access Delay. The service access delay is the

time period between sending a short message to the network

and receiving a send confirmation from the network at the

origin side.

SMS Successful SMS Ratio. This ratio describes the

probability that a user can send a short message from a ter-

minal to the Short Message Center and is applicable for

service providers offering the SMS. Therefore, it is recom-

mended that result statistics should return the percentage

of successfully sent short messages, the number of obser-

vations and its calculated absolute accuracy limits for 95%

confidence.

Measurements should be scheduled so as to reflect accu-

rately traffic variations over the hours of a day, the days

of the week and the months of the year. The parameter

is intended to measure a combination of network acces-

sibility and congestion in the signalling channels of the

SMS system throughout service operator’s claimed area of

coverage.

Completion Rate for SMS. This describes the ratio of

correctly received short messages to those sent, and is also

applicable to SMS service providers. It is recommended

that the results that provide statistics should return the ratio

of sent and correctly received short messages, the number

of observations used and its calculated absolute accuracy

limits for 95% confidence.

Measurements should be scheduled so as to reflect accu-

rately traffic variations over the hours of a day, the days of

the week and the months of the year.

End-to-End Delivery Time for SMS. This time is the pe-

riod that elapses between the sending of a short message

from the sender’s terminal equipment to a Short Message

Centre and the receiving of that message on the receiver’s

terminal equipment, and is applicable for service operators

providing the SMS. Result-providing statistics should return

the mean value in seconds for sending and receiving short

messages, the time in seconds within which the fastest 95%

4

The New Metric and the New Software Tool for Determining QoS in the Short Messages Service in Mobile Networks

of short messages are sent and received and the number of

observations performed.

Measurements should be scheduled so as to reflect accu-

rately traffic variations over the hours of a day, the days of

the week and the months of the year. The above-mentioned

measurements for estimating QoS parameters should be

done either on real traffic, on test calls for short messages

sent among a representative population of local exchanges

or on a combination of the above.

A measurement system can be designed for these QoS pa-

rameters that takes “Completion Rate for SMS” and the

“End-to-End Delivery Time for SMS” into account.

Research of current QoS measurement techniques for

the SMS has culminated in a large professional solution

called QualiPoc Android, from SwissQual AG, a Rohde

& Schwarz company [8]. An alternative system, QoSCalc

(SMS), is presented in this work. The next section intro-

duces the new metric for evaluating the QoS in SMS.

3. New Metric for QoS in SMS

The conceptual idea is to introduce the SMS Quality Factor

(SMSQ-F), which takes completion rate and end-to-end de-

livery time into account. The formulas of this metric have

the following definition:

SMSQ−F=CRResult ·CRWeighting+DTResult ·DTWeighting , (1)

where:

CRResult =

1

n∑Completicon Rate(n)

Completation Rate Threshold, (2)

DTResult =Delivery Time Threshold

1

n

n

∑n=1

Delivery Time(n)

. (3)

Both thresholds should be chosen judiciously depending on

individual needs. Usually, completion rate threshold is 1,

so that every short message should be successful delivered.

The threshold in seconds for the delivery time will reflect

professional requirements, e.g. alarm systems ≤ 3 or online

banking ≤ 7.

The respective weightings must have the sum of 1 and

should be chosen to reflect how the evaluator gauges their

importance.

The following example is chosen:

• completion rate threshold = 1,

• delivery time threshold = 10 s,

• weighting completion rate = weighting delivery time

= 0.5.

Table 1 gives some example results.

From Table 1 it can be seen that the proposed metric, based

on Eq. (1), works well. Furthermore, it can be seen that

Table 1

Example results of the new metric

n Average CR Average DT [s] CRResult DTResult SMSQ-F

31 1 10 1.00 1.00 1.00

31 1 11 1.00 0.91 0.95

31 1 4 1.00 1.00 1.00

31 1 40 1.00 0.25 0.63

31 1 22 1.00 0.45 0.73

31 0.9 15 0.90 0.67 0.78

31 0.8 16 0.80 0.63 0.71

31 0.7 10 0.70 1.00 0.85

31 0.6 4 0.60 1.00 0.80

31 0.5 19 0.50 0.53 0.51

31 0.4 7 0.40 1.00 0.70

the values of SMSQ-F are within the interval [0, 1]. To be

able to adapt these values to a respective (Quality of Expe-

rience; subjective evaluation) QoE scale, the following rela-

tionship has been chosen, based on the QoS measurement

technique “Apdex-Index” for the WWW service [9]–[10]

(see Fig. 2).

Fig. 2. Translation of QoS values into QoE values for the

SMSQ-F.

In general, the proposed metric can be described as follows:

CRResult =

1 for1

n

n

∑n=1

Completion Rate(n) =

= Completion Threshold

0 for n=0∪Completion Rate Threshold=0

else

1

n

n

∑n=1

Completion Rate(n)

Completion Rate Threshold

,

(4)

5

Marcus Rompf and Tadeus Uhl

DTResult =

1 for1

n

n

∑n=1

Delivery Time(n) =

= Delivery Time Threshold

0 for n=0∪Delivery Time=0

elseDelivery Time Threshold

1

n

n

∑n=1

Delivery Time(n)

. (5)

Furthermore, this metric has been implemented in the

new tool QoSCalc (SMS), neatly customizing it for the

QoS evaluation for SMS. The next chapter introduces this

tool.

4. The New Tool for Analysing QoS

in SMS

The tool that was developed in the course of this work

to analyze QoS in the SMS service has been called

QoSCalc(SMS). It can be used for both objective and sub-

jective analysis of service quality of SMS in mobile net-

works. Therefore, a concept for a new measurement system

has been formulated that takes both the completion rate

for SMS and the end-to-end delivery time for SMS into

account. It is therefore necessary to understand the idea

behind merging these QoS parameters.

Once a short message has been correctly sent and delivered,

the completion rate for SMS provides statistics about the

ratio of correct short messages received to short messages

sent from all observations, while end-to-end delivery time

for SMS provides statistics about the delivery time of short

messages. By merging these QoS parameters it is possible

to provide statistics about the QoE using the SMS.

Fig. 3. Conceptual measurement environment.

In example users will be unsatisfied when a sent short

message may be correctly received but delivered only af-

ter 10 minutes. They might be equally upset by promptly

delivered but error-laden messages. In professional imple-

mentations, time-criticalness is just as important factor as

reliability. When it comes to security and alerting sys-

tems, business implementations stand or fall depending on

time-criticalness and reliability. This concept introduces an

approach to estimate such appropriate QoS parameters.

The general architecture is described in Fig. 3.

The test engineer is responsible for the measurement setup

and controls the test control station, which in turn controls

the SMS send and receive stations. The send station and the

receive station are synchronized and communicate via the

service provider. The chosen hardware platform is the mo-

bile phone HTC Desire 620G dual SIM on Android 4.4.2.

The implementation is based on the block diagram shown

in Fig. 4.

Fig. 4. Conceptual description for the implementation.

The entire solution was written in the programming lan-

guage Java and developed in Android Studio 2.1.2. The

compiled SDK Version is API 23: Android 6.0 and the

Build Tools Version is 23.0.0. The dependencies are set as

follows:

• support: AppCompat 23.1.1 and Design 23.1.1,

• visualization: MPAndroidChartLibrary 2.1.6,

• time synchronization via Network Time Protocol

(NTP): Apache Commons Net 3.4,

• several permissions, such as Receive SMS,

Send SMS and Access Network State, need to

be set.

The basic implemented functionalities are:

• Time synchronization. The time synchronization is

done via the NTP and the server involved is the de-

fault one configured in the system. Then the differ-

ence between the system and server time is calculated

as an offset for the SMS timestamp. Note: it is not

necessary to use the offset correction between send-

ing station and receiving station time synchronization

because the environment does not need to be accurate

to within less than 10 ms.

• Send SMS. Each SMS starts with the string “Mes-

sage”, followed by a zero-initialized, pre-increment

counter and the respective synchronized timestamp.

After 20 short messages have been sent the sending

station goes into the idle state.

6

The New Metric and the New Software Tool for Determining QoS in the Short Messages Service in Mobile Networks

• Receive SMS. Each received SMS is stored as

a structure with the message body, originating ad-

dress, receive time and sending time.

• Analysis. The analysis is done for the “End-to-End

Delivery Time for SMS”, i.e. the difference between

sending and receiving time, the “Completion Rate for

SMS”, i.e. the difference between successful sent and

received short messages, and finally the calculation

of the SMSQ-F.

After being defined, the functionalities have been linked

to the control elements and are ready for the first tests, as

shown in Figs. 5–8.

Fig. 5. Start-up screen QoSCalc(SMS).

Fig. 6. Receive station QoSCalc(SMS) – LTE.

Start-up screen. The Start-up screen allows the user to

choose between sending station and receive station. By de-

fault it is displayed whenever a synchronization error should

occur.

Sending/receive station. The two stations can be time syn-

chronized with the Sync button. Of course, the receive sta-

tion must already have been started. After a measurement,

Fig. 7. Sending station QoSCalc(SMS) – LTE.

the analysis section can be initialized with the analysis

button.

Analysis section. At the current state of development, this

section is only active for the receive station. It provides the

following information:

• “End-to-End Delivery Time for SMS” as a function

of the short message number,

• average “End-to-End Delivery Time for SMS”,

• “Completion Rate for SMS” for one measurement

scenario,

• the calculated “SMSQ-F”, based upon the user-

specified thresholds for the completion rate and de-

livery time and their weightings.

Finally, the new tool has to be tested in a real testing envi-

ronment to verify if it is working properly.

5. Measurement Environment and

Measurement Results

The measurement environment for the chosen measure-

ment scenario consists of two HTC 620G dual SIMs, each

plugged with a SIM card from Swisscom with LTE func-

tionality. Test location was 47◦12’25.2” N, 7◦33’17.3” E.

Unfortunately, the HTC 620G is not suitable for LTE test-

ing. Therefore, two Samsung Galaxy S6 Edge+ were been

included in the environment. Then, the following measure-

ment scenario was chosen.

7

Marcus Rompf and Tadeus Uhl

Fig. 8. Analysis section QoSCalc(SMS) – LTE.

Fig. 9. Results QoSCalc(SMS) – EDGE.

Fig. 10. Results QoSCalc(SMS) – UMTS.

Technology testing. Technology testing uses different

mobile network technologies. These are 4G (LTE), 3G

(UMTS) and 2G (EDGE). Each test consists of a set of

20 short messages sent at intervals of 10 s. The defined

thresholds were as follows:

• completion rate for all technologies: 1,

• delivery time: LTE – 1.3 s; UMTS – 4 s; EDGE –

5 s,

• the weightings are to be considered as equal.

8

The New Metric and the New Software Tool for Determining QoS in the Short Messages Service in Mobile Networks

Fig. 11. Results QoSCalc(SMS) – LTE.

Being successful in the measurement environment, the new

tool has delivered the results shown in Figs. 9 to 11.

EDGE results. These results show an average delivery

time of 5.06 s within an interval of [4.48; 5.73] and a com-

pletion rate of 1. With the chosen threshold, the SMSQ-F

is 0.99.

UMTS results. These results are showing an average de-

livery time of 3.83 s within an interval of [2.98; 4.28] and

a completion rate of 1. With the chosen threshold, the

SMSQ-F is 1.

LTE results. These results are showing an average delivery

time of 1.35 s within an interval of [0.82; 2.09] and a com-

pletion rate of 1. With the chosen threshold, the SMSQ-F

is 0.98.

Having analyzed all measurement results, one can conclude

that the new tool QoSCalc(SMS) is robust and suitable for

evaluating the QoS of SMS in different mobile networks,

such as LTE, UMTS and EDGE. The implemented metric

works properly and provides valuable results. It is possible

to adjust the thresholds to suit the testers’ needs, which

increases the flexibility of the tool considerably.

6. Summary and Outlook

This paper has been devoted to the subject of QoS mea-

surement within the SMS. It has focused on the creation of

a tool for measuring QoS in SMS for mobile network test-

ing. The new tool has been given the name QoSCalc(SMS).

It is based upon a new metric “SMSQ-F”, described in

Section 3, which takes both end-to-end delivery time for

SMS and completion rate for SMS into account. The im-

plemented tool has been tested in a real mobile network

environment. As far as the technology is concerned the

underlying measurement scenario and the respective results

display both a high degree of diversity, e.g. LTE, UMTS

and EDGE, and a high degree of flexibility regarding the

SMS quality factor, which is easily adjustable to meet the

end users’ needs when choosing the best service provider.

With that, the tool has proved its practical capabilities and

is ready to be used effectively in determining the quality of

the SMS in mobile networks.

In future work it would be very worthwhile to com-

pare the SMS service with OTT content messaging with

regard to reliability and especially availability. In ad-

dition, the tool should be improved to be more suitable

for test automation. Work in this direction is already in the

offing.

References

[1] Directives 2009/136/WE (Recom.EU L.337/11) [Online]. Available:

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=

OJ:L:2009:337:0011:0036:en:PDF

[2] Official Journal of the European Communities, The European Par-

liament and Council, 07 03 2002 [Online]. Available:

http://www.ofcom.org.uk/static/archive/oftel/ind info/eu directives/

authorisation.pdf

[3] Official Journal of the European Communities, the European Parlia-

ment and Council, 12.07.2002 [Online]. Available:

http://www.dataprotection.ro/servlet/ViewDocument?id=201

[4] Official Journal of the European Union, The European Parliament

and Council, 27 10 2004 [Online]. Available: http://www.wipo.int/

wipolex/en/text.jsp?file id=199676

[5] ETSI TS 102 250-2 V2.2.1, “Speech and multimedia Transmission

Quality (STQ), QoS aspects for popular services in mobile networks,

Part 2: Definition of Quality of Service parameters and their com-

putation”, 2011.

[6] ETSI EG 202 057-2 V1.3.2, “Speech and multimedia Transmission

Quality (STQ). User related QoS parameter definitions and measure-

ments. Part 2: Voice telephony, Group 3 fax, modem data services,

and SMS”, 2011.

[7] Portio Research, “Mobile Messaging Futures 2014–2018” [Online].

Available: http://www.portioresearch.com/en/messaging-reports/

mobile-messaging-research/mobile-messaging-futures-2014-

2018.aspx

[8] SwissQual AG – A Rohde & Schwarz Company” [Online]. Avail-

able: http://www.swissqual.com/en

[9] Application Performance Index – Apdex Technical Specification Ver-

sion 1.1 [Online]. Available: http://www.apdex.org/specs.html

[10] T. Uhl and M. Rompf, “New Tool for Investigating QoS in the WWW

Service”, J. Telecommun. and Inform. Technol., no. 1, pp. 1–7,

2015.

9

Marcus Rompf and Tadeus Uhl

Marcus Rompf received his

M.Sc. in Information Technol-

ogy from the Kiel University

of Applied Sciences (Germany)

in 2016. Since 2011, he is a

voluntary Research Scientist at

the Institute of Communications

Technology, Flensburg Univer-

sity of Applied Sciences (Ger-

many), headed by Prof. Tadeus

Uhl in the field of performance

analysis of communication systems and evaluation of the

QoS and QoE in Triple Play services. Today he works as

Senior Research Engineer for Mobile Network Testing at

SwissQual AG (Zuchwil, Switzerland). His main activities

cover the following areas: Development of QoE and QoS

measurement applications for telecommunication services,

analysis of video, audio signals and data communication

in mobile networks, implementation of algorithms under

Linux and Android and support in design of measurement

frameworks in automated test systems.

E-mail: rompfmarcus@gmail.com

Flensburg University of Applied Sciences

Kanzlei st 91-93

D 24943 Flensburg, Germany

Tadeus Uhl received his M.Sc.

in Telecommunications from

the Academy of Technology

and Agriculture in Bydgoszcz,

Poland in 1975, Ph.D. from

Gdańsk University of Technol-

ogy, Gdańsk, Poland in 1982

and D.Sc. from University at

Dortmund, Germany in 1990.

Since 1992 he has worked as

Professor at the Institute of

Communications Technology, Flensburg University of Ap-

plied Sciences, Germany and in addition since 2013 as

Professor at the Institute of Transport Engineering and Eco-

nomics, Maritime University of Szczecin, Poland. His main

activities cover the following areas: traffic engineering, per-

formance analysis of communications systems, measure-

ment and evaluation of communication protocols, QoS and

QoE by Triple Play services, Ethernet and IP technology.

He is author or co-author of five books and some 130 pa-

pers on the subjects of LAN, WAN and NGN.

E-mail: t.uhl@am.szczecin.pl

Maritime University of Szczecin

Henryka Pobożnego st 11

PL 70-507 Szczecin, Poland

10