Embed Size (px)
Citation preview
Measurement and AnalysisofVoIP Server Performance
by
Yusong WangB.E., Huangzhong University of Science and Technology, 1990
B.Sc., Simon Fraser University, 2005
A PROJECT SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE
In the School ofComputing Science
© Yusong Wang 2008
SIMON FRASER UNIVERSITY
Spring 2008
All rights reserved. This work may not bereproduced in whole or in part, by photocopy
or other means, without permission of the author.
Name:
Degree:
Title of Project:
Examining Committee:
Chair:
APPROVAL
Yusong Wang
Master of Science
Measurement and AnalysisofVoIP Server Performance
Dr. Janice ReganLecturer of school of Computing Science
Dr. Jiangchuan (JC) LiuSenior SupervisorAssistant Professor of school of Computing Science
Dr. Mohamed HefeedaSupervisorAssistant Professor of school of Computing Science
Dr. Jie LiangSFU ExaminerAssistant Professor of school of Engineering Science
Date Defended/Approved:
ii
J7 I Lo07,
SIMON I~RASER UNIVERSITYLIBRARY
Declaration ofPartial Copyright LicenceThe author, whose copyright is declared on the title page of this work, has grantedto Simon Fraser University the right to lend this thesis, project or extended essayto users of the Simon Fraser University Library, and to make partial or singlecopies only for such users or in response to a request from the library of any otheruniversity, or other educational institution, on its own behalf or for one of its users.
The author has further granted permission to Simon Fraser University to keep ormake a digital copy for use in its circulating collection (currently available to thepublic at the "Institutional Repository" link of the SFU Library website<www.lib.sfu.ca> at: <http://ir.lib.sfu.ca/handle/1892/112>) and, without changingthe content, to translate the thesis/project or extended essays, if technicallypossible, to any medium or format for the purpose of preservation of the digitalwork.
The author has further agreed that permission for multiple copying of this work forscholarly purposes may be granted by either the author or the Dean of GraduateStudies.
It is understood that copying or publication of this work for financial gain shall notbe allowed without the author's written permission.
Permission for public performance, or limited permission for private scholarly use,of any multimedia materials forming part of this work, may have been granted bythe author. This information may be found on the separately cataloguedmultimedia material and in the signed Partial Copyright Licence.
While licensing SFU to permit the above uses, the author retains copyright in thethesis, project or extended essays, including the right to change the work forsubsequent purposes, including editing and publishing the work in whole or in part,and licensing other parties, as the author may desire.
The original Partial Copyright Licence attesting to these terms, and signed by thisauthor, may be found in the original bound copy of this work, retained in theSimon Fraser University Archive.
Simon Fraser University LibraryBurnaby, BC, Canada
Revised: Fall 2007
ABSTRACT
Voice over IP (VoIP) and Instant Messaging (1M) are rapidly changing our daily
communication landscape. SIP and 1M protocols are two of the most widely deployed
protocols for these applications. However, there are still many practical challenges
toward implementing SIP and 1M in commercial products, and their performances in real
large-scale systems have yet to be understood and optimized.
In this project, we applied two open source benchmark tools, Jabsimul and SIPp,
to investigate the performance of state-of-the-art VolP and 1M systems. We closely
examined two core server products, XMPP Server and SIP Proxy Server, from Eyeball
Networks, a pioneer in this industry. We applied the user behaviour models that well
capture the capacity and dynamics of these two systems to study their system
performance. Based on our test results, we further diagnosed their performance
bottlenecks, and proposed effective solutions toward enhancing their scalability and
reliability.
Keywords: VoIP; System Performance; Benchmark tools; Performance study; SIP; XMPP
Subject Term: Internet Telephony; TCP/IP; Multimedia systems; Computer network protocols
iii
ACKNOWLEDGEMENTS
I would like to give my special thanks to my senior supervisor, Dr. Jiangchuan Liu for his
support during my studying of this graduate program. Without his guidance, support and
encouragement, it is impossible for me to finish this project.
I would like to thank Dr. Jie Liang, Dr. Mohamed Hefeeda, and Dr. Janice Regan for
their kindness to be able to serve on my examining committee.
I would also like to thank Eyeball Networks and MATACS to provide me with this
invaluable internship opportunity, so that I can conduct my project research with their test bed.
Without their support and help, this project could never have been done.
I also would like to give my sincere thanks to my family. Only with their endless support
and backup, I can finish study of this program.
iv
TABLE OF CONTENTS
Approval iiAbstract iiiAcknowledgements ivTable of Contents vList of Figures viiList of Tables viii
Chapter 1: Introduction 1
Chapter 2: Overview of XMPP Protocol and Session Initial Protocol 4
2.1 Extensible Messaging and Presence Protocol 42.1.1 Introduction to extensible messaging and presence protocol 42.1.2 XMPP network architecture 52.1.3 Establishment of XMPP session 62.1.4 XMPP stanza 7
2.2 Session Initial Protocol 82.2.1 Introduction to session initial protocol 82.2.2 SIP network architecture 92.2.3 SIP registration procedure 102.2.4 Routing messages with SIP proxy server 11
Chapter 3: Load Testing Benchmark Tools 14
3.1 Introduction to Load Test Benchmark Tools 143.2 Benchmark Tools for XMPP Servers and SIP Servers 153.3 Jabsimul Benchmark Tool 163.4 SIPp Benchmark Tool 18
3.4.1 Introduction to SIPp 183.4.2 SIPp scenario files 20
3.5 Enhancement of Benchmark Tools 233.5.1 Support load-testing multiple servers 233.5.2 Control online message distribution for Jabsimul test tool 243.5.3 Integrate user registration with call set up into one scenario 25
Chapter 4: Benchmark Architecture 28
4.1 Overview of Eyeball XMPP Server System 284.2 Overview of Eyeball SIP Server System 294.3 Benchmark Architecture for System Performance Studies 30
Chapter 5: MySQL Database Optimization 32
5.1 Optimizing MySQL Statements 325.2 Optimize MySQL Storage Engines 33
v
5.3 MySQI Server Tuning 36
Chapter 6: Measurement and Analysis Of System Performance 406.1 System Configuration for Benchmark System 406.2 Limitations of the System Performance Study 436.3 Performance Study for Eyeball XMPP Server system 44
6.3.1 Jabsimul test tool configuration 446.3.2 Eyeball XMPP server system configuration 456.3.3 XMPP edge server incoming buffer size .466.3.4 Message response time for XMPP server system .476.3.5 Memory usage for XMPP server components 496.3.6 CPU usage of Eyeball XMPP server components 506.3.7 Bandwidth usage of Eyeball XMPP server components 52
6.4 Performance Study for Eyeball SIP Server System 536.4.1 RPS and CPS of Eyeball SIP server system 536.4.2 Transaction response time of Eyeball SIP server system 566.4.3 Memory and CPU usage of Eyeball SIP server components 576.4.4 Bandwidth utilization of Eyeball SIP server components 60
Chapter 7: Conclusions and Future Work 61
Appendices 63Appendix 1: Jabsimul Configuration File for Eyeball XMPP Server 63Appendix 2: SIPp Call Scenario File of UAC for Eyeball SIP Server 65Appendix 3: SIPp Call Scenario File of UAC for Eyeball SIP Server 68
Reference List 70
vi
LIST OF FIGURES
Figure 2.1
Figure 2.2
Figure 2.3
Figure 2.4
Figure 2.5
Figure 3.1
Figure 3.2
Figure 3.3
Figure 3.4
Figure 3.5
Figure 4.1
Figure 4.2
Figure 4.3
Figure 5.1
Figure 6.1
Figure 6.2
Figure 6.3
Figure 6.4
Figure 6.5
Figure 6.6
Figure 6.7
Figure 6.8
Figure 6.9
XMPP Client-Server Based Network Architecture 5
Message flow involved in establishing an XMPP Session 7
A SIP Network Architecthure 9
Call flow for SIP Registration 11
Call Flow for SIP Call setup and teardown involving a SIP Proxy 13
A screenshot of Jabsimul user interface 18
A screenshot of SIPp scenario screen 19
A screenshot SIPp statistic screen 20
A successful simple SIP to SIP call flow 21
A Call flow including registration, call setup and teardown 26
Eyeball XMPP Server Overview 28
Eyeball SIP Server Overview 30
Benchmark System Architecture for Eyeball XMPP/SIP Server Systems 31
A screenshot for MySQL SHOW PROCESSLIST command 34
System configuration for Eyeball XMPP Server benchmark system .41
System configuration for Eyeball SIP Proxy Server benchmark system .41
Message Response Time of different loads for one-XMPP edge serversystem 48
Message Response Time of different loads for two-XMPP edge serversystem 48
XMPP Server Free Memory under different traffic loads 50
A Call flow with Registration, Call setup and teardown 55
Transaction Response Time under different traffic load 57
SIP edge server free memory under different traffic loads 59
CPU Idle Time of SIP Server components 59
vii
LIST OF TABLES
Table 6.1
Table 6.2
Table 6.3
Table 6.4
Table 6.5
Table 6.6
Table 6.7
Table 6.8
Table 6.9
Table 6.10
Table 6.11
Table 6.12
Maximum system throughput with different server configurations .45
Maximum system throughput with different Incoming Message Buffer Size .46
Memory utilization for one-XMPP server system with 120K TCPconnections , 49
Memory utilization for two-XMPP server system with 240K TCPconnections 49
CPU Usage for components of one-XMPP edge server system with 120thousand concurrent TCP connections 51
CPU Usage for components oftwo-XMPP edge server system with 240thousand concurrent TCP connections 51
Bandwidth utilization for One-XMPP edge server system with 120Kconcurrent TCP connections (Kbits/s) 52
Bandwidth utilization for two-XMPP edge servers system with 240K TCPconnections (Kbits/s) 52
CPU Utilization and Transaction Response Time for maximum RPS 53
CPU Utilization and Transaction Response Time for maximum CPS 54
Memory utilization for SIP Server system with 120K TCP connections 58
Bandwidth utilization of SIP server components with lOOK connections(Kbits/s) 60
viii
CHAPTER 1: INTRODUCTION
Since the Internet has burgeoned into a very convenient and effective infonnation super
highway worldwide in the past ten years, it has brought many brand new ways for people to
communicate with each other and to facilitate their daily life. Two most prominent applications
among these are Voice over IP (VoIP) and Instant Messaging (1M), which have a very quick
development recently.
VoIP is an Internet-based voice communication technology. Over the past few years,
Voice over IP networks has made significant improvements on providing quality and reliable
voice conversations with attractive pricing plans, and has become a promising competitor in the
telephone market because of its potential benefit for both end users of traditional Public Switched
Telephone Network (PSTN) and VoIP service providers. From the end users' point of view, it is
much cheaper for them to make an internet telephone call than to make a traditional circuit
switched phone call, so that they can achieve monetary saving for the same type of phone
services. VoIP also provides the flexibility to allow an end user to keep the same phone number
when he moves the IP Phone around with the available Internet access. In addition, many new
services beyond the traditional PSTN capacities become possible with VoIP. For example, VoIP
allows users to use web-based call control and contact management, to set up multi-conference
phone calls etc. On the other hand, several compelling reasons impel service providers to consider
providing VoIP service over Internet. First, VoIP technology provides them new revenue sources.
Secondly, VoIP uses the Internet as communication media and consumes less bandwidth than
circuit-switched phone calls, so the VoIP service providers can compete with their PSTN service
providers at a much cheaper service price. Lastly, VoIP gives the service carriers the ability to
integrate voice and data into a single network, and provide various multimedia services that
PSNT providers cannot provide or that are very expensive for them to do so.
Session Initial Protocol (SIP) [1] is a signal protocol that provides the basic Internet
based multimedia communications architecture for VoIP applications. SIP protocol controls the
initiation, modification, and tennination of interactive multimedia sessions among two or more
participants, and allows users to locate and contact one another-regardless of media content or
number of participants-using disparate computers, phones, televisions, or hand-held devices.
Designed to serve as a mechanism to establish a wide variety of sessions, SIP protocol IS
independent of the underlying transport protocols.
Instant Messaging (1M) [2] is a form of allowing instantaneous messages and presence
information to exchange among a number of parties at the same time. Recently, many Instant
Messaging services also integrate VoIP and web conference features. Because Instant Messaging
can transmit information quickly and efficiently, and can receIve near real-time
acknowledgements or replies, this form of digital communication has been widely accepted and
used by individuals of variety of age groups. 1M are also widely used by enterprises as an
alternate way of communications among employees in the company besides email or office phone
lines, so that they can communicate faster and more efficiently,
As we know, many simulation studies of VoIP have focused on a number of network
level performance measurement that are known to impact on QoS of VoIP service, such as the
end-to-end packet delay [3] [4], delay variation (jitter) [5] [6] [7] and packet loss between the
source and destination [8] [9] etc.. However, researchers have not spent enough effort analyzing
and studying the overall system performance of the VoIP signalling server systems. With Voice
over IP and Instant Message applications being widely developed, there are many challenges in
deploying large-scaled VoIP services and Instant Messaging applications (see [10]). While many
VolP and 1M servers usually could have good performance and provide required QoS service
under light weight traffic load, most performance issues often arise when the server is stressed
with very heavy loads, and these performance issues can only be explored when we really apply
loads on such a server. However, simulating enough users to load test a server is very resource
consuming, so it is important for the simulation tools to be able to generate a large group of user
traffic with limited resources to load test a server effectively.
Eyeball Networks is one of the world's leading companies in providing Internet
telephony and Instant Message (1M) software products. Under their cooperation, I used two open
source benchmark tools, Jabsimul [II] and SIPp [12], to measure and study the system
performance of two Eyeball core server products, Eyeball XMPP Server system and Eyeball SIP
Server system, in this project. An Eyeball XMPP Server is a scalable, distributed Instant
Messaging server fully compliant to the IETF standard of Extensible Messaging and Presence
Protocol, and an Eyeball SIP Proxy Server is a fully SIP compliant stateless server.
This thesis is organized as follows. In Chapter 2, we provide a brief review of XMPP
protocol and Session Initial Protocol that have been used for Eyeball XMPP Server and Eyeball
SIP Server systems. In Chapter 3, we introduce two open source benchmark tools, namely
2
Jabsimul and SIPp, and describe the enhancements that we have made with these two test tools,
so that they can satisfy our performance study purpose for these two Eyeball server systems.
Chapter 4 describes the system architectures of two Eyeball Networks server systems, and
introduce the benchmark architecture that we have used for our system performance study. In
Chapter 5, we describe how to optimize the backend MySQL server perfomlance used for
Eyeball XMPP and SIP Server systems from three different aspects, so that we can improve the
overall system performance of Eyeball XMPP Server and SIP Server. In Chapter 6, we describe
our experimental result and performance analysis of Eyeball XMPP and SIP Server systems. We
conclude and provide future research directions in Chapter 6.
3
CHAPTER 2: OVERVIEW OF XMPP PROTOCOL ANDSESSION INITIAL PROTOCOL
2.1 Extensible Messaging and Presence Protocol
Instant messaging applications were first developed to facilitate the communication
among users logged into the same multi-users operating system in 1970s. From middle of 1990s,
with the development of Internet, many companies have developed Internet-wide Instant
Messaging applications, such as ICQ (1996), AOL Instant Messenger (1997), Yahoo Messager
and MSN etc, each with its own proprietary protocol and clients, so they could not communicate
with each other. Since then, several major efforts have been emerged to define global standards
for Instant Messaging and Presence protocols. One such outcome is the XMPP/Japper protocol
formalized by IETF based on the open-source Jabber protocol. Now XMPP/Jabber is the most
widespread open source platform with approximately 170,000 servers deployed around the world,
and nearly 100 million users.
2.1.1 Introduction to extensible messaging and presence protocol
The xEtensible Messaging and Presence Protocol (XMPP) is an IETF adaptation of the
open-source Jabber protocol for instant messaging and presence. The core technology of XMPP
protocol was originally invented by Jeremie Miller in 1998. In 1999 and 2000, Jabber open
source community refined the technology and made it an open-source jabber protocol. In 2002
and 2003 the IETF formalized the standards which resulted in publication of RFC 3920 (see [13])
and RFC 3921 (see [14]) in 2004.
The XMPP is a pure XML-based application-layer protocol that provides near real-time
communication for instant messaging and presence service. An XMPP application is usually
deployed in client-server architecture. First, it requires establishing a session between client and
server by negotiating an XML stream using the TLS (Transport Layer Security) and SASL
(Simple Authentication and Security Layer) protocol for secure data transmission and user
authentication. Once the session is created, XML stanza can be sent over the stream between the
client and server to engage in the common XMPP functionalities of exchanging instant message
and presence information, such as sending messages, chatting, updating presence status etc.
4
2.1.2 XMPP network architecture
Although XMPP is not specified to use any specific network architecture, XMPP
applications are usually deployed in the client-server architecture, as shown in Figure 2.1. Under
such system architecture, a TCP connection is required between the client and server, and the
recommended port for the connection is 5222. As a simple extension of client-server architecture,
it is also possible to communicate between XMPP servers over TCP connections, and the
recommended port is 5269.
••••1+----------------+Server sends XML stanza to client
=~
XMPP f--~'"
ServerlClient sends XML stanza to server
Internet
Client sends XML stanza to server
Server sends XML stanza to client
•••
XMPPServer2
Figure 2.1 XMPP Client-Server Based NetworkArchitecture
In this client-server based architecture, XMPP clients are XMPP users who apply XMPP
protocol to talk to a XMPP server over a TCP connection. One client may have multiple
resources, for example, the laptop and desktop, a computer at home or at office. These resources
may connect simultaneously to a server on behalf of the authorized user, with each resource
identified by the resource identifier of an XMPP address, e.g. <node@domain/home> vs.
node@domain/work>. On the other hand, as an intelligent abstraction layer for XMPP
communication, an XMPP server is mainly responsible for managing TCP connections and
maintaining session information, routing appropriate-addressed XML stanzas among XMPP
entities over XML streams. An XMPP server also stores client data, such as contact block lists for
5
users etc. Thus, the XMPP server with such kind of information can process the XML data
directly on behalf of the client. Further, one XMPP system usually consists of a network of
XMPP-compliant servers, and they can inter-communicate with each other over a TCP
connection on port 5269.
The exchanges of instant messages and presence information between a client and a
server are over an XML stream in an instant messaging and presence session, and the session is
identified by two XML streams flowing in two directions between the client and the server.
Before a client and a server can start an instant message and presence session, they need to
establish a secure connection using TLS protocol, and authenticate each other with SASL
negotiation (Use of SASL). Once the session is created, client and server may exchange an
unbounded number of instant messaging and presence XML stanzas over the stream in the
session. Next, we will briefly describe how to establish an XML session, and what kinds of
XMPP stanza have been specified in the standard.
2.1.3 Establishment of XMPP session
To create a session between a client and server, we need to follow these steps:
I. A client conducts stream authentication
2. Client binds a resource to the stream so that the client's address is in the form of
<user@domain/resource>
3. Establish an instant messaging and presence session
Figure 2.2 describes an example message flow of creating an instant message and
presence session with no errors. Actually, there exist many alternate flows to deal with various
errors and alternations for a session establishment. We can find more information for many other
alternative flows with various alternatives or flows with errors in [13] and [14].
6
Client Server
MLI crlent opens a X stream to server
2. Server response with a XML stream to client~
3. Server makes an encryption proposal
•4. Client request an TLS negotiation
5. Server acknowledges the TLS negotiation
6. Client opens an TLS stream
7. Server opens an TLS stream
8. Server sends a list of features available
•9. SASL negoriation between Client and Server
10. Client opens an authenticated stream
~
11. Server opens an authenticated stream
12. Client request initiation of an 1M session
13. Server reDorts success or failure of client reauest
Figure 2.2 Message flow involved in establishingan XMPP Session
2.1.4 XMPP stanza
Once an instant message and presence session has been established, the client and server
can exchange an unbounded number of instant messaging and presence XML stanzas over the
stream in the session by using one of the following XML stanzas:
7
• Message stanza: Message stanzas are used to send infonnation to another entity.
Message types could be single chat messages, group chat messages, headline, and other
alerts or errors.
• Presence stanza: Presence stanzas are used to express an entity's current network
availability, such as offline or online, and to notify other entities of that availability.
Presence stanzas can also be used to negotiate and manage the presence subscription of
other entities in the system.
• IQ stanza: XMPP protocol use IQ stanzas, or Info/Query semantics, to provide a
structured request-response mechanism between two entities. Each entity can track the
corresponding request and response pairs by using "id" attribute in IQ stanza. In [13], it
also extended two namespaces to use IQ stanza to manage roster list and block
communications.
2.2 Session Initial Protocol
2.2.1 Introduction to session initial protocol
Session Initial Protocol (SIP) is initially defined by the IETF Multiparty Multimedia
Session Control Working Group (MMUSIC WG) in 1999 in RFC 2543, and then updated by the
SIP WG in 2002 in RFC 3261. SIP is a signal protocol that provides the basic Internet-based
multimedia communications architecture for VoIP applications. It controls the initiation,
modification, and tennination of interactive multimedia sessions among two or more participants,
and allows users to locate and contact one another, regardless of media content or number of
participants, using disparate computers, phones, televisions, or hand-held devices. Because SIP is
designed to serve as a mechanism to establish a wide variety of sessions, it does not dictate the
details within a session but instead negotiates interaction based on the capabilities of participants,
and is independent of the underlying transport protocols. This simplicity means that SIP IS
scalable, extensible, and fits comfortably into different architectures and deployment scenarios.
Similar to the HTTP protocol, SIP is a text-based protocol and adopts client/server
architecture. A SIP Client sends a request to a SIP server, and the server processes the request and
sends a response to the client to indicate the status of the SIP request. There are six SIP requests
defined in the context, which are REGISTER, INVITE, ACK, BYE, CANCEL and OPTIONS. A
SIP response is numbered and grouped as 1xx, 2 xx and so on through 6xx and classified as a
provisional or final response. A provisional response indicates that the server is processing the
request, and final response indicates the tennination of a request and the final result of the request
8
In the next three sections, we first briefly discuss the SIP network architecture and
available SIP functionality, and then we give a detailed description about how to register an SIP
endpoint to a SIP system to allow user mobility. Lastly, we describe a typical call flow procedure
to set up and release a call between two SIP end users.
2.2.2 SIP network architecture
In this section, we briefly describe the network architecture of a typical SIP system and
corresponding functionalities of each component. A simple paradigm of such architecture is
shown in Figure 2.3.
,,,I
III,,,, ,
SIP proxy, redirectAnd location servers
-------
Sip user agents(VAs)
Figure 2.3
SIP proxy, redirectAnd location servers
II
II,
A SIP Network Architecthure
A SIP network typically consists of devices of User Agents (UAs) and various SIP
servers. A User Agent (UA) is the SIP's endpoint that initiates or responds to a SIP transaction. It
can function either as a user agent client (UAC) that initiates SIP requests or as a user agent
server (UAS) that receives requests and responds on behalf of the user.
There usually exist four kinds of SIP servers in a SIP network, which are proxy servers,
redirect servers, location servers and registrar servers. These four types of SIP servers have to
work together to handle a SIP request. A proxy server is an intermediate entity in a SIP network,
9
and mainly responsible for routing a SIP request or SIP response to the target UA or another SIP
proxy on behalf of the UAC. When a UAC sends an INVITE request to the proxy server, the
proxy server contacts the location server to get the address of the target server, and then forwards
the INVITE request directly to the retrieved address of called party. Different from a proxy server
is when a redirect server accepts the INVITE request from the calling party UA, and retrieves the
address of called party through a location server. Rather than forwarding the INVITE request
directly, a redirect server returns the address to the calling party UA and allows the calling party
to send a new INVITE request by itself to the redirect address. Registrar servers accept
REGISTER requests from user agents and co-work with either proxy servers or redirect servers to
allow user mobility. Finally, a SIP endpoint can be bridged with other non-SIP terminals through
a gateway.
2.2.3 SIP registration procedure
Each SIP user typically has a public SIP or SIPS URL, known as AOR. AOR is the way
to identifY a user in SIP system. The AOR of the user is usually configured on a UA such as an IP
phone. Because the IP address of a UA is typically provided by DHCP, the user agent may vary
its IP address. So a UA need to create a time-limited binding a user's AOR with its current IP
address at the SIP registrar server, and this binding process is called registration.
Figure 2.4 describes the procedure about how a UA register itself to the SIP registrar. A
UA first sends a SIP REGISTER request to the SIP registrar. In this request, it specifies its AOR
in the "To" header, and its current IP address in the "Contact" header. The SIP registrar then
updates SIP location server database to bind the UA's AOR with its current IP address, but this
binding is valid only within the limited time duration, which is indicated by the "expires"
parameter of "Contact" header in the REGISTER request. The UA needs to refresh the
registration within the specified time. Otherwise, the binding will be removed from the database.
This registration procedure enables SIP to support user mobility.
10
IP Phone SIP Proxy SIP Registrar
REGISTER sip:eyeball.comTo:<sip:al [email protected]>
REGISTER sip:eyeball.comContact:<sip: [email protected]>;expire=7200To:<sip:[email protected]>
r r()nt"('t'<~in'l ?1inl 10 10 1 1>'pl<n;rp=7?OO
200 OK
ZOO OK~
Figure 2.4 Call flow for SIP Registration
2.2.4 Routing messages with SIP proxy server
A SIP proxy server is an element to route SIP requests from one UAC to the UAS, and to
route SIP responses from UAS to the UAC. Because a proxy server typically implements the SIP
registrar functions, it has access to the location database. By consulting the location database, a
SIP proxy determines the next-hop address or the contact address of the user to where one SIP
message needs to be forwarded, and forwards the message. Further, a SIP proxy can indicate its
willingness to be in the path of subsequent requests within the dialogue by inserting a Record
Route header in the request to establish the dialogue, such as INVITE or SUBSCRIBE messages,
and thus can be used to enforce policies or as checking point in the SIP network.
Figure 2.5 is the call flow example to exemplifY how a call is established and tom down
between two end users, two IP phones of Alice and Bob, via one proxy server. In this call flow
example, we assume that both SIP phones of Alice and Bob have already been registered with the
proxy server in the domain of company.com. Alice first initiates a call set up to Bob through one
SIP proxy. After they finish talking, Bob ends the call first. Messages exchanges between these
two end users are shown as follows:
I. The UA of Alice sends an INVITE message with Request URI sip:[email protected] to
the proxy server.
2. The proxy server accepts the INVITE and sends a 100 Trying back to the UA of Alice.
3. The proxy server forwards the INVITE to the user agent of Bob.
4. The Bob's IP phone accepts the INVITE request and sends back 100 Trying to the proxy.
11
5. Bob's VA starts ringing to alert the user about the incoming call, and also sends 180
Ringing to the proxy to indicate the ringing state.
6. The proxy forwards the 180 Ringing to the VA of Alice, and Alice's VA starts playing a
ring back tone to Alice.
7. Bob picks up the phone, and the VA of Bob sends 200 OK to the proxy.
8. The proxy forwards the 200 OK to the VA of Alice.
9. The VA of Alice acknowledges the 200 OK and sends an ACK directly to the VA of Bob.
At this time, the SIP dialogue is established. The dialogue identifiers are Call-ID, From
Tag, and To-Tag. Because the proxy did not insert a Record-Route header in the INVITE
message, all the subsequent messages in the dialogue will go directly between two end
users, Bob and Alice, and not via the proxy.
10. Alice and Bob start their conversation. Audio packets are sent directly between the
phones using RTP over VDP.
II. Now assuming Bob decides to disconnect the call, he hangs up the phone. His VA sends
a BYE request directly to the VA of Alice, and the flow of audio packets between the two
phones is also stopped.
12. The VA of Alice acknowledges the BYE transaction by sending 200 OK. Her phone also
initiates call disconnection. Now the call between two end users is now terminated.
12
IP PhoneOf Alice
I INVITE sip:[email protected]:<sip:[email protected]>;tag=FAEIITo:<sip: [email protected]>Call-IO: [email protected]: 1 INVITEContact:<sip:alice@ I0.1 0.1.1>;expire=7200Content-Type:application.sdp
_ 2. 100 Trying
6. 180 Ringing
8:200 OK with SOP body
9.ACK
SIP ProxyServer
3. INVITE sip:[email protected]:<sip:al [email protected]>;tag=FAEl ITo:<sip:[email protected]>Call-IO:[email protected]: I INVITEContact:<sip:[email protected]>;expire=7200Content-Type:application.sdp
4. 100 Trying
5.180 Ringing
7.200 OK with SOP bodyTo:<sip:[email protected]>tag=FAE22Contact:<sip:[email protected].\.2>
....
IP Phoneof Bob
Note I: Without inserting a Record-Route header in the INVITE message, ACK is sent directlyto Bob's IP Phone
Note2: SIP dialog is identified by following parameters:Call-IO: 11-22-33@1 0 1.10.1.1From Tag:FAEIITo Tag:FAE22
-----------------------------------1-------------------------------------
~O. Audio packets are sent directly between IP Phones of Alice and Bob ~~ V
II. BYE sip:[email protected]:<sip:[email protected]>tag=FAEIIFrom:<sip: [email protected]>tag=FAE22Call-IO: [email protected]:1 BYEContent-Length:O
r----------------------------------------------------- --------------------,: Note I: Bob hang up the phone first, and IP phone initiates BYE trasaction to tear down the session. II I
:_~.?~el~ _~~~_~:s:~g_e_i: :~I~ ::':t_d~r~~I~ !~o~ ~~~'s .?~.?~: ~o_ ~~i~:': .?~~~e :
12.200 OK
Figure 2.5 Call Flow for SIP Call setup andteardown involving a SIP Proxy
13
CHAPTER 3: LOAD TESTING BENCHMARK TOOLS
3.1 Introduction to Load Test Benchmark Tools
In this project, we load tested two Eyeball Network core server products, the Eyeball
XMPP Server system and Eyeball SIP Proxy Server system, and studied their system
performance. It can be difficult, if not impossible, to organize load tests with a group of real users.
The right way is to use advanced automatic load and stress testing tools. A good benchmark tool
should provide a framework for developing a traffic generator that produces massive, realistic
network payloads, and should function correctly and work effectively.
To function correctly, the test tool should possess the following features:
• Protocol compatible: The benchmark should be able to verify proper protocol
implementation by the tested product, and ascertain their standard compliance by
generating and injecting required messages compatible to corresponding protocol of the
server. A more general benchmark could even support multiple protocols using a plug-in
system
• Be able to verify that the system reacts as expected under non-legal conditions.
• Be able to verify that functions work correctly under stress and that operation is
maintained consistently over a long period of time
To work effectively, the test tool should also have the following characteristics:
• High Performance: The benchmarking should be able to simulate a huge number of
simultaneous users with a single physical computer or a single CPU. Each simulated
client usually needs to make its own connection to a server, and the cost of these
connections is usually very expensive. Because traditional benchmark tools can usually
only simulate fewer than 1000 users per server, it is very difficult and costly to set up
benchmark for testing large applications, which usually need hundreds of thousand
simulated users for stress testing, but good benchmark tools should be able to simulate
tens of thousands of virtual users.
• Distributed: Even though with high performance of benchmarks, one single computer can
still only simulate a limited number of users, for example, thousands of simulated users,
but usually hundreds of thousand, or even millions, of simulated users are required in
14
order to make a large system become under the load. Thus, generated user load should be
able to be distributed on a cluster of client machines
A good load and stress test tool should also be flexible, and provide some building
functionality to facilitate the test, such as:
• OS monitoring: A good benchmark could provide built-in monitoring mechanism for the
CPU usage, memory usage or even network traffic statistics so that users can easily
collect these important testing data.
• QoS metric parameter monitoring: It is also important for a benchmark to provide
various metric parameter about VolP QoS information.,
• Configurable system: A benchmark allows users to create various scenarios with a simple
configuration file, written in XML or plain text file, or script languages, such as php, perl
etc.
3.2 Benchmark Tools for XMPP Servers and SIP Servers
Many commercial or open source benchmark tools are available for load testing and
performance studying for XMPP servers and SIP servers. Among them, Tsung [17], Jabber Test
Suite [18] and Jabsimul [11] are the benchmark test tools for load testing a XMPP server; and
SIPSim [19] and SIPp [12] are the benchmark test tools for load testing a SIP server.
Tsung [17] was originally developed by IDEALX in 2000, and now process-one IS
responsible for its maintenance, development and commercial support. Tsung is an open-source
load-testing tool for Jabber/XMPP server. Tsung is implemented in Erlang, which is a
concurrency-oriented programming language. Developed as a distributed load-testing tool with an
extensible framework, Tsung can generate very a large realistic traffic payload on a limited
number of servers. By simulating the behaviour of a large number of users for a given application,
it can measure the system performance, collect measurement data, and provide performance
reports of the stressed system under a heavy load. Thus, developers can analyze the system
capability and pinpoint the performance bottleneck based on this information to improve the
target system.
The Jabber Test Suite (JabberTest) [18] is another collection of softwares that tests the
performance of Jabber-based services. It is developed by Dustin Puryear. The software includes
several tools to perform various tests, including timing of account creation, message delivery, and
concurrent session limits. Compared with Tsung, it is a relatively simple test tool.
15
Jabsimul [II] is a test tool for benchmarking Jabber/XMPP servers. It is very effective
and flexible for load testing XMPP servers. With simple xml configuration files, one can easily
configure the number of simulated users and concurrent connections for the XMPP server. It can
also configure user login and log out rate, message sending rate, the updating frequency of a user
for his buddy list or presence status etc. In this project we used this test tool to load-test Eyeball
XMPP Server system, and we will provide more detailed description about Jabsimullater.
On the other hand, SIPSim [19] is a Sip-Based commercial Services Simulator developed
by Sygnus Data Communications Ltd, which is devoted to network test, analysis, load generation,
simulation and monitoring solutions and services. SIPSim simulator focuses on the high-stress
load testing of any SIP application. It can generate high volume various SIP-based services, such
as video and audio stream, presence and instant messaging, as well as all relevant RTP
streams. By simulating a number of SIP terminals, SIPSim simulator can initiate one or more SIP
sessions, receive corresponding network SIP responses and terminate established calls. As a load
testing tool, the SIPSim is capable of fully stressing large carrier-grade network elements, such as
registration servers; sip proxy servers and video servers etc.
Another Commercial-grade loading test open source under GPL license for SIP
applications is Sipp [12]. SIPp is a performance-testing tool for the SIP protocol. Its main features
include support for basic xml based SIPStone [20] and customized scenarios, TCP/UDP transport
and dynamic adjustment of call-rate, and provide a comprehensive set of real-time statistics. It
can also generate media (RTP) traffic for audio and video calls. In this project, we are going to
use SIPp to load-test Eyeball SIP Proxy server, and we will also discuss SIPp test tool in more
detail in the later sessions.
3.3 Jabsimul Benchmark Tool
Jabsimul [II] is a test tool originally used for benchmarking jabberdlA, jabberd2 and
ejabberd instant messaging XMPP server. It is a very effective and flexible tool for load testing
an XMPP server.
With simple xml configuration files, Jabsimul can easily simulate real world user
behaviors to generate expected traffic pattern for load testing XMPP server. We can see such a
configuration file used in this project in Appendix I. Jabsimul test tool is compliant with XMPP
Core 1.0.and XMPP 1M 1.0 standard. With a proper configurable xml file, one can easily
configure the following parameters that influence the user behavior for the load testing:
16
• Number of users for the XMPP server
• Login/logout rate of users for XMPP server
• Message sending rate for text message
• Percentage of messages sending to online and offline users
• Rate for adding/deleting items from a user's roster list
• Frequency for updates on a user's presence status
• Sending raw data to a peer to simulate noise for the system
• Dynamically choosing one XMPP server to connect in case of multiple available XMPP
servers
Jabsimul also provides a very good user interface to allow a tester to observe various
system running status information and performance metric when it is running against one XMPP
server. Figure 3.1 is a screen shot for this user interface. From this user interface, we can obtain a
variety of information such as:
• Connection statistics, including total connections created, current established connections,
killed connections as expected and unexpected
• Message information statistics, including total messages sent, received, received offline,
forwarded and normal messages, and the average time for received an response for
echoing messages
• Roster list update information, including total messages for updating roster list, and their
corresponding average response time for the messages
• Presence update messages that have been sent and received so far
• Packets that have been created, sent and canceled so far by the test tool, and the number
of messages in the test tool sending queue
17
88:80.25Conn stat: conns: total: 241
kills: total: 0~lessages: tot. sent: 18
rcvd.offline: 0rcvd.normal: 18di ff check: 8
Roster: tot.adds: 0tot.dels: 0
Presences: tot. sent: 246Packets: created: 1228
canceled: 0
estabilished: 241unexpected: 0tot. rcvd : 18rcvd.admin: 0fwd: 9 avg.time:stability: 8avg.time: 8 [ms]avg.time. 8 [ms)tot. rcvd: 5
sent: 1227in queues: 1
68 [ms J
glob rost: 6521
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user11378: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user24288: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user11491: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user53363: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:featllres>User user50367: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://Jabber.or-g/features/iq-auth'/></stream:features>User user24976: Przyszlo cos nieznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user45263: Przyszlo cos nieznanego bez ida r<stream: fcatures><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user56584: Przyszlo cos nleznanego bez ida I
<stream:features><auth xmlns='http://jabber.org/features/iq-auth'/></stream:features>User user58037: Przyszlo cos nieznanego bez ida r<stream:features><auth xmlns='http://Jabber.org/features/iq-auth'/></stream:features>
11J ::lfOollt P"""':·IWt. O""I\~'I'" O·...~·~1Il." I'M! Ctitu:·ha l:!ml'L tl)IJ: .... ~'#IIat~tt•••~l
(J Applo. to '..... 1 'i~i.·1I'1l1 :)t.} 'ru"
Figure 3.1 A screens hot of Jabsimul user interface
3.4 SIPp Benchmark Tool
3.4.1 Introduction to SIPp
SlPp [12] is a free open source test tool and traffic generator for SIP protocol. As a
powerful performance testing tool for the SIP protocol, SIPp can be distributed on a cluster of
client machines and allowed to emulate thousands of user agents (UAs) calling our SIP server
system and load-test it. We can run a SIPp executable in client mode or in server mode, where
one UA simulated by the SIPp executable running in client mode will always initiate a call setup
request, and the UA simulated by the SIPp executable running in server mode then responses the
call setup request. One SIPp executable has to run against an embedded or a customized call flow
scenano.
SIPp provides several statistic and report screens to display call scenanos together with
various statistics for the running tests. We can switch screens by pushing I to 9 keys in the
keyboard. Among those screens, two important screens are scenario screen and statistics screen.
18
Scenario screen displays a call flow of the scenario as well as some important information, such
as call-rate, local call port, remote host and port, and protocol that is using etc., as shown in
Figure 3.2
Call-rate(length)5.0(0 ms)I1.000s
Po rt6060
Total-time Total-calls Remote-host23.01 s 115 10.50.1.32 :5060(TCP)
5 new calls during 1.001 s period30 calls (limit 15000)o Running, 30 Paused, 0 Woken upo out-of-call msg (discarded)31 open sockets
3 ms scheduler resolutionPeak was 35 calls, after 21 s
~'essages RetransREGISTER - - - - - - - - --> 115 0
401 <- - - - - - - - -- 115 0REGISTER - - - - - - - - --> 115 0
200 <. - - - - - - - _. 115 0Pause [1000ms/5000ms] 115
INVITE - - - - - - - - --> 100 0100 <- - - - - - - - -- 100 0180 <- - - - - - - - -- 100 0200 <- - - - - - - - -- E-RTD1 100 0ACK - - - - - - - - - -> 100 0
Pause [1000ms/5000ms] 100BYE - - - - - - - - --> 85 0200 <- - - - - - - - -- 85 0
Timeouto
o
o
o
Unexpected-Msg
o
oo
ooo
o
o
Pause [ 0ms] 85 0[+1-1*11]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic -----
._~ ~.!FE.IQ]
.-: .'.:t..!-.l~
Figure 3.2 A screenshot of SlPp scenario screen
A screenshot for Statistics screen is shown in Figure 3.3, and it displays the maIn
statistics counters, including call rate, incoming and outgoing calls created, successful and failed
calls, call length and response time in "periodic value" and "cumulative value" columns. The
"Cumulative value" column gathers all statistics since srpp has been launched, and the "Periodic
value" column gives the statistic value for the period that is specified by -f frequency command
line parameter.
19
----------------------------- Statistics Screen ------- [1-9]: Change Screen --Start Time I 2007-08-24 10:46:56Last Reset Time I 2007-08-24 10:47:46Current Time I 2007-08-24 10:47:47
._~~---------------------+---------------------------+--------------------------
Counter Name 1 Periodic value I Cwnulative value- - - + ~ - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - -
Elapsed Time 1 00:00:01:000 1 00:00:51:083Call Rate 1 5.000 cps I 4.992 cps
- - - - - - - - - - - - - - - - - - - - - - - - -+- - - - - - - - - - - - - - - - - - - - - - - - - - -+- - - - - - - - --Incoming call created 0 0OutGoing call created 5 255Total Call created 255Current Call 30
- - - - - - - - - - - - - - - - - - - - - - - - -+- - --
Successful callFailed call
5o
- - - - - - - - - - - - - - - - -+- --
225o
- - - - - - - - - - - - - - - - - . ~ - - - - ~ - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - -Response Time 1 1 00:00:03:505Call Length 1 00:00:06:281
[+1-1*1/]: Adjust rate ---- [q]: Soft exit
00:00:02:94900:00:05:911[p]: Pause traffic
A screens hot SIPp statistic screen
E::::==-~=--==-------=-__=- ;::::"'-'" u rr ... /III"'..... • • "':J ~:~ l ..rt'fl~~ ., •.e K,r",':'
(jA I- ".. 1"'... ~1Jt::.i
Figure 3.3
3.4.2 SIPp scenario files
!.Ll§::lLL!j• ::JI.oIl"""';,)
To run SIPp correctly, one SIPp executable has to run against a proper call scenario file
which is speci fled by "-sf' option in the command line. This scenario file specifies the scenario
file of a call flow. There are several pre-defined scenarios embedded in the SIPp executable. A
SIPp user can also write customized scenario files. These scenario files consist of a set of
commands defined in SIPp implementation, which includes send, recv, sendCmd, recvCmd,
pause, ResponseTimeRepartition and CallLengthRepartition commands. Each command may also
have one or more attributes.
IETF provide many examples of SIP call flows for IP telephony in [31].
Figure 3.4 show such a simple SIP-to-SIP call scenarioin, in which SIPp UAC
successfully initiates a call set up, pauses for some random time, and then tears down the call. To
run SIPp with such a call flow, we need to write two scenario files, one scenario file (uac.xml) is
for UAC, and the other (uas.xml) is for UAS. We can check Appendix 2 and 3 for contents of
these two files.
20
SIPP UAC SIPP UAS
(1) INVITE
(2) 100 (optional)
(3) 180 (optional)
....
(4) PAUSE
(5) 200 OK
(6) SIP ACK
(7) SIP BYE-
(8) 200 OK
Figure 3.4 A successful simple SIP to SIPcall flow
We should know that a SIPp scenario is written in XML, and always starts with:
<?xml version=" 1.0" encoding="ISO-8859-1" ?>
<scenario name="Basic Sipstone UAC">
And end with:
</scenario>
For the client scenario specified in uac.xm1, we need to start the scenario with a "send"
command:
<scenario name="Basic Sipstone UAC">
<send>
<
INVITE sip:[service]@[remotejp]:[remote~ort] SIP/2.0
Via: SIP/2.0/[transport] [locaUp]:[local~ort]
From: sipp <sip:sipp@[locaUp]: [Iocal~ort]>;tag=[call_number]
To: sut <sip:[service]@[remote_ip]:[remote~ort]>
Call-ID: [caIUd]
Cseq: I INVITE
Contact: sip:sipp@[locaUp]: [local~ort]
Max-Forwards: 70
Subject: Performance Test
Content-Type: application/sdp
Content-Length: [len]
v=O
o=userl 536557652353687637 IN IP[locaUp_type] [locaUp]
s=-
t=O 0
c=IN IP[media_ip_type] [media_ip]
m=audio [media~ort] RTP/AVP 0
a=rtpmap:O PCMU/8000
]]>
</send>
Inside a "send" command, all the SIP message are closed between the "<! [CDATA" and
the "]]>" tags, and everything between those tags is going to be sent to the remote system.
Once the UAC sends the INVITE message using the "send" command, it can wait for an
answer by using the "recv" command.
<recv response=" 100"> optional="true" </recv>
<recv response=" 180"> optional="true" </recv>
<recv response="200"> </recv>
22
Here, sequences of responses are received with "recv" commands, which may include
one or more optional messages, such as "100 Trying" or "180 Ring", but must include one
mandatory message, such as "200 OK". Following this call flow, ACK and BYE messages are
sent with "send command", and corresponding 200 OK message is received by "recv" command.
In this client scenario file, we see there are many SIPp keywords enclosed by a pair of
bracket [ ], such as [locaUp_type], [locaUp] etc. These keywords represent certain values of a
SIP message filed for the current call. The value of these keywords may be specified by
command line options, or defined by SIPp executable, or come from the specified fields in the
external inject file indicated by the command line -inf option.
On the other hand, the server scenario specified in uas.xml has exactly the same syntax
and the list of available commands as for the client scenario, except that the server scenario
should always start with a "recv" command. Further, we see that many [Iast_*] keywords are used
in the server side scenarios, which will be replaced automatically by the specified header as it was
presented in the last received message from the client.
With provided commands, users can write a very complicated call scenario, and you can
get more infonnation about how to write such a call scenario from the SIPp reference
documentation (see [8]).
3.5 Enhancement of Benchmark Tools
Although both Jabsimul and SIPp are commercial-grade open source load-testing tools,
because they are on-going projects, so many bugs need to fix. Further, there still exist problems
and limitations in their implementation, which prevent them from working properly for load
testing Eyeball XMPP Server and SIP Proxy Server, which have their own special design
considerations. Therefore, we have to modify these two test tools, and customize them to fit our
test purpose. In the following few sessions, I am going to describe several main modifications
that I have made for Jabsimul and SIPp test tools.
3.5.1 Support load-testing multiple servers
For both SIPp and Jabsimul test tools, all the simulated users from one runnmg
executable are originally allowed to only connect to one server at a time. But for both Eyeball
XMPP Server and SIP Proxy server, we can configure them to run on several hosts to fonn a
server cluster. In this case, we hope both Jabsimul and SIPp test tools can evenly distribute their
23
load by connecting simulated users to any host of the server cluster randomly. After adding only
a few lines of codes, both two test tools now allow users to specify a cluster of server by using a
domain name on the command line, or specifying a list of IP addresses of hosts used by the
server cluster in the XML configuration file, so that the test tools can randomly connect to one
host of the cluster server for each single simulated user, and evenly distribute traffic load among
the server hosts.
3.5.2 Control online message distribution for Jabsimul test tool
Original implementation for Jabsimul just sent messages to any randomly chosen users in
the list, no matter whether the receiving user was online or not at the moment when the message
was sending, so there was no control about how many messages should be sent to online users, or
how many messages should be sent to offline users. Especially at the beginning of a load test,
most users are still offline, so sending messages to a randomly chosen user causes too many
messages to offline users. As we know in the real world, people usually want to send messages to
online users, only very few messages will go to offline users. For those messages to offline users,
the XMPP Server system needs to store them in the MySQL database first, and then forward them
to those receivers once they become online later. So sending too many messages to offline users
heavily influenced the overall system performance. In order to closely simulate the real word user
behavior of XMPP server, we need the Jabsimul test tool to be able to control how to distribute
sent messages between online users and offline users.
We modified the Jabsimul implementation to allow the Jabsimul program to specify what
percentage of messages should go to online users. When a simulated user in Jabsimul wants to
send a message to its peer, Jabsimul first decides whether the sending message should go to an
online user or an offline user based on a specified percentage distribution for online and offline
messages (later on we will discuss how to calculate this value). Then the program chooses a user
with correct online or offline status, and sends the message to it. This small change makes the
Jabsimul to be able to simulate the real world user behavior more closely.
However, there is a little tricky about how to specify the percentage of messages going to
online users for a simulated user. In Jabsimul implementation, when a simulated user sends a
message to an online user, the peer will echo the messages back to the sender, so echoed
messages will always go to online users because only online users can send messages first. Thus,
the actual final percentage of messages sent to online users is different from the percentage of
messages to online users sent by a simulated user. Now we assume that X percentage of messages
24
will finally go to online users, and Y is the percentage of messages that simulated users will send
to some other online users. Then we can have the following equation for Y value:
2*Xy=--
l+X
And then we will use value of Y to calculate whether a simulated user should send a
message to an online user or an offline user for Jabsimul .implementation.
3.5.3 Integrate user registration with call set up into one scenario
In this project, we are mainly interested in the system performance with TCP connections,
in which case SIPp load generator will connect each simulated user agent to the SIP server with a
separate TCP connection. Thus, we need to integrate user registration call flow with call setup
and call release call flow into one SIPp call scenario file, as shown in Figure 3.5.
25
UAC SIP UAS
Register
40 I Unauthorized
Register
200 OK
INVITE
100 TryingINVITE
180 Ringing
180 Ringing200 OK
200 OK
SIP ACKSIP ACK
SIP BYE SIP BYE~
200 OK
200 OK
Figure 3.5 A Call flow including registration, callsetup and teardown
Unfortunately, even most up-to-date version of the SIPp implementation still does not
support SIPp server and SIPp client to run against each other using the above call scenarios, and
the SIPp server always complains it is receiving a call INVITE message with an unknown Call-
10.
Call-IO is used in SIPp to identitY messages belonging to the same call transaction. By
analyzing the SIPp source code, we see that a SIPp executable will consider itself running in
26
client mode when it finds that the first command in scenario file is "send" command, and a Call
ID for the call is generated by the SIPp client. The SIPp client stores this Call-ID and then used it
to identify the following answers to the message from SIPp server as being part of an existing call.
On the other hand, a SIPp executable will consider itself running in server mode when it finds
that the first command in its scenario file is "recv" command. When a SIPp process is running in
server mode, it will not generate any Call-ID of its own, rather than retrieving the Call-ID of a
call from the messages received by its first "recv" command, and using it to match the following
received message to the same call. In above call scenario file, both SIPp client and SIPp server
need first to register their users to SIP server before they can establish calls between them, so the
first command in the call scenario for both SIPp client and server is "send" command, which
makes both SIPp executable running in SIPp client mode.
Now we can understand why we have this "unknown Call-ID" problem when we try to
run two SIPp instances against above call scenarios. We previously assumed that one SIPp
process that initiated the first "INVITE" message was running in client mode, and the other one
that first received the first "INVITE" message, was running in server mode. Actually, both SIPp
executables were running in client mode. So when one SIPp process that we assumed to be
running in server mode finished registering one user and received the first "INVITE" message, it
could not recognize this "INVITE" message by the Call-ID in this received "INVITE" message,
because this Call-ID was generated by its peer SIPp process. Since it could not match this Call-ID
to any existed Call-ID generated in its own side, the SIPp process considered this incoming
"INVITE" message as a message with unknown call-ID and rejected this "INVITE" message.
Therefore, original SIPp implementation will always fail when it tries to run against the above
call scenario.
To solve this problem, we modified SIPp implementation to allow it to check whether the
first received message is "INVITE" message after it registers a user. If so, it will consider itself as
a virtual SIPp server, and will simply use the Call-ID in "INVITE" message received from the
peer and store the value in its own side to identify the following call messages belonging to the
same call transaction. With this simple change, we extended SIPp test tool to be able to run
against above scenarios.
27
CHAPTER 4: BENCHMARK ARCHITECTURE
In this project, we studied the system performance of two Eyeball Networks core server
products for Voice over IP and Instant Messaging services, which are Eyeball XMPP Server
system and Eyeball SIP Proxy Server system. In this chapter, we will first briefly overview these
two Eyeball server systems, and then introduce the system architecture that we use to study the
system performance for these two server systems.
4.1 Overview of Eyeball XMPP Server System
An Eyeball XMPP Server is a scalable, distributed server for Instant Messaging. It
consists of three components: XMPP edge server, state server and MySQL database server, as
shown in Figure 4.1. XMPP Clients can only connect to edge servers; state servers are internal
servers and should not be accessible directly from the Internet. Edge servers and state servers
communicate with each other and with the database server.
\,-------,/ jState ServerEdge Server
,I. _. _. _. _. _. _ . _. _. _. _ . _. _ . _. _ . _. _. _. _ . . _. _.1.'
~ ....---...:
Clients Eyeball XMPP Server Database Server
Figure 4.1 Eyeball XMPP Server Overview
28
At least one XMPP edge and one state server are required to form the simplest Eyeball
XMPP Server configuration, and they can run on the same machine or on different machines.
Both edge and state server components interface with the database server to obtain user
information, perform user activity registration and retrieve the status and location of the other
edge server and state components within the Eyeball XMPP Server configuration. We can simply
start additional edge or state server components during run-time on the same machine or on
additional computers to scale an existing Eyeball XMPP server, giving the database as a
parameter in the server's configuration file. The new server(s) will automatically be integrated
into the existing server components without interrupting the running service. Once a new edge
server is started, it can immediately process requests from clients, and once a new state server is
started, it will take load off the already existing state server components. On the other hand, it is
possible to take out one or more server components from an Eyeball XMPP server configuration
dynamically for maintenance reasons etc. This will not lead to an interruption of the service, the
remaining server components in the server system will automatically take over the load from the
server that has been removed.
4.2 Overview of Eyeball SIP Server System
The Eyeball SIP Proxy Server is fully SIP compliant, and is a scalable and distributed
server to support SIP-based call and message forwarding. It uses MySQL database as its backend
for administration and statistics. The Eyeball SIP Proxy Server system also consists of three
components: SIP edge server, state server and MySQL database server, as shown in Figure 4.2.
We see that the Eyeball SIP Server system has the same system architecture as an Eyeball XMPP
Server system. Similarly, Eyeball SIP Clients connect only to edge servers; state servers are
internal servers and should not be accessible directly from the Internet. Edge servers and state
servers communicate with each other and with the database.
29
IJ' .....---...:
.......f-------l.~'
Edge Server State Server
~--
Clients
I. _ . _. _. _. _. _. _. _. _ . _. _ . _. _. _. _. _. _. _. _. _. _. _.1
Eyeball SIP Server Database Server
Figure 4.2 Eyeball SIP Server Overview
At least one SIP edge and one state server are required to fonn a simplest Eyeball SIP
Server configuration, and they can run on the same machine or on different machines. Both edge
and state server components interface with the database server to obtain user infonnation,
perfonn user activity registration and retrieve the status and location of the other edge server and
state components within the Eyeball SIP Server configuration. Further, in the exactly same
manner as in an Eyeball XMPP Server, we can dynamically add one or more server components
to, or remove one or more server components from an Eyeball SIP Server configuration without
interrupting the service of the system.
4.3 Benchmark Architecture for System Performance Studies
In order to start an Eyeball XMPP Server system, or an Eyeball SIP Server system, we
should first start MySQL database server, then we need to start at least one state server before we
can start an XMPP edge server or a SIP edge server. From section 4.1 and 4.2, we see both
Eyeball XMPP Server system and Eyeball SIP Server system have the same system architecture.
Thus, we summarized the system architecture in Figure 4.3 that we have used to load test either
of these two Eyeball server systems in this project.
In this diagram, clients represent the computers that are going to run Jabsimul or SIPp
test tools to generate required traffic load, and these computers running clients are in the same
LAN as the computers running the server components of Eyeball Server systems. Further, we
30
only studied the system perfonnance of one Eyeball Server system at a time. In addition, the
corresponding benchmark test tools will generate required traffic load to test either Eyeball
Network XMPP or SIP server systems based on our assumed user behaviour model.
State StateServer ~ Server
1 X.-----------,lSateServer
StateServer
1-------------------------IIIIIIIII
I I
HH I
II,I
IIIIIrII I1 ------------------_.
XMPP/SIP edge server(Multithreads)
XMPP/SIP edge server(Multithreads)
Figure 4.3 Benchmark System Architecture forEyeball XMPP/SIP Server Systems
31
CHAPTER 5: MYSQL DATABASE OPTIMIZATION
Eyeball XMPP server and SIP server are currently using MySQL as their back end data
and statistics storage and administration. The Eyeball Network XMPP server and SIP server
performance heavily depend on the performance of MySQL database server. In the beginning of
this project, default parameters had been used to run MySQL database server, and not much effort
had ever been done to optimize its performance. Although using default MySQL configuration
and parameters in most applications are good enough, our tests showed that these two Eyeball
Network server systems performed badly because of the poor performance ofMySQL server with
default system configuration and parameters. Before we had done any optimization for MySQL
database server, the XMPP server system could only support about 60 thousand concurrent TCP
connections, and SIP server system could only support about 50 thousand concurrent TCP
connections under our assumed user behaviour model.
According to the suggestions of MySQL database optimization in [23], we have tried to
improve MySQL database performance from the following several aspects:
• Optimizing MySQL Statements
• Choosing proper MySQL table types
• Tuning MySQL server parameters
After optimization from these three aspects, we have significantly improved the overall
system performance of these two server systems. Eyeball Network XMPP server system can now
support at least 240,000 concurrent TCP connections, and Eyeball Network SIP server system can
support more than 100,000 concurrent TCP connections with the same user behaviour model.
5.1 Optimizing MySQL Statements
As experts said that "tuning queries (and schemas) can often give you 1000-fold
performance increase" [24], query optimization is the first and most important task to optimize
MySQL performance.
However, for an application with tens or even hundreds of SQL queries, where should we
start? The most effective way for query optimization is first to find out whether there exist some
32
slow queries in the application. We can tum on slow query log in the MySQL configuration file
my.cnf as follows:
log-slow-querieslong-query-time=2
The other way is to use SHOW PROCESSLIST to catch most typical queries used in one
application:
mysql> show processlist;
One major cause for a slow query is often the result of badly defined index or non
existent indexes, so adding appropriate index for a slow query often leads to a significant
performance improvement. One useful tool for such a purpose is MySQL EXPLAIN command.
When we precede a SELECT statement with the keyword EXPLAIN, MySQL will display the
information about the query execution plan from the optimizer. With the help of EXPLAIN,
MySQL can explain how it would process the SELECT query. From the output table of
EXPLAIN, we can find out whether the query has used index or has used index effectively. We
can also use EXPLAIN to check whether the optimizer joins the tables in an optimal order.
Further, we can use EXPLAIN to examine UPDATE and DELETE queries like SELECT queries,
with the extra overhead of writing for UPDATE and DELETE queries.
With the help of those tools, we examined the important queries used by Eyeball XMPP
Server and Eyeball SIP Server systems and found out that a few queries failed to use proper
indexes, such that the queries have to examine every line of some large tables, and resulted in
slow response time for these queries. After adding the proper indexes for corresponding tables,
we see that these queries are optimized.
5.2 Optimize MySQL Storage Engines
Before we made any changes to original MySQI database, when XMPP server or SIP
server was under the load, we ran SHOW PROCESSLIST command in MySQL database server,
and we got some output, as shown in Figure 5.1:
33
Ill't,
•)1
.('~n."
tI
"n
,....
.,(.~t
I·•
~l:;
-';'
=l-l
~b:"
i;~
'=,r
:-:,
~\'l
::t
I I•
till
Irn
aIl
lU :.n;o
Jt1~.·O
::l'I:l:~'~':~<::~~:~o:-.:
:or
'J;n
1:-'
J~I:
:Cl)
J:·.
:O:t
t,o;
t··
,:r.
r;j'
;~i'
,;;T
ot~,
t::.
o;h)
·~'o
.l.;
-".:
,;.n
:ort
:~:o~~~f;:o~~cc:
::-
:'::0
1::
-''li
ar:'!
"lI
',1'
\0;ourc~-
.:::
t:r'
•:-
}-o
-'corm:~
0J'
•!.d
ue:::
-'.:
,:(,
/liL
t.~
I,,;
lr-d
·~":
Ii·~
r",:
t··,
..)(
r:n·
·-l"
'''~
I'I·
·f'1
'fl"
"~n
;;tr
~')l
Il.i
't'l
-r
fr"
1f"I
_lr)
='n,
-"r'
IIf~
':~
:flr
t:;
"!'l
.~()
"\)
..,~
WI
I.IlIJ
L
,.-I
-rl
'.::d
::~;
ctcd
o u
I.'~
~
:::.:c
pQ
u:~1'
Il\l
:r:'
:"",
,11
.....,.~.
'T",
II:;
"rp
-;ro
t"I
:;-"
'I'
"_r'
l,II
::-'
I'"'
,-,1
.]1
S:=<
v
"',~JJ
:>:"'tV
c>;:
)11
cr.':
)11
:"lo
11
1(;
t'I
,'.,
11.
I.I,
(i:
"11
;~.
i
IG
:;39;
.:.~~fJd
Ic..:
1.~,
8::~
11~c
IC:3~j
!~.
J.'.
t'~
IC.:
),:.1:i:~;127
Iti5
:10:
;:1~
Qr.1
(.:]
.•.S
:;5)1~2
IlD
Oi
;:1'
,OC
!C.~
).U
:;63
12)
Ib~JO,
I;~
J~('
CI
lL.:
J.;.
6_:~
jIJ,
1';')
11,-
",,~
II.-I•.
f:,;f.
·!.>1
IIi"
'II,
,-~.
rri
Il~
-II.
,ti
::',
LI,
11:,.
111'
;0."
I,.
~I
II'.II.~.X,~II
IGJl
Il...
i$'t"
dIL
I.'.
f;.~
1'1,
ItD~:
•,.',~J'
IC.:
J.:.
B:i~
j1')
3
I6:if
lll~
~,1
'.t"J
le:1
.:.S
UJI
J?It
1~L
I~l"Od
IC.:.
J.1.
S::5
3J'!']
I65J
J:I
;"3'
OC
LC.5
).d
::m
·t!IO
:.!J.
:-71
'''~
Ll.";
·.I,
J,C
:.:bJ
I1.
Ili-
J'
t-"~r
II'.
-.1.
.~;
:b'1~
.
I,,,'
J;""
""t.
~IL
',J.
'.li
~:b
Hr,I
-:·t
,.IJ
QI'-
,),
III
I...d
",,J
~u
p..
,I,..
i.rl
jJlJ
.-..
...l
".\'
~~I
~t,.._
I11
1-l-
..../~,lh~~).-~
.u.U.1.}:~
~._~;./
IEllt.
..~...
\·~
l..d
l..~!-·:
~r.,
~•.
e:'·c
:',ll
i:~,:~·
I0
I-,}
J.t..
_,...
.'1",
Iv~'
Yr'=
.h-,..
_~._
V!,'
·"..
\U~
'l.I
:'.:
',,'.G
Bi'"
,:vi,I
,-;'_
'1'.
",:n
t1i
,:"'
ll,;
I.:S
c:"l
;.~~..
,.7'
..:':
/~:'
lll
Qu~;l'
I0
I:.d
..oJ
1J~I}Jk
Z""'I-':t}'~I
...~·
(l;'
t3'
,:,'
1;'
":'.
JU
.:,,
!••
',,l,ol
;~~H
•,0
.:0,
1,8
::(1
:C':,
'.'IE
~~t'
:;~g
't.l
;Joj
J~~·
~::
r.:,:.:
[~tl11
:;::c
pI
0I
IM_L
('I:
bll
VJt:
r<'
I0
I·::d~t:.-'1:
U\l
:OIC
!:!l
iPr,
r~~'
l!:c
:l;c
t:;
':-~
.'e:
nm
rl::
',1l
.E~,
:~·
.0.:0
.J.b
,:&
::C
::·,E
c~Q~
(r"c
~l.j
dl~o
~-.:
6:,l.:
;blll
;',:
(0I
~I
IlU.l.
':r'
,1\
;;'r
rI
III
flUI
~:r'
r\11
~.
'f't
III
IJlr1
.li;,l~.
)."
.JIf.·I..f:i:~I!,,
,:,.
!dJ
':"\'
IIII
till
I.
IiiI
'!',
'1,
,'/';,
1>'"
-:"h
lli~
"'I)
Ifl
ll'l
ll",[
,v:r
,-.!
"h
"
le:3
1.iJ
""U
1(,
:').
:,1
2;)
):1
:t"
!~:'
~]l
S:el
:\lI
01II
lLl
IC:M
1iJ
'i-.J
lCJ.:
J.:.
);:1
J~,:
':!
e'l,t
..11
s::~,
Il~
9I
Im
uIt
i5~)J
!.:v:
1C.:').J.~):.~.~'
E'1
?:';]
!$.~.p
I::
'I
IlU.l
+_----.-+----.-.--~-----
..-.
---.
----
.---
---.
---.
+_-
---.
+_-
.---
----
---.
.---
---.
.---
----
----
--..
.---
----
----
----
----
----
----
----
----
----
-·--
----
----
----
---t
~-r)
'I::t
..:
::ta
O:
=:~/
Fig
ure
5.1
Asc
reen
shot
for
My
SQ
LS
HO
WP
RO
CE
SS
LIS
Tco
mm
and
34
From this output, we see many query processes are in "locked' state. Why is that? Before
we can answer this question, we need to understand the concept of MySQL storage engines, and
how to choice correct table type (see [25]).
As we know, MySQL uses storage engines, or table types, to decide how to store tables,
and how and where a database table is to be stored. Currently there are four types of storage
engine defined in MySQL database:
• MyISAM
• InnoOB
• Memory
• NOB.
Memory storage engine type utilizes only computer RAM. Because it keeps all tables and
data in memory, it is very fast, but it requires large enough RAM to store those tables and data.
Thus for a large database such as used in our two server systems, memory storage engine is not a
feasible choice for the tables used in this database. NOB is a storage engine designed for MySQL
Cluster database, so it is also not the choice for our current MySQL database configuration.
MyISAM is MySQL default storage engine, so was the storage engine for all the tables
used in our MySQL database. MyISAM is a disk-based storage engine, and does not support
transactions in order to achieve low overhead. In order to achieve a very high lock speed,
MyISAM uses table-level locking, with which many threads are allowed to read a table at the
same time, but a thread must obtain a table lock to get exclusive access to write to the table, and
all other threads that want to access this same table must wait until the update is done. So
MyISAM storage engine for a table may not be a good choice if there are many mixed select and
update operations on the same table, one may be able to improve database performance by
converting some tables from MyISAM storage engine to InnoOB storage engine.
InnoOB is also disk based storage engine with fully ACID transactional capabilities.
InnoOB use a concept of table space to store all structures, table data and indexes, and requires
about three times as much disk space as MyISAM. Further in order to obtain high speed, InnoOB
use memory caching aggressively. InnoOB does not require any lock for a SELECT. As for
updates, only row-level lock is used. This makes it is possible for extremely high concurrent table
operations even when there are mixed selection and update operations on one table.
Arjen Lentz suggests some general rules to decide whether MyISAM storage engine or
InnoOB storage engine should be used for a MySQL database table [25]:
35
• If we require multi-statement transactions, advanced isolation levels and row-level
locking, foreign key constraints, or otherwise have a requirement for ACID features, go
for InnoDB. Otherwise, simply use MyISAM.
• If we have an application with lots of selects, inserts as well as updates, InnoDB is
probably the generally appropriate choice.
For Eyeball XMPP and SIP servers, originally all the tables in MySQL database use
MySQL default MyISAM storage engine. After closely examining the table operations used in
XMPP server and SIP server applications, we found that for some large tables, there exist mixed
insertion, updating or deletion operations on those tables. So in such a heavily-loaded multi
threaded system, mixed insertion, updating or deletion operations on one table require one thread
to obtain a table lock in order to exclusively access the table, and all other threads have to wait for
the availability of this lock. That is why we saw so many SQL processes are in "locked" status
when we run SHOW PROCESSLIST above when the system is under the load. The table-level
lock for concurrent insertions, updating and deletion operations on one table with MyISAM
storage engine thus cause very poor overall system performance, and this has been verified by our
performance test results. Our tests showed that with default MyISAM storage engine for these
tables, both XMPP server and SIP server would have become overloaded when each server tried
to handle less than 60 thousand concurrent users under our assumed user behavior model.
Once we change those tables to use InnoDB storage engine, under the same test
environment, both XMPP server and SIP server system could handle about more than 90
thousand concurrent users even without tuning any parameters for the MySQL server. However,
we know we can achieve much better system performance of MySQI server if we can tune the
MySQL Server configuration properly.
5.3 MySQl Server Tuning
As the most popular Open Source SQL database management system, the MySQL server
has many features that can be configured to best fit the system hardware and applications.
Although the default configurations are fine for most applications, it proves that many of them
need to be tuned carefully in order to improve the over all system performance. MySQL provides
many ways for us to check the MySQL server configuration and runtime statistics so that we can
tune MySQL server parameters based on these runtime values [26] [27].
For a MySQL server that is currently running, we can run the following statement and see
the current values of its system variables:
36
mysql> SHOW VARIABLES;
We can also examine various statistical and status information for a running MySQL
server by issuing the fol1owing command:
mysql> SHOW STATUS;
System variable and status information for a runmng MySQL database also can be
obtained using mysqladmin in a shel1 terminal
shel1> mysqladmin variablesOr
shel1> mysqladmin extended-status
For InnoDB table, we can also issue the fol1owing command to view internal INNODB
runtime stats of MySQL server
mysql> SHOW INNODB STATUS
By watching these statistical and status information for a runtime MySQL server
provided by these tools, we can tune related system parameters. Even though there are quite a lot
of variables that could be tuned for a MySQL server, only a few of them are the most important
variables that need to be tuned for a large performance gain. Once we feel confident that we have
these variables set appropriately, changing any other variables wil1 mostly offer incremental
performance improvement. So next we will examine these parameters in more detail, and explain
how to tune them to optimize a MySQL server.
key-buffer_size
key_buffer_size IS very important if we have MyISAM tables. It is used by all the
indexes, and should be large enough to contain them, namely, the total size of al1 the .MYI files in
the server. It is usual1y set up to 30-40% of a computer's physical memory size. We can check the
following few status variables to find out whether we need to increase this value:
• Key_readJequests : The number of requests to read a key block from the cache.
• Key_reads: The number of physical reads of a key block from disk.
• Key_writeJequests : The number of requests to write a key block to the cache.
• Key_writes: The number of physical writes of a key block to disk.
We should keep the ratio of KeyJeads : KeyJead_requests to be less than 1: 100 and the
ration of Key_writes : Key_write_requests always be less than 1. Otherwise, it is the time to
increase key_buffer_size.
37
table cache
Opening tables can be expensive, so it is always good to size table_cache to be large
enough to keep most of tables open in cache. We can check whether we need to have this value
increased by checking the following status variables during peak time:
• open_tables: the number oftables opened in cache
• opened_tables: the total number of tables opens.
If open_table equals table_cache and opened_tables is very high, you need to increase
table_cache to reduce the number of opened_tables.
Because Innodb tables are used in our database, we also need to tune the following
variables that are dedicated to Innodb tables:
Innodb_buffer-pool_size
It is the size of the memory buffer that InnoDB uses to cache data and indexes of its
tables. The larger we set this value, the fewer disks I/O is needed to access data in tables.
Innodb_log_file_size
It is very important for write intensive workloads especially for large data sets. Larger
sizes often offer better perfonnance but increase recovery times.
Here is an example to show how we decide whether the value of innidb_buffer~ool_size
and innodb_log_file_size is proper.
We once ran "SHOW INNODB STATUS" during the test, and looked at "FILE 10"
section of the output, it shows:
520155 OS file reads, 146300 OS file writes, 7609 OS fsyncs, 1106.36 reads/s, 17133
avg bytes/read, 301.87 writes/s, 15.05 fsyncs/s
We noticed that 1106.36 reads/s is very high amount of reads. The reason is default
lnnodb_buffer~ool is only 8M, so we increased it to 1000M for a computer with 2GB physical
memory.
Further, in "BUFFER POOL AND MEMORY" section, it showed:
Pages read 1277679, created 16248, written 1922267
85.24 reads/s, 4.03 creates/s, 642.08 writes/s Buffer pool hit rate 999 / 1000
38
89275 log i/o's done, 73.81 log i/o's/second
We have nearly eight times write rate than read rate, which was not expected. The reason
is that we used default value ofinnodb_log_file_size=5MB, which is too small, so we increased it
to 5l2M.
innodb_flush_logs_at_trx_commit
innodb_flush_logs_at_trx_commit is another important variable for Innodb table. It can
be value of 0, 1 or 2. Default value of 1 means each update transaction commit will need to flush
log to the disk, which is rather expensive. When it is set to 2, the log buffer is written out to the
file at each commit, but the flushing on the log file takes place once per second. When it is set to
0, the log buffer is written out to the log file once per second and the flush to disk operation is
performed on the log file, but nothing is done at a transaction commit. Using value °is a bit faster,
but less secure than using value 2. Because in both cases, the log is flushed to the disk each
second, so we normally would not lose more than 1-2 sec worth of updates, but get a much
performance gain. In our project, we set this value to 2 to obtain a much better performance for
InnoDB tables.
Tuning the variables properly is critical to achieve a good MySQL performance. Of
course, they are not the only parameters we should tune; many other parameters might need to be
further tuned in order to achieve a better MySQL server performance. For more information
about how to tune a MySQL server, we can further reference to [24] [28]
39
CHAPTER 6: MEASUREMENT AND ANALYSIS OFSYSTEM PERFORMANCE
In this chapter, we start to describe various performance measurements and studies that
we have done for Eyeball XMPP Server and SIP Proxy Server systems.
6.1 System Configuration for Benchmark System
All benchmark tools and Eyeball Server systems are running on the computers of the
same LAN, which is Eyeball Networks Peer I Collocation network. This network contains 8
computers. Each computer has the following hardware configuration with installed operating
system:
• Dual-processor with inter Xeon CPU speed 3.00GHz and 2M cache
• 2GB physical memory with the same amount of swap space.
• Installed OS of Linux Fedora core 6 with kernel version 2.6. I8
In order to study Eyeball XMPP Server system performance with currently available
hardware resources, we assigned one computer to run MySQL database server, and one computer
to run state servers. Further, four computers were used run labsimul benchmark tool as XMPP
clients to generate required user traffic load. For the other two computers in the network, we
could run only one Eyeball XMPP edge server on one of these two computers, or two Eyeball
XMPP edge servers, each running on one computer. For simplicity, in the rest of the project, we
call it one-XMPP edge server system if we only run one XMPP edge server in the system, and we
call it two-XMPP edge server system if we run two XMPP edge servers in the system. The
system configuration for benchmarking Eyeball XMPP Server system is shown in Figure 6.1.
Similarly, in order to study Eyeball SIP Proxy Server system performance with available
hardware resources, we assign one computer to run MySQL database server, one computer to run
state servers, and one computer to run SIP Proxy edge server. Further, the other five computers
were used to run SIPp benchmark tool as SIP clients to generate required user traffic load. The
system configuration for benchmarking Eyeball SIP Proxy Server system is shown in Figure 6.2.
40
-r------------------.: Te\1 CU(>1l1') :, ,: Ruunillg .Jnb\ilnlll :I ,I ,, I, ,, ,I ,: _~_-----J
,
-
-<"- --
-~-
X'IPP edge ,el'Wl' 1
X'JPP Nlg~ "e,t"' ...
'hSQL Dnl~b;)~e
Figure 6.1 System configuration for EyeballXMPP Sel-ver benchmark system
T4?''\t C'lteuh
Runnln!/. SIPp
Figure 6.2 System configuration for Eyeball SIPProxy Sel"Ver benchmark system
41
There are many default Linux System parameters that might limit maximizing the XMPP
and SIP Server performance. Therefore, before we start the system performance test, we need to
consider tuning the following kernel parameters of Linux operating system:
#max open files
shell>echo 262144> /proc/sys/fs/file-max
# kernel thread
shell>echo 131072 > /proc/sys/kernel/threads_max
#port range
shell>echo "1024 65535" > /proc/sys/net/ipv4/ip_Iocal---'portJange
#netdev backlog
shell>echo 4096 > /proc/sys/net/core/netdev_max_backlog
#socket buckets
shell>echo 131072 > /proc/sys/net/ipv4/tcp_max_tw_buckets
# socket buffer
shell>echo 111616 > /proc/sys/net/core/wmem_default
shell>echo 4194304 > /proc/sys/net/core/wmem_max
shell>echo 111616> /proc/sys/net/core/rmem_default
shell>echo 4194304 > /proc/sys/net/core/rmem_max
Another important kernel parameter that might limit XMPP and SIP server performance
is ip_conntrack_max, which controls the maximum number of allowed conntrack entries for
netfilter [30]. We can check the value of this parameter as follows after Linux kernel version
2.4.23:
shell>cat /proc/sys/net/ipv4/netfiIter/ip_conntrack_max.
42
For systems with more than IGB of RAM, default ip_conntrack_max value is set to
65536. We should set this parameter to be a larger number since we might expect hundreds of
thousand TCP connections to one XMPP or SIP server. In this project, we set ip_conntrack_max
value as follows:
shell>echo 131072 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
Ifwe set these parameters this way, all setting will be lost once we reboot a computer. So
we can edit /etc/sysctl.cnf to provide automatic reboot setting if we decide to use these parameter
settings systematically.
6.2 Limitations of the System Performance Study
The goal of this project is to study the system performance of two Eyeball Networks core
product servers, namely, XMPP server and SIP Proxy server. However, our performance studies
have the following two limitations:
• Limitation of physical memory: Because of the limitation of physical memory on the
computers that run the XMPP server, one XMPP server can only support about maximal
130,000 concurrent TCP connections. Operating system OOM application will kill
XMPP server if Jabsimul clients try to add more TCP connections to the server beyond
that number. In order to avoid this "out of memory" problem, we limit the number of the
maximum concurrent TCP connections to one XMPP server to be 120,000. Further,
because each Jabsimul process simulates 30,000 concurrent active users, as we
mentioned before, in the following tests, we need four Jabsimul test clients to simulate
120,000 concurrent TCP connections to load-test one XMPP server, and eight Jabsimul
test clients to simulate 240,000 concurrent TCP connections to load test two XMPP
servers. Generated traffic load for each TCP connection follows our assumed user
behavior model.
• Limitation of available computers: Because there are only eight computers available in
our test bed, we did not have enough computers to generate enough simulated active
users to overload our XMPP server and SIP server; therefore, all the tests done in this
project have to be under this resource limitation.
In next two sections, we present detailed descriptions about the various performance
studies that we have done with Eyeball XMPP Server system and SIP Proxy Server systems,
within these two limitations.
43
6.3 Performance Study for Eyeball XMPP Server system
6.3.1 Jabsimul test tool configuration
We used Jabsimul benchmark tool to study the system performance of Eyeball XMPP
Server system. The system configuration for this benchmark test has been shown in Figure 6.2.
The Jabsimul sends XMPP stanzas that comply with Internet Engineering Task Force (IETF)
Requests For Comments (RFC) 3920 and 3921 [13] [14]. Each message that the Jabsimul
benchmark tool sends is an XMPP "events", and is configured in its XML format configuration
file. In this performance study, Jabsimul benchmark tool generates login and logout events, sends
chat message events, changes presence events, and updates roster list events with defined the
frequency according to the following user behaviour model:
• Avg. Session Time: 3600 seconds
• Avg. Message Sending: 0.01 Sent/Sec/Subscriber
• Avg. Presence Updates: 6 Updates/Session/Subscriber
• Avg. Roster Updates: 0.5 Updates/Session/Subscriber
Based on this user behaviour model, we can calculate corresponding parameters used in a
Jabsimul configuration file. Here we give an example to show how to calculate login frequency in
Jabsimul configuration file.
Jabsimul will create a separate TCP connection to XMPP server for each simulated user.
If we assume each Jabsimul process manages 60,000 users, when it enters its stable stage, at
which the Jabsimul process have the same login and logout rate for simulated users, there should
be 30,000 concurrent active users within each Jabsimul process.
Assume login rate at the stable stage is R logins/second. Because we have 30000
concurrent active users with average session time of 3600 seconds for each user at the stable stage,
by Little's law [29], we can have:
R logins/second * 3600 seconds =' 30000 logins
So R =' 8.3 logins/second
With total 60,000 managed users within this Jabsimul process, the login frequency for
each simulated user is:
60000users / 8.3users/second =' 7200 seconds.
44
It should be straightforward to calculate other parameters in this configuration file. The
labsimul configuration file for Eyeball XMPP Server system based on our user behaviour model
is shown in appendix I.
6.3.2 Eyeball XMPP server system configuration
To achieve the best system performance for Eyeball XMPP Server, we can specify the
number of threads running within one XMPP server, and the number of state servers. In this test,
we studied how to configure Eyeball XMPP Server system so that it could support the maximum
concurrent TCP connections under assumed user behaviour model. We measured how many
concurrent TCP connections the system could support before XMPP edger sever reported "server
busy" in its log file with different combinations of numbers of thread and state servers, and the
test result is shown in Table 6.1. In this table, the farthest left column shows the thread number
within one XMPP server, and the top row shows the running number of the state server. Further,
because of the "out of memory" limitation, we only load tested one XMPP server up to maximum
of 120k concurrent TCP connections.
Table 6.1 Maximum system throughput with different serverconfigurations
Number of Number of State ServersThreads I 5 10 20 40
I 15 K 15 K 20K 20K 20K
4 35 K 60K lOa K 100 K 100 K
8 55 K 80K 110 K 120 K 120 K
16 65 K 100 K 120 K 120 K 120 K
32 85 K lOa K 120 K 120 K 120 K
From this table, we see that if we run fewer than four threads within one XMPP server, or
run fewer than five state servers, the XMPP server system performance is always suboptimal, and
will not be able to reach 120k concurrent TCP connections. In other words, we should run at least
16 threads within I XMPP server and 10 state servers in the system in order to achieve better
system performance. To be safely, in our subsequent tests, we will run 32 threads within 1 XMPP
server, and 20 state servers correspondingly to avoid this suboptimal system performance because
of bad server system configuration.
45
6.3.3 XMPP edge server incoming buffer size
Eyeball XMPP edge server implementation maintains an incoming message queue buffer,
which is used to store incoming requests from clients. When an XMPP Server system is under the
load, it may not be able to handle the incoming XMPP requests fast enough; the incoming
message will fill the buffer queue very quickly. Once the server detects that the incoming
message buffer queue is almost full, it considers the system is in busy status, and then refuses to
handle more traffic load by resetting any new incoming TCP connection requests. At the same
time, the XMPP edge server logs "server busy" messages in the server's log file. Therefore, in
our system perfonnance study, we have used this log infonnation to decide whether the XMPP
Server system is under the load.
The incoming message buffer size is hard-coded in XMPP source code, but we do not
know whether the default value is set properly. In this test, we want to know whether different
incoming message buffer sizes may influence the XMPP server perfonnance. Because we wanted
to check how different incoming buffer sizes influence the overall system perfonnance, we ran an
XMPP edge server with only 4 threads and 10 state servers in these tests. Then we tested how
many concurrent TCP connections the Eyeball XMPP Server system can support with different
predefined incoming message buffer size, as shown in Table 6.2. The last column in the table
shows the average response time of sending message when the server has been overloaded.
Table 6.2 Maximum system throughput with different IncomingMessage ButTer Size
Incoming Message Maximum TCP Send MessageBuffer Size Connections Response Time
300 102 K 200 ms ~300 ms
600 102 K 300 ms~600 ms
1200 101 K 1 s ~ 1.5s
2400 104K 2s ~ 4s
4800 104 K 20s ~ 30 s
9600 103K 60s ~ 70s
From above table, we can see that with different message buffer sizes, the Eyeball XMPP
Server system can support almost the same number of concurrent TCP connections, but message
46
response time increased accordingly. Therefore, we can conclude that incoming message buffer
size does not affect the system throughput, but it does influence the message response time when
the XMPP Server system is under the load. So in order to obtain reasonable response time even
when the XMPP Server system is under the load, we define the default buffer size for incoming
message queue as holding 600 messages.
6.3.4 Message response time for XMPP server system
As we mentioned above, labsimul test tool can measure average response time of sending
messages, which is the time from labsimul sending a message to one online user until it receives
the echoed message from the peer. In this study, we measured average message response time in
an one-XMPP-edge-server system when labsimul test clients increase traffic load generated from
o up to 120 thousands concurrent TCP connections. The test result is shown in Figure 6.3. We
also measured average response time of sending messages in a two-XMPP-edge-server system
when labsimul test tools increased traffic load generated from zero to 240 thousands concurrent
TCP connections, and the test result is shown in Figure 6.4.
From these two figures, we see that the average response time for sending messages
increases linearly with the number ofTCP connections. We also noticed that the average message
response time for one-XMPP-edge-server-system with traffic load generated by 120,000
concurrent TCP connections is less than l40ms; and the average message response time for two
XMPP-edge-server system with traffic load generated by 2400,000 concurrent TCP connections
is less than 160ms, which are both still reasonably small. Further, we did not have any "server
busy" log messages in XMPP server's log file. Therefore, it is safe to say that Eyeball one
XMPP-edge-server system can support at least 120,000 concurrent TCP connections, and two
XMMP-edge-server system can support at least 240,000 concurrent TCP connections, while each
TCP connection has traffic load compliance to our assumed user behaviour model.
47
160
140----if]
-5 120(j)
60'-;
100E-
(j)if]
c 8000.if](j)
0:: 60(j)b/)()jif] 40if](j)
:::;;:
20
00 20 40 60 80 100
Concurrent rcp Connections (*1000)120 140
Figure 6.3 Message Response Time of different loads forone-XMPP edge server system
160
140----if]
-5 120(j)
s0'-;
100E-
(j)if]
c 800e-if](j)
0:: 60(j)b/)()jif] 40if](j)
::?
20
00 50 100 150 200 250 300
Concurrent rcp Connections (*1000)
Figure 6.4 Message Response Time of different loads fortwo-XMPP edge server system
48
6.3.5 Memory usage for XMPP server components
In this test, we used Linux "top" tool to measure the memory usage for different server
components in Eyeball XMPP Server system. When one-XMPP-edge-server system reaches its
stable stage with 120k concurrent TCP connections under assumed user behaviour model, the
total virtual memory utilization of corresponding server components is shown in Table 6.3.
Table 6.3 Memory utilization for one-XMPP server system with120K TCP connections
Components Virtual Memory Utilization
XMPP edge server 987 MB
State server 1396 MB
MySQL server 1658 MB
Further, when a two-XMPP-edge-server system reaches its stable stage with 240k
concurrent TCP connections under assumed user behaviour model, the total virtual memory
utilization of corresponding server components is shown in Table 6.4.
Table 6.4 Memory utilization for two-XMPP server system with240K TCP connections
Components Virtual Memory Utilization
XMPP edge server! 987MB
XMPP edge server! 992MB
State server 1918 MB
MySQL Server 1672 MB
Because for each simulated user, labsimul creates a TCP connection to XMPP edge
server, we want to know what the average memory usage of one TCP connection is for the
Eyeball XMPP edge server. We used Linux "free" tool to measure the free memory changes in
the computers running Eyeball XMPP edge servers, and the result is shown in Figure 6.5.
49
2500000
2000000 ~~""'"
00::oc:::'--' -(fJ 1500000
Q.l
;...,0EQ.l
~ 1000000Q.lQ.l;...,
CL.
500000
oo 20
Figure 6.5
40 60 80 100Concurrent TCP Connections (*1000)
XMPP Server Free Memory underdifferenttrafiftcloads
120 140
From Figure 6.5, we can see:
I. The memory usage of the computer running XMPP edge server are decreased linearly
with the number of concurrent TCP connections
2. The average memory usage for each TCP connection in XMPP edge server is about:
(l998MB-1610MB)
= 388MB /120000 TCP connections
=3.2KB/ per TCP connection
We should notice that memory usage per TCP connection depends on many factors, such
as size of roster, type of authentication etc., and different usage patterns may result in different
outcomes.
6.3.6 CPU usage of Eyeball XMPP server components
Another important parameter related to XMPP server system performance is the CPU
usage of each component in the XMPP server system when they are running under a certain
traffic load. Here we used Linux tool "vmstat" to measure the CPU idle time of computers
running XMPP server, State server and MySQL database server. Table 6.5 shows the CPU usage
50
for one-XMPP-edge-server system under the traffic load generated by 120,000 concurrent TCP
connections, and Table 6.6 shows the CPU usage for two-XMPP-edge-server system under the
traffic load generated by 240,000 concurrent TCP connections. The values under "Initial" column
is the CPU idle time of the computer running the corresponding system component before any
traffic has been generated by labsimul test clients, and the values under "Final" column is the
corresponding CPU idle time when the system has been in the stable stage with expected traffic
load.
From these two tables, we should have noticed that all computers running XMPP edge
server, state server and MySQL database server still have much free CPU time even at their final
stable stage.
Table 6.5 CPU Usage for components of one-XMPP edge serversystem with 120 thousand concurrent TCPconnections
Component Initial CPU Idle Final CPU Idle
XMPP server 100% 63%
State server 100% 73%
Database server 99% 82%
Table 6.6 CPU Usage for components of two-XMPP edge serversystem with 240 thousand concurrent TCPconnections
Component Initial stage CPU Idle Final stage CPU Idle
XMPP server I 100% 52%
XMPP server2 100% 48%
State server 100% 36%
Database server 99% 56%
51
6.3.7 Bandwidth usage of Eyeball XMPP server components
In our test environment, all computers running labsimul clients and Eyeball XMPP
Server system components are in the same LAN of a switched 100Mb/s Ethernet, and there is
only measurement-related traffic and no other traffic in the LAN during our XMPP performance
tests. Is it possible that that generated network traffic loading become a bottleneck for our
performance tests? We measured the bandwidth usage for each Eyeball XMPP Server system
component when they were loaded with our expected traffic load with Linux tool "IpTraf'. Table
6.7 shows the bandwidth usage for one-XMPP-edge-server system under the traffic load
generated by 120,000 concurrent TCP connections, and Table 6.8 shows the bandwidth usage for
two-XMPP-edge-server system under the traffic load generated by 240,000 concurrent TCP
connections.
From these two tables, we see that the bandwidth usage for each system component in the
two-XMPP server system with 240K concurrent TCP connections is about twice as much as the
bandwidth usage for one-XMPP server system, as we expected. Further, the maximum bandwidth
is the outgoing traffic generated by state server, which is about 20Mbits/second, and is still much
less than 100Mbits/s. Therefore, it confirms that the generated network traffic in our tests so far
has not saturated our Ethernet LAN, and should not be a bottleneck for our XMPP server system
performance tests
Table 6.7 Bandwidth utilization for One-XMPP edge server system with120K concurrent TCP connections (Kbits/s)
XMPP server State Server Database Server
Incoming outgoing Incoming outgoing Incoming outgoing
4907 5858 6607 10482 1891 1556
Table 6.8 Bandwidth utilization for two-XMPP edge serverssystem with 240K TCP connections (Kbits/s)
XMPP server 1 XMPP server 2 State Server Database Server
Incoming outgoing Incoming outgoing Incoming outgoing Incoming outgoing
10272 11238 10067 11215 13839 19208 3516 3002
52
6.4 Performance Study for Eyeball SIP Server System
In this study, SIPp benchmark tool has been used to load test Eyeball SIP Server system
and study its system performance. The system configuration for this benchmark test has been
shown in Figure 6.2. Even though Eyeball SIP Proxy server supports UDP and TCP protocol, we
are mostly interested in testing SIP Server using TCP protocol, which is supposed to consume
many more resources than using UDP protocol.
6.4.1 RPS and CPS of Eyeball SIP server system
Several important SIP server performance metrics have been defined in [20]. Among
them we are interested in Registrations per second (RPS), Calls per second (CPS) and
Transaction Response Time (TRT) for Eyeball SIP Server system. It also described how to
measure and report these values. However, in Eyeball SIP Server implementation, the SIP server
use the same way as Eyeball XMPP server to detect whether the system is getting busy, and also
report this information to users in its log file. Therefore, in the subsequent tests of this system
performance study, we used this log information to determine the RPS and CPS values. Namely,
we increased the request rate generated by SIPp benchmark tool until the SIP server reported
"server busy" in its log file, and then the highest sustained throughput is reported as the
corresponding RPS and CPS value.
In Table 6.9 and Table 6.10, we display the values of RPS and CPS together with the
CPU usage and response time corresponding to RPS and CPS. With current system configuration,
we see that RPS is about 780 registrations/seconds, and CPS is 350 Calls/second.
Table 6.9 CPU Utilization and Transaction Response Time formaximumRPS
Registration rate SIP serverState
Database(Registrations/second) CPU idle
serverCPU idle
Response TimeCPU idle
780 72% 95% 40% 340 ms
53
Table 6.10 CPU Utilization and Transaction Response Time formaximum CPS
Call rate SIP server StateDatabase
serverCPU Idle
Response Time(Calls/second) CPU idle CPU Idle
350 32% 90% 36% 280ms
To further study the Eyeball SIP system performance in this project, we are going to
assume the following user behaviour model:
• REGISTER transactions per second: 20.8 per lOOk users
• INVITE transactions per second: III per lOOk users
• ACK transactions per second: III per lOOk users
• BYE transactions per second: III per lOOk users
Each simulated user in SIPp benchmark tool will connect to Eyeball SIP edge server with
a separate TCP connection and generate traffic load compliance to above user behaviour model.
As we mentioned before, we have modified the SIPp test tool so that it can integrate REGISTER
transaction, call establishing and releasing transactions in a single call scenario and the call flow,
as shown in Figure 6.6. Based on our user behaviour model, each user will make an average of
five calls after its registration to the SIP proxy server, and each single call will include both the
INVITE and corresponding BYE transaction. Here in Figure 6.6 we only show one call setup and
teardown transactions after each registration transaction for simplicity.
54
UAC SIP Proxy UAS
RegisterI,
40 I Unauthorized
Register
200 OK
INVITE~
100 Trying
INVITE
180 Ringing
180 Ringing
200 OK
200 OK
SIP ACK
SIP ACK
SIP BYESIP BYE
200 OK
200 OK
Figure 6.6 A Call flow with Registration, Callsetup and teardown
Since the computer running SIPp test tool has dual processors, and the SIPp executable is
single-threaded, we can run two SIPp executables in one client computer, one running as a UAC,
and the other one running as a UAS at the same time. However, each SIPp executable can only
simulate about 10,000 concurrent TCP connections because the processor running the SIPp
executable will have run out of its CPU resources with 10,000 concurrent TCP connections.
Therefore, one computer running two SIPp executables can totally simulate 20,000 concurrent
55
TCP connections. Based on our limited hardware resources, with only five computers available to
run SIPp clients, we can run 10 SIPp clients, and in total simulate about 100 thousand concurrent
registered users with TCP connection to our Eyeball SIP Server.
Based on our user behaviour model, for 100 thousand concurrent users, there will be 20.8
registrations/second, and we have totally 10 SIPp clients, so each SIPp client should have 2
logins/second.
Thus, we should run one SIPp executable as UAS with following command:
>.Isipp -sf server.xml -inf server.cvs -I localhost -p 5060 sipp.eyeballchat.com-I 12000 -max socket 65536 -t tn -r 2
In addition, run the other SIPp executable as UAC with following command:
>.Isipp -sf c1ient.xml -inf c1ient.cvs -I localhost -p 6060 sipp.eyeballchat.com -I12000 -max socket 65536 -t tn -r 2
Under this test configuration, SIPp generates the following traffic load for the SIP Server
system, which is very close to our assumed user behaviour model, and we will use this test
configuration for all tests in our system performance study for Eyeball SIP Server system:
• 20 registrations per second per lOOk registered users.
• 100 invitations per second per lOOk registered users.
• 100 ACK messages per second per lOOk registered users
• 100 BYE messages per second per lOOk registered users.
6.4.2 Transaction response time of Eyeball SIP server system
As defined in [20], the TRT of a transaction is "the time elapsed from the first byte sent
by a UAC to initiate a transaction until the last byte received by the UAC that ends the
transaction". SIPp benchmark tool provides a way to allow users to define a timer to measure the
TRT of a transaction. In this test, we measured TRT of registration transaction and invite
transaction when Eyeball SIP Proxy Server system has been loaded with traffic load generated by
10k, 20k up to lOOk concurrent TCP connections, and the test result is shown in Figure 6.7:
56
-------- ---45~
40lfJE
'-/
Q) 35E
.-<b 30
Q)
lfJc
250QlfJQ)
200:::
C0 15+-'ucD 10lfJCcDH 5b
00 20 10 60 80
Concurrent rcp Connections (*1000)
,I.
100 120
-+-TRT for registrations TRT for Call s
Figure 6.7 Transaction Response Time underdifferent traffic load
From Figure 6.7, we see that TRT for both registration transaction and invite transaction
are still less than 50ms even when the SIP server reaches 100,000 concurrent TCP connections,
which is much smaller than the TRT constraint of 2.0 seconds for a failed transaction defined in
[20]. Further, it shows that all transactions in SIPp executables succeed; and the Eyeball SIP edge
server does not report that the server is getting busy in its log file. Thus, we can conclude that
under the current system configuration, the traffic load generated by 100 thousand concurrent
TCP connections does not overload the Eyeball SIP Server system, and the system should be able
to support more than 100 thousand concurrent TCP connections under our assumed user
behaviour model.
6.4.3 Memory and CPU usage of Eyeball SIP server components
As requested in [20], to study a SIP system performance, we also need to measure CPU
and memory utilization of the components in SIP server system with various loads. First we
measured the maximum virtual memory utilization for each component in Eyeball SIP Server
system when the server system reaches its stable stage of lOOk concurrent TCP connections under
our assumed user behaviour model, and the test result is shown in Table 6.1 I.
57
Table 6.11 Memory utilization for SIP Server system with lOOKTCP connections
Components Virtual Memory Utilization
SIP edge server 980 MB
State server 720MB
MySQL Server 1660 MB
Because each simulated user in SIPp clients connects to the Eyeball SIP edge server with
a separate TCP connection, we also want to know what the average memory usage of one TCP
connection is for the Eyeball SIP edge server. We use Linux "Free" tool to measure the free
memory in the computer running Eyeball SIP edge server, and the result is shown in Figure 6.8.
From Figure 6.8, we see that memory usage of Eyeball SIP edge server almost linearly
increases with the number of TCP connections, and the average memory usage for one TCP
connection is about:
(1987MB - 1162 MB)/1 00,000 TCP connections
= 8.25 Kbytes/TCP connection
CPU utilization of Eyeball SIP edge server, state server and MySQL server under the
di fferent test loads generated from 10K, 20K up to lOOK concurrent TCP connections with our
user behaviour model is also shown in Figure 6.9. We should notice that all computers running
Eyeball SIP edge server, state server and MySQL database server still have much free CPU time.
58
§ 2500000 r~ 2000000~;>...(J)
if; 1500000CL>-<Ul
"-<oA 1000000'-oE(J)
~
())(j)H
LL
.'lOOOOO
o 20 40 60 80 100
Concurrent rcp Connections (*1000)
120
Figure 6.8 SIP edge server free memory underdifferent traffic loads
12010040 60 80Concurrent rcp Connections (*1000)
20
==-.:=tl----ll------·~ --l...... I__..- ____
120
t-100
~80
())
E
f-
Q!60
~
-0
:.:J ItO0-C!
20
00
--.-Sip Server CPU IdJe-~-State Server CPU IdleMysql Databse CPU Idlr
Figure 6.9 CPU Idle Time of SIP Servercomponents
59
6.4.4 Bandwidth utilization of Eyeball SIP server components
In our test bed, all computers running SIPp clients and SIP server components are
connected by a switched 100Mb/s Ethernet. There is only measurement-related traffic, and no
other traffic in the LAN. We measured the bandwidth utilization of incoming and outgoing traffic
for the computers running SIP proxy server, state server and MySQL database server, which have
been loaded with the traffic generated by lOOK concurrent TCP connections under our user
behaviour model. The measured result is shown in Table 6.12. From this table we can see that the
maximum bandwidth is the outgoing traffic generated by SIP proxy server, which is only about
5.6Mbits/second. Thus, we can conclude that the network traffic is not a bottleneck in our
performance test for SIP server system.
Table 6.12 Bandwidth utilization of SIP server components withlOOK connections (Kbits/s)
SIP server State Server Database Server
Incoming outgoing Incoming outgoing Incoming outgoing
3019 5560 1315 1417 1118 609
60
CHAPTER 7: CONCLUSIONS AND FUTURE WORK
In this project, we applied two open source benchmark tools, labsimul and SIPp, to study
the system performance of two Eyeball Network core server products: Eyeball XMPP Server
system and SIP Proxy Server systems. We customized benchmark tools so that they can work
properly and effectively for the purpose of our system performance study in this project. We
applied our user behaviour model to these two benchmark tools so that they can generate required
user traffic to load test these two server systems. We conducted various tests to measure the
overall system performance of these two server systems. Based on our test results, we further
diagnosed their performance bottlenecks, and proposed effective solutions toward enhancing their
scalability and reliability.
This project has focused on the system performance studies of Eyeball XMPP and SIP
server system under our assumed user behaviour model. However, because of hardware resource
and time limitation for this project, we could not have the opportunities do the following
performance study for these two VoIP server systems:
• We actually have not had enough resources to generate user traffic loads to put both
XMPP and SIP server systems to be under the load. Therefore, we actually have not
found out the peak throughput of these two server systems under our current system
configuration.
• As we mentioned above, we can configure both XMPP server and SIP server systems to
run in a cluster configurations, but how well will each system scale with different cluster
configurations? Namely, if we can have enough hardware resources to run more edge
servers and state servers within the system, together with properly configured MySQL
database servers, we can further study how the peak throughput of these two server
systems change with these different system configurations.
• In a paper present by [32], it shows that the system performance of a SIP server varies
greatly with different protocols. In this project, we mainly focused on the system
performance study of Eyeball SIP server system with TCP protocol, but Eyeball SIP
server can support UDP, TCP and TLS protocols, so how does the Eyeball SIP Proxy
Server system performance, such as peak throughput, average response time, RPS, CPS
etc., vary with different protocols?
61
Answers to these questions are very important and useful for us to better understand the
overall system performance of these two Eyeball server systems, and to help us to develop a more
scalable VoIP system. The best way to answer these questions is to do more experimental
performance tests, which require more hardware resources.
Moreover, in this project we set up all our experiments based on the user behaviour
model provided by Eyeball Networks. Therefore, the scalability of Eyeball Networks' servers
may need to be further analyzed with respect to a variety of different user behaviour models and
traffic types etc. The statistic data of system performance gained from this project, can be further
extended to derive additional performance requirements related to the actual audio and video
traffic. For example, we can apply the same user behaviour model with the signalling traffic to
create a model for various call completion methods, such as peer-to-peer, across relay, across
relay through proxies, etc. and study the various Quality of Service metrics (e.g. round trip delay,
packet loss, packet delay and delay jitter) for actual media traffic. The analysis of actual media
transmission is useful in order to assess the requirements on the actual networks and in particular
the capacity requirements of relay servers (session border controllers or media relays) used to
complete calls across firewalls. Such results will help Eyeball Networks to develop more scalable,
reliable and practical VoIP systems, as well as to design appropriate pricing and revenue models
for this new type of Internet service.
62
APPENDICES
Appendix 1: Jabsimul Configuration File for Eyeball XMPP Server
<! -- eyeballchat_configuration.xml -->
<dpsm>
<events after> I</events after>
<user_names_generator>
<range><from> I</from><to>60000</to></range>
<name>user%(num:u)</name>
<server>eyeballchat.com</server>
</user_names_generator>
<!-- Wlasciwosci uzytkownikow wybranych z przestrzeni nazw -->
<user-properities>
<filter><name>user.*</name></filter>
<properities>
<fullname>user%(num*2:u)-%(3+num%50000 I/( I+3)+7:u)</fullname>
<password>passwd%(num:u)</password>
<resource>Tester</resource>
<host> IO.50.1.32</host>
<host> IO.50.1.82</host>
<!-- Pelny log socketow -->
<Xsniff>/tmp</Xsniff>
<!-- Polacz sie z serwerem -->
<event><name>connect</name><frequency>7200000</frequency>
<counter> IOO</counter>
<!--Mozemy sie logowac digestem a nie plainem: <digest!> -->
</event>
<!-- Dodajemy do rostera nowego uzytk. wylosowanego z podanego zakresu -->
<event><name>addJoster</name><frequency> I4400000</frequency>
<user><range><from> 1</from><to>60000</to></range></user>
<max roster count>30</max roster count>- -
</event>
63
<!-- Usuwamy z rostera uzytk. wylosowanego z rostera -->
<event><name>delJoster</name><frequency>14400000</frequency></event>
<!-- Wysylamy wiadomosc -->
<event><name>send_message</name><frequency> 199000</frequency>
<user><range><from> 1</from><to>60000</to></range></user>
<prepend_with_debug_info/>
<Xfile>wiadomosc.txt</Xfile>
<text>Wiadomosc testowa krotka</text>
</event>
<!-- Zmieniamy status -->
<event><name>change_status</name><frequency>600000</frequency>
<!-- Mozna podac dokladnie co, a domyslnie losuje-->
<! --<status>Jezdem tera dostepny</status><show>chat</show>-->
</event>
<!-- Zamykaj ladnie polaczenie -->
<event><name>logout</name><frequency>7200000</frequency></event>
<!-- Zabijaj polaczenie -->
<event><name>kill_connection</name><frequency>1500000</frequency>
</event>
<!-- Tez powinno zabic polaczenie -->
<event><name>sendJaw_bytes</name><frequency>33000</frequency>
<random stream len=" 1000"/>
</event>
</properities>
</user-properities>
</dpsm>
64
Appendix 2: SIPp Call Scenario File of UAC for Eyeball SIP Server
<?xml version=" 1.0" encoding="ISO-8859-1 " ?>
<!OOCTYPE scenario SYSTEM "sipp.dtd">
<scenario name="Basic Sipstone UAC">
<send retrans="500">
<![COATA[
INVITE sip: [service]@[remotejp]:[remote---'port] SIP/2.0
Via: SIP/2.0/[transport] [locaUp]:[local---'port];branch=[branch]
From: sipp <sip:sipp@[locaUp]:[local---.port]>;tag=[call_number]
To: sut <sip:[service]@[remote_ip]:[remote---'port]>
Call-ID: [caIUd]
CSeq: I INVITE
Contact: sip:sipp@[locaUp]:[local---'port]
Max-Forwards: 70
Subject: Performance Test
Content-Type: application/sdp
Content-Length: [len]
v=O
o=userl 536557652353687637 IN IP[locaUp_type] [locaUp]
s=-
c=IN IP[mediajp_type] [media_ip]
t=O 0
m=audio [media---.port] RTP/AVP 0
a=rtpmap:O PCMU/8000
]]>
</send>
<recv response=" I00"
optional="true">
</recv>
<recv response=" 180" optional="true">
</recv>
<recv response="200" rtd="true">
</recv>
<!-- Packet lost can be simulated in any send/recv message by -->
65
<!-- by adding the 'lost = "10"'. Value can be [1-100] percent. -->
<send>
<![CDATA[
ACK sip:[service]@[remote_ip]:[remote-port] SIP/2.0
Via: SIP/2.0/[transport] [locaUp]: [Iocal-port];branch=[branch]
From: sipp <sip:sipp@[locaUp]:[local-port]>;tag=[call_number]
To: sut <sip:[service]@[remotejp]:[remote-port]>[peer_tag-'param]
Call-ID: [call id]
CSeq: 1 ACK
Contact: sip:sipp@[local_ip]:[local-port]
Max-Forwards: 70
Subject: Performance Test
Content-Length: °]]>
</send>
<pause/>
<send retrans="500">
<![CDATA[
BYE sip:[service]@[remote_ip]:[remote-port] SIP/2.0
Via: SIP/2.0/[transport] [locaUp]: [Iocal-port];branch=[branch]
From: sipp <sip:sipp@[locaUp]:[local-port]>;tag=[call_number]
To: sut <sip: [service]@[remote_ip]:[remote-port]>[peer_tag-param]
Call-ID: [caIUd]
CSeq: 2 BYE
Contact: sip:sipp@[local_ip]:[local-port]
Max-Forwards: 70
Subject: Performance Test
Content-Length: °]]>
</send>
<recv response="200" crlf="true">
</recv>
<!-- definition of the response time repartition table (unit is ms) -->
<ResponseTimeRepartition value=" 10, 20, 30, 40, 50, 100, 150, 200"/>
<!-- definition of the call length repartition table (unit is ms) -->
66
<CallLengthRepartition value=" 10, 50, 100, 500, 1000, 5000, 10000"/>
</scenario>
67
Appendix 3: SIPp Call Scenario File of UAC for Eyeball SIP Server
<7xml version=" 1.0" encoding="ISO-8859-l " 7>
<!DOCTYPE scenario SYSTEM "sipp.dtd">
<scenario name="Basic VAS responder">
<recv request="INVITE" crlf="true">
</recv>
<send>
<![CDATA[
SIP/2.0 180 Ringing
[last Via:]
[last_From:]
[last_To: ];tag=[call_number]
[Iast_Call-ID:]
[last CSeq:]
Contact: <sip: [local_ip] :[local-port];transport=[transport]>
Content-Length: 0
]]>
</send>
<send retrans="500">
<![CDATA[
SIP/2.0 200 OK
[last_Via: ]
[last_From:]
[Iast_To:];tag=[call_number]
[last_Call-ID:]
[last_CSeq:]
Contact: <sip: [locaUp]:[local_port]; transport=[transport]>
Content-Type: application/sdp
Content-Length: [len]
v=O
o=userl 536557652353687637 IN IP[locaUp_type] [locaUp]
s=-
68
c=IN IP[media_ip_type] [media_ip]
t=O °m=audio [media--l'ort] RTP/AVP °a=rtpmap:O PCMU/SOOO
]]>
</send>
<recv request="ACK"
optional="true"
rtd="true"
crlf="true">
</recv>
<recv request="BYE">
</recv>
<send>
<![CDATA[
SIP/2.0 200 OK
[Iast_Via:]
[last_From:]
[last_To:]
[last Call-ID:]
[last_CSeq:]
Contact: <sip: [local_ip]: [local--l'ort] ;transport=[transport]>
Content-Length: °]]>
</send>
<pause milliseconds="4000"/>
<!-- definition of the response time repartition table (unit is ms) -->
<ResponseTimeRepartition value=" 10, 20, 30, 40, 50, 100, 150, 200"/>
<!-- definition of the call length repartition table (unit is ms) -->
<CallLengthRepartition value=" 10, 50, 100, 500, 1000, 5000, 10000"/>
</scenario>
69
REFERENCE LIST
[I] J. Rosenberg, H. Schulzrinne, G. Camarillo, A. Johnston, J. Peterson, R. Sparks, M.Handley, E. Schooler, "SIP: Session Initiation Protocol", IETF RFC 3261, June 2002
[2] M. Day, S. Aggarwal, G. Mohr, J. Vincent, "Instant Messaging/Presence Protocolrequirements", http://www.ietf.org/rfc/rfc2779.txt, February 2000
[3] ITU-R Recommendation G.114, "General Characteristics ofInternational TelephoneConnections and International Telephone Circuits: One-way Transmission Time", February1996.
[4] ETSI TIPHON, "End-to-end Quality of Service in TIPHON Systems; Part 2: Definition ofQuality of Service (QoS) Classes", TS 101 329-2, July 2000.
[5] Mansour J. Karam, Fouad A. Tobagi, "Analysis of the Delay and Jitter of Voice Trafficover the Internet", in Proc. of IEEE INFOCOM 200 I, pp. 824-833, Anchorage, Alaska,USA, Apr 22-26, 200 I.
[6] L. Zhang, L. Zheng, K.S. Ngee, "Effect of Delay and Delay Jitter on VoicelVideo over IP",Computer Communications, Vol. 25, pp. 863-873,2002
[7] Li Zheng, Liren Zhang, Dong Xu, "Characteristics of Network Delay and Delay Jitter andits Effect overIP (VoIP)", in Proc. of IEEE International Conference on Communications(ICC) 2001, Vol. I, pp. 122 -126,11-14 June 2001
[8] R. J. B. Reynolds, A. W. Rix, "Quality VoIP - an engineering challenge", BT Techno!.Journal, Vol 19, No.2, pages 23-32, April 2001.
[9] Tadeus Uhl, "Quality of Service in VoIP Communication", Int. J. of Electronics andCommunications,Vo!. 58, pp. 178-182, 2004.
[10] R. G. Cole and J. H. rosebluth, "Voice over IP Performance Monitoring," ACM ComputerCommunications Review, vol. 31, no. 2, pp. 9-24, 2001
[11] Benchmarking Jabber/XMPP Servers with Jabsimul, http://www.ejabberd.im/benchmark
[12] SIPp reference documentation, http://sipp.sourceforge.net/doc/reference.html
[13] P. Saint-Andre, "Extensible Messaging and Presence Protocol (XMPP): Core",http://www.ietf.org/rfc/rfc3920.txt, October 2004
[14] P. Saint-Andre, "Extensible Messaging and Presence Protocol (XMPP): Instant Messagingand Presence", http://www.ietf.org/rfc/rfc3921.txt, October 2004
[15] S. Karapantazism, F. Pavlidou, "VoIP: A Comprehensive Survey On A PromisingTechnology"
[16] J. Davidson, J. Peters, M. Bhatia, S. Kalidindi, S. Mukherjee, "Voice IP fundamentals",Cisco Press, Second edition, 2007
70
[17] Tsung User's manual, http://www.process-one.netJdocs/tsung/user_manual.html. 7thOctober, 2006
[18] The Jabber Test Suite (JabberTest), http://sourceforge.netJprojects/jabbertest
[19] SIPSim Sip-Based Services Simulator,http://www.sygnusdata.co.uklpartners_radcom_sipsim.htm
[20] sipstone - Benchmarking SIP Server Performance
[21] Eyeball XMPP Server 6.5 Administration Guide
[22] Eyeball SIP Proxy Server 6.5 Administration Guide
[23] MySQL 5.1 Reference Manual, Chapter 6. Optimizationhttp://dev.mysql.com/doc/refman/5.I/en/optimization.html
[24] By Jeremy Zawodny, Derek J. Balling, "High Performance MySQL Optimization, Backups,Replication, Load Balancing & More", First Edition, April 2004
[25] MySQL Storage Engine Architecture, Arjen Lentz, MySQL AB, 28 April 2004
[26] MySQL online Manual, http://www.mysql.com/doc
[27] Peter Zaitsev, "MySQL Server Settings Tuning", MySQL Conference and Expo 2007, April23-26,2007
[28] MySQL 5.0 Reference Manual, Chapter 6. Optimizationhttp://dev.mysql.com/doc/refman/5.0/en/optimization.html
[29] D. Bertsekas and R. Gallager. Data Networks. Prentice Hall, Englewood Cliffs, NJ, 1987
[30] H. Eychenne ([email protected]), "Netfilter conntrack performance tweaking"http://www.wallfire.org/misc/netfilter_conntrack~erf.txt
[31] Alan Johnston, Steve Donovan, "SIP Call Flow Examples", http://tools.ietf.org/id/draft-ietfsip-call-flows-OS .txt
[32] E. Nahum, J. Tracey, and C. P. Wright. "Evaluating SIP Server Performance", ResearchReport 24183, IBM TJ. Watson Research Center, Feb. 2007.
71
