CommuniGatePro Best Practices Bench Marking

8/6/2019 CommuniGatePro Best Practices Bench Marking

1/12

CommuniGate Pro Best Practices

Benchmarking

June 11, 2007


2/12

Best Practices: Benchmarking

Page 2 of 12

IntroductionThis document is provided by CommuniGate Systems Services as a helpful Best

Practices guide to benchmarking a CommuniGate Pro Dynamic Cluster.Following is a simplified overview diagram of a CommuniGate Pro DynamicCluster:

Testing EnvironmentPrior to installing and configuring CommuniGate Pro, it is recommended to setupan environment for testing. This would include the following:

1. DNS environment: a good, fast DNS server (such as BIND 9) providinginternal-only, fake domains for use in testing:


3/12


Page 3 of 12

Example.lan: your CGP local domain

Remote.lan: a remote domain, for relaying outbound messages to.

It is critical that the DNS zones used in these tests include proper A-

records and PTR-records for all systems in the benchmark environment,including the load generators. Improper DNS setup will greatly decreaseyour test results.

2. Load generators: the test environment will include, at a minimum, thefollowing systems for the benchmark load generators:

a. Benchmark manager the main testing box, could also function asother roles

b. Load generator SPECmail or mstone, or others

c. SMTP sink the postfix smtp-sink is a good, free toolThe benchmark manager, load generators, and sinks can run on one boxor many. Generally speaking, 2-4 is sufficient.

3. Tuning: Depending on your test platform, some tuning may be required.Please reference the following Scalability page of the CommuniGate ProGuide for recommendations on tuning:

http://www.communigate.com/CommuniGatePro/Scalability.html

Dynamic Cluster OverviewThe foundation building block of the CommuniGate Pro Dynamic Cluster is theShared File System mounted on all CommuniGate Pro Backend Servers. TheShared File System represents one or more file systems which aresimultaneously accessible by all Backend Servers. It is critical to note that this filesystem must be shared (read and written) by multiple servers concurrently.

The Shared File System can be of one of two types:

1. A Network File System (NFS) on a Network-Attached Storage (NAS)2. A Cluster File System (CFS) on a Storage-Attached Network (SAN)

System Layout

Every CommuniGate Pro Frontend server performs all of its high-I/O operationsin /var/CommuniGate, which is called the BASE directory. This directory may beplaced on a high-performance disk system for best performance. If located


4/12


Page 4 of 12

anywhere except /var/CommuniGate, then /var/CommuniGate should be asymbolic link to the real location.

Every CommuniGate Pro Backend server uses two main directory locations:

/var/CommuniGate/var/CommuniGate/SharedDomains

/var/CommuniGate provides the configuration functions for each system, and isgenerally stored on the local disk (although, can be placed on the NAS or SANfor some highest-level performance requirements). However, /var/CommuniGateCANNOT be shared across multiple systems if locating /var/CommuniGate onthe SAN, each system must have its own separate directory.

/var/CommuniGate/SharedDomains must always be located on the sharedfilesystem, whether on NFS or when using a Cluster File System on a SAN. AllBackend servers must have access to this shared filesystem. A symbolic linkshould be used to link this location to other real directory locations.

Here are two example layouts then:

Layout #1: Configuration directory on local disk, SharedDomains on NAS/SAN:

System1: /var/CommuniGate

System1: /data (Shared File System mount)

System1: /var/CommuniGate/SharedDomains ->

/data/SharedDomains (symbolic link)

System2: /var/CommuniGate




Layout #2: Both the configuration directory and the SharedDomains onNAS/SAN:


System1: /var/CommuniGate -> /data/CommuniGate-system1

(symbolic link)




System2: /var/CommuniGate -> /data/CommuniGate-system2

(symbo2ic link)




5/12


Page 5 of 12

Installation

Please reference these basic installation instructions here for specifics oninstalling on your specific platform:

http://www.communigate.com/CommuniGatePro/Install.html

Once installed, you log into the WebAdmin Interface (GUI) on each system at thislocation:

http://:8010

or

https://:9010

Upon initial startup of CommuniGate Pro, and once youve connected to theWebAdmin Interface, you will be prompted to enter a new postmaster password.This password has complete administration rights to each system, and should be

created securely.

CommuniGate Pro Cluster Configuration

Once you've gotten this far, you want to configure your system something like thefollowing steps, for best performance. All paths below are clicks within theCommuniGate Pro WebAdmin GUI.

Under Settings section:

4. Domain Name

Settings->General->Domain Name

Set to .example.com on each host (or similar, if using a differentdomain). Depending on your systems true host and domain names, this may befilled in correctly by default. This domain name entry MUST match the DynamicCluster license, such as *.example.com.

5. Licenses

Settings->General->Licenses

Install all license keys on all CommuniGate Pro servers. This is important, pleaseremember to install all licenses on all servers.

6. Cluster IP AddressesSettings->General->Cluster

Add all Frontend and Backend IPs on all hosts:

Example:

Cluster Members


6/12


Page 6 of 12

This Server Cluster Address: On restart Active

[first IP_____] [192.168.1.3]

Backend Server Addresses:

192.168.1.3_____________________________

192.168.1.4_____________________________

________________________________________

________________________________________

Frontend Server Addresses:

192.168.1.1_____________________________

192.168.1.2_____________________________

________________________________________

________________________________________

7. Stop the CommuniGate Pro servers on each host.

8. Add a Startup file on each server. Additional tuning recommendations forvarious operating systems for CommuniGate Pro are always available in theScalability section of the CommuniGate Pro manual, available by defaulthere:

http://www.communigate.com/CGatePro/Scalability.html

For every server you are testing, you will need to add this file (default location):

/var/CommuniGate/Startup.sh

For Frontends:

#### Startup.sh

SUPPLPARAMS=-ClusterFrontend --DefaultStackSize 131072 \--useNonBlockingSockets --closeStuckSockets \

--CreateTempFilesDirectly 10

ulimit n 16384

#### end Startup.sh

For Backends:

#### Startup.sh

SUPPLPARAMS=ClusterBackend --DefaultStackSize 131072 \

--useNonBlockingSockets --closeStuckSockets \

--CreateTempFilesDirectly 10

ulimit n 16384

#### end Startup.sh

9. Start all servers


7/12


Page 7 of 12

Be sure to start at least one Backend server first, as the first Backend serverautomatically becomes the Cluster Controller. A Cluster Controller is requiredbefore any other servers can join the cluster:

# /etc/init.d/CommuniGate start

10.If the restart was successful, the Cluster Controller will show in the log therest of the service:

http://:8010

Monitor -> Logs -> Display

Confirm that all systems join the cluster. This page will display the entirecluster:

MONITORS->Cluster

(Note: The Cluster monitoring page is also a great place to manage all of yoursystems from. If you open the WebAdmin Interface on your Cluster Controller,then use the system links on the MONITORS->CLUSTER page, you can theneasily login to the WebAdmin Interface for all your systems).

Tuning optimizations

You must set these settings on all servers for most of these options. Clustersettings (as part of any Cluster-Wide setting) only need to be set on one server,and will take affect for all servers within the cluster.

1. Server-Wide and Cluster-Wide Client IPs

Be sure to add your local IP ranges to the Cluster-Wide Client IPs

Settings->Network->LANIPs->Cluster-Wide

2. Queue

Settings->Queue->Settings->Message Queue->Queue Foldering =

100

Settings->Queue->Settings->Message Enqueuer->Processors =

50

Settings->Queue->Settings->Message Dequeuer->Processors =

10

3. DNS Resolver


8/12


Page 8 of 12

Settings->Obscure->Domain Name Resolver->Concurrent

Requests=25

4. SMTP Sending

Settings->SMTP->Sending->Channels=1000

Settings->SMTP->Sending->Channels/Host->Other=50

5. SMTP Receiving

Settings->SMTP->Receiving->Channels=1000

Settings->SMTP->Receiving->Disconnect after: [5] errors and

Deny Access for: [15 minutes]

6. LOCAL

Settings->LOCAL->other=50

7. Access

Settings->Access->POP=1000

Settings->Access->IMAP=1000

Settings->Access->PWD clients=50

# (set PWD channels to at least 5 * )

Also note here, if testing the WebUser Interface, it would probably be best to set

the HTTPU->Listener ports to listen on 80 for HTTP and 443 for HTTPS, ratherthan the default 8100/9100.

8. Cluster-Wide Domain Defaults

Users->Domain Defaults->Cluster-Wide->Account Storage-

>Foldering Method = Hashed 2 Levels


>Rename in Place =Yes


>Generate Index =Yes

9. TextMailbox or MaildirMailbox

One can use with two options - different filesystems perform different with each.MaildirMailbox format uses more small files and many more filesystem metadataoperations, and additionally can take advantage of single-instance messagestore (which especially benefits Enterprises, Educational facilities, and ASPs


9/12


Page 9 of 12

where one message may be sent to thousands of recipients). TextMailbox formatuses bigger mailbox files but these files are indexed for high performance. Acombination using MaildirMailbox in the INBOX and TextMailbox is alsopossible this method can be easily scripted in provisioning scripts.

For most Cluster File Systems and NFS, TextMailbox format may perform betterdue to less overall metadata file operations and good indexing. The default isTextMailbox, and in most cases it is recommended.

When using the TextMailbox (default) format, it is recommended to reduce thedefault mailbox index size and increase the concurrent large buffers, which areconfigured in the Settings->General->Other settings page:

Settings->General->Other->TextMailbox Manager-

>Concurrently used large buffers=50

Settings->General->Other->TextMailbox Manager->Index

Mailboxes larger than=1024K (1 Mbyte)

Optionally:

Objects->Domains->Account Defaults->Server-Wide->New

Mailboxes=TextMailbox (or MailDirMailbox)

Objects->Domains->Account Defaults->Cluster-Wide->New

Mailboxes=TextMailbox (or MailDirMailbox)

Domain Setup

Finally, add your clustered domain(s), and proceed with adding accounts.

Domain setup is managed in the WebAdmin Interface at the location:

Users->Domains

Be sure to create all your domains as Clustered Domains.

Benchmark ToolsThere are many tools available for benchmarking and testing e-mail, calendaring,and voice services. We provide some details of one of these, SPECmail2001,here.

SPECmail2001SPECmail2001 is a standardized benchmark developed by messaging vendorsand research organizations that measures mail server performance using a real-world workload. CommuniGate Systems is a member of the SPEC organization"The Standard Performance Evaluation Corporation, at http://www.spec.org) anda voting member of the SPECmail, SPECimap, and SPECsip committees. As


10/12


Page 10 of 12

well, CommuniGate Pro is the current world-record holder in the SPECmail2001benchmark, beating all previous records by 67% or more at 2.5-million users.These results provide a good overview of what a large SPECmail test looks like:

http://www.spec.org/mail2001/results/mail2001.html

The CommuniGate Systems Services department has used the SPECmail2001

application to test CommuniGate Pro implementations prior to going live inproduction. The purpose of using this application in validation testing is that theinitialization and benchmark test stages generate significant mail delivery andaccess load on the mail system.

Since 2004, CommuniGate Systems Professional Services has regularly testedmost new implementations with SPECmail2001 to ensure that the system is


11/12


Page 11 of 12

prepared for deployment. While SPECmail2001 is not able to completelyduplicate the 'real world' environment, it is capable of demonstrating that theinitial design of the system is fundamentally solid, and that the hardware used isstable and capable of providing sufficient performance.

The types of problems which SPECmail2001 can discover includes the following:

Faulty or insufficient hardware

DNS server errors or configuration problems

Load Balancer mis-configuration

Network performance, routing, or stability issues

Shared storage performance

External authentication or performance problems with integratedapplications

Content filtering (anti-spam, anti-virus) and plugin performance

SPECmail Preparation

SPECmail2001 is a Java-based application, and runs best on a relatively recentversion of Sun Java (1.4.2 or later). Other java-like implements (such as GNUJava) are not sufficient - we highly recommend you download the latest versionof Java from http://java.sun.com.

The SPECmail2001 application should be capable of running on any Sun Java-supported platform, including but not limited to Linux, Solaris, Windows, and MacOSX. Generally speaking, at least two "load-generator" systems are required forSPECmail testing, one acting as the SMTP/POP load generator, and the otheracting as the SMTP sink for outbound delivery. These (minimum) two systemscannot be part of the CommuniGate Pro Dynamic Cluster, and for best results,should be placed in a location to route connections through any relevant load-balancers, routers, and firewalls in order to verify not just the CommuniGate Prosystem(s), but also the entire production environment.

SPECmail2001 Overview from SPEC.org

SPECmail2001 is a standardized mail server benchmark designed to measure asystem's ability to act as a mail server servicing email requests, based on the

Internet standard protocols SMTP and POP3. The benchmark characterizesthroughput and response time of a mailserver system under test with realisticnetwork connections, disk storage, and client workloads. The benchmark focuseson the ISP as opposed to Enterprise class of mail servers, with an overall usercount in the range of approximately 10,000 to 1,000,000 users. The goal is toenable objective comparisons of mail server products.


12/12


Page 12 of 12

SPEC and the benchmark name SPECmail are registered trademarks of theStandard Performance Evaluation Corporation. Competitive benchmark resultsstated above reflect results published on www.spec.org as of August 15, 2005. The comparison presented above is based on StandardPerformance Evaluation Corp.'s benchmark that measures a system's ability toact as a mail server servicing email requests, based on the Internet standardprotocols SMTP and POP3. For the latest SPECmail benchmark results, visithttp://www.spec.org/mail2001.

Documents

CommuniGatePro Best Practices Bench Marking