© Blackboard, Inc. All rights reserved. Deploying Tomcat Clusters in an Advanced Blackboard TM Environment Speaker: Steve Feldman Director, Software Performance

© Blackboard, Inc. All rights reserved.

Deploying Tomcat Clusters in an Advanced Blackboard TM Environment

Speaker: Steve FeldmanDirector, Software Performance Engineering and [email protected]

mailto:[email protected]

2

Agenda and Introductions

» Goals, Objectives and Outcomes» Background Knowledge» Hands on Learning» Benchmark Statistics» Questions…Questions…Questions» Total Time: 50 Minutes

3

Note to Attendees» Tomcat clustering is not a standard configuration

of the Blackboard Academic Suite.» Clusters have been set-up and tested by Blackboard

Engineers, but have not been certified as a valid configuration.

» The technology is proven and used throughout the world.

» The same can be said of server virtualization» Many Blackboard clients use and deploy virtual servers.» The guidance provided in this presentation is to help

clients take advantage of the power of clustering, but help clients understand that they must have knowledge on setting up and managing clusters.

4

Presentation Goals» The goals of this presentation are:

» Introduce the concepts and techniques for planning and deploying Tomcat Application Clusters.

» Help administrators understand the value of Tomcat clusters for performance and availability.

» Show administrators how to set-up their very own Tomcat cluster.

5

Presentation Objectives» Define what are clusters. » Define what are nodes.» Explain the differences between cluster nodes and

load balanced nodes.» Review the configuration requirements for setting

up a Tomcat cluster» Review the configuration requirements for setting

up web server distribution to the cluster.» Explain issues that might come up with clusters

and provide techniques for diagnosing and resolving the issues.

» Convince every customer to go home and start setting up clusters in their development environments.

6

Presentation Outcomes

» At the end of the session, administrators will be able to do the following:» Define the difference between a load-balanced

configuration and a tomcat cluster. » Set-up a Tomcat cluster or a load-balanced

nodes» Configure clustering for quality of service.

7

What is a Tomcat Cluster?» A cluster is a collection of two or more

Tomcat instances running in parallel and working together to provide robust scalability, performance and reliability.

» Clusters can be multi-homed and reside physically on the same hardware.

» Clusters can be distributed and reside on distributed hardware.

» Clusters are indistinguishable to end users presenting a seamless experience

8

Why Cluster?» Clusters distribute workload

» Minimize unnecessary queuing» When configured for a small JVM it can reduce the amount

of JVM pause times.» Clusters can help better utilize the hardware you

run on» Clusters enable a scale-up model to go along with load-

balanced installations (scale-out model)» Clusters can provide additional availability

» Not recommend using Tomcat Clusters as the primary failover solution

» Clusters are cool…and because they are cool you should consider them!» Just one person’s opinion!

9

Visual Look at a Tomcat Cluster

Web ServerWeb ServerInstanceInstance

Node 1Node 1

Node 2Node 2

Node NNode N

Traffic is directed by the Traffic is directed by the Web Server like a software load balancerWeb Server like a software load balancer

Nodes know littleNodes know littleabout there partnersabout there partners

Recommend 1 nodeRecommend 1 nodePer CorePer Core

10

What is an Application Node?

» An application node is a single instance of Tomcat.

» Nodes can be part of a cluster.» Means they are aware of each other» Part of a group» Failover

» Nodes can be part of a load-balanced configuration.» A single physical server can have multiple logical

nodes not in a cluster, but technically load-balanced by JK

11

Clustered Nodes versus Load-Balanced Nodes

» A load-balanced node is the following:» Physically distinct from a hardware perspective.» Has its own web server managing traffic.» Has a dedicated configuration and resource definition.» Has no understanding of failover definition.

» Relies on third-party resources like load-balancers.

» A clustered node is the following:» Can be physically or logically distinct from a hardware

perspective.» Shares a web server with potential N+ nodes.» Shares configuration and resource definition with other

nodes.» Has failover definition as part of the application configuration.

12

Load-Balanced and Clustered Nodes


Node 1Node 1

Node 2Node 2

Node NNode N


Node 1Node 1

Node 2Node 2

Node NNode N

Hardware or SoftwareHardware or Software Load BalancerLoad Balancer

ClusterCluster Load-BalancedLoad-Balanced

Nodes unaware of each other.

Nodes aware of each other.

13

Session Replication» To enable session replication in Tomcat,

three different paths can be followed to achieve the exact same thing:» Using session persistence, and saving the

session to a shared file system (PersistenceManager)

» Using session persistence, and saving the session to a shared database (JDBCManager)

» Using in-memory-replication, using the SimpleTcpCluster that ships with Tomcat 5 (server/lib/catalina-cluster.jar)

» Recommend Using SimpleTcpCluster

14

Guidelines for Setting Up a Cluster» The cluster configuration is completely described in

the server.xml file under $BLACKBOARD_HOME/apps/tomcat/conf

» You can set-up a true cluster or settle for load-balanced JVMs.» Either way you win.

» Create multiple clusters in a multi-homed configuration initially» Each multihomed cluster is capable of failing over to each

other rather then across physical systems» Eliminates the need for cross-over connections between

servers.

15

Ingredients of a Tomcat Cluster» Class Name= Must have class package

» org.apache.catalina.cluster.tcp.SimpleTcpCluster

» Cluster Identification and Listen Port» Multicast Communication Address» JK Connector Port» JK Configuration Definition for Load Distribution» A Few Directory Copies

» $BLACKBOARD_HOME/apps/tomcat/bin» $BLACKBOARD_HOME/apps/tomcat/conf» $BLACKBOARD_HOME/apps/tomcat/work» $BLACKBOARD_HOME/apps/tomcat/log

16

Example Cluster Code<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster" managerClassName="org.apache.catalina.cluster.session.DeltaManager" expireSessionsOnShutdown="false" useDirtyFlag="true" service.mcastBindAddress="127.0.0.1" notifyListenersOnReplication="true">

<Membership className="org.apache.catalina.cluster.mcast.McastService" mcastAddr="228.0.0.4" mcastBindAddress="127.0.0.1" mcastPort="45564" mcastFrequency="500" mcastDropTime="3000"/>

<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener" tcpListenAddress="auto" tcpListenPort="4000" tcpSelectorTimeout="100" tcpThreadCount="6"/>

<Sender className="org.apache.catalina.cluster.tcp.ReplicationTransmitter" replicationMode="pooled" ackTimeout="15000"/>

<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/> </Cluster>

17

Easiest Way to Skin a Tomcat Cluster

» Copy $BLACKBOARD_HOME/apps/tomcat to $BLACKBOARD_HOME/apps/tomcat_node2

» Change all references of $BLACKBOARD_HOME/apps/tomcat to now use $BLACKBOARD_HOME/apps/tomcat_node2

» Modify server.xml in new directory to reflect unique server port for AJP13 connector and shutdown.» Cluster ports need to be unique in order to create a socket

unless IP Addresses are defined and uniquely defined.

» Modify CATALINA_PID in blackboard-tomcat.sh or .bat file to point to the right node and log file directory.

18

Setting Up JK Load Distribution» Apache and IIS will route traffic based on JK

definition.» Best to set-up a new workers.properties

» Located under $BLACKBOARD_HOME/apps/tomcat/conf/jk/workers.properties

» Change httpd.conf to point to new workers.properties (Unix)

» Managed from within IIS» Blackboard Web Site Jakarta» Simply copy workers.properties and rename old file

19

Example Workers.Properties# One virtual workerworker.list=loadbalancer

# Configure the virtual worker as a load balancerworker.loadbalancer.type=lbworker.loadbalancer.balanced_workers=tomcat1,tomcat2worker.loadbalancer.method=Request

# Attempt to worker.loadbalancer.sticky_session=1

# Actual workers, not available directlyworker.tomcat1.port=8009worker.tomcat1.host=localhostworker.tomcat1.type=ajp13worker.tomcat1.lbfactor=50worker.tomcat2.port=8209worker.tomcat2.host=localhostworker.tomcat2.type=ajp13worker.tomcat2.lbfactor=50

20

Web Server Configuration» Apache manages Tomcat worker

information in httpd.conf» Define workers.properties» Establish JK Mount (Bottom of File)» Probably called new worker list “loadbalancer”

» Windows manages Tomcat worker information un uriworkermap.properties» Must note new worker name for Cluster (aka:

worker list)

21

Things to Consider» Apache has a limit of 1024 maxclients

» More then enough for a 2, 3 or 4 node cluster» Must always keep maxclients = (Tomcat Max

Server Processes) * (# of Tomcat Nodes per Cluster)

» Recompiling Apache is one way to solve the problem

22

Advanced Feature: Setting Up QOS Models

Application Server

Tomcat

DBIJDBC

Tomcat

Modperl / PerlEx

Apache / IIS

Change JKMount

**JK Mount says /webapps/* but can be changed to any single webapp if you choose.

23

Example Benchmark StatisticsT2000 Analysis and Clustering

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

Configuration

Ses

sio

ns

Per

Ho

ur

Single Server 2 Nodes

Load Balanced 1 Node Each

Load Balanced 2 Nodes Each

24

Example Benchmark StatisticsWorkload R7.1 Entry-

LevelR7.1 Mid-

LevelR7.1 High-

LevelR7.1 HL

ClusteredR1 (Workload of

120 Possible Concurrent Simulations

Learning System/Communit

y System)

7238 Sessions/Hr19 UPL/Second

311,656 Bytes/Second

51,888 Transactions

8080 Sessions/HR22 UPL/Second


53,780 Transactions

8212 Sessions/HR22 UPL/Second


54,049 Transactions

10455 Sessions/HR

25 UPL/Second544,673 Bytes/Second59,239

Transactions(2 Nodes)

R2 (Workload of 240 Possible Concurrent Simulations


y System)

12459 Sessions/HR

31 UPL/Second640,958

Bytes/Second87,433

Transactions

13341 Sessions/HR


Bytes/Second90,353

Transactions

14913 Sessions/HR


Bytes/Second94,181

Transactions

16063 Sessions/HR


Bytes/Second106,659

Transactions(4-Nodes)

R3 (Workload of 360 Possible Concurrent Simulations


y System)

17288 Sessions/HR


Bytes/Second118,754

Transactions

18455 Sessions/HR

50 UPL/Second1,102,667

Bytes/Second130,811

Transactions

20343 Sessions/HR


Bytes/Second145,287

Transactions

24034 Sessions/HR


Bytes/Second157,629

Transactions(6-Nodes)

25

Example Benchmark StatisticsWorkload R7.1 Entry-

LevelR7.1 Mid-Level R7.1 High-

LevelR7.1 HL

ClusteredR7 (Workload of

200 Possible Concurrent

Simulations Full Academic Suite)



37,313 Transactions



84,004 Transactions

12974 Sessions/Hr


Bytes/Second84,970

Transactions

13804 Sessions/HR

36 UPL/Second763,955 Bytes/Second90,941


R8 (Workload of 400 Possible Concurrent




77,553 Transactions


1,157,486 Bytes/Second

118,353 Transactions

14668 Sessions/Hr


Bytes/Second96,742



1,392,037 Bytes/Sec157,629


R9 (Workload of 600 Possible Concurrent




64,289 Transactions


1,196,553 Bytes/Sec149,709

Transactions

20207 Sessions/Hr

47 UPL/Second1,014,189 Bytes/Sec130,907



1,527,433 Bytes/Sec181,121


26

References

» Tomcat: The Definitive Guide by Jason Brittain » http://www.onjava.com/pub/a/onjava/2004/01/28/tomcat5.ht

ml?CMP=ILC-VP9761447835&ATT=804

» http://tomcat.apache.org/tomcat-5.0-doc/balancer-howto.html

» http://www.onjava.com/pub/a/onjava/2002/07/17/tomcluster.html?page=1

» http://www.onjava.com/pub/a/onjava/2004/03/31/clustering.html

» http://www.onjava.com/pub/a/onjava/2004/04/14/clustering.html

» http://arstechnica.com/articles/paedia/cpu/hyperthreading.ars» http://lists.samba.org/archive/samba-technical/2004-Decemb

er/038305.html» http://tomcat.apache.org/tomcat-5.0-doc/cluster-howto.html

http://www.onjava.com/pub/a/onjava/2004/01/28/tomcat5.html?CMP=ILC-VP9761447835&ATT=804

http://www.onjava.com/pub/a/onjava/2004/01/28/tomcat5.html?CMP=ILC-VP9761447835&ATT=804

http://tomcat.apache.org/tomcat-5.0-doc/balancer-howto.html

http://tomcat.apache.org/tomcat-5.0-doc/balancer-howto.html

http://www.onjava.com/pub/a/onjava/2002/07/17/tomcluster.html?page=1

http://www.onjava.com/pub/a/onjava/2002/07/17/tomcluster.html?page=1

http://www.onjava.com/pub/a/onjava/2004/03/31/clustering.html




http://arstechnica.com/articles/paedia/cpu/hyperthreading.ars

http://arstechnica.com/articles/paedia/cpu/hyperthreading.ars

http://lists.samba.org/archive/samba-technical/2004-December/038305.html

http://lists.samba.org/archive/samba-technical/2004-December/038305.html

http://tomcat.apache.org/tomcat-5.0-doc/cluster-howto.html

27

Past Presentations of Note» B2 2006: How We Size the Academic Suite, Benchmarking at Blackboard

» B2 2006: Deploying Tomcat Clusters in an Advanced Blackboard Environment

» 2006 BbWorld Presentation: Practical Guide to Performance Tuning and Scaling (2 Hour Workshop)

» B2 2005: Introduction to Load Testing, A Blackboard Primer

» B2 2005: Performance Testing Building Blocks

» Users Conference 2005: Managing Your Blackboard Deployment for Growth and Performance

» Users Conference 2005: Applied Software Performance Engineering

» B2 2004: Introduction to Software Performance Engineering

» B2 2004: Profiling Building Blocks for Performance Analysis

28

Questions?

Documents

© Blackboard, Inc. All rights reserved. Deploying Tomcat Clusters in an Advanced Blackboard TM Environment Speaker: Steve Feldman Director, Software Performance