23 September 2004
Evaluating Adaptive Middleware Load Evaluating Adaptive Middleware Load
Balancing Strategies for Middleware Balancing Strategies for Middleware
SystemsSystems
Department of Electrical Engineering & Computer Science
Vanderbilt University, Nashville
Jaiganesh Balasubramanian
Ossama Othman
Douglas C. Schmidt
Lawrence Dowdy
8th EDOC Conference, Monterey, California
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesR&D Motivation: Scaleable Distributed
Systems•Heterogeneous environments
•Large number of bursty clients
•Stringent QoS requirements, e.g.:
– 24x7 availability
– Low latency & high throughput•Examples
– Online trading systems– Mission-critical systems for critical infrastructure
•e.g., air transportation, power grid control
Local AreaNetwork
NYSE
NASDAQ
Our R&D goal is to assure the scalability of these types of distributed systems
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesGeneral Approach: Scalability via Load
Balancing•Goal: Improve the efficiency &
scalability of distributed systems
•Approach: Partition & balance application tasks across many computers via load distribution strategies
•Load balancing be implemented at multiple layers, e.g.:
– Network layer
– OS layer
– Middleware layer
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Our Approach: Middleware Load Balancing•Capabilities of middleware
– Control end-to-end resources & QoS
– Leverage hardware/software advances
– Handle new environments & requirements
– Provide reusable COTS services
•Motivation for middleware load balancing
– Can take into account distributed system state, system runtime behavior, & request content
– Application-level control over load balancing policiesHardware
Domain-SpecificMiddleware Services
CommonMiddleware Services
DistributionMiddleware
Host InfrastructureMiddleware
Operating Systems & Protocols
Applications
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
TAO & Cygnus: Our Middleware Environment
•Based on The ACE ORB (TAO) – an open-source, high-performance, real-time CORBA Object Request Broker
– deuce.doc.wustl.edu/ Download.html
•Cygnus is a middleware load balancing service implemented atop TAO
•TAO also provides many other middleware services
LOAD LOAD BALANCINGBALANCING
RT-CORBA
EVENT SERVICE SECURITYTRANSACTIONS
DYNAMIC/STATICSCHEDULING
AV STREAMS
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Cygnus Load Balancing Architecture •Multiple Clients can invoke
requests
– Potentially non-deterministic
– Duration called a session
•Members
– Multiple instances of the same object implementation
•Object Groups
– Collections of members among which loads will be distributed equitably
•Load Balancer
– Transparently distributes requests to members within an object group
LoadBalancer
Clients
Req
uest
s
Rep
lies
MembersObject Groups
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
•Load Monitor– Provides load feedback
•Load Analyzer– Determines location and
member load conditions•Member Locator
– Binds client to appropriate object group member
•Load Alert– Facilitates load control (load
shedding)•Load Manager
– Mediates all interactions between load balancer components
Client
Location/Node
LoadManager
MemberLocator
LoadAnalyzer
next_member
push_loads
POArequests
member
requests
*
* LoadMonitor
LoadAlert alert
Cygnus Load Balancing Components
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Cygnus Load Balancing Strategies•Non-adaptive strategies
– RoundRobin & Random strategies
•Adaptive strategies
– LeastLoaded strategy rejects new sessions so that load is balanced
•RejectThreshold controls runtime rejection decision
– LoadMinimum strategy migrates client sessions from one machine to another to balance load
•MigrationThreshold controls runtime migration decision
Network
Clients
Req
uest
s
Rep
lies
IDC
Servers
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Cygnus Adaptive Load Balancing Interactions
Client LoadManager MemberLocator LoadAnalyzer LoadMonitor LoadAlert Member
1: send_request
2: send_request
3: next_member
4: LOCATION_FORWARD
5: send_request
6: push_loads
7: is_overloaded
8: alert
9: LOCATION_FORWARD
Acceptance strategies
Migration strategies
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Evaluating Load Balancing Strategies with LBPerf
We use LBPerf to empirically determine appropriate load balancing strategies for applications with various workload characteristics
We use LBPerf to empirically determine appropriate load balancing strategies for applications with various workload characteristics
•Open-source tool suite for evaluating middleware load balancing strategies
•Can tune different configurations of middleware load balancing
– e.g., systematically choose different load balancing strategies & runtime parameters associated with those strategies
•Evaluate strategies using different metrics
– e.g., throughput, latency, CPU utilization, overhead, etc.
Client code generated by LBPerf
Server code generated by
LBPerf
Load balancer code with option to change strategies
& run-time parameters
Client workloads can vary!!
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Summary of Testbed & Experiments•Emulab is a NSF-sponsored testbed
for conducting large-scale network/system experiments
– www.emulab.net
•LBPerf experiments conducted with 16 clients & 32 servers
•RedHat Linux 7.1, Pentium III 850Mhz & 600Mhz machines
•LBPerf experiments run in Emulab
– Overhead measurements
– Throughput measurements
– CPU utilization
– Migration threshold values
– Reject threshold values
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Measuring Overhead of Cygnus (1/2)
CPU-intensive requests
Single server
•Single-threaded clients generate CPU-intensive requests on one server
•Repeat experiment with & without load balancer
•Measure throughput, defined as # of requests processed per-second by server
Without load balancer
With load balancer
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Measuring Overhead of Cygnus (2/2)•Single-threaded
clients generate CPU-intensive requests on one server
•Repeat experiment with & without load balancer
•Measure throughput, defined as # of requests processed per-second by server
Overhead of using Cygnus is negligible
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Load Balancing Effectiveness
(1/3)
Load balancer code with option to change strategies
& run-time parameters
CPU-intensive requests
•Single-threaded clients generate CPU-intensive requests on 4 servers
•Repeat experiment, choosing each of four strategies
•Measure – Throughput,
defined as # of requests processed per-second by server
– CPU utilization at each server
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Load Balancing Effectiveness
(2/3)•Single-threaded
clients generate CPU-intensive requests on 4 servers
•Repeat experiment, choosing each of four strategies
•Measure – Throughput,
defined as # of requests processed per-second by server
– CPU utilization at each server
Adaptive strategies
Non-adaptive strategies
Adaptive strategies behave better than non-adaptive strategies with non-uniform
CPU intensive workloads
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Load Balancing Effectiveness
(3/3)
Adaptive strategies ensure uniform utilization of servers
•Single-threaded clients generate CPU-intensive requests on 4 servers
•Repeat experiment, choosing each of four strategies
•Measure – Throughput,
defined as # of requests processed per-second by server
– CPU utilization at each server
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Effect of Migration Threshold
(1/2)•Single threaded
clients generate CPU-intensive requests on 4 servers
•Use LoadMinimum strategy
•Repeat experiments for different MigrationThreshold values
•Measure throughput, defined as # of requests processed per-second by server
Load balancer code with LoadMinimum strategy
Change MigrationThreshold
values
CPU-intensive requests
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
•Single threaded clients generate CPU-intensive requests on 4 servers
•Use LoadMinimum strategy
•Repeat experiments for different MigrationThreshold values
•Measure throughput, defined as # of requests processed per-second by server
Migration’s not always effective
Measuring Effect of Migration Threshold (2/2)
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Effect of Reject Threshold
(1/2)•Single threaded
clients generate CPU-intensive requests on 4 servers
•Use LeastLoaded strategy
•Repeat experiments for different RejectThreshold values
•Measure throughput, defined as # of requests processed per-second by server
Load balancer code with LeastLoaded strategy
Change RejectThreshold
values
CPU-intensive requests
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing StrategiesMeasuring Effect of Reject Threshold
(2/2)•Single threaded
clients generate CPU-intensive requests on 4 servers
•Use LeastLoaded strategy
•Repeat experiments for different RejectThreshold values
•Measure throughput, defined as # of requests processed per-second by server
Reject sessions to ensure QoS for admitted sessions
Vanderbilt University
Balasubramanian et all Evaluating Load Balancing Strategies
Concluding Remarks•Cygnus adds little overhead for load
balancing, while providing increased scalability & utilization
•Adaptive load balancing strategies can perform much better than non-adaptive strategies in presence of non-uniform loads
•Different run-time parameters associated with adaptive strategies must be tuned to perform well
•Future work: Devise self-adaptive load balancing architectures
– Take QoS specifications as input
– Tune run-time parameters dynamically to achieve those specifications
Client
Location/Node
LoadManager
MemberLocator
LoadAnalyzer
next_member
push_loads
POArequests
member
requests
*
* LoadMonitor
LoadAlert alert