Effective Use of Load Testing Tools to Tune Web Applications

04/08/2304/08/23 Copyright 2009 The Thomas Consulting GrCopyright 2009 The Thomas Consulting Group, Inc.oup, Inc.

11

Effective Use of Load Testing Tools Effective Use of Load Testing Tools to Tune Web Applications to Tune Web Applications

Tom Igielski, CSTE, CPC, CITom Igielski, CSTE, CPC, CI

The Thomas Consulting Group, Inc.The Thomas Consulting Group, Inc.

[email protected]@thomasconsultgrp.com

January 8January 8thth, 2009, 2009


Discussion TopicsDiscussion Topics

Web Terminology and ArchitecturesWeb Terminology and Architectures

Common Problem AreasCommon Problem Areas

Suggested Testing ApproachesSuggested Testing Approaches

How Does Architecture Affect ScalabilityHow Does Architecture Affect Scalability

Tuning Your ApplicationTuning Your Application


Application ScalabilityApplication ScalabilityWeb applications are considered “scalable” if a graph depicting

Web page response times compared to the number of concurrent users accessing the system increases in a linear fashion

Users

Res

pon

seT

ime

Expected

Acceptable

Crash

Ideal


BottleneckBottleneck

A limiting resource that affects the system under testA limiting resource that affects the system under test

Typically the cause of slow or unacceptable Typically the cause of slow or unacceptable performanceperformance

Affects performance and scalability of a systemAffects performance and scalability of a system

Only one in existence at any point in timeOnly one in existence at any point in time


Typical Web Problem Areas

ApplicationServer

WebServer

DatabaseServer

Firewall

HubHub Router

ISP1

Router

Internet bottlenecks

Application server issues

End-user Connection issues

LoadBalancer Web

Server

Load balancerproblems

Faulty web server within a cluster

Server HW, device and OS issues

ISP2

ISP performance and peering

Network vs. Server issues

ExternalDatafeed

Content issues

Errors


Candidate Tuning AreasCandidate Tuning Areas

25%

20%

23%

27%

5%

Network

Web Server

A[pplication

Database

Other


Why Do Bottlenecks Exist?Why Do Bottlenecks Exist?ObviousObvious Inadequate HardwareInadequate Hardware Inadequate Network CapacityInadequate Network Capacity Poor Implementation / Tuning of DatabasesPoor Implementation / Tuning of Databases

Not So ObviousNot So Obvious Poor Architecture and System DesignPoor Architecture and System Design Limited Expertise in System Design / DevelopmentLimited Expertise in System Design / Development Changing RequirementsChanging Requirements Changing Project PlansChanging Project Plans Not Thinking About “Load” or “Scalability” or “Performance” Not Thinking About “Load” or “Scalability” or “Performance”

During Des & DevDuring Des & Dev


Suggested Testing ApproachSuggested Testing Approach

Plan the Testing ActivitiesPlan the Testing Activities Develop a StrategyDevelop a Strategy Clearly State Your ObjectivesClearly State Your Objectives Define the Scope of the Performance TestDefine the Scope of the Performance Test Identify the Performance RequirementsIdentify the Performance Requirements Develop Necessary Test CasesDevelop Necessary Test Cases Line Up the Necessary Testing ResourcesLine Up the Necessary Testing Resources Define Needed Test Data and Process to MaintainDefine Needed Test Data and Process to Maintain Identify any Necessary ToolsIdentify any Necessary Tools



Build the Test TeamBuild the Test Team QA / Testing PersonnelQA / Testing Personnel Database AdministratorsDatabase Administrators Network EngineersNetwork Engineers Application ArchitectsApplication Architects DevelopersDevelopers AnalystsAnalysts SME’sSME’s



Develop the Test ScriptsDevelop the Test Scripts Select Busiest PagesSelect Busiest Pages

Always Include the Home PageAlways Include the Home Page Test Most Common Business Test Most Common Business

TransactionsTransactions Test Most Critical Business FunctionsTest Most Critical Business Functions Test Pages that are SuspectTest Pages that are Suspect



Establish the Test EnvironmentEstablish the Test Environment Procure the Load Test Tool(s)Procure the Load Test Tool(s)

# Vusers Needed# Vusers Needed

Protocols to SupportProtocols to Support

Monitors NecessaryMonitors Necessary Procure / Configure Test ServersProcure / Configure Test Servers Establish Necessary DatabasesEstablish Necessary Databases Create / Maintain Test DataCreate / Maintain Test Data


Running Load TestsRunning Load Tests

Start load testing software and allow the system to Start load testing software and allow the system to stabilize before adding many usersstabilize before adding many users

Start ramping up users until you see some system Start ramping up users until you see some system slowdown appearslowdown appear

Check system components for problemsCheck system components for problems Could be servers, application, networkCould be servers, application, network


Ideal Business ScenarioIdeal Business Scenario

12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Customer Search250 users, 50 / hr / user

Customer Referrals100 users, 5 / hr / user

Credit Checks500 users, 5 / hr / user

Recommend Product300 users, 5 / hr / user

Loan Approval150 users, 2 / hr / user

BusinessProcess

am pm

Peak Load = BPMix for Scenario


Hits / Sec vs. # VUsers

0

10

20

30

40

50

60

70

80

0:00

1:04

2:08

3:12

4:16

5:20

6:24

7:28

8:32

9:36

10:4

011

:44

12:4

813

:52

14:5

616

:00

17:0

418

:08

Relative Time

Val

ues Hits / Sec

# Vusers


Load Test CasesLoad Test Cases

Generally 3 CategoriesGenerally 3 Categories End-to-endEnd-to-end ModularModular Single or low number of pagesSingle or low number of pages

Modular tests of single / low number of pages work best for isolating Modular tests of single / low number of pages work best for isolating bottlenecksbottlenecks

Create short tests that focus narrowly on problem pages or functionsCreate short tests that focus narrowly on problem pages or functions

Run scripts one at a time, with either fixed delays or no delaysRun scripts one at a time, with either fixed delays or no delays


Successful Load TestsSuccessful Load Tests

MetricsMetrics Hits / SecHits / Sec # Virtual Users# Virtual Users Transaction Response TimeTransaction Response Time Transactions / SecondTransactions / Second Pages / SecondPages / Second % CPU Utilization% CPU Utilization


Desired Metric: Hits / SecondDesired Metric: Hits / Second

The number of requests made to the web server per secondThe number of requests made to the web server per second

How many should the application support?How many should the application support?

Determine peak based on the assumption that a % of activity in a Determine peak based on the assumption that a % of activity in a day comes within a window of a few hoursday comes within a window of a few hours

Let H = peak hits per second

Let h = # hits received over a one month period

Let a = % of activity that comes during peak time

Let t = peak time in hours

then

H = h * a / (days * t * minutes * seconds)

H = h * a / (108,000 * t)


Desired Metric: # Virtual Users

How many users should the application support?Determine the peak Virtual Users

Peak hits/second + page view times

Let U = peak virtual users

Let H = peak hits per second

Let p = average number of hits / page

Let v = average time a user views a page

U = (H / p) * v


Metrics - ExampleMetrics - Example

h = 150,000,000 hits per month

a = 10% of traffic occurs during peak time

t = peak time is 2 hours

p = a page consists of 6 hits

v = the average view time is 30 seconds

H = (h X a) / (108,000 * t)H = (150,000,000 * .1) / (108,000 X 2)H = 48

U = (H / p) * vU = (48 / 6) * 30U = 8 * 30U = 240

Desired Metric – 48 Hits / Sec or 240 Virtual Users


V1280 Average Transaction Response Time

0.5

1

1.5

2

2.5

3

0:00

1:04

2:08

3:12

4:16

5:20

6:24

7:28

8:32

9:36

10:4

0

11:4

412

:48

13:5

2

14:5

6

16:0

0

17:0

4

18:0

8

19:1

2

Relative Time After Ramp Up

Res

po

nse

Tim

e (S

eco

nd

s)

20 Vusers 40 Vusers 80 Vusers

How # VUsers Affect Response Time


Performance RequirementsPerformance Requirements

Number of “registered” usersNumber of “registered” usersNumber of simultaneous usersNumber of simultaneous usersData growth rateData growth rateTransaction RatesTransaction Rates Pages / SecPages / Sec Hits / SecHits / Sec Business Functions / Unit of TimeBusiness Functions / Unit of Time

Response time limitsResponse time limitsHow users will use the systemHow users will use the system Business scenariosBusiness scenarios

Impending events that will drive useImpending events that will drive use


Architecture and TuningArchitecture and Tuning

ServersServers System Utilization System Utilization

(CPU)(CPU) Memory UtilizationMemory Utilization ThroughputThroughput Disk I/ODisk I/O Current ConnectionsCurrent Connections QueueQueue Queue Wait TimeQueue Wait Time

DatabasesDatabases Table ScansTable Scans Table LocksTable Locks Cache HitsCache Hits IndexesIndexes


Architecture and TuningArchitecture and Tuning

Load BalancersLoad Balancers Total Concurrent Total Concurrent

ConnectionsConnections System Utilization System Utilization

(CPU)(CPU) ThroughputThroughput Load DistributionLoad Distribution Load Balancing Load Balancing

MechanismMechanism

FirewallsFirewalls Total Concurrent Total Concurrent

ConnectionsConnections Total Current SSL Total Current SSL

ConnectionsConnections System Utilization System Utilization

(CPU)(CPU) ThroughputThroughput


Example ProblemsExample Problems

High server CPU usage but goals not metHigh server CPU usage but goals not met Have servers been tuned?Have servers been tuned? Has application code been optimized?Has application code been optimized? SSL overhead?SSL overhead? Additional HardwareAdditional Hardware

Low server memory availabilityLow server memory availability Sufficient memory for applicationSufficient memory for application Memory leak?Memory leak?


%CPU Utilization vs. Hits / SecTest Environment

0

10

20

30

40

50

60

70

80

90

0:00

1:04

2:08

3:12

4:16

5:20

6:24

7:28

8:32

9:36

10:40

11:44

12:48

13:52

14:56

16:00

17:04

18:08

19:1

220:

1621:

2022

:24

Relative Time

%C

PU

/ H

its

pe

r S

ec

%CPU

Hits/Sec


Comparison of Hits / Sec and % CPU Utilization

0

20

40

60

80

100

120

0 10 20 30 40 50 60

Hits / Sec

% C

PU

1 Instance

2 Instances


Comparison of %CPU vs. Hits / Sec

0

20

40

60

80

100

120

0 2 4 6 8 10 12 14 16 18 20

Hits / Sec

% C

PU 1 Instance

2 Instances

3 Instances


Comparison of Hits / Sec vs. CPU UtilizationRunning 1 and 2 Instances

0

10

20

30

40

50

60

70

80

90

100

0:00

0:16

0:32

0:48

1:04

1:20

1:36

1:52

2:08

2:24

2:40

2:56

3:12

3:28

3:44

4:00

4:16

4:32

4:48

5:04

5:20

5:36

5:52

6:08

6:24

6:40

6:56

7:12

7:28

Relative Time

% C

PU

/ H

its

pe

r S

ec

%CPU 1 Instance

%CPU 2 Instances

Hits / Sec 1 Instance

Hits / Sec 2 Instances



Low activity on systems but increased transaction Low activity on systems but increased transaction response timeresponse time Look at requests queuedLook at requests queued Too few connections availableToo few connections available

Application server activity Application server activity ↓↓ but response time and but response time and timeouts timeouts ↑ as ↑ as observed by clientobserved by client Check server configurationCheck server configuration Look at execute threads availableLook at execute threads available Network problemsNetwork problems



Response times “spike” during the runResponse times “spike” during the run Set up application servers in “cluster” modeSet up application servers in “cluster” mode

Eliminates problems with garbage collectionEliminates problems with garbage collection Garbage CollectionGarbage Collection

Heap size large, incorrect garbage collection Heap size large, incorrect garbage collection settingssettings

Disk I/O and CPU high on Application ServersDisk I/O and CPU high on Application Servers Application or server logging levels too highApplication or server logging levels too high


Affects on Response timeAffects on Response time

Affect of Garbage Collection on Response Time

0

0.5

1

1.5

2

2.5

0:00

0:30

1:00

1:30

2:00

2:30

3:00

3:30

4:00

4:30

5:00

5:30

6:00

6:30

7:00

7:30

8:00

8:30

9:00

9:30

10:0

010

:30

11:0

011

:30

12:0

0

Relative Time

Re

sp

on

se

Tim

e

Response Time



Poor Java server performancePoor Java server performance Optimize the JVMOptimize the JVM Check JVM switches for startingCheck JVM switches for starting

Memory consumptionMemory consumptionGarbage collectionGarbage collectionThread countsThread counts

Consider Using Multiple InstancesConsider Using Multiple Instances




V1280 CPU %

17

19

21

23

25

27

29

31

0:00

1:04

2:08

3:12

4:16

5:20

6:24

7:28

8:32

9:36

10:4

0

11:4

412

:48

13:5

2

14:5

6

16:0

0

17:0

4

18:0

8

19:1

2


CP

U %

90 Threads 60 Threads 30 Threads

Comparing Thread Counts


V1280 Average Transaction Response Time

1.2

1.3

1.4

1.5

1.6

1.7

1.8

0:00

1:04

2:08

3:12

4:16

5:20

6:24

7:28

8:32

9:36

10:4

0

11:4

412

:48

13:5

2

14:5

6

16:0

0

17:0

4

18:0

8

19:1

2


Res

po

nse

Tim

e (S

eco

nd

s)

90 Threads 60 Threads 30 Threads

Comparing Thread Counts



Web servers being used unequally with Web servers being used unequally with load balancer in placeload balancer in place Problem with the load balancing Problem with the load balancing

algorithm being usedalgorithm being used Servers sending “false” busy stats back Servers sending “false” busy stats back

to load balancerto load balancer ““Run away” processes on select serversRun away” processes on select servers


Comparison of Response Times

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0:16

0:48

1:20

1:52

2:24

2:56

3:28

4:00

4:32

5:04

5:36

6:08

6:40

7:12

7:44

8:16

8:48

9:20

9:52

10:2

410

:56

11:2

812

:00

Relative Time

Re

sp

on

se

Tim

e

Instance 1

Instance 2

Load Balancing Problems



Slow database query responseSlow database query response Use profiling tool to identify slow queriesUse profiling tool to identify slow queries Optimize queriesOptimize queries

Eliminate unnecessary sortingEliminate unnecessary sortingEliminate unnecessary triggersEliminate unnecessary triggersMinimize number of temp tablesMinimize number of temp tablesUse stored procedures where possibleUse stored procedures where possible

Create necessary indexesCreate necessary indexes


Database ProblemsDatabase Problems

Impacts on Database

0

5

10

15

20

25

30

Tim

e0:

160:

481:

201:

522:

242:

563:

284:

004:

325:

045:

366:

086:

407:

127:

448:

168:

489:

209:

52

Relative Time

Hit

s /

Se

c Baseline

Test1

Test2


Monitoring ActivityMonitoring Activity


When Things Go WellWhen Things Go Well

Hits Per Second / CPU %

0

10

20

30

40

50

60

70

0:00

0:48

1:36

2:24

3:12

4:00

4:48

5:36

6:24

7:12

8:00

8:48

9:36

10:2

4

11:1

212

:00

12:4

8

13:3

6

14:2

4

Relative Time

Hit

s P

er

Se

co

nd

/ C

PU

%

Hits Per Second - 403.0.21 CPU % - 403.0.21

Hits Per Sec - 403.0.24 CPU % - 403.0.24


When Things Go WellWhen Things Go Well

Average Transaction Response Time

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0:00

0:48

1:36

2:24

3:12

4:00

4:48

5:36

6:24

7:12

8:00

8:48

9:36

10:2

4

11:1

212

:00

12:4

8

13:3

6

14:2

4

Relative Time

Ave

rag

e Tr

ansa

ctio

n R

esp

on

se

Tim

e (S

eco

nd

s)

403.0.21 403.0.24


SummarySummary

Web architectures are complex and provide Web architectures are complex and provide many areas that can be “tuned”many areas that can be “tuned”

Develop a specific process for Load TestingDevelop a specific process for Load Testing

Take a surgical approach to address problems – Take a surgical approach to address problems – one at a timeone at a time


QuestionsQuestions??

Documents

Effective Use of Load Testing Tools to Tune Web Applications