Load Testing ApproachLoad Testing Approach

Quick Guide to Plan and Quick Guide to Plan and Execute Load Test Using Load Execute Load Test Using Load Runner!Runner!

-Ranjit-Ranjit

jcrvs@hotmail.comjcrvs@hotmail.com

Why Is a Approach Why Is a Approach Needed?Needed?

• Quality approach Quality approach work smarter to be work smarter to be faster.faster.

• Do it right first time, Do it right first time, every time.every time.

• Detailed approach Detailed approach that prevent from that prevent from getting lost in the getting lost in the “load testing sea”.“load testing sea”.

• Use of smart tools Use of smart tools load runner.load runner.

Conventional ApproachConventional Approach

• Random Random requirements from requirements from management fro load management fro load testtest

• In-experienced staffIn-experienced staff• No requirements No requirements

analysis performed analysis performed for load testfor load test

• Easy to get carried Easy to get carried away in load testingaway in load testing

The ApproachThe Approach

• Perform requirement Perform requirement analysis to get analysis to get information asinformation as– Type of load testType of load test– User load estimationUser load estimation– User load distributionUser load distribution– User activity analysisUser activity analysis– Production environment Production environment

analysisanalysis– Database sizeDatabase size– Report generationReport generation– Simple metric collectionSimple metric collection

Type of Load TestType of Load Test

• Determine the goal Determine the goal of the testof the test– Do we intent to find Do we intent to find

the synchronization the synchronization problems?problems?

– Do we intend to Do we intend to perform a load test? perform a load test? Or a stress test? Or a Or a stress test? Or a performance test?performance test?

Synchronization Problem Synchronization Problem Tracing Load TestTracing Load Test

• Goal of this test is to determine the Goal of this test is to determine the synchronous issues and trouble some areas.synchronous issues and trouble some areas.

• This test may need vu-generated scripts with This test may need vu-generated scripts with lots of rendezvous points or rendezvous points lots of rendezvous points or rendezvous points at each post/action.at each post/action.

Load/stress/performanceLoad/stress/performance

• Different type of tests :Different type of tests :– Load test determines how is the performance of Load test determines how is the performance of

application under the concurrent user sessions for application under the concurrent user sessions for typical user scenario. The think time taken into typical user scenario. The think time taken into consideration in these test scripts.consideration in these test scripts.

– Stress test examines how application behaves under Stress test examines how application behaves under maximum load. In simple terms find the upper maximum load. In simple terms find the upper threshold for the application below which it can work threshold for the application below which it can work normally. Think time ignored in these tests.normally. Think time ignored in these tests.

– Performance test indicates response time for the Performance test indicates response time for the entire application from the user’s perspective.entire application from the user’s perspective.

What Do U Want?What Do U Want?

• Do determine what Do determine what kind of test do you kind of test do you want.want.

• Then plan ahead.Then plan ahead.

User Load EstimationUser Load Estimation• A detailed feed back form the A detailed feed back form the

marketing/business marketing/business development will give a idea of development will give a idea of the user load or the number of the user load or the number of users using the product. This users using the product. This will determine the load to be will determine the load to be used against the product in used against the product in testing.testing.

• An inexperienced staff may An inexperienced staff may configure a load test to simulate configure a load test to simulate 1000 v-user but user base for 1000 v-user but user base for the application may be not the application may be not more than 400. This may result more than 400. This may result in licenses being lost for the v-in licenses being lost for the v-user and time and efforts as user and time and efforts as well.well.

User Load DistributionUser Load Distribution

• A detailed feed back form A detailed feed back form the marketing/business the marketing/business development will give a development will give a idea of the user load idea of the user load distribution. distribution.

• The user load may be The user load may be peak in the morning or peak in the morning or afternoon and very less afternoon and very less in the evening.in the evening.

• This may also help This may also help determine the concurrent determine the concurrent users and simultaneous users and simultaneous users.users.

User Load DistributionUser Load Distribution

• This factor will input to This factor will input to the scenarios to be used the scenarios to be used and configured in the and configured in the load test.load test.

• E.G. BD may come up E.G. BD may come up with a user load of 1000 with a user load of 1000 users per day and each users per day and each grouped to perform grouped to perform certain activity as say certain activity as say 30% in section A of 30% in section A of application rest in section application rest in section B of the application etc.B of the application etc.

User Activity AnalysisUser Activity Analysis• A detailed discussion with A detailed discussion with

marketing/BD may reveal marketing/BD may reveal the user activity details on the user activity details on which detailed scripts can which detailed scripts can be written.be written.

• E.G. In morning there may E.G. In morning there may be 200 forms submitted or be 200 forms submitted or in evening most of the users in evening most of the users may login and perform may login and perform results generation activity results generation activity so in morning scenario out so in morning scenario out of 1000 concurrent user, of 1000 concurrent user, 70% may be performing this 70% may be performing this activity.activity.

Production Environment Production Environment AnalysisAnalysis

• A discussion with the I.T. A discussion with the I.T. Department will throw Department will throw light on deployment light on deployment environment.environment.

• This may input valuable This may input valuable information as does the information as does the environment have environment have enough hardware? Is it enough hardware? Is it running in a clustered running in a clustered environment?environment?

Production Environment Production Environment AnalysisAnalysis

• Mirror the production Mirror the production environment into a environment into a test bed.test bed.

• Alternatively create Alternatively create test bed and after test bed and after load test suggest load test suggest production production environment to environment to management.management.

Database SizeDatabase Size

• The database size does matter in the The database size does matter in the load test.load test.

• Bulky the database, more the latency.Bulky the database, more the latency.• Define the database size prior to load Define the database size prior to load

test e.G. Conducting first set of test on a test e.G. Conducting first set of test on a 50K sized DB then a 100K sized DB and 50K sized DB then a 100K sized DB and so forth.so forth.

Report GenerationReport Generation

• Generate two types of reportsGenerate two types of reports– Load runner detailed reports for engineering Load runner detailed reports for engineering

departmentdepartment– Generic reports for the activities performed Generic reports for the activities performed

in the scenario and response time details in the scenario and response time details with other observation and conclusions for with other observation and conclusions for the managementthe management

• Generating graphs also helpsGenerating graphs also helps– Helps in tracking per load cycle resultsHelps in tracking per load cycle results– Easy interpretation for peopleEasy interpretation for people

Simple Metric CollectionSimple Metric Collection

0

20

40

60

80

100

1st

Cycle

4th

Cycle

CPUUtilization

Totalhits

Perpagebyte

• Basic metrics can be Basic metrics can be collected as follows:collected as follows:– CPU utilization : CPU utilization :

should not exceed should not exceed 60%.60%.

– Per page size. This is a Per page size. This is a indicator of the bulky-indicator of the bulky-ness of the page. ness of the page. Response time for Response time for each transaction.each transaction.

– Configuration details.Configuration details.– Test bed details.Test bed details.

Simple Metric CollectionSimple Metric Collection

0

20

40

60

80

100

1stCycle

Memory

Processes

Failedtransactions

– Total hits and Total hits and hits/second, should hits/second, should not be greater than 20 not be greater than 20 or request queue or request queue details need to be details need to be collected.collected.

– % Failed transactions, % Failed transactions, should not be greater should not be greater than 5%.than 5%.

– Number of processes Number of processes running on server.running on server.

– Memory details.Memory details.

Simple Metric CollectionSimple Metric Collection

• Load runner can be used to collect the above Load runner can be used to collect the above metrics.metrics.

• Other option is to use O.S. Specific tools.Other option is to use O.S. Specific tools.

Plan on the Above Analysis – Plan on the Above Analysis – Clubbing All TogetherClubbing All Together

• After the requirement analysis, feed all the After the requirement analysis, feed all the data into a test plan.data into a test plan.

• Test plan will contain the type of test to Test plan will contain the type of test to perform, test bed details, load/user details and perform, test bed details, load/user details and activity scenarios.activity scenarios.

Plan on ScenariosPlan on Scenarios

• Plan scenarios for execution based on the Plan scenarios for execution based on the above input and different daytime for the day above input and different daytime for the day as morning scenario, afternoon scenario etc.as morning scenario, afternoon scenario etc.

• Each scenario may scale to 3 to 4 hours. So Each scenario may scale to 3 to 4 hours. So activity analysis as number of forms submitted activity analysis as number of forms submitted per day etc and time may help to set the per day etc and time may help to set the number of iteration for each scenario.number of iteration for each scenario.

Follow up Action:scriptsFollow up Action:scripts

• Create test scripts with vu-generator for the Create test scripts with vu-generator for the interactive activities to be performed with interactive activities to be performed with application.application.

• Unit test in vu-generator debug mode.Unit test in vu-generator debug mode.• Also run the same in 10 user 10 iteration load Also run the same in 10 user 10 iteration load

runner scenario mode. Use the option “show v-runner scenario mode. Use the option “show v-user” to check for the activities being user” to check for the activities being performed.performed.

• Parameterize scripts and co-related them.Parameterize scripts and co-related them.• Make scripts dynamic, more efforts to be put in Make scripts dynamic, more efforts to be put in

for making dynamic than just co-relating the for making dynamic than just co-relating the scripts.scripts.

More on ScriptsMore on Scripts

• Add comments and queries to fetch the Add comments and queries to fetch the parameterized data from database to make it parameterized data from database to make it user independent.user independent.

• Logs may reveal error which at time may be Logs may reveal error which at time may be hidden.hidden.

• In runtime settings, use option as mark each In runtime settings, use option as mark each step as transaction to prevent adding manually step as transaction to prevent adding manually transaction function for each step.transaction function for each step.

Configuring ScenariosConfiguring Scenarios

• Recommended to unique create groups, each group is Recommended to unique create groups, each group is represented by one script.represented by one script.

• Use ramp-up and ramp-down features to prevent Use ramp-up and ramp-down features to prevent instantaneous loads on load test server. Load runner instantaneous loads on load test server. Load runner gives one click access to pre-configured “slow ramp-up” gives one click access to pre-configured “slow ramp-up” and “ramp up” scenario.and “ramp up” scenario.

• If scripts are dynamic then recommended to use If scripts are dynamic then recommended to use duration based scenario.duration based scenario.

• More refined control can be achieved for scenario by More refined control can be achieved for scenario by scheduling by group rather than scheduling by scenario.scheduling by group rather than scheduling by scenario.

Configuring ScenariosConfiguring Scenarios

• Enable standard logging.Enable standard logging.• One click generator configuration can be performed One click generator configuration can be performed

using load runner.using load runner.• In runtime settings, use option as mark each step as In runtime settings, use option as mark each step as

transaction to prevent adding manually transaction transaction to prevent adding manually transaction function for each step.function for each step.

• The think time in-between two iterations is The think time in-between two iterations is recommended in load tests.recommended in load tests.

• Proxy may or may not be needed. Recommended not to Proxy may or may not be needed. Recommended not to use proxy to get a benchmark results first, then run the use proxy to get a benchmark results first, then run the same scripts though proxy and compare results.same scripts though proxy and compare results.

Configuring ScenariosConfiguring Scenarios

• Simulating browser cache option may be Simulating browser cache option may be turned off to make sure each request is fetched turned off to make sure each request is fetched form the server.form the server.

• Bandwidth throttling may or may not be used Bandwidth throttling may or may not be used as per requirements.as per requirements.

• Enable web performance graphs option in load Enable web performance graphs option in load runner to monitor at test runtime the results.runner to monitor at test runtime the results.

Configuring ScenariosConfiguring Scenarios

• Disable or enable “continue on error” option. Disable or enable “continue on error” option. Recommended to enable it for cluster fail-over Recommended to enable it for cluster fail-over tests and also regular load test.tests and also regular load test.

• The option to run v-user as a process or thread The option to run v-user as a process or thread may be used as per requirements. may be used as per requirements. Recommended to use option is thread to Recommended to use option is thread to prevent overhead of the memory and system prevent overhead of the memory and system resources. But also more threads per process resources. But also more threads per process makes the process unstable. Load runner makes the process unstable. Load runner allows configuration of number of thread per allows configuration of number of thread per process for stability.process for stability.

Execute Tests and ReportExecute Tests and Report

• Prepare test bed and Prepare test bed and execute planned execute planned scenarios.scenarios.

• Recommended Recommended server restart before server restart before each test cycle.each test cycle.

• Generate reports for Generate reports for each cycle.each cycle.

In a NutshellIn a Nutshell

• Planned approach make Planned approach make life simple and makes life simple and makes load test simple too.load test simple too.

• Requirement analysis is Requirement analysis is important activity before important activity before starting load test.starting load test.

• Metrics are the end Metrics are the end results so save results for results so save results for each cycle and analyze.each cycle and analyze.

• FLY HIGH with simple FLY HIGH with simple minds and simple minds and simple practices !!practices !!

GlossaryGlossary

• LR- load runnerLR- load runner• O.S. – Operating systemO.S. – Operating system• BD – business developmentBD – business development

Q & AQ & A