Software Performance TestingSoftware Performance Testing

Load Runner TrainingLoad Runner Training

Network issues:IP connections and the use of HTTPExcessive use of security featuresType and characteristic of load balancersPossible over-interaction between application servers and database serversSizing of packets (size in application versus size on network)Connection pools and connection sharingLocation of servers on the networkIncreased latency and jitterApplication issues:Excessive memory allocation and de-allocationImproper task initialization and housekeepingImproper garbage collection, especially after a fault or failureLoss of sessions or sessions kept alive too long (timers)Persistent applications like BlackBerry devicesApplication configuration parameter conflictsHigh resource consumption features (CAD/CAM, etc.)Database issues:Index designUse of dynamic indexesPotential deadlocks from locking contentionInefficient use of cacheOveruse of cacheUse of resource-intensive features such as referential integrityUse of stored procedures or trigger eventsTable fragmentation (overuse of third normal form)Improper timing of table and index reorganization

• Performance Testing determines how fast a system Performance Testing determines how fast a system performs under a particular workload to validate the performs under a particular workload to validate the quality attributes of the system such as scalability quality attributes of the system such as scalability and reliability and Resource usage of the Application and reliability and Resource usage of the Application Under Test.Under Test.

• It explores several system qualities, that can be It explores several system qualities, that can be simplified to:simplified to:

Speed - does the system respond quickly enough?Speed - does the system respond quickly enough?

Capacity - is the infrastructure sized satisfactorily ?Capacity - is the infrastructure sized satisfactorily ?

Scalability - can the system grow to handle future Scalability - can the system grow to handle future volumes?volumes?

Stability - does the system behave correctly under Stability - does the system behave correctly under load?load?

WHAT IS PERFORMANCE TESTING

Why Do Performance Testing? Why Do Performance Testing?

• Obtain an accurate picture of end-to-end system Obtain an accurate picture of end-to-end system performance before going live.performance before going live.

• How The System is Capable of handling future growth.How The System is Capable of handling future growth.

• Whether Application requires a performance Whether Application requires a performance improvement/hardware upgrade prior to release. improvement/hardware upgrade prior to release.

• Revenue losses or damaged brand credibility due to Revenue losses or damaged brand credibility due to scalability or stability issues.scalability or stability issues.

• Customers being dissatisfied with application response Customers being dissatisfied with application response time. time.

• Analyzing the behavior of the application at various load Analyzing the behavior of the application at various load levels. levels.

• Identifying bottlenecks in the application.Identifying bottlenecks in the application.

Why do performance testing?Why do performance testing?

• May not detect some functional defects May not detect some functional defects that only appear under load. that only appear under load.

• Does the application respond quickly Does the application respond quickly enough for the intended users? enough for the intended users? • Will the application handle the expected • Will the application handle the expected user load and beyond? user load and beyond? • Will the application handle the number • Will the application handle the number of transactions required by the business? of transactions required by the business? • Is the application stable under expected • Is the application stable under expected and unexpected user loads? and unexpected user loads?

Performance Tester Must Performance Tester Must KnowKnow• What is the type of communication between the client and What is the type of communication between the client and

server?server?• Requirements to monitor & Transaction Response Time or Requirements to monitor & Transaction Response Time or

server utilization (SLA).server utilization (SLA).• Is the application client-side?, And the AUT architecture.Is the application client-side?, And the AUT architecture.• Is there client-side activities involved in the entire Is there client-side activities involved in the entire

application? application? • What is the authentication mechanism?What is the authentication mechanism?

• Does the application allow multiple login of a single user? Does the application allow multiple login of a single user? • How does the application maintain a session?.How does the application maintain a session?.• Environment specifications including Software and Environment specifications including Software and

hardware.hardware.• How many users or Volumetric client expecting?How many users or Volumetric client expecting?• How does the load balancer distribution the load? How does the load balancer distribution the load? • What are the parameters been sent to the Server? What are the parameters been sent to the Server?

Loadrunner BenifitsLoadrunner Benifits• HP LoadRunner supports performance testing for a wide range HP LoadRunner supports performance testing for a wide range

of application environmentsof application environments and protocols and protocols• Decreases the risk of deploying systems that do not meet performance Decreases the risk of deploying systems that do not meet performance

requirementsrequirements• Reduces hardware and software costs by accurately predicting system Reduces hardware and software costs by accurately predicting system

capacity.capacity.• Load Runner reduces the personnel requirements Load Runner reduces the personnel requirements

b) Load Runner reduces the hardware requirements b) Load Runner reduces the hardware requirements c) The controlling will be accomplished effectively by the controller c) The controlling will be accomplished effectively by the controller d) Because Load Runner tests are fully automated, we can easily d) Because Load Runner tests are fully automated, we can easily repeat them as many times we need repeat them as many times we need

• Lets you begin intelligent service-level management before services go Lets you begin intelligent service-level management before services go livelive

• Shortens test cycles to accelerate delivery of high quality applicationsShortens test cycles to accelerate delivery of high quality applications• Pinpoints end-user, system-level and code-level bottlenecks rapidly Pinpoints end-user, system-level and code-level bottlenecks rapidly

and with ease.and with ease.• Reduces the cost of defects by early testing in the development cycle Reduces the cost of defects by early testing in the development cycle

Loadrunner VersionsLoadrunner Versions• LR Version 6.0--- Used a separate thread per user, which required almost 10 LR Version 6.0--- Used a separate thread per user, which required almost 10

times more i/o and CPU cycles than 6.5. times more i/o and CPU cycles than 6.5. • Version 6.5 available June 2000 –Turbo Load technology-thousands of vusers Version 6.5 available June 2000 –Turbo Load technology-thousands of vusers

using a single operating system thread.using a single operating system thread.• Version 7.8 became available September 2003.Version 7.8 became available September 2003.• Version 7.8 Feature Pack 1 added support for Windows XP. Version 7.8 Feature Pack 1 added support for Windows XP. • Version 8.0 became available August 2004 –Additional Runtime Settings & Version 8.0 became available August 2004 –Additional Runtime Settings &

Diagnostics and tuning Capabilities.Diagnostics and tuning Capabilities.• Version 8.1 became available October 2005– Workflow view and workflow Version 8.1 became available October 2005– Workflow view and workflow

wizard renamed as Execution log and replay log.wizard renamed as Execution log and replay log.• Version 8.1 Feature Pack 3 –Web services Enhancements.Version 8.1 Feature Pack 3 –Web services Enhancements.• Version 8.1 Feature Pack 4—Upgraded recorder version.Version 8.1 Feature Pack 4—Upgraded recorder version.• Version 8.1 Feature Pack 4 --January 2, 2007.Version 8.1 Feature Pack 4 --January 2, 2007.• Version 9.10 installer, available Feb. 2008.Version 9.10 installer, available Feb. 2008.• Version 9.50 available June 2009.Version 9.50 available June 2009.• Version 9.51 patch to 9.50 --July 6, 2009, enables better AJAX recognition Version 9.51 patch to 9.50 --July 6, 2009, enables better AJAX recognition

and specification of DOM element properties. which require Vista users to and specification of DOM element properties. which require Vista users to allow. allow.

Features of Load runner 9.5Features of Load runner 9.5• Protocol Advisor : Protocol Advisor : Protocol Advisor helps you to determine an Protocol Advisor helps you to determine an

appropriate protocol for recording a Vuser script. The Protocol appropriate protocol for recording a Vuser script. The Protocol Advisor scans your application for elements of different protocols Advisor scans your application for elements of different protocols and displays a list of the detected protocols. and displays a list of the detected protocols.

• We can export the Test Result from VUGen scripts to HTML and We can export the Test Result from VUGen scripts to HTML and open Quality Center defects directly from the report. open Quality Center defects directly from the report.

• Load Runner 9.5 supports Windows Vista SP1. Load Runner 9.5 supports Windows Vista SP1. • Provides support uptoProvides support upto .NET Framework 3.5 .NET Framework 3.5 versionversion  and supports and supports

.NET and Web Services protocols..NET and Web Services protocols.• LoadRunner is now integrated with a 3rd party software LoadRunner is now integrated with a 3rd party software Shunra, Shunra, to to

provide you access to WAN emulation. provide you access to WAN emulation. • LR controller has a new utility called LR controller has a new utility called Host Security Manager Host Security Manager

utility utility that will allow you to create secure communication that will allow you to create secure communication channels between the LoadRunner Controller and load generators. channels between the LoadRunner Controller and load generators.

• There is a new Analysis API that will allow you to create an There is a new Analysis API that will allow you to create an analysis session from test results analysis session from test results

• LR analysis now supports LR analysis now supports SQL Server 2005SQL Server 2005. . • LoadRunner 9.5 no longer supports Windows 2000. LoadRunner 9.5 no longer supports Windows 2000. • LR 9.5 is compatible with Quality Center QC) 9.2, 10.0 and QTP 9.5 LR 9.5 is compatible with Quality Center QC) 9.2, 10.0 and QTP 9.5

and 10.0 versions. and 10.0 versions.

Key Types of Performance TestingKey Types of Performance Testing • Performance test :Performance test :

A performance test is a technical investigation done A performance test is a technical investigation done to determine or validate the response, speed, to determine or validate the response, speed, scalability, stability characteristics of the product scalability, stability characteristics of the product under test.under test.

• Focuses on determining if the user of the system will Focuses on determining if the user of the system will be satisfied with the performance characteristics of be satisfied with the performance characteristics of the application. the application.

• Identifies mismatches between performance-related Identifies mismatches between performance-related expectations and reality. expectations and reality.

• Supports tuning, capacity planning, and optimization Supports tuning, capacity planning, and optimization efforts.efforts.

PurposePurpose : : To determine or validate speed, scalability, and/or stability.To determine or validate speed, scalability, and/or stability.

• Load test:Load test: A load test enables you to measure response times, A load test enables you to measure response times, throughput rates, and resource-utilization levels, and to throughput rates, and resource-utilization levels, and to identify your application’s breaking point, assuming that identify your application’s breaking point, assuming that the breaking point occurs below the peak load condition.the breaking point occurs below the peak load condition.

Performance objectives are often specified in a service Performance objectives are often specified in a service level agreement (SLA).level agreement (SLA).

PurposePurpose: : To verify application behavior under normal and peak load To verify application behavior under normal and peak load conditions.conditions.

• Determines the sufficiency of a hardware environment. Determines the sufficiency of a hardware environment. • Evaluates the adequacy of a load balancer. Evaluates the adequacy of a load balancer. • Detects concurrency issues.Detects concurrency issues.• Determines how many users can application handle. Determines how many users can application handle.

• Detects functionality errors under load.Detects functionality errors under load.

• Stress testStress test :: SStress testing enables you to identify your application’s weak tress testing enables you to identify your application’s weak points, and shows how the application behaves under extreme points, and shows how the application behaves under extreme load conditions.load conditions.

Purpose:Purpose: To determine or validate an application’s behavior when it is pushed To determine or validate an application’s behavior when it is pushed

beyond normal or peak load conditions.beyond normal or peak load conditions. Determines the side effects of common hardware or supporting application failures. Determines the side effects of common hardware or supporting application failures. Determines if data can be corrupted by overstressing the system. Determines if data can be corrupted by overstressing the system. Helps to determine what kinds of failures are most valuable to plan for. Helps to determine what kinds of failures are most valuable to plan for. Challenges :Challenges : Because stress tests are unrealistic by design, some stakeholders may Because stress tests are unrealistic by design, some stakeholders may

dismiss test results. dismiss test results. It is often difficult to know how much stress is worth applying. It is often difficult to know how much stress is worth applying.

• Capacity test:Capacity test: Capacity testing helps you to identify a scaling strategy in order Capacity testing helps you to identify a scaling strategy in order

to determine whether you should scale up or scale Down.to determine whether you should scale up or scale Down. Capacity testing is conducted in conjunction with capacity planning, which Capacity testing is conducted in conjunction with capacity planning, which

you use to plan for future growth, such as an increased user base or you use to plan for future growth, such as an increased user base or increased volume of data. For example, to accommodate future loads, you increased volume of data. For example, to accommodate future loads, you need to know how many additional resources (such as processor capacity, need to know how many additional resources (such as processor capacity, memory usage, disk capacity, or network bandwidth) are necessary to memory usage, disk capacity, or network bandwidth) are necessary to support future usage levels. support future usage levels.

SOAK TEST/Endurance SOAK TEST/Endurance TestTest• Soak testing involves testing a system with Soak testing involves testing a system with

a significant load extended over a significant a significant load extended over a significant period of time, to discover how the system period of time, to discover how the system behaves under sustained use. behaves under sustained use.

• For Example: a system may behave exactly For Example: a system may behave exactly as expected when tested for 1 hour. as expected when tested for 1 hour. However, when it is tested for 3 hours, However, when it is tested for 3 hours, problems such as memory leaks cause the problems such as memory leaks cause the system to fail or behave randomly. system to fail or behave randomly.

Performance Testing Performance Testing ApproachApproach

Performance Test ApproachPerformance Test Approach

Performance Testing ToolsPerformance Testing Tools

• Load RunnerLoad Runner

• Rational RobotRational Robot

• Silk PerformerSilk Performer

• Web LoadWeb Load

• OpenSTAOpenSTA

• Visual Studio Team SystemVisual Studio Team System

• JMeterJMeter

• Comparision of features of Loadrunner, Silk Performer and RPT • Features Features

Sl no 1: Protocol SupportSl no 1: Protocol Support Load Runner :Load Runner : Supports over 51 protocols Supports over 51 protocols Silk Perfomer:Silk Perfomer: Supports around 40 protocols Supports around 40 protocols RPT :RPT :Very few - HTTP, SAP, Sockek API, Citrix Very few - HTTP, SAP, Sockek API, Citrix

Sl no 2 : Selection of multiprotocol for scripting Sl no 2 : Selection of multiprotocol for scripting Load Runner :Load Runner : Supported Supported Silk Perfomer:Silk Perfomer: Not supported Not supported RPT : RPT : Not supported Not supported

Sl no 3 : Data corelationSl no 3 : Data corelation Load Runner : Load Runner : Much simpler and easier Much simpler and easier Silk Perfomer : Silk Perfomer : Clumsy Clumsy RPT : RPT : Simpler Simpler

Sl no 4 : Fune tuningSl no 4 : Fune tuning Load Runner :Load Runner :HP Sitescope and Diagnostics available for troubleshooting the identified bottlenecks HP Sitescope and Diagnostics available for troubleshooting the identified bottlenecks Silk Perfomer:Silk Perfomer:Not possible, rely on 3rd party tool Not possible, rely on 3rd party tool RPT :RPT :To use IBM's Tivoli To use IBM's Tivoli

Sl no 5 : Identifiation of bottlenecksSl no 5 : Identifiation of bottlenecks Load Runner :Load Runner :Easier and faster Easier and faster Silk Perfomer:Silk Perfomer:Not easy Not easy RPT :RPT :Not easy Not easy

Sl no 6 : AnalysisSl no 6 : Analysis Load Runner :Load Runner :Around 100 types garphs present. Cross garph result.Help in finding the bottle neck. Around 100 types garphs present. Cross garph result.Help in finding the bottle neck. Silk Perfomer:Silk Perfomer:Supports cross graphs Supports cross graphs RPT :RPT :Has limitation Has limitation

Sl no 7 : Analysis reportSl no 7 : Analysis report Load Runner :Load Runner :Repors - html,excel and word Repors - html,excel and word Silk Perfomer:Silk Perfomer:Only html Only html RPT :RPT :Only html Only html

Sl no 8 : Industry-standard scripting languageSl no 8 : Industry-standard scripting language Load Runner : Load Runner : Load Runner:C,Java,VB Load Runner:C,Java,VB Silk Perfomer: Silk Perfomer: bdl bdl RPT : RPT : Java Java

Sl no 9 : Support levelSl no 9 : Support level Load Runner : Load Runner : Three levels of support available. Good support provided Three levels of support available. Good support provided Silk Perfomer: Silk Perfomer: Not too good in providing support Not too good in providing support RPT : RPT : Not too good in providing support Not too good in providing support

Sl no 10 : Latency Time analysisSl no 10 : Latency Time analysis Load Runner : Load Runner : End to end response time as well as response time at each level supported. End to end response time as well as response time at each level supported. Silk Perfomer: Silk Perfomer: Displays response time at end to end as well as in the method level Displays response time at end to end as well as in the method level RPT : RPT : Displays response time end to end Displays response time end to end

Sl no 11 : Distributed Load Generation Sl no 11 : Distributed Load Generation Load Runner : Load Runner : Can control multiple load generators and collect results. Can control load generators located at remote networks (through a firewall) if required Can control multiple load generators and collect results. Can control load generators located at remote networks (through a firewall) if required Silk Perfomer: Silk Perfomer: Single MMC can control upto 256 agents (load injectors) provided sufficient physical memory is available on the MMC. Single MMC can control upto 256 agents (load injectors) provided sufficient physical memory is available on the MMC. RPT : RPT : Not Supported Not Supported

Sl no 12 : LAN/WAN The ability to emulate the behaviour of different network infrastructures during a test. Sl no 12 : LAN/WAN The ability to emulate the behaviour of different network infrastructures during a test. Load Runner : Load Runner : New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults & dynamic routing effects over the LAN used in a test. Requires a special license. New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults & dynamic routing effects over the LAN used in a test. Requires a special license. Silk Perfomer: Silk Perfomer: SilkPerformer integrates with Shunra Storm Traffic to provide WAN emulation (Add-on) SilkPerformer integrates with Shunra Storm Traffic to provide WAN emulation (Add-on) RPT : RPT : Not supported Not supported

Sl no 13 : Record and Replay featureSl no 13 : Record and Replay feature Load Runner :Load Runner :Extended logging supports view of parameter values and Server messages. Also view and comparison with ‘recorded’ version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints. Extended logging supports view of parameter values and Server messages. Also view and comparison with ‘recorded’ version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints. Silk Perfomer:Silk Perfomer:Animated try Script option helps in debugging the script. Animated try Script option helps in debugging the script. RPT :RPT :NO script, Custom code debugging in Java NO script, Custom code debugging in Java

Sl no 14 : ParametrisationSl no 14 : Parametrisation Load Runner :Load Runner :Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests. Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests. Silk Perfomer:Silk Perfomer:Customize user data' wizard helps in visual parameterizing user data. There are special bdl functions which can be used to fetch data from DBs. Customize user data' wizard helps in visual parameterizing user data. There are special bdl functions which can be used to fetch data from DBs. RPT :RPT : Automatic datapooling Automatic datapooling

Sl no 15 : URLSl no 15 : URL Load Runner :Load Runner :www.hp.com www.hp.com Silk Perfomer:Silk Perfomer:www.segue.com www.segue.com RPT :RPT :http://www.ibm.com/

Sl no 16 :Caching - The ability to emulate the caching of pages as performed by a web browser. Sl no 16 :Caching - The ability to emulate the caching of pages as performed by a web browser. Load Runner :Load Runner :Can control browser cache emulation during playback and control setting for each individual vuser. Can control browser cache emulation during playback and control setting for each individual vuser. Silk Perfomer :Silk Perfomer : TrueCachetechnology simulates the exact behaviour of a browser caching files. Has bdl functions which helps in controlling cache behaviour. TrueCachetechnology simulates the exact behaviour of a browser caching files. Has bdl functions which helps in controlling cache behaviour. RPT :RPT :Not Supported Not Supported

Sl no 17 :Release - patch and new version Sl no 17 :Release - patch and new version Load Runner :Load Runner :Every 6 months Every 6 months Silk Perfomer:Silk Perfomer:Every 1 year Every 1 year RPT :RPT :Every 1 year Every 1 year

Sl no 18 :New EnhancementSl no 18 :New Enhancement Load Runner :Load Runner :LR V9.5 - Protocol Analyser LR V9.5 - Protocol Analyser Silk Perfomer : Silk Perfomer : nil nil RPT :RPT :nil nil

•From the information it seems you have not even bothered to read the RPT product documentation.From the information it seems you have not even bothered to read the RPT product documentation.

I have been using RPT for many years and almost everywhere you claim RPT does not support something, when in fact it does.I have been using RPT for many years and almost everywhere you claim RPT does not support something, when in fact it does.

Some examples:Some examples:Point #11 & #12 are completely incorrect, RPT supports as many load generators as you like and it provides the capability to control line speed to emulate LAN WANs...Point #11 & #12 are completely incorrect, RPT supports as many load generators as you like and it provides the capability to control line speed to emulate LAN WANs...Point #1: "Protocol Support" you list only 4 while RPT supports about 12 protocols and also provides an open API to develop support for anything you choose.Point #1: "Protocol Support" you list only 4 while RPT supports about 12 protocols and also provides an open API to develop support for anything you choose.Point#2: Multiprotocol you claim "not supported" yet RPT allows you to execute any number of different protocols all from one schedule (or suite).Point#2: Multiprotocol you claim "not supported" yet RPT allows you to execute any number of different protocols all from one schedule (or suite).Point#3: Data correlation you claim "Simpler", yet you don't mention that RPT has one of the most powerful automatic correlation engines provided, where 95% of the correlation & parameterization is done for you.Point#3: Data correlation you claim "Simpler", yet you don't mention that RPT has one of the most powerful automatic correlation engines provided, where 95% of the correlation & parameterization is done for you.

Understanding LoadRunner LicensingUnderstanding LoadRunner Licensing

Load Runner licenses fall into one of the following categories:Load Runner licenses fall into one of the following categories:• permanent license-Annual maintenance fee required permanent license-Annual maintenance fee required

• Term license -either 1 month or 3 monthsTerm license -either 1 month or 3 months • Virtual User Days –ex:1000 vusers for one dayVirtual User Days –ex:1000 vusers for one day• Software as a Service :- Hp will provide the controller and chargeSoftware as a Service :- Hp will provide the controller and charge

When you purchase a Load Runner license, you will need to purchase a license for a When you purchase a Load Runner license, you will need to purchase a license for a Load Load Runner ControllerRunner Controller and for a certain number of and for a certain number of Virtual UsersVirtual Users of a particular type. of a particular type.

Performance Testing Performance Testing ProcessProcessThe following steps are involved in load-testing:The following steps are involved in load-testing:

• Step 1 - Identify performance acceptance Step 1 - Identify performance acceptance criteria criteria

• Step 2 - Identify key scenarios Step 2 - Identify key scenarios • Step 3 - Create a workload model Step 3 - Create a workload model • Step 4 - Identify the target load levels Step 4 - Identify the target load levels • Step 5 - Identify metrics Step 5 - Identify metrics • Step 6 - Design specific tests Step 6 - Design specific tests • Step 7 - Run tests Step 7 - Run tests • Step 8 - Analyze the resultsStep 8 - Analyze the results

Loadrunner Testing ProcessLoadrunner Testing Process

Performance and Technical Performance and Technical Testing ApproachTesting Approach

Load testing typically consists of five phases: planning, script Load testing typically consists of five phases: planning, script creation, scenario definition, scenario execution, and results creation, scenario definition, scenario execution, and results analysis. analysis.

Define your performance testing requirements, for Define your performance testing requirements, for example, number of concurrent users, typical business example, number of concurrent users, typical business processes and required response times. processes and required response times.

Create Vuser Scripts:Create Vuser Scripts: Capture the end-user activities into Capture the end-user activities into automated scripts. automated scripts.

Define a Scenario:Define a Scenario: Use the LoadRunner Controller to set up the Use the LoadRunner Controller to set up the load test environment. load test environment.

Run a Scenario:Run a Scenario: Drive, manage, and monitor the load test from Drive, manage, and monitor the load test from the LoadRunner Controller. the LoadRunner Controller.

Analyze the Results:Analyze the Results: Use LoadRunner Analysis to create Use LoadRunner Analysis to create graphs and reports, and evaluate the performance. graphs and reports, and evaluate the performance.

Loadrunner Testing ProcessLoadrunner Testing Process

LoadRunner Internal Architecture

Keylabs Training.com Key for IT Key for IT SolutionsSolutions•QA TrainingQA Training