Software Performance Testing ---Sathireddy MandadiSatish.mandadi@gmail.com

Network issues:IP connections and the use of HTTPExcessive use of security featuresType and characteristic of load balancersPossible over-interaction between application servers and database serversSizing of packets (size in application versus size on network)Connection pools and connection sharingLocation of servers on the networkIncreased latency and jitterApplication issues:Excessive memory allocation and de-allocationImproper task initialization and housekeepingImproper garbage collection, especially after a fault or failureLoss of sessions or sessions kept alive too long (timers)Persistent applications like BlackBerry devicesApplication configuration parameter conflictsHigh resource consumption features (CAD/CAM, etc.)Database issues:Index designUse of dynamic indexesPotential deadlocks from locking contentionInefficient use of cacheOveruse of cacheUse of resource-intensive features such as referential integrityUse of stored procedures or trigger eventsTable fragmentation (overuse of third normal form)Improper timing of table and index reorganization

Performance Testing determines how fast a system performs under a particular workload to validate the quality attributes of the system such as scalability and reliability and Resource usage of the Application Under Test.It explores several system qualities, that can be simplified to:Speed - does the system respond quickly enough?Capacity - is the infrastructure sized satisfactorily ?Scalability - can the system grow to handle future volumes?Stability - does the system behave correctly under load?

WHAT IS PERFORMANCE TESTING

Why Do Performance Testing? Obtain an accurate picture of end-to-end system performance before going live. How The System is Capable of handling future growth.Whether Application requires a performance improvement/hardware upgrade prior to release. Revenue losses or damaged brand credibility due to scalability or stability issues.Customers being dissatisfied with application response time. Analyzing the behavior of the application at various load levels. Identifying bottlenecks in the application.

Why do performance testing?May not detect some functional defects that only appear under load. Does the application respond quickly enough for the intended users? Will the application handle the expected user load and beyond? Will the application handle the number of transactions required by the business? Is the application stable under expected and unexpected user loads?

Performance Tester Must KnowWhat is the type of communication between the client and server?Requirements to monitor & Transaction Response Time or server utilization (SLA).Is the application client-side?, And the AUT architecture.Is there client-side activities involved in the entire application? What is the authentication mechanism? Does the application allow multiple login of a single user? How does the application maintain a session?.Environment specifications including Software and hardware.How many users or Volumetric client expecting?How does the load balancer distribution the load? What are the parameters been sent to the Server?

Loadrunner BenifitsHP LoadRunner supports performance testing for a wide range of application environments and protocolsDecreases the risk of deploying systems that do not meet performance requirementsReduces hardware and software costs by accurately predicting system capacity.Load Runner reduces the personnel requirements b) Load Runner reduces the hardware requirements c) The controlling will be accomplished effectively by the controller d) Because Load Runner tests are fully automated, we can easily repeat them as many times we need Lets you begin intelligent service-level management before services go liveShortens test cycles to accelerate delivery of high quality applicationsPinpoints end-user, system-level and code-level bottlenecks rapidly and with ease.Reduces the cost of defects by early testing in the development cycle

Loadrunner VersionsLR Version 6.0--- Used a separate thread per user, which required almost 10 times more i/o and CPU cycles than 6.5. Version 6.5 available June 2000 Turbo Load technology-thousands of vusers using a single operating system thread.Version 7.8 became available September 2003.Version 7.8 Feature Pack 1 added support for Windows XP. Version 8.0 became available August 2004 Additional Runtime Settings & Diagnostics and tuning Capabilities.Version 8.1 became available October 2005 Workflow view and workflow wizard renamed as Execution log and replay log.Version 8.1 Feature Pack 3 Web services Enhancements.Version 8.1 Feature Pack 4Upgraded recorder version.Version 8.1 Feature Pack 4 --January 2, 2007.Version 9.10 installer, available Feb. 2008.Version 9.50 available June 2009.Version 9.51 patch to 9.50 --July 6, 2009, enables better AJAX recognition and specification of DOM element properties. which require Vista users to allow.

Features of Load runner 9.5Protocol Advisor : Protocol Advisor helps you to determine an appropriate protocol for recording a Vuser script. The Protocol Advisor scans your application for elements of different protocols and displays a list of the detected protocols. We can export the Test Result from VUGen scripts to HTML and open Quality Center defects directly from the report. Load Runner 9.5 supports Windows Vista SP1. Provides support upto .NET Framework 3.5 versionand supports .NET and Web Services protocols.LoadRunner is now integrated with a 3rd party software Shunra, to provide you access to WAN emulation. LR controller has a new utility called Host Security Manager utility that will allow you to create secure communication channels between the LoadRunner Controller and load generators. There is a new Analysis API that will allow you to create an analysis session from test results LR analysis now supports SQL Server 2005. LoadRunner 9.5 no longer supports Windows 2000. LR 9.5 is compatible with Quality Center QC) 9.2, 10.0 and QTP 9.5 and 10.0 versions.

Key Types of Performance Testing Performance test :A performance test is a technical investigation done to determine or validate the response, speed, scalability, stability characteristics of the product under test.Focuses on determining if the user of the system will be satisfied with the performance characteristics of the application. Identifies mismatches between performance-related expectations and reality. Supports tuning, capacity planning, and optimization efforts.

Purpose : To determine or validate speed, scalability, and/or stability.

Load test: A load test enables you to measure response times, throughput rates, and resource-utilization levels, and to identify your applications breaking point, assuming that the breaking point occurs below the peak load condition.Performance objectives are often specified in a service level agreement (SLA).

Purpose: To verify application behavior under normal and peak load conditions.Determines the sufficiency of a hardware environment. Evaluates the adequacy of a load balancer. Detects concurrency issues.Determines how many users can application handle. Detects functionality errors under load.

Stress test : Stress testing enables you to identify your applications weak points, and shows how the application behaves under extreme load conditions. Purpose: To determine or validate an applications behavior when it is pushed beyond normal or peak load conditions.Determines the side effects of common hardware or supporting application failures. Determines if data can be corrupted by overstressing the system. Helps to determine what kinds of failures are most valuable to plan for. Challenges :Because stress tests are unrealistic by design, some stakeholders may dismiss test results. It is often difficult to know how much stress is worth applying. Capacity test: Capacity testing helps you to identify a scaling strategy in order to determine whether you should scale up or scale Down. Capacity testing is conducted in conjunction with capacity planning, which you use to plan for future growth, such as an increased user base or increased volume of data. For example, to accommodate future loads, you need to know how many additional resources (such as processor capacity, memory usage, disk capacity, or network bandwidth) are necessary to support future usage levels.

SOAK TEST/Endurance TestSoak testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use. For Example: a system may behave exactly as expected when tested for 1 hour. However, when it is tested for 3 hours, problems such as memory leaks cause the system to fail or behave randomly.

Performance Testing Approach

Performance Test Approach

Performance Testing ToolsLoad RunnerRational RobotSilk PerformerWeb LoadOpenSTAVisual Studio Team SystemJMeter

Comparision of features of Loadrunner, Silk Performer and RPT Features Sl no 1: Protocol Support Load Runner : Supports over 51 protocols Silk Perfomer: Supports around 40 protocols RPT :Very few - HTTP, SAP, Sockek API, Citrix Sl no 2 : Selection of multiprotocol for scripting Load Runner : Supported Silk Perfomer: Not supported RPT : Not supported Sl no 3 : Data corelation Load Runner : Much simpler and easier Silk Perfomer : Clumsy RPT : Simpler Sl no 4 : Fune tuning Load Runner :HP Sitescope and Diagnostics available for troubleshooting the identified bottlenecks Silk Perfomer:Not possible, rely on 3rd party tool RPT :To use IBM's Tivoli Sl no 5 : Identifiation of bottlenecks Load Runner :Easier and faster Silk Perfomer:Not easy RPT :Not easy Sl no 6 : Analysis Load Runner :Around 100 types garphs present. Cross garph result.Help in finding the bottle neck. Silk Perfomer:Supports cross graphs RPT :Has limitation Sl no 7 : Analysis report Load Runner :Repors - html,excel and word Silk Perfomer:Only html RPT :Only html Sl no 8 : Industry-standard scripting language Load Runner : Load Runner:C,Java,VB Silk Perfomer: bdl RPT : Java Sl no 9 : Support level Load Runner : Three levels of support available. Good support provided Silk Perfomer: Not too good in providing support RPT : Not too good in providing support Sl no 10 : Latency Time analysis Load Runner : End to end response time as well as response time at each level supported. Silk Perfomer: Displays response time at end to end as well as in the method level RPT : Displays response time end to end Sl no 11 : Distributed Load Generation Load Runner : Can control multiple load generators and collect results. Can control load generators located at remote networks (through a firewall) if required Silk Perfomer: Single MMC can control upto 256 agents (load injectors) provided sufficient physical memory is available on the MMC. RPT : Not Supported Sl no 12 : LAN/WAN The ability to emulate the behaviour of different network infrastructures during a test. Load Runner : New feature added to version 7.6. Allows the emulation of latency, packet loss, link faults & dynamic routing effects over the LAN used in a test. Requires a special license. Silk Perfomer: SilkPerformer integrates with Shunra Storm Traffic to provide WAN emulation (Add-on) RPT : Not supported Sl no 13 : Record and Replay feature Load Runner :Extended logging supports view of parameter values and Server messages. Also view and comparison with recorded version of web page view and client response messages. Debugging facilities in script generator, step and breakpoints. Silk Perfomer:Animated try Script option helps in debugging the script. RPT :NO script, Custom code debugging in Java Sl no 14 : Parametrisation Load Runner :Extensive facilities for data entry including wizard interface to DB interrogation. No standard function to lock data sources and maintain uniqueness of concurrently accessed data across distributed tests. Silk Perfomer:Customize user data' wizard helps in visual parameterizing user data. There are special bdl functions which can be used to fetch data from DBs. RPT : Automatic datapooling Sl no 15 : URL Load Runner :www.hp.com Silk Perfomer:www.segue.com RPT :http://www.ibm.com/ Sl no 16 :Caching - The ability to emulate the caching of pages as performed by a web browser. Load Runner :Can control browser cache emulation during playback and control setting for each individual vuser. Silk Perfomer : TrueCachetechnology simulates the exact behaviour of a browser caching files. Has bdl functions which helps in controlling cache behaviour. RPT :Not Supported Sl no 17 :Release - patch and new version Load Runner :Every 6 months Silk Perfomer:Every 1 year RPT :Every 1 year Sl no 18 :New Enhancement Load Runner :LR V9.5 - Protocol Analyser Silk Perfomer : nil RPT :nil From the information it seems you have not even bothered to read the RPT product documentation. I have been using RPT for many years and almost everywhere you claim RPT does not support something, when in fact it does. Some examples: Point #11 & #12 are completely incorrect, RPT supports as many load generators as you like and it provides the capability to control line speed to emulate LAN WANs... Point #1: "Protocol Support" you list only 4 while RPT supports about 12 protocols and also provides an open API to develop support for anything you choose. Point#2: Multiprotocol you claim "not supported" yet RPT allows you to execute any number of different protocols all from one schedule (or suite). Point#3: Data correlation you claim "Simpler", yet you don't mention that RPT has one of the most powerful automatic correlation engines provided, where 95% of the correlation & parameterization is done for you.

Understanding LoadRunner Licensing

Load Runner licenses fall into one of the following categories:permanent license-Annual maintenance fee required Term license -either 1 month or 3 months Virtual User Days ex:1000 vusers for one daySoftware as a Service :- Hp will provide the controller and chargeWhen you purchase a Load Runner license, you will need to purchase a license for a Load Runner Controller and for a certain number of Virtual Users of a particular type.

Performance Testing ProcessThe following steps are involved in load-testing: Step 1 - Identify performance acceptance criteria Step 2 - Identify key scenarios Step 3 - Create a workload model Step 4 - Identify the target load levels Step 5 - Identify metrics Step 6 - Design specific tests Step 7 - Run tests Step 8 - Analyze the results

Loadrunner Testing Process

Performance and Technical Testing Approach

Load testing typically consists of five phases: planning, script creation, scenario definition, scenario execution, and results analysis. Define your performance testing requirements, for example, number of concurrent users, typical business processes and required response times. Create Vuser Scripts: Capture the end-user activities into automated scripts. Define a Scenario: Use the LoadRunner Controller to set up the load test environment. Run a Scenario: Drive, manage, and monitor the load test from the LoadRunner Controller. Analyze the Results: Use LoadRunner Analysis to create graphs and reports, and evaluate the performance.

Loadrunner Testing Process

LoadRunner Internal Architecture