Load Runner User Guide

Software Quality Technicians

ABN: 99 070 946 986 www.qualtech-int.com.au email: [email protected]

QualTech International

LoadRunner User’s Guide

A White Paper by Ivan Maclaine

Contents

Introduction

LoadRunner Overview

Preparation

Development

Execution

Analysis

Results

Level 2, 222 Pitt Street Sydney NSW 2000 Phone: (02) 9264 0055 Fax: (02) 9264 0033

Level 28, 303 Collins Street Melbourne VIC 3000 Phone: (03) 9614 8555 Fax: (03) 9614 8666

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

Software Quality Technicians Commercial-in-confidence

Introduction

Purpose

The Client/Server approach to providing business solutions, while at the same time delivering more functional and powerful systems, is also introducing new risks. As well as the functional concerns associated with any computer systems, client/server solutions have a much greater associated risk in terms of the performance of the overall system. LoadRunner is a software tool that facilitates in measuring the performance of systems prior to deployment. This allows performance issues to be addressed before rollout.

This paper is a guide to implementing a LoadRunner based load-testing solution. It also provides some theoretical background into the area, defining several key concepts. It is recommended that this paper be used in conjunction with the LoadRunner Implementation Guide available from Mercury Interactive. This paper should not be seen to replace Mercury’s guide, but as a supplement to it.

Load/Stress Testing

Load and stress testing are two different, but related forms of testing. Stress testing involves testing a system under extreme cases, which will rarely occur under normal conditions. Such testing can involve such scenarios as running many users concurrently for short times, short transactions repeated quickly, and extraordinarily long transactions.

Load testing places a more realistic load on the system. Scenarios are modelled on the demand that the system is expected to face under real conditions. Load testing starts by placing a low demand on the system, and gradually increasing the load. This process allows measuring the performance of the system under different loads, as well as determining the maximum load that under which the system can respond acceptably.

Load and stress testing are clearly two interrelated processes, generally performed concurrently.

It is important to note what load/stress testing isn’t. They are not functional tests. In order to perform performance testing, it must be assumed that a reasonably stable and functional system is in use, otherwise results are meaningless. With that said, it is still common to uncover functional problems in a system while performing load testing. However, uncovering excessive functional problems can be a sign of inadequate functional testing.

Black Box Testing

Strictly speaking, black box testing involves testing a system as a whole, not as a system of connected parts. Realistically, however, black box testing still provides some differentiation between clients, the server, and the network connecting them. This means that when performing black box testing, it is possible to determine whether time is being spent in the client, the network, or the server, and hence determine where bottlenecks lie.

The limitation of this approach is that knowing, for example, that the server is the bottleneck of a system is not necessarily sufficient. Is the bottleneck caused by a lack



of memory, a lack of CPU performance, poor algorithms, or one of many other causes? To answer these questions, diagnostics need to be performed on the server while generating a load.

White Box Testing

White box testing treats the system as a collection of many parts. During white box testing, many diagnostics will be run on the server(s), the network, and even on clients. This allows causes of bottlenecks to be much more ably identified, and therefore addressed.

White box testing requires much greater technical knowledge of the system than does black box testing. To perform white box testing, knowledge is needed as to the components that make up the system, how they interact, the algorithms involved with the components, and the configuration of the components. It must also be known how to perform diagnostics against these components, and even what diagnostics are needed and appropriate. This knowledge can generally only be provided by someone intimate with the system.

A problem with white box testing is the effect that running diagnostics has on system performance. Ideally, running diagnostics should have no impact on performance. However, with the exception of the network, this is rarely possible, running diagnostics consumes some system resources. When considering the appropriateness of various diagnostics, one must consider the benefit vs. the impact on performance.

Black Box vs White Box Testing

Black box testing is generally much easier to perform than white box testing, requiring system-specific expertise. However, while black box testing may identify that problems do exist, white box shows with much more clarity where they exist.

Black box simply shows whether or not performance is acceptable. This is still useful, as it can show which transactions are taking the most time, as well as highlight problems with certain numbers of transactions. A common use of black box testing is to identify knees in the response of a system against the number of concurrent transactions. Such a knee indicates a limitation in the system that once passed, provides exponentially decreasing system performance, a bad thing if reached.

End to End Testing

End to end testing is testing from the point of view of the client. It involves manually or automatically using the actual client software to perform business processes, and measures the response time of the application. End to end testing measures system performance from the end users perspective

LoadRunner Overview

LoadRunner is a tool, or more accurately a collection of tools, which facilitates in performing load testing on a system. LoadRunner is used to create a load or stress on a system, and can be used to measure the responsiveness of the system to that load. The key concept behind LoadRunner is the use of virtual users, or vusers.

Virtual Users



To perform a load test, a system needs many users to access the system concurrently, often one hundred or more. Performing such a test with people is a logistical nightmare. When considering that the test cycle is generally iterative, this problem is compounded.

LoadRunner overcomes this problem through the use of vusers. A vuser performs the actions that a real user would perform, except that a vuser is controlled by LoadRunner, and the actions are performed automatically.

Vusers are created to perform specific transactions in the system. Once created a vuser can be replicated as many times as required, thus producing an effective load as large as is required. With the exception of GUI vusers, virtual users don’t use the client to perform their actions; they take the place of the client. This allows in many vusers to be run on one workstation, as the overhead associated with the client is not required. This does mean, however, that non-GUI vusers do not test the performance associated with the client side of a system, only the network and server associated performance.

Vusers can be of four main types.

DB Virtual Users

A DB, or database vuser performs operations against a database server. LoadRunner supports all the most popular interfaces to database servers including but not limited to ODBC, SQL, and various Oracle interfaces. It is also possible to create LoadRunner interfaces that allow recording against unsupported databases.

Recording a DB vuser is as simple as performing a business process while having launched the LoadRunner Virtual User Generator. This creates a script which is used by LoadRunner to perform the recorded business process.

Once recorded, a vuser script is then parameterised. This process involves replacing the hard coded data values in the script with parameters, which refer to tables external to the script. Values can then be placed in these tables allowing vusers to perform a single business process, but with different data each time.

Web Virtual Users

A web virtual user is one that performs actions against a web based system. A web vuser is essentially identical to a DB vuser, except that where the DB vuser is recorded against a client application that communicates directly with a database of some description, a web vuser is recorded against a web based application.

RTE Virtual Users

Remote Terminal Emulation virtual users are used to emulate users that would be operating on a lightweight, character based terminal.

SAP Virtual Users

SAP R/3 virtual users are created somewhat differently than the others, and require the SAP R/3 LoadRunner add-on. Essentially, a SAP R/3 virtual user is one that performs actions in a SAP R/3 environment.



GUI Virtual Users

Unlike the previous types of vusers, GUI virtual users actually perform actions using the client. This means that fewer GUI vusers can be run per workstation than the other types of vusers due to the overhead of the client, and is in fact limited to one GUI vuser per PC workstation.

GUI vusers are used not so much to generate load on a system, but to test end-to-end performance associated with various business processes. Although LoadRunner can control GUI vusers, it is unable to create them. For this purpose, WinRunner is required.

Controller

The LoadRunner controller is used to create and run scenarios. Once vuser scripts have been created, they can be added into the controller in varying quantities to produce the demand on the system that is required.

When running a scenario, the controller coordinates all the vusers, graphical or otherwise. It starts vusers as required, and can force them to rendezvous with each other, thus producing a maximum peak demand.

Agent

When running a scenario, every workstation that is to be used to generate load must have the agent software installed. This software allows a remote (or local, but this is not advised) controller to send vusers to the workstation to be run. The agent communicates directly with the controller, reporting performance data of the vuser, as well as ensuring that the vuser runs as specified in the scenario.

Virtual User Generator

The virtual user generator (VUGen) is used to generate scripts for all non-GUI vusers. The VUGen records the behaviour of the client software, and automatically generates a script that can be used by the controller to run vusers.

Once a script is created, the VUGen can then be used to parameterise the script. This involves replacing hard-coded values in the script with references, which refer to external tables. The VUGen provides a semi-automated procedure for doing this.

Load Analyser

The load analyzer utility presents the results of running a scenario in a variety of graphical and tabular forms. These graphs and tables can be used to determine the acceptability of a system, and to determine areas of concern. The data presented can be exported in a variety of popular formats, allowing it to be used outside LoadRunner.

Preparation

Before load testing can begin, a certain amount of preparation is needed, however the exact time required varies considerably from system to system. A guide to the issues that should be addressed follows.



Planning

This phase identifies the goals of testing, how the goals may be reached, and a timeline for doing so. A typical goal is to conduct load tests to determine where bottlenecks lie in a system, and to suggest ways the system could be improved to overcome these. Note here that this goal implies some degree of white box testing, as it asks us to determine where the bottlenecks lie inside the system.

In this phase, resources are allocated for the load test. This includes people, hardware and time. Project managers in conjunction with testing consultants generally perform this phase. This stage can generally be performed quickly, in no more than a few days.

Analysis/Design

This phase involves identifying transactions performed in a system, rates of performing transactions, and scenarios that model expected use of the system. It involves performing a risk assessment of all transactions of a system to determine which are the most critical, in terms of:

Frequency of use

Demand placed on system

Importance to the system

Any other criteria deemed appropriate

A transaction is a series of steps a real user would perform to achieve a business process in the system. Typical transactions include such things as performing a customer query, entering an order, and invoice printing. The exact transactions, and complexity thereof vary considerably between systems. It is preferable to limit the complexity of transactions, especially compound transactions, while ensuring that meaning isn’t lost. This is for practical reasons; complex transactions are more difficult to model, maintain, and reduce flexibility when creating scenarios.

It is important to determine exactly what transactions should be used in load testing; creating them all as vuser scripts is time consuming and inefficient. Experience shows that modelling 80% of a user population provides accurate load tests, and this is typically achieved by about 20% of transactions. This typically amounts to 5-10 transactions per application module, although this figure can vary depending on the modularity of a system.

The previously mentioned risk assessment is used to determine which transactions should be used in the load test. When determining the transactions to implement, the tradeoff of accuracy of the load test against the time involved in getting results needs addressing. The risk assessment is best performed by both project managers and experienced testing consultants, and generally takes two or more days.

Once identified, the transactions to be performed during load testing must be detailed in a step-by-step fashion. These can later be passed on to test engineers who can then work unassisted to create LoadRunner vusers. When detailing the transactions, it is advised to extract the data that can change between runs of the transaction from the rest of the script. This makes it clear to the test engineer exactly what is to be



parameterised.

Instead of detailing transactions in the above fashion, it is also possible to make available to test engineers people knowledgeable with the system that can show them how to perform transactions as required. This approach requires less planning, and up-front work, however is more likely to lead to transactions which do not accurately reflect the processes of real uses.

With transactions identified, it is also necessary to design the scenarios to be run. The scenarios should be designed with the goals of the load testing in mind, and place demand on the server and network as required to meet the goals of the test.

Data Dependencies and Consumption

Transactions can be divided into two categories; those that do not have any need of information already in the system, and those that do. The first type are easily implemented, but are in a minority. These transactions can simply be recorded, parameterised as appropriate, and run as many times as required.

The second type has a dependency on other data in the system. This dependency can be quite weak, such as simply needing an unused customer ID number, or as strict as requiring a specific customer ID number. There are also transactions which consume data. Consider a delete customer transaction; it is dependent upon a pre-existing customer ID, but once the transaction is performed with that ID, that ID becomes unavailable, it has been consumed.

It is important that in this analysis phase, that data dependencies and data consumption are identified. This allows the system to be pre-populated with sufficient data that scenarios can be run several times before requiring a refresh.

Testing Environment

The environment used for testing can vary considerably depending on the goals of the load test. If the load test aims to simply determine whether the performance of a server is satisfactory, then the test environment can consist of the server and a high-powered workstation connected by a high-speed network.

If, however, the impact of the network is a larger concern, then it is necessary to have a test environment that runs over a network identical to the one used when the system goes live. This is especially a concern when testing a system that will be run over a WAN.

Isolated Environment

Generally, it is preferable to have a testing environment that is isolated from other users. This allows precise measurement of the impact of different loads on the system, with no glitches caused by external network or server access.

However, in systems that are to be run over the same network as other systems, which will have users accessing the server for reasons outside the system under test, this is not always desirable. In these cases, it may be preferable to perform load testing with the test environment connected to other systems, as it would be when live. The danger of this approach is that spikes can enter into performance measures, and that the load



test can render other activities unusable, due to network and server congestion.

An isolated environment is recommended for testing unless there are very good reasons to the contrary.

Realistically sized database

In many cases, the responsiveness of a system is related to the amount of data in the system. Therefore, it is meaningless to perform load tests on a system unless it contains a realistic amount of information. In this phase, it is important to identify the expected size of the system when it goes live, so that it can be later populated to reach this size.

Identify LoadRunner Workstations

Depending on the exact goals of the load testing, a number of workstations to run vusers must be identified. If the load test aims to do no more than place a heavy load on the server, then one high powered workstation is all that is required. If the effects of various network connections is to be tested, then workstations will be required at the end of these connections, or least at the end of identically simulated connections.

LoadRunner Installation

At this point, LoadRunner will need to be installed on at least one machine for development purposes. This install will include all components of LoadRunner. Workstations that have been identified as those to run vusers can be installed with the LoadRunner agent at this stage, but this is not necessary until just before execution of the scenarios.

Development

Development is one of the more time consuming aspects to load testing. Development can easily take one month or more, but this depends entirely on the number and complexity of the transactions to be modelled, as well as the number of people working in development.

A proficient test engineer is generally able to produce about 2 scripts per day, with less capable and experienced people capable of about half this.

LoadRunner Scripts

LoadRunner vuser scripts form the backbone of a load test. Every transaction must be recorded, parameterised, and tested prior to executing load scenarios.

Recording Vuser scripts

Recording vuser scripts is normally as simple as performing actions in the client application while recording with the virtual user generator. The VUGen creates a script file containing all the data access that occurs between the client and the server.

In some cases, this is not sufficient, as the client application may communicate with an application such as Excel, which then communicates with the server. The VUGen does not record this secondary communication. If this is the case, it is possible that an experienced test engineer can first record against the client, and then against the secondary application, and combine the two scripts. This is a complicated procedure,



and should not be attempted by inexperienced test engineers.

Parameterising Vuser scripts

Once a vuser script is recorded, it must then be parameterised. This involves extracting the hard-coded values from within the script, and placing them in an external data table. This allows a single vuser to be run many times, and to use different data on each run. This provides a more realistic demand on the system, as it prevents the server from caching the results of many transactions.

Scripts can be parameterised in a variety of ways. Some parameters will be defined to act as global variables, specifying such things as the database name, while others will take on a new value on every iteration of the vuser. LoadRunner is extremely flexible in this respect, and can model most requirements that may be placed on the load test.

Testing Scripts

Test scripts, like any form of program, need to be tested. Initial testing involves running a test in isolation, ensuring that it performs the transaction as intended, that no unforeseen data is consumed, and that all necessary data is available for it.

Once this is achieved, multiple instances of a test are run concurrently, ensuring that they don’t conflict with each other, and that their parameters are correctly set so as not to cause deadlocks. It is also preferable to run different scripts concurrently, ensuring that one script cannot cause another to deadlock. If the analysis phase was performed with due care, this will not be a problem.

WinRunner Scripts

WinRunner scripts are not used so much to generate load, but to measure the end to end performance of the system when it is under load. As a result of this, it is common for only a subset of LoadRunner implemented transactions to be implemented as WinRunner scripts. This is a practical consideration; scripting all transactions as both GUI and non-GUI vusers is time consuming. However, it is important that sufficient transactions are scripted using WinRunner so as to provide a clear assessment of the end to end performance of a system, and occasionally this does involve scripting all transactions. The tradeoff here is again the time required vs. benefits gained in terms of the usefulness of the performance results.

Scripting transactions as GUI vusers is considerably different to non-GUI vusers, and is explained fully in the associated WinRunner implementation documentation.

Data Preparation

Preparation of data in the system must be completed before scenario execution can commence. This phase ensures that the system contains enough data that the most demanding load scenarios can be executed at least once, but preferably three or more times, before the system requires refreshing.

This phase can be performed in parallel to script development, and can normally be achieved quite quickly if the data demands of the transactions was correctly identified in the planning phase.



Scenario Creation

With transactions scripted as both GUI and non-GUI vusers, scenarios that place the system under load can be implemented. These scenarios should have been previously identified in the earlier planning phase, and all that is required here is creating them in the LoadRunner controller.

As a rule of thumb, most scenarios should run for about one hour. This allows the results of the run to be more statistically reliable than a shorter run, and is not so long as to consume too much time. Shorter scenarios should be looked upon unfavourably, and longer scenarios implemented only if there is good cause to do so.

Golden Area

Before execution is commenced, it is important to create a golden area, or a place where all scripts, scenarios, and a database baseline can be placed. This golden area should be treated as write-once read-many. This means that scripts placed here must be bug-free, as modifications made to files in the golden area can greatly confuse future testing.

The database baseline should contain all the data required for performing the most strenuous of the load tests, and is used to refresh the system as required. Preferably, the database baseline placed here should allow several load tests to be performed before requiring the system to be refreshed. As these refreshes are generally time consuming activities, it is ideal if refreshes can be performed overnight, after every day of load testing.

It is important to note that it is not always possible or feasible to create database baselines, due to the complexity of a system and its batch processing. Where this is the case, it is important to pre-determine this, and make allowances in creating the data that load tests can consume. This type of scenario is far from ideal, as it is difficult to compare results between runs, as too many system variables have changed. All that can be done is to raise the issue, and continue as best as possible. It is, however, worth noting that systems in which it is "impossible" to revert to baselines, often through ingenuity techniques can be created allowing just that to be done.

Execution

When this phase is commenced, the testing environment must be correctly set-up, and the scenarios for testing must be created and prioritised. The exact scenarios to be run vary widely depending on the goals set prior to planning the load test. Some common scenarios that are recommended follow.

Baseline Load

The baseline or light load scenario involves running small numbers of users, ensuring that the testing environment behaves as expected. As the scripts have been tested under the development environment, problems here are likely to indicate configuration problems in the testing environment. If these occur, they must be addressed before more demanding testing can begin.

A second reason for executing a light load scenario is to use the results as a benchmark. Performance results obtained under the baseline load should be more than



satisfactory, otherwise the system has no hope of performing adequately under a higher load.

Assuming no performance problems were noticed during this scenario, the results obtained can be used as "best case" results. These results indicate how the system performs when it is not under any real form of stress, allowing conclusions to be drawn about the performance of the system during the higher load tests.

Medium Load

Depending exactly on the requirements of the load testing project, a medium load may constitute what is considered a real-world worse case load, or as little as 50% of this. Where the load that is applied is less than the expected worse case load, the scenario is used to identify major failings in a system, but does so in a way that doesn't highlight many of the more minor failings, allowing an easier analysis of results. Where the load is equivalent to the expected real-world worse case load, actual performance of the system can be measured, and associated problems clearly identified.

Running a medium load before moving directly to a high load allows the more major problems of a system to be identified separately of the smaller issues, by providing a simpler result analysis.

Heavy Load

Heavy load scenarios test a system at and beyond the worse expected demand it is likely to face. The majority of major deficiencies in the system will have already been identified in the medium load execution, so this phase deals more with assessing the impact of these issues under a heavy load. However, heavy scenarios also identify many other system bottlenecks not previously noticed, which may in fact be partially responsible for earlier identified problems.

Heavy load scenarios are generally designed to be far more than a system can handle. They are used not to identify if a system fails, but where it fails first, by how badly, and why. By answering the why question, it can be determined whether a system is as good as it should be.

Heavy scenarios often involve starting a scenario running, and incrementally adding more and more load onto the system, until it either fails altogether, or becomes totally unusable. At this point, the load can be slowly decremented. This allows the effect of different loads to be easily analysed, and plotted graphically.

The analysis performed in this phase can vary depending on the exact goals and objectives of the load testing. Results obtained from within LoadRunner are often used in conjunction with results obtained directly from the system, where white-box testing has been employed.

The majority of scenario execution time will be spent with heavy load scenarios, as these provide the most information about the performance of a system.

Analysis

The entire reason of performing load testing is so a better understanding of a system’s performance can be obtained. The results obtained during LoadRunner scenario



execution are plentiful, and this is compounded if white-box metrics have also been measured. The analysis phase typically occurs in parallel with the execution phase, as both phases are iterative in nature, although analysis occurs with an offset from the execution, and can continue after execution has completed.

White Box vs Black Box Analysis

Where white-box metrics have been recorded, they can provide much more insight into where problems exist in a system, as well as to highlight where improvements are most necessary.

Black box analysis can show the effects of running different numbers of simultaneous transactions, as well as showing the effect of different combinations of transactions, but the results. However, black box testing will only highlight performance issues on a per transaction basis.

White box analysis can show exactly where in a system time is being spent, as well as show other information not directly related to time, such as the percentage of transactions that result in more than 10 rows being fetched. Clearly, this information requires a lower level of knowledge about a system than LoadRunner can gain.

Black box analysis provides the big picture of a system’s performance, which is often all that is required. White box analysis concentrates on looking at the system as a collection of co-operating parts, allowing finer detail to be obtained.

Results

Many load test projects iterate between the execution and analysis phases many times, with system modifications being made between iterations, thus the load tests are used to improve the system. Other load test projects are simply performed to measure the performance of a system, preparing the systems owners for its performance, and allowing them to determine if changes may be required. Depending on the situation, the writing of results may vary.

The results of a load test project should directly relate to the original goals and objectives of the load test. They should focus on what system requirements are met, how well they are met, and which fail. Any issues encountered that do not directly relate to the objectives of the project should be mentioned in full, allowing further action to be taken where this is deemed necessary.

A write-up of results should include at least part of the analysis, but this is more often desirable as an appendix. Load testing is often performed as part of user acceptance testing, even if only implicitly, and this should be recognised when load test results are being tabulated.

Documents

Load Runner User Guide