Overview of Performance Testing Concepts

8/8/2019 Overview of Performance Testing Concepts

1/7

Overview of Performance Testing Concepts

Performance Testing : There are lot of Definitions available but the one mentionedin IEEE Glossary is as follows:

Testing conducted to evaluate the compliance of a system or component withspecified performance requirements. Often this is performed using an automatedtest tool to simulate large number of users. Also known as "Load Testing".

Or

The testing performed to determine the degree to which a system or componentaccomplishes its designated functions within given constraints regardingprocessing time and throughput rate.

The purpose of the test is to measure characteristics, such as response times,

throughput or the mean time between failures (for reliability testing)

Performance testing tool:A tool to support performance testing and that usually has two main facilities:load generation and test transaction measurement. Load generation can simulateeither multiple users or high volumes of input data. During execution, responsetime measurements are taken from selected transactions and these are logged.Performance testing tools normally provide reports based on test logs and graphsof load against response times.

Features or characteristics of performance-testing tools include support for: generating a load on the system to be tested; measuring the timing of specific transactions as the load on the system varies; measuring average response times; producing graphs or charts of responses over time.

Load test: A test type concerned with measuring the behavior of a component or system with increasing load, e.g. number of parallel users and/or numbers oftransactions to determine what load can be handled by the component or system.

While doing Performance testing we measure some of the following:

Characterisitics (SLA) Measurement (units)Response Time SecondsHits per Second #HitsThroughput Bytes Per SecondTransactions per Second (TPS) #Transactions of a Specific Business ProcessTotal TPS (TTPS) Total no.of TransactionsConnections per Second (CPS) #Connections/SecPages Downloaded per Second (PDPS) #Pages/Sec


2/7

Some Definitions and importance of the Above:

Response Time :

What is TransactionR

esponse Time?

Transaction Response Time represents the time taken for the application tocomplete a defined transaction or business process.

Whyis important to measure Transaction Response Time?

The objective of a performance test is to ensure that the application is workingperfectly under load. However, the definition of perfectly under load may varywith different systems.By defining an initial acceptable response time, we can benchmark theapplication if it is performing as anticipated.

The importance of Transaction Response Time is that it gives the project team/application team an idea of how the application is performing in themeasurement of time. With this information, they can relate to theusers/customers on the expected time when processing request or understandinghow their application performed.

What does TransactionResponse Time encompass?

The Transaction Response Time encompasses the time taken for the request

made to the web server, there after being process by the Web Server and sent tothe Application Server. Which in most instances will make a request to theDatabase Server. All this will then be repeated again backward from the DatabaseServer, Application Server, Web Server and back to the user. Take note that thetime taken for the request or data in the network transmission is also factored in.

To simplify, the TransactionResponse Time comprises of the following:1. Processing time on Web Server2. Processing time on Application Server3. Processing time on Database Server4. Network latency between the servers, and the client.

The following diagram illustrates Transaction Response Time.


3/7

TransactionResponse Time = (t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9) X 2Note:Factoring the time taken for the data to return to the client.

How do we measure?

Measuring of the Transaction Response Time begins when the definedtransaction makes a request to the application. From here, till the transaction

completes before proceeding with the next subsequent request (in terms oftransaction), the time is been measured and will stop when the transactioncompletes.

Differences withHits Per Seconds

Hits per Seconds measures the number of hits made to a web server. Thesehits could be a request made to the web server for data or graphics. However,this counter does not represent well to users on how well their applications isperforming as it measures the number of times the web server is being accessed.

How can we use Transaction Response Time to analyze performanceissue?

Transaction Response Time allows us to identify abnormalities whenperformance issues surface. This will be represented as slow response of thetransaction, which differs significantly (or slightly) from the average of theTransactionResponse Time.With this, we can further drill down by correlation using other measurementssuch as the number of virtual users that is accessing the application at the pointof time and the system-related metrics (e.g. CPU Utilization) to identify the rootcause.

Bringing all the data that have been collected during the load test, we cancorrelate the measurements to find trends and bottlenecks between the responsetime, the amount of load that was generated and the payload of all thecomponents of the application.

How is it beneficial to the Project Team?


4/7

Using Transaction Response Time, Project Team can better relate to their usersusing transactions as a form of language protocol that their users cancomprehend. Users will be able to know that transactions (or business processes)are performing at an acceptable level in terms of time.Users may be unable to understand the meaning of CPU utilization or Memory

usage and thus using a common language of time is ideal to convey performance-related issues.

Relation between Load, Response Time and Performance:

1. Load is Directly Proportional toResponse Time2. Performance is inversely proportional toResponse Time.

So, As and When the Load increases the Response Time Increases. As ResponseTime Increases, the Performance Decreases.

Hits Per Second

A Hit is a request of any kind made from the virtual client to the application beingtested (Client to Server). It is measured by number of Hits. The higher the HitsPer Second, the more requests the application is handling per second.

A virtual client can request an HTML page, image, file, etc. Testing theapplication for Hits Per Second will tell you if there is a possible scalability issuewith the application. For example, if the stress on an application increases but theHits Per Second does not, there may be a scalability problem in the application.

One issue with this metric is that Hits Per Second relates to all requests equally.Thus a request for a small image and complex HTML generated on the fly will both be considered as hits. It is possible that out of a hundred hits on theapplication, the application server actually answered only one and all the restwere either cached on the web server or other caching mechanism.

So, it is very important when looking at this metric to consider what and how theapplication is intended to work. Will your users be looking for the same piece ofinformation over and over again (a static benefit form) or will the same numberof users be engaging the application in a variety of tasks such as pulling upimages, purchasing items, bringing in data from another site? To create the

proper test, it is important to understand this metric in the context of theapplication. If youre testing an application function that requires the site towork, as opposed to present static data, use the pages per second measurement.

Pages Per Second

Pages Per Second measures the number of pages requested from the applicationper second. The higher the Page Per Second the more work the application is


5/7

doing per second. Measuring an explicit request in the script or a frame in aframeset provides a metric on how the application responds to actual workrequests. Thus if a script contains a Navigate command to a URL, this request isconsidered a page. If the HTML that returns includes frames they will also beconsidered pages, but any other elements retrieved such as images or JS Files,

will be considered hits, not pages. This measurement is key to the end-usersexperience of application performance.

There is a correlation between the Page Per Second and the stress inflicted on anapplication. If the stress increases, but the Page Per Second count doesnt, theremay be a scalability issue. For example, if you begin with 75 virtual usersrequesting 25 different pages concurrently and then scale the users to 150, thePage Per Second count should increase. If it doesnt, some of the virtual usersarent getting their pages. This could be caused by a number of issues and onelikely suspect is throughput.

Throughput

The amount of data transferred across the network is called throughput. Itconsiders the amount of data transferred from the server to client only and ismeasured in Bytes/sec.

This is an important baseline metric and is often used to check that theapplication and its server connection is working. Throughput measures theaverage number of bytes per second transmitted from the application beingtested to the virtual clients running the test agenda during a specific reportinginterval. This metric is the response data size (sum) divided by the number ofseconds in the reporting interval.

Generally, the more stress on an application, the more Throughput. If the stressincreases, but the Throughput does not, there may be a scalability issue or anapplication issue.

Another note about Throughput as a measurement it generally doesnt provideany information about the content of the data being retrieved. Thus it can bemisleading especially in regression testing. When building regression tests, leavetime in the testing plan for comparing returned data quality.

Rounds

Another useful scalability and performance metric is the testing ofRounds.Rounds tells you the total number of times the test agenda was executed versusthe total number of times the virtual clients attempted to execute the Agenda.The more times the agenda is executed, the more work is done by the test and theapplication.The test scenario the agenda represents influences the rounds measurement.


6/7

This metric can provide all kinds of useful information from the benchmarking ofan application to the end-user availability of a more complex application. It is notrecommended for regression testing because each test agenda may have adifferent scenario and/or length of scenario.

H

it TimeHit time is the average time in seconds it took to successfully retrieve an elementof any kind (image, HTML, etc). The time of a hit is the sum of the Connect Time,Send Time, Response Time and Process Time. It represents the responsiveness orperformance of the application to the end user. The more stressed theapplication, the longer it should take to retrieve an average element. But, likeHits Per Second, caching technologies can influence this metric. Getting the mostfrom this metric requires knowledge of how the application will respond to theend user.This is also an excellent metric for application monitoring after deployment.Using baseline data, a test insert probe can alert test or QA when the applicationslows to a certain response time.

Time to First Byte

This measurement is important because end users often consider a sitemalfunctioning if it does not respond fast enough. Time to First Byte measuresthe number of seconds it takes a request to return its first byte of data to the testsoftwares Load Generator.For example, Time to First Byte represents the time it took after the user pushesthe enter button in the browser until the user starts receiving results. Generally,more concurrent user connections will slow the response time of a request. Butthere are also other possible causes for a slowed response.

For example, there could be issues with the hardware, system software ormemory issues as well as problems with database structures or slow-respondingcomponents within the application.

Page Time

Page Time calculates the average time in seconds it takes to successfully retrieve apage with all of its content. This statistic is similar to Hit Time but relates only topages. In most cases this is a better statistic to work with because it deals with thetrue dynamics of the application. Since not all hits can be cached, this data ismore helpful in terms of tracking a users experience (positive or frustrated). Its

important to note that in many test software application tools you can turncaching on or off depending on your application needs.

Generally, the more stress on the site the slower its response. But since stress is acombination of the number of concurrent users and their activity, greater stressmay or may not impact the user experience. It all depends upon the applicationsfunctions and users. A site with 150 concurrent users looking up benefit


7/7

information will differ from a news site during a national emergency. As always,metrics must be examined within context.

Failed Rounds/FailedRounds Per Second

During a load test its important to know that the application requests perform asexpected. The Failed Rounds and Failed Rounds Per Second tests the number ofrounds that fail.

This metric is an indicator metric that provides QA and test with clues to theapplication performance and failure status. If you start to see Failed Rounds orFailed Rounds Per Second, then you would typically look into the logs to see whattypes of failures correspond to this metric report. Also, with some software testpackages, you can set what the definition of a failed round in an application.

Sometimes, basic image or page missing errors (HTTP 404 error codes) could beset to fail a round, which would stop the execution of the test agenda at that pointand start at the top of the agenda again, thus not completing that particularround.

FailedHits/FailedHits Per Second

This test offers insight into the applications integrity during the load test. Anexample of a request that might fail during execution is a broken link or a missingimage from the server. The number of errors should grow with the load size. Ifthere are no errors with a low load, the number of errors with a high load shouldremain zero. If the percentage of errors only increases during high loads, theapplication may have a scalability issue.

Failed Connections

This test is simply the number of connections that were refused by theapplication during the test. This test leads to other tests. A failed connectioncould mean the server was too busy to handle all the requests, so it startedrefusing them. It could be a memory issue. It could also mean that the user sentbogus or malformed data to which the server couldnt respond so it refused theconnection.

Documents

Overview of Performance Testing Concepts