22
"How To" manual for planning, setting up and executing Online Performance testing using Oracle Application Test Suite (OATS) to test the RPAS Fusion Front end

Online Performance Testing With OATS

Embed Size (px)

DESCRIPTION

Online Performance Testing With OATS

Citation preview

Page 1: Online Performance Testing With OATS

"How To" manual for planning, setting up and executing Online Performance testing using Oracle Application Test

Suite (OATS) to test the RPAS Fusion Front end

Page 2: Online Performance Testing With OATS

2

Purpose

• The purpose of this slide deck is to provide Accenture projects and resources with first hand project experience using the Oracle Application Testing Suite (OATS) and in particular the Oracle Load Tester (OLT) when performance testing Oracle Retail Predictive Application Server (RPAS)

• This guide will provide all the information necessary to properly plan, set up, execute and report on RPAS Oracle Fusion Client performance tests using OATS

• This is not an installation guide and will not provide installation steps for all the elements, for this type of information please refer to the reference section at the end of this document

• Although not every single procedure and step needed to create an end to end performance test is listed in this document, it does provide practical advice that will allow the reader to fast track the overall process and avoid some potential pitfalls

• This guide should answer the most relevant questions a project team may have with regards to testing the Oracle Fusion Client using OATS

• The results of RPAS MFP tests conducted can be found at the following location https://kx.accenture.com/Repositories/ContributionForm.aspx?path=C22/39/4&mode=Read

Page 3: Online Performance Testing With OATS

3

Agenda

Overview Oracle RPAS Fusion Client and ADF explained The Oracle Application Testing Suite Why Oracle Load Tester? Oracle Load Tester architecture System requirements for OATS Other performance testing considerations Practical OpenScript Advice Parameterisation of scripts Performance testing process Test execution Performance test reporting Room for improvement Lessons learnt References

Page 4: Online Performance Testing With OATS

4

Overview

Intended AudienceThis document is intended for all Accenture TA teams involved with Performance Testing/Performance Engineering activities related to Oracle RPAS. The focus of the document is Oracle RPAS but the same principle could be used in testing other web based applications.

Pre requisite skill or knowledgeThere are some skills/knowledge that would be beneficial to achieving the desired outcome in terms of performance testing the Oracle RPAS Fusion Client using OATS• Oracle RPAS (preferable)• Java development (preferable)• Oracle ADF (preferable) • A basic understanding of application testing, load testing, scalability testing etc. (required)

Page 5: Online Performance Testing With OATS

5

Oracle RPAS Fusion Client and ADF explained

Oracle RPAS Fusion Client • The original RPAS Classic Client (thick client) requires a lot of network bandwidth and was not

suitable for RPAS implementations where users had network constraints• Oracle therefore developed the Oracle Fusion based thin client for Oracle RPAS to provide

users with a web based alternative• The Fusion Client is built using the Oracle Application Development Framework (ADF),which

makes scripting online performance testing scripts very difficult using standard performance testing tools

• The Fusion Client is hosted on an Oracle Weblogic platformOracle Application Development Framework (ADF) • The Oracle Java development framework used to develop enterprise applications and in

particular feature rich web applications• Oracle Fusion Client has been developed using ADF• Simplifies and accelerates development of Java based applications by reducing the amount of

coding required

There are two key components when referring to the RPAS “thin client”, the first is the Oracle Fusion Client and second is Oracle ADF

Page 6: Online Performance Testing With OATS

6

The Oracle Application Testing Suite

Oracle OpenScript / Functional Tester • This is Oracles script development tool which is based on the Eclipse IDE• OpenScript provides an easy to use GUI for recording functional and load testing scripts• OpenScript includes a plug in specifically for ADF scripting, amongst others• This is a required component for load testing activitiesOracle Load Tester (OLT)• OLT simulates multiple virtual users accessing an application using the load testing scripts

developed in OpenScript• OLT is the single control point for all load testing agents and reporting of test execution• OLT is able to monitor the underlying infrastructure during testing, using ServerStats and

monitoring agents• Provides post execution analysis capability including transaction analysis• This is a required component for load testing activitiesOracle Test Manager• Test Manager is used to manage the overall testing process• This includes management of test requirements, defect tracking, test planning etc• The Test Manager integrates with OATS allowing you to launch and execute scripts from within

the application• This is not a required component for load testing activities

The complete Oracle Application Testing Suite consists of three components, OpenScript, Load test and Test Manager

Page 7: Online Performance Testing With OATS

7

Why Oracle Load Tester?

• Performance testing can be done manually by getting real users to test the system. The main problem here is getting the sufficient numbers of users required to test in a uniform way and at the same time recording the responses

• Taking a small subset of users and extrapolating for larger numbers of users is not an accurate method of forecasting application performance

• OLT is able to simulate as many users as required and execute all activities in a uniform manner, at the same time extracting vital data from each session

• The Oracle Application Testing Suite is the only tool with an ADF plug in, which can be used to script against ADF based web applications such as the RPAS Fusion Client

• HP Loadrunner is one of the more popular load testing tools but it can only execute directly against RPAS using a specific dll and therefore does not test Fusion Client performance

• At the time of writing it is the only solution, when testing the actual RPAS Fusion Client, not just RPAS performance

• OLT provides real time monitoring of infrastructure during testing and can therefore accurately correlate the performance of the infrastructure and the transactions being executed

The Oracle Application Testing suite has many features and benefits. There are other tools with similar capabilities but OLT comes into its own when testing ADF based applications

Page 8: Online Performance Testing With OATS

8

Oracle Load Test architecture

OLT Controller Module

OLT ServerStats

Module

WebLogic Server

OLT DB

Agent 1

Agent 2

Agent 3

Agent n

Target Application

Monitoring Agent

OS, Host, DB ...

The OLT architecture consists of a server component, which makes us of an Oracle database, as well as load testing and monitoring agents

Virtual users

Page 9: Online Performance Testing With OATS

9

System Requirements for OATS

Component Hardware Software

OLT Load Testing • 2 x 3 GHz processor• 4 GB RAM• 50 GB HDD• IE 8 or 9• Oracle XE db (or full Oracle db)

• Windows Server 2003 R2 / Windows Server 2008

Oracle Test Manager • 2 x 3 GHz processor• 4 GB RAM• 50 GB HDD• IE 8 or 9• Oracle XE db (or full Oracle db)

• Windows Server 2003 R2 / Windows Server 2008

OpenScript • 1 x 3 GHz processor• 2 GB RAM• 50 GB HDD• IE 8 or 9

• Windows Server 2003 R2 / Windows Server 2008

OLT Load TestingAgent

• 1 x 3GHz processor• 2 GB RAM• 50 GB HDD• IE 8 or 9

• Windows XP• Windows Server 2003 R2 / Windows

Server 2008

OLT Load TestingMonitor Agent

• 1 x 3GHz processor• 2 GB RAM• 50 GB HDD• IE 8 or 9

• Windows Server 2003 R2 / Windows Server 2008

Oracle provides system requirements for each of the OATS components but the requirements below are based on actual testing experience. All of the components can be hosted on a single OATS server if required

Page 10: Online Performance Testing With OATS

10

Other performance testing considerations

Monitoring • Although OATS has its own monitoring capability, it would be advisable to make use of other

tools such as nmon or Enterprise Manager to collect and graph statistics• In particular the following must be included when monitoring RPAS performance

• If the clients are accessing the RPAS server across the WAN some utilisation stats should be gathered to determine what the effect the users have on available bandwidth

Cost• Licensing for OATS can be expensive, the cost for load testing is calculated using the number

of concurrent users on the system• The Functional Tester is charged by the number of users developing scripts

Component Area Metric

Fusion Front End Server • JVM Heap size• JVM Garbage collection• CPU utilization percentage

RPAS Server • CPU utilization percentage• Memory (utilization, swapping/paging)

Storage Sub system • Server disk I/O

Page 11: Online Performance Testing With OATS

11

Performance testing process

Define scope

Gather requirement

sPrepare

environmentModel scripts Dry run Execute Report

1. Define scope Determine concurrent user loads Determine performance test type Determine user scenarios Define test environment

2. Gather requirements Develop and document test scripts Determine user allocations Determine data requirements Define testing pre requisites

3. Prepare environment Load data Create system users in RPAS Develop and execute any prep scripts Set up load testing agents in OLT Set up ServerStats in OLT Set up any additional monitoring

4. Model scripts Build and test basic scripts in OpenScript for each

scenario Parameterise and test each script in OpenScript Define think times for each completed script

5. Dry run Build complete scenarios in OLT Execute all scenarios with 25% of planned user

volume load Correct any errors and iterate until error free Complete an hour long dry run

6. Execute Execute all scenarios with user volume loads defined

in scope Monitor and capture all infrastructure stats Execute multiple iterations of test

7. Report Gather all transactional statistics Gather all infrastructure statistics Develop final report

The performance testing process can be broken down into 7 high level activities. These activities are further broken down into major tasks that must be completed

Page 12: Online Performance Testing With OATS

12

Practical OpenScript AdviceCreating Custom Step Groups

• As you record your scripts OLT will number and name your scripts step groups

• This naming and numbering is generally always incorrect, so its best to switch auto step creation off

• This is done by going to the Script Properties

• Ensure that the Step Group is set to “Do not create/number/name steps”

• The step groups are what you will report against in terms of transactions so its vital they are correct

Page 13: Online Performance Testing With OATS

13

Practical OpenScript AdviceCreating Custom Step Groups

• Create a new Step Group for each transaction that you will be recording

• Number the step using curly bracket and give it an appropriate title e.g. [1] Open webpage

Page 14: Online Performance Testing With OATS

14

Practical OpenScript AdviceCreating Custom Step Groups

• Click on the record button to initiate the script recording, a browser window should open

• As you complete recording one step you should click pause and create the next Step Group e.g. [2] Login

• Once completed the script should look similar to the figure on the left

Page 15: Online Performance Testing With OATS

15

Parameterisation of ScriptsUsing Databanks

• Before attempting to parameterise scripts, the data being injected into the scripts should be identified and a csv file with the relevant data must be created and saved to [INSTALLDIR]\OFT\DataBank

• The format of the file is the name and corresponding data of each of the elements separated by a comma

• This databank file is then added to the script during the parameterisation set up

• Initially it would be a good idea to make a copy of your working script before you start altering it with parameterisation

After the initial script is recorded and tested, the script will need to parameterised, so that it can be executed by multiple users or have different inputs during execution

• The most obvious place you would need multiple inputs is during logon, this will allow multiple users to logon to the system by injecting data from the databanks

• The databank depicted injects the username and domain the users are logging on to. It also injects specific departments for each user during the execution of the test

Page 16: Online Performance Testing With OATS

Parameterisation of ScriptsSubstitute Databank Variables

16

After the initial script is recorded and tested, the script will need to parameterised, so that it can be executed by multiple users or have different inputs during execution

Look for “Post Data” where data is being sent

• Add the Databank created previously to the new script as per the user guide. Save the script and the Databank should now be available.

• To parameterise the script, expand the script in the step group where values are being substituted

• A good way to find the correct variable is to look for the data originally used during the recording e.g. pftest051

• Right click the appropriate parameter under Post Data and select substitute

• This method of parameterisation will work when OpenScript adequatley identifies the window in which the data is being entered like a logon screen

Page 17: Online Performance Testing With OATS

17

Parameterisation of ScriptsCustom Variables

• In some instances OpenScript cannot substitute the variables you are looking to replace because OpenSscript may not recognise the field you are replacing

• In these instances you need to directly substitute the parameters with the details of your databank and the data you are replacing

• Right click the appropriate parameter under Post Data and select properties

• Replace the recorded data e.g. D+129+Watches

• The reference to the databank would be {{db.MFP_edgars_users.Dept,D 129 WATCHES}}

• The databank in this example is called MFP_edgars_users

• The variable is Dept and the actual example of the data would be D 129 WATCHES

• Double curly brackets are used to wrap this custom parameter

After the initial script is recorded and tested, the script will need to parameterised, so that it can be executed by multiple users or have different inputs during execution

Page 18: Online Performance Testing With OATS

18

Test execution

• Create a new scenario in OLT and add all the scripts developed in OpenScript to the scenario

• Allocate the number of users to the scenario, allocate the scripts to a load testing agent (system) and configure any additional parameters needed

• Once all the scripts have been added and configured add them to the Autopilot

All load tests are set up and executed via the OLT user interface

• Configure when the load test is started (specific time or when button is pressed)

• Configure how the load test will be stopped (either after the delay or once the button is pressed)

• Configure the VU ramp up• Select the ServerStats configuration previously set up

(verify the correct monitors are in place)• Click the green start button

Page 19: Online Performance Testing With OATS

19

Performance test reporting

Transaction reporting• OLT records each transaction as well as the responses

gathered during the test • A transaction report breaks down all of the session

statistics as well as calculating the standard deviation and 90th% for transactions of users

• This is key in providing detailed transaction analysis

Infrastructure reporting• If ServerStats has been configured infrastructure metrics

will be gathered during test execution• These metrics can then be used for creating graphs

relating to the metrics that have been configured• It is still recommended to gather additional stats relating to

infrastructure monitoring e.g. Nmon

As the test is being executed different statistics are being logged by OLT. These statistics can be used to develop overall test reports

Page 20: Online Performance Testing With OATS

20

Room for improvement

• OLT is not able to execute scripts in a step change ramp up, this means that you have to add users to the load in a uniform fashion at predefined intervals. It becomes difficult to gauge the effect of smaller subsets of users on load without creating smaller individual load tests

• Scripts may need to be split and then allocated to different load testing agents to control the execution more accurately

• Troubleshooting script errors can be difficult depending on your knowledge of java• There was a limitation on script length - long scripts may need to be split into two different

functions but this may have been addressed in OATS 9.30• OpenScript scripts can be unpredictable, the same script may be successful on a single run

and then generate errors afterwards. In depth testing is required at all stages of script development.

• OpenScript is not as very stable for basic testing and is known to hang after multiple iterations• The infrastructure monitoring component ServerStats is not very polished. It lacks detail in

terms of what stats it can gather. We found that the data being retrieved was sometimes inaccurate

• The graphing component is not user friendly or configurable

Although Oracle Load Tester is an advance product and the only one of its kind that can test against ADF applications, there are some areas where the product falls short

Page 21: Online Performance Testing With OATS

21

Lessons learnt

Several lessons learnt using OATS

• A good understanding of the application and what it does is needed before diving into developing of scripts. This understanding is needed to know what is possible/practical in OLT as well as what can be parameterised or not

• Ensure the functional team is intimately involved with the development of the test scenarios/scripts and that they understand what is practical in terms of performance testing using OLT

• Think times are critical to generating load, ensure the think times specified are as realistic as possible

• As much as possible, the RPAS code should be stable to ensure that there are no changes to the environment as this may require re recording of scripts

• Any updates to the Fusion environment i.e. Fusion Client can result in working OLT scripts failing after the upgrade, this may require re recording of scripts

• Make use of OpenScript wherever possible to set up users by creating simple scripts to perform basic set up activities in the test environment

• Setting up and preparing for testing takes up the majority of performance testing time, allocate enough time for these activities. This talks specifically to the point below

• Once all data has been set up in RPAS e.g. users created, passwords reset, views created, a copy of the domain must be taken prior to testing. In the event that the domain is corrupted there is no need to re run all the set up activities

Page 22: Online Performance Testing With OATS

22

References

• Oracle Application Testing Suite Getting Started Guidehttp://download.oracle.com/otn/nt/apptesting/oats-docs-9.31.0036.zip

• Oracle Load Testing Load Testing User’s Guidehttp://download.oracle.com/otn/nt/apptesting/oats-docs-9.31.0036.zip

• Oracle ADF overviewhttp://www.oracle.com/technetwork/developer-tools/adf/adf-11-overview-1-129504.pdf

The following documents were used as reference material when compiling this document