Stress Testing Web Intelligence and Crystal Reports

7/25/2019 Stress Testing Web Intelligence and Crystal Reports

http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 1/15

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com

© 2010 SAP AG 1

Stress Testing Web Intelligenceand Crystal Reports

Applies to:

SAP BusinessObjects XI 3.1. For more in formation, visit the Business Objects homepage.

Summary

This paper discusses stress testing an SAP BusinessObjects Enterprise system to compare the efficiency of

Web Intelligence and Crystal Reports. For both Web Intelligence and Crystal Reports it looks at howdocument size and number of users can affect the impact on system resources. The results demonstrate thatCrystal Reports is a better fit for deployments where there are a large number of users who are viewi ng and

refreshing pre-built reports.

Author: Al Gulland

Company: gulland

Created on: 8 November 2010

Author Bio

Al Gulland is a senior BI consultant for IBM Global Services and has been active in Business Intelligence forover 15 years, specializing in SAP BusinessObjects and data warehouse solutions. Al maintains a blog atwww.gulland.com and has posted many articles on SAP BusinessObjects.

https://www.sdn.sap.com/irj/boc






Stress Testing Web Intelligence and Crystal Reports


© 2010 SAP AG 2

Table of Contents

Introduction.......................... ....................................... ................................................ ............................ 3

Test Environment .......................................................................... ................................................ .......... 3

Test Documents Used ............................................................................... ........................................... 3

Stress Test Approach ................................... .................................................. ......................................... 4

Performance Monitoring .............................................. ................................................. ........................ 4

Test Results................................................................ ................................................ ............................ 5

Conclusion.................. .............................................. ......................................... ..................................... 5

Appendix 1: Server Configuration ..... ............ ......... ...... ... ......... ......... ......... ... ......... ......... ......... ...... ... ........ 6

Web Intelligence Server Configuration ....................................... .................................................. .......... 6

Crystal Reports Server Configuration .......................................... ................................................ .......... 7

Properties for Crystal Reports Processing Server ................................................................................................................... 7

Appendix 2: Detailed Results ... ......... ......... ....................... ......... ......... ... ...... ......... ......... ......... ... ......... ..... 7

Test: Web Intelligence Small Document .............................................. ............................................... ... 7

Test: Web Intelligence Medium Document ............................... ........................................ ...................... 9

Test: Crystal Report Small Document ................................................... ............................................. .. 10

Test: Crystal Report Medium Document .................................... ................................................ .......... 12

Related Content ............................................................................ ................................................ ........ 14

Boest………………………………………………………………………………………………………………….14

Linear Regression Analysis ......................................... ............................................... ........................ 14

Other Related Topics ................................ ................................................ ......................................... 14

Disclaimer and Liability Notice................................ .............................................. .................................. 15





© 2010 SAP AG 3

Introduction

The overall objective of this exercise was to compare Web Intelligence to Crystal Reports under identical

stress test conditions. Here the stress test was to have a set number of users viewing and refreshing adocument while at the same time measure the memory and CPU consumption on the SAP BusinessObjectsEnterprise (BOE) server. The test was executed for both a Web Intelligence document and a Crystal Report

of the same size. In both cases the number of users was increased and measures of memory and CPU

usage were taken. The Boest stress testing application was used to execute the stress tests and to analyzethe results.

This article reviews the testing environment, how the stress tests were executed, discussion of the testresults and finally conclusions.

Test Environment

The same BOE server was used for both the Web Intelligence and Crystal Reports stress tests. As the testswere a comparison of the two report formats it wasn't necessary to have a high end capacity server and so a

virtual machine was used to host the server. The environment specification is as follows.

OS Windows 2003 Server

CPU 2 cores from a Intel Core i5 750 of 2.67 Ghz

RAM 2 GB

Virtualisation Technology VM Player 3.1

SAP BOE Version XI 3.1 sp3

Database SQL Server Express 2005

Boest was installed on a Windows 7 desktop that had access to the BOE Server. The same desktop washost to the virtual machine running the BOE server.

Test Documents UsedThe test documents used for both Web Intelligence and Crystal Reports were created to be as much aspossible of the same size and complexity in order to ensure a valid comparison test. In this case the same

SQL statement was used in both the Web Intelligence document and Crystal Report. This ensured that notonly the database execution of the SQL statement would be the same but that the number of rows in thereport would be the same. In addition the display and formatting of the data in the reports was the same for

both document types which was a simple table of data.

In both cases a small and a medium sized document was created where the difference in the document sizeis purely based on the number of rows returned by the SQL statement.

The SQL used by all reports is,

SELECT

ID, Store, Product, Purchase_Date, Quantity, On_SaleFROM

SalesDataMartDemo.dbo.TBL_LARGE2

WHERE

ID <= 10000

ORDER BY

Store, Product, Purchase_Date





© 2010 SAP AG 4

The only difference between small and medium reports is the value used in the condition. For small reportsthe condition is ID <= 10000 and for medium it is ID <= 100000.

The following reports are used in these tests.

Name Type Size on disk (kb) Number of Rows

PerfTestWebi_Small Web Intelligence 224 9,991

PerfTestWebi_Medium Web Intelligence 1,921 99,991PerfTestCrystal_Small Crys tal Report 168 9,991

PerfTestCrystal_Medium Crys tal Report 1,214 99,991

For comparison this is same amount of data saved as CSV where the file was generated from a SQL Serverexport.

Name Type Size (kb) Number of rows

Small CSV 522 9,991

Medium CSV 4,266 99,991

Stress Test Approach

Boest is an application designed to stress test an SAP BOE system by simulating multiple parallel users allviewing and refreshing reports. It was used here to stress test both Web Intelligence and Crystal Reportswhere the method used was the same in both cases and is described below.

The test involved logging on an initial set of 5 users and once they have all logged on they thensimultaneously view and refresh a test document. Once complete the users were all logged off. While thistest is being executed Windows Performance Monitor was used to track the memory and CPU being used by

the Web Intelligence or Crystal Report processes on the server. For the test the maximum memory reachedis noted along with maximum percentage CPU reached. As well as maximum CPU the average percentageCPU was also measured during the test in order to obtain a measure of overall load.

The test of five users was repeated five times, and each time the memory and CPU usage was logged. Thepurpose of this was to obtain an average of the memory and CPU so as to eliminate any unexpected spikeswhich could be caused by other processes on the system.

After the test of five users was complete the test ing was repeated for 10 users and this was also repeatedfive times to take an average. The test was repeated again for 15 users and then 20 and so on up to 50users in total.

This process was performed for both the small and medium Web Intelligence documents and then again forthe small and medium Crystal Reports.

Performance Monitoring

Windows Performance Monitor was used to measure the amount of memory and percentage CPU used bythe Web Intelligence and Crystal Reports processes. The server process that is used by Web Intelligence isthe WIServerProcess and the counters tracked were "% Processor Time" for Percentage CPU and

"Working Set" for memory.

For Crystal Reports the main Crystal Report process crproc will instantiate a further 5 processes acrosswhich the processing is spread. For each of these processes (identified by crproc#1, crproc#2 etc) the "%

Processor Time" and "Working Set" were measured from which the results were aggregated together.





© 2010 SAP AG 5

Test Results

The results were analyzed by first taking an average across the 5 repeated tests. This average was thenplotted against the number of users to produce a chart of memory usage by number of users and also a

chart of CPU usage by number of users (see appendix). The chart of number of users versus memory was agood fit for a straight line and so in each case linear regression analysis was used to determine the best fitfor a straight line through these points. This straight line can then be used to determine a relation between

number of users and memory which can then be extrapolated to find a maximum load for the system.

The table below summarizes the results. The initial memory is the intersect of the straight line with the

vertical axis and is equivalent to the amount of RAM consumed prior to start of test. The maximum memoryis the maximum memory reached during the test and this was always when the full load of 50 users wasreached. The next column displays the gradient of the straight line and this is the increase in RAM per user.The final column is the maximum percentage CPU reached during the test, note this is out of 200% as the

system was configured with two CPUs.

Test Initial Memory(mb)

MaximumMemory (mb)

Increase in RAMper user

Maximum CPU

Web Intelligence Small Document 63.6 168.7 2.17 190.31

Web Intelligence Medium Document 44.2 502.5 8.31 195.02

Crystal Report Small Document 257.5 292.5 0.78 66.87

Crystal Report Medium Document 255.5 291.9 0.81 78.75

From this it can be seen that although Web Intelligence starts from a lower base the increase in memory peruser is much greater than Crystal Reports and for the medium doc about 0.5GB of RAM is reached for 50

users. For Crystal Reports 50 users viewing a report of equivalent size consumes 291 MB of RAM.

It should also be noted that Web Intelligence places more demand on CPU usage indicating moreprocessing required to render the displayed report.

Conclusion

The above analysis clearly shows that there is, as you would expect, a direct correlation between increasingusers and an increase in memory for Web Intelligence and Crystal Report types. However although Crystal

Reports has a higher initial memory the rate of increase in memory is much less than that of WebIntelligence such that for a large number of users Crystal Reports uses less memory than Web Intelligence.

CPU usage is also different and by inspection of the graphs it can be seen that Crystal Reports is less CPU

intensive than Web Intelligence.

From this it can be concluded that Crystal Reports is a better solution for deployments with a high number ofusers who need to view and refresh prebuilt reports and that Web Intelligence is more efficient for a low

number of users.

Does this mean then that Web Intelligence should never be used and that Crystal Reports is best used in allscenarios? No. These are two different products that meet two similar but different requirements. Web

Intelligence is designed for a business user to create new reports and perform some basic data analysis.Crystal Reports on the other hand has been designed for large volume report generation and distribution.

This exercise was designed to compare Crystal Reports to Web Intelligence in a tightly controlled

environment. The above values are not expected in a real world scenario and the results should not be

interpreted as a realistic measure of demand in a production system.

Report size has a significant impact on system demand and in a production environment it is expected that

there will be a wide range of reports of different sizes and complexity. Furthermore, user demand will not beregular and there will be peak times when users are accessing the system.

These factors and more will affect the maximum load that a system can comfortably handle and a more

detailed capacity planning exercise should be undertaken when designing a deployment.





© 2010 SAP AG 6

Appendix 1: Server Configuration

For both Crystal Reports and Web Intelligence caching was reduced so that memory usage can bemeasured per user per document. Full server settings are as follows.

Web Intelligence Server Configuration

The following are the settings used for the Web Intelligence Processing Server.

Property Setting

Document cache cleanup interval (minutes): 120

Binary Stream Maximum Size (MB): 50

Cache Timeout (minutes): 4370

Memory Maximum Threshold (MB): 1800

Idle Document Timeout (seconds): 100

Server Polling Interval (seconds): 100

Universe Cache Maximum Size (Universes): 20

Disable Cache Sharing: Off

Images Directory:

Maximum Document Cache Size (KB): 1000000

Output Cache Directory:

Maximum Documents per User: 5

Allow Document Map Maximum Size Errors: On

Maximum Documents Before Recycling: 50

Maximum Connections: 100

Idle Connection Timeout (minutes): 5

Maximum List Of Values Size (entries): 50000

Enable List Of Values Cache: On

Enable Real-time Cache: Off

Maximum document cache reduction space (%): 70.00%

Maximum Documents in Cache: 1

Memory Upper Threshold (MB): 1500

Enable Memory Analysis Off

Enable Document Cache On

List Of Values Batch Size (entries): 1000

Maximum Custom Sort Size (entries): 100

Memory Lower Threshold (MB): 1000

Timeout Before Recycling (seconds): 1200

Maximum Character Stream Size (MB): 5





© 2010 SAP AG 7

Crystal Reports Server Configuration

Properties for Crystal Reports Processing Server

Property Setting

IdleJobTimeout 1

Database Records Read When Preview ing or Refreshing 0

Maximum Lifetime Jobs Per Child: 1000

Viewer Refresh Always Yields Current Data On

Share Report Data Betw een Clients On

Idle Connection Timeout (minutes): 20

Maximum Concurrent Jobs (0 for automatic): 0

Oldest On-Demand Data Given to Clients (seconds): 0

Maximum Number of Prestarted Children 1

Allow Report Jobs to Stay Connected to the Database untilthe Report Job is Closed

Off

Appendix 2: Detailed Results

Below are full results for each test. Each test of a set number of users is repeated five times in order toobtain the averages. The average CPU is the average of the average percentage CPU during the fiverepeated test runs and the average maximum CPU is the average of the maximum percentage CPU

obtained during each test. The average memory is the average of the maximum memory reached duringeach test.

Test: Web Intelligence Small Document

The test results for users viewing and refreshing the small Web Intelligence document are displayed in thetable and graphs below. The increase in memory is linearly correlated to the increase in number ofsimultaneous users where each user increases memory usage by 2.174MB on average.

Percentage CPU used by the Web Intelligence process quickly reaches 180% after 20 simultaneous userswhich is pretty much full load with the remaining percentage used by other system processes. The averagepercentage CPU used slowly increases indicating that more CPU cycles are required to handle the increased

user demand.

Number Users Average CPU Average Max CPU Average Memory

5 14.08 88.44 74.34

10 24.28 158.12 86.30

15 30.51 171.25 96.21

20 35.20 183.75 106.89

25 39.36 180.31 114.55

30 43.24 188.12 130.41

35 45.78 185.00 137.69

40 48.47 188.43 155.01

45 51.28 190.31 163.68

50 52.51 189.06 168.73





© 2010 SAP AG 8

Linear regression analysis of number of users versus average memory gives the following metrics,

Gradient: 2.174

Intersect: 63.598

Correlation Coefficient: 0.997

Graph of memory consumption vs. number of users for the small Web Intelligence document

Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Web

Intelligence document





© 2010 SAP AG 9

Test: Web Intelligence Medium Document

The test results for users viewing and refreshing the small Web Intelligence document are displayed in thetable and graphs below. In this test the gradient is higher which is due to the fact that a larger document is

being used and so each additional user viewing a larger document has a higher demand on system memory.

A second difference is that the percentage CPU usage is also much more than with the smalle r document inboth maximum and average percentage CPU used.

Number Users Average CPU Average Max CPU Average Memory5 48.96 183.74 101.54

10 77.52 190.62 138.43

15 90.11 191.24 163.31

20 98.9 190.96 207.18

25 104.56 193.43 248.38

30 110.52 192.49 273.9

35 115.17 195.02 325.73

40 114.25 193.75 355.32

45 115.7 192.81 410.15

50 118.1 194.68 502.53

Linear regression metrics,

Gradient: 8.306

Intersect: 44.236


Graph of memory consumption vs. number of users for the medium Web Intelligence document





© 2010 SAP AG 10

Graph of average CPU (lower line) and maximum CPU vs. number of users for the medium Web

Intelligence document

Test: Crystal Report Small Document

The test results for users viewing and refreshing the small Crystal Report are displayed in the table andgraphs below. Here the increase in memory per increase in user is much less than that of the equivalentreport in the Web Intelligence tests. The increase per user is 0.780MB compared to 2.174MB for Web

Intelligence. The initial memory usage is higher than the maximum memory reached in the Web Intelligencetest using the small document.

CPU usage is also significantly lower than the equivalent test in Web Intelligence where the maximum

percentage CPU reaches a peak of about 70%.


5 3.16 17.19 257.43

10 4.04 25 264.48

15 5.71 35.94 270.04

20 6.66 46.56 274.37

25 8.33 56.25 279.28

30 8.88 64.37 283.61

35 11.24 66.87 286.48

40 8.41 55.62 289.37

45 10.88 67.81 291.71

50 10.94 63.13 292.53


Gradient: 0.780

Intersect: 257.469






© 2010 SAP AG 11

Graph of memory consumption vs. number of users for the small Crystal Report

Graph of average CPU (lower line) and maximum CPU vs. number of users for the small CrystalReport





© 2010 SAP AG 12

Test: Crystal Report Medium Document

The test results for users viewing and refreshing the medium Crystal Report are displayed in the table andgraphs below.


5 2.67 14.37 255.24

10 2.98 19.69 263.06

15 4.56 28.75 268.57

20 6.13 34.06 273.48

25 8.56 57.19 277.41

30 8.48 52.18 283.42

35 9.55 55.31 286.22

40 10.51 60.94 289.05

45 10.28 71.56 290.8

50 12.81 78.75 291.94


Gradient: 0.814

Intersect: 255.547


Graph of memory consumption vs. number of users for the small Crystal Report





© 2010 SAP AG 13

Graph of average CPU (lower line) and maximum CPU vs. number of users for the small CrystalReport





© 2010 SAP AG 14

Related Content

Boest

Boest is a Java application built specifically for stress testing the execution of Crystal Reports and Web

Intelligence documents in an SAP BusinessObjects Enterprise deployment. Further details of the applicationand download can be found at Sourceforge.net.

Linear Regression Analysis

The linear regression analysis was calculated using Web intelligence using the method discussed in thearticle Linear Regression Analysis and Web Intelligence.

Other Related Topics

BusinessObjects Enterprise XI Release 2 Sizing Recommendations Guide. Although this was written for XIr2much of the content remains valid for XI3.

Selecting the Right Reporting Tool – A comparison of Web Intelligence and Crystal Reports

Business Objects Community: Crystal Reports

Business Objects Community: Webi

For more information, visit the Business Objects homepage

http://sourceforge.net/projects/boestresstester/



http://www.gulland.com/wp/?p=534



http://www.sdn.sap.com/irj/boc/bi-solution-architecture?rid=/library/uuid/e01588cd-c73e-2b10-018c-fe083bc6e05e


http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/d0674568-e242-2c10-02ad-ea55edb44c43?QuickLink=index&overridelayout=true


http://www.sdn.sap.com/irj/boc/crystalreports


https://www.sdn.sap.com/irj/boc/webi















SAP COMMUNITY NETWORK SDN - sdn sap com | BPX - bpx sap com | BOC - bocsap com | UAC - uac sapcom

Disclaimer and Liability Notice

This document may discuss sample coding or other information that does not include SAP official interfaces and therefore is notsupported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade.

SAP w ill not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document,and anyone using these methods does so at his/her own risk.

SAP offers no guarantees and assumes no responsibility or liability of any type w ith respect to the content of this technical article orcode sample, including any liability resulting f rom incompatibility between the content within this document and the materials andservices offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable w ith respect to the content of thisdocument.