- Oracle...• Sign-on to Apps with System Administrator responsibility • Navigate to Concurrent -> Program -> Define • Query the concurrent program

<Insert Picture Here>

Tuning the E-Business Suite

Ahmed AlomariApplications Performance GroupSession ID: S281393

Agenda

• Architecture

• Application Services Tier• Forms

• Reports

• Concurrent Manager

• Apache

• Jserv/JVM

• Portal

• Discoverer

• Network Performance

• Data Server Tier

• Application Tuning

• 11i10 Performance Improvements

• Q & A


Tuning the Applications Tier

Application Services Tier

• Forms• Deploy with socket mode for internal users:

• connectMode=socket (appsweb.cfg)

• Enable Forms Dead Client Detection

• Value specified in minutes

• Terminates f60webmx processes for dead clients.

• FORMS60_TIMEOUT=10

• Enable Forms Abnormal Termination Handler

• Do not set FORMS60_CATCHTERM

• If you are running 11.5.10 and Jinitiator 1.3, you should apply patch 4122271.

• Without this patch, a new Jinitiator session is open for every form.


• Forms

• Upgrade to the latest Forms patch set (patch set 18)

• Refer to MetaLink document 125767.1 for Forms patchset upgrades.

• Forms – Cancel Query

• Should not be enabled unless you are on Forms patchset 14 (Forms version 6.0.8.23.x or higher).

• Cancel Query increases middle-tier CPU as well as DB CPU.

• Refer to MetaLink Note 138159.1 for instructions on how to enable and tune Cancel query related parameters.

• To Disable Cancel Query

• Set the Profile “FND: Enable Cancel Query” to No.


• Forms

• Ensure users are optimally utilizing the professional Forms interfaces.

• Avoid Blind queries

• Provide selective criteria in Find windows and LOVs.

• Avoid opening and closing forms across transactions.

• Users which need to switch responsibility in order to complete a business transaction should instead keep the relevant Forms open.


• Forms (Tracing)

• If the form is slow, or a specific flow in the Form is slow, first generate a complete SQL trace using the Trace menu in Help-Diagnostics.

• If the SQL trace does not account for the majority of the response time experienced by the user, then generate a Forms Runtime Diagnostics (FRD) Trace.


• Forms (Tracing – SQL Trace)


• Forms (Tracing – FRD Trace)

• Logon to Applications 11i with the following parameters appended to the URL: play=&record=collect&log=/tmp/oeform.trc

• Navigate through the flows which you would like to trace.

• Exit from Oracle Applications completely.

• Review the trace file from the application server (/tmp/oeform.trc).


• Reports

• Ensure users provide selective parameters to the standard Reports.

• Review the Report output:

• Number of rows returned to the report.

• Number of report pages.

• SQL Trace can be generated at Request Submission Time (SRS form – Debug Options) or by setting the “Enable Trace” flag in the Concurrent Program Definition Form.


• Reports (Tracing)• If the SQL trace for the report does not account for the

elapsed time of the request, then generate a Reports trace by appending tracing parameters the to concurrent program command line.

• The following are the trace options:

• TRACE_ALL : log all possible trace information in the trace file. (DEFAULT)

• TRACE_APP : log trace information on all the report objects in the trace file.

• TRACE_BRK : list breakpoints in the trace file.

• TRACE_ERR : list error messages and warnings in the trace file.

• TRACE_PLS : log trace information on all the PL/SQL objects in the trace file.

• TRACE_PRF : log performance statistics in the trace file.

• TRACE_SQL : log trace information on all the SQL in the trace file.

• TRACE_TMS : enter a timestamp for each entry in the trace file.


• Reports (Tracing)• Enabling Reports Trace (pre 11i10)

• Sign-on to Apps with System Administrator responsibility

• Navigate to Concurrent -> Program -> Define

• Query the concurrent program (report) which you want to trace.

• Locate the “Options” field and enter the following values :

• TRACEOPTS=<trace option> TRACEFILE=<path/file>

• Save the concurrent program

• Run the report and a trace file will be generated.

• Remember to clear the trace options after running the report.

• Enabling Reports Trace (11i10)

• Use the Debug Options page at request submission.


• Reports (Trace Output)

Report: /u02/appl/ar/11.5.0/reports/US/ARXAGM.rdf

Logged onto server:

Username:

. . . . . . . .

11:38:59 APP ( Computed Column C_Risk_Option_Meaning

11:38:59 PLS . ( Function: cf_claim_option_meaningformula

11:38:59 PLS . ) Function: cf_accounting_flexfieldformula

11:38:59 PLS . ) Function: c_bal_segment_lpromptformula

11:38:59 PLS . ( Function: set_meaningsformula

11:38:59 PLS . ) Function: set_meaningsformula

11:38:59 PLS . ( Function: rp_ftr_lblformula. . . . . . . .

. . . . . . .

+-------------------------------------+

| Report Builder Profiler statistics |

+-------------------------------------+

TOTAL ELAPSED Time: 2243.04 seconds

Reports Time: 1196.62 seconds (53.34% of TOTAL)

ORACLE Time: 1046.41 seconds (46.65% of TOTAL)

UPI: 1046.00 seconds

SQL: 0.40 seconds

TOTAL CPU Time used by process: 904.71 seconds



• Avoid enabling an excessive number of standard or specialized managers.

• Use specialization rules and work shifts to bind specific jobs to specific time windows.

• Helps avoid scheduling resource intensive batch requests during peak activity.

• For jobs which spawn parallel workers such as Auto Invoice or Payroll, set the sleep time of the Conflict Resolution Manager (CRM) to null (i.e. 10 seconds). The default value is 60 seconds.


• Concurrent Manager• Transaction Managers

• Used for Inventory Transactions as well as other synchronous online processing.

• Ensure enough transaction managers exist to service the request load.

• Set the profile “Concurrent:Wait for Available TM” to 1 (second).

• Set “TP:INV Transaction processing mode” to “On-line processing” for small inventory requests from the UI.

• Set the sleep time on the transaction managers to a high number (e.g. 10 minutes).

• Avoids constant polls to check for shutdown requests.

• Transaction Managers can also be traced.

• Pipes be gone!

• Apply RUP3 (4334965)

• Set the profile “Concurrent:TM Transport Type”

• Avoids the need to run a transaction manager instance per database instance for RAC deployments.



• Enabling Tracing

• Program Level

• Enable the trace option in the concurrent program definition form.

• Request Level (new in 11i10)

• Use the Debug Options button in the Request Submission Form

• The profile “Concurrent: Allow Debugging” should be set to “Yes.”


• Concurrent Manager• Enabling Trace at the Program Level


• Concurrent Manager• Enabling Trace at the Request Level


• Concurrent Manager• Enabling Trace at the Request Level


• Apache

• On UNIX, Apache is process based (httpd), and mods such as mod PL/SQL run within the process address space of the httpd processes.

• On Windows, Apache is multi-threaded (Apache.exe).

• Tune the number of processes and number of clients (httpds.conf):

• MinSpareServers 5

• MaxSpareServers 10

• StartServers 5

• MaxClients 512


• Apache

• Minimize levels of logging (httpds.conf):

• LogLevel warn

• SSLLogLevel warn

• Enable Caching of non-HTML resources including images, style sheets, and Java script.

• Caches content in the browser cache.

• Reduces network round-trips (non SSL)

• Reduces network bandwidth utilization (SSL)

• Included in AutoConfig templates in 11.5.8 (or higher).


• Apache

• Enable Caching of non-HTML resources (httpd.conf or apps.conf)

<Directory "<physical_path_corresponding_to_the_alias_/OA_HTML/>">

#enable the generation of the Expires header for files under /OA_HTML/

ExpiresActive On

#expire images one month after last client access

ExpiresByType image/gif "access plus 1 month"

#expire stylesheets one week after the last client access

ExpiresByType text/css "access plus 1 weeks"

#expire javascript libraries one day after the last client access

ExpiresByType text/javascript "access plus 1 days“

ExpiresByType application/x-javascript "access plus 1 day"

</Directory>


• Apache

• Review Apache Access Log file to ensure images are being satisfied from the browser cache.

• Http code 200 (request for document)

• Http code 304 (request for time stamp)

130.35.127.106 - - [21/Oct/2006:14:00:21 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 200 821

130.35.127.106 - - [21/Oct/2006:14:38:23 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:38:23 -0700] "GET /OA_MEDIA/FNDWATHS.gif HTTP/1.1" 200 190

130.35.127.106 - - [21/Oct/2006:14:38:23 -0700] "GET /OA_MEDIA/FNDREDPT.gif HTTP/1.1" 200 70

130.35.127.106 - - [21/Oct/2006:14:39:23 -0700] "GET

/OA_HTML/OA.jsp?page=/oracle/apps/icx/por/rcv/pages/ReceivingHomePage&OAHP=ICXP

OR_MENU&OASF=ICXPOR_RCV_HOME_PAGE&dbc=ap107fam_ipdev11i&language_code=US&transactionid=70A5819F04C0F411

HTTP/1.1" 200 14

130.35.127.106 - - [21/Oct/2006:14:39:34 -0700] "GET /servlets/PoolMonitor?jvm HTTP/1.1" 200 7954

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_HTML/US/ICXINDEX_ipdev11i.htm HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_MEDIA/logo.gif HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_MEDIA/appslogo.gif HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRL.gif HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -

130.35.127.106 - - [21/Oct/2006:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRR.gif HTTP/1.1" 304 -


• Apache Mod PL/SQL

• Configure a dedicated mod PL/SQL Listener

• Improves performance and scalability

• Significantly reduces overall number of sessions/connections.

• Reduces latency of web requests.

• Improves cursor sharing.

• Documented in “Oracle 9i Application Server Using the PL/SQL Gateway Release 1 (v1.0.2.2)”

• http://download-west.oracle.com/docs/cd/A97335_01/apps.102/a90099/apptroub.htm#634165


• Apache Mod PL/SQL (dedicated listener)

1. For the main Listener running on Port 7000, edit the file

$IAS_HOME/Apache/modplsql/cfg/plsql.conf as follows:

Disable the mod PL/SQL service from the main listener by commenting out the

lines between the two Location parameters as follows:

# <Location /pls>

# SetHandler pls_handler

# Order deny,allow

# Allow from all

# </Location>

Comment out the following line as follows:

# LoadModule plsql_module /d1/ias/Apache/modplsql/bin/modplsql.so


• Apache Mod PL/SQL (dedicated listener)

2. Configure the main listener to forward all mod_plsql requests to the

dedicated mod PL/SQL listener by adding the following line:

ProxyPass /pls/ http://sechost.us.oracle.com:8888/pls/

For the dedicated mod PL/SQL Listener running on Port 8888, configure each

DAD to override the default CGI environment variables in order to allow

redirects. Edit the file $IAS_HOME/Apache/modplsql/cfg/wdbsvr.app and add

the following line for each DAD:

cgi_env_list=SERVER_NAME=mainhost.us.oracle.com,SERVER_PORT=7000,HOST=mainhos

t.us.oracle.com:7000


• Apache Jserv / JVM• Minimize Jserv logging

• ApJServLogLevel warn

• log=false

• log.channel.warning=true

• log.file=/d1/ias/Apache/Jserv/logs/jserv_7000.log

• Review error log files for exceptions.

• Disable auto reload in production environments:

• autoreload.classes=false

• autoreload.file=false

• Use Jserv Auto Load Balancing

• Configure Multiple Zones.

• Provides higher availability and improves scalability.


• Apache Jserv / JVM

• Upgrade to the latest JDK

• JDK 1.3 has been in end of life status since October 2004.

11.5.10 CU1 or

higher

11.5.9 or

higher

Applications Version

304099.1JDK 5.0 Update 91.5.0

246105.11.4.2_12 1.4.2

MetaLink NoteLatest UpdateJDK Release



• Do not disable hotspot or the JIT

• Do not set -DCOMPILER=NONE or –Xint

• Enable verbose GC in order to optimally tune the heap sizes based on the GC traffic.

• Minor Collections

• Major Collections

• Elapsed Time of GC



• Enable Verbose GC Output (pre 11.5.10)

• Overload wrapper.bin with a shell wrapper in order to redirect stdout to a file.

Existing:

wrapper.bin=/apps/jdk/jdk1.4.2/bin/java

Change To:

Wrapper.bin=/apps/scripts/java.sh

java.sh:

=========

/apps/jdk/jdk1.4.2/bin/java -verbosegc $* >> /apps/logs/java.log



• Verbose GC Output (11.5.10 or higher)

• In 11.5.10, JVM logs re-directed to the directory $APACHE_TOP/Apache/Jserv/logs/jvm

• File name format is

• <coregroupname>.<core#>.stdout | stderr

DiscoGroup.0.stderr

DiscoGroup.0.stdout

FormsGroup.0.stderr

FormsGroup.0.stdout

OACoreGroup.0.stderr

OACoreGroup.0.stdout

XmlSvcsGrp.0.stderr

XmlSvcsGrp.0.stdout



• Verbose GC Data

• Can use the GC Timestamp and GC Details to print additional information (JDK 1.4 or higher):

• -XX:+PrintGCTimeStamps

• -XX:+PrintGCDetails

0.000: [GC [PSYoungGen: 43008K->3698K(50176K)] 43008K->3698K(160768K), 0.0770625 secs]













• Utilize the verbose GC output to tune the JVM heaps (-Xmx and –Xms) accordingly.

• Review the frequency of collections, especially major collections (i.e. Full GC).

• Start with:

• -Xms512M and –Xmx512M

• -XX:NewRatio=2

• Above settings have been incorporated in 11i10.


• Java Memory and Connection Leaks

• Apply ATG Rollup 4 (Bug 4676589)

• Refer to MetaLink note 365228.1

• As of JDK 1.5 (Update 7) and 1.4.2_12, a new option was added to generate a heap dump.

• Add –XX:+HeapDumpOnOutOfMemoryError –XX:HeapDumpPath=/tmp to the JVM startup options

• If an OutOfMemoryError occurs, a heap dump will be generated in the path specified by –XX:HeapDumpPath

• The heap dump file can be read by the Heap Analysis Tool (HAT), or jhat (in JDK 6), or a profiler that supports the binary heap dump format (e.g. YourKit Java Profiler)


• Apache Jserv / JVM• Use Garbage Collection Tools to display the data graphically

• HPjtune

• -verbose:gc -Xloggc:/logs/gc.out

• http://www.hp.com/products1/unix/java/java2/hpjtune/

• visualgc (jvmstat)

• http://java.sun.com/performance/jvmstat/

D E M O N S T R A T I O N

HPjtune Tool


• JDBC Connections

• Process Identification (V$SESSION)

• Specify the property –DCLIENT_PROCESSID in the JVM startup shell script.

• Set automatically in 11i10.

Existing:

wrapper.bin=/apps/jdk1.4.2/bin/java

Change To:

Wrapper.bin=/apps/scripts/java.sh

java.sh:

=========

#!/bin/sh

/apps/jdk1.4.2/bin/java -verbosegc -DCLIENT_PROCESSID=$$ $* >> /d2/logs/java.log


• JDBC Connections• Process Identification (V$SESSION)

• Allows you to map the JDBC session from v$session to a particular JVM process.

• Set automatically in 11i10

SID MACHINE PROCESS MODULE LOGON

---- ------------------------- ------------ ------------------ -----------------

41 aptier1.us.oracle.com 28806 JDBC Thin Client 10/22/06 17:26:43




. . . . . . .

. . . . . . .

aptier1{apps_a}-> ps -ef | grep 28806

apps_a 28806 28561 0 17:26:39 pts/20 0:00 /bin/sh ./java.sh

apps_a 28807 28806 53 17:26:40 pts/20 8:55 /jdk1.4.2/bin/../bin/sparc/native_threads/java


• Apache Jserv / JVM - Upgrade to JDK 5.0• Refer to MetaLink note 304099.1 upgrade instructions.

• ~15% performance improvement.

• Parallel Collector is used

• Adaptive Heap Sizing

• Improved scalability

• New Manageability framework and monitoring tools

• Jconsole• http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html

• jps

• jstat

• visualgc

• Download from http://java.sun.com/performance/jvmstat/


• Apache Jserv / JVM• JDK 1.5 provides new JVM monitoring tools (visualgc)


visualGC Tool


jconsole Tool


• Java Server Pages (JSPs)

• Precompile the JSPs to avoid dynamic compilation.

• Users experience poor performance for the initial page loads.

• Potential deadlocks if multiple users attempt to compile the same JSP.

• Potential JVM death due to OutOfMemoryException during concurrent compilation.

• MetaLink Document 215268.1 provides the instructions and the patch reference for an automated script to perform the precompilation.

• ojspCompile Script

• With the latest version of ojspCompile, it takes 15 minutes to compile all the JSPs.


• Java Server Pages (JSPs)

• Use a separate JVM to perform the JSP compilation.

• Automatically set in 11i10.

root.properties:

servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,

unsafe_reload=false,

page_repository_root=/appl_top/115/common/html/jsp/pagecache,

alias_translation=true,developer_mode=false,

javaccmd=/usr/jdk142/bin/javac,send_error=true


• Portal Tuning

• Tune the number of content fetcher threads for PPE (poolSize). Default is 25.

• Adjust the fetch request timeout (requesttime).

• Adjust the fetch connection request timeout (stall).

zone.properties:

servlet.page.initArgs=poolSize=50

servlet.page.initArgs=requesttime=200

servlet.page.initArgs=stall=100


• Portal Tuning

• Ensure that the mod PL/SQL cache is enabled and sized appropriately:

$APACHE_TOP/modplsql/cfg/cache.cfg:

[PLSQL Cache]

enabled=yes

total_size=100000000

cleanup_size=75000000

cleanup_interval=86400

;

[Cookie Cache]

enabled=yes

total_size=25000000

cleanup_size=15000000

cleanup_interval=86400

max_size = 0


• Discoverer 4i Viewer Tuning

• Disable Query Prediction by setting QPPEnable=0 in pref.txt.

• Set ObjectsAlwaysAccessible=1 to avoid extra workbook SQL validation, and related dictionary SQL.

• Set the Query Governor option “Limit retrieved query data to:” to 100 rows.

• Set the option “After opening a worksheet:” to “Don’t run query.”

• Ensure Custom workbooks define mandatory parameters.


• Upgrade to Discoverer 10g (10.1.2) • Refer to MetaLink Note 313418.1

• Discoverer 10g Tuning• Enable SQL Flattening Feature

• SQLFlatten=1

• Significantly reduces sharable memory and parse time of SQL statements generated by Discoverer

• Enable Item trimming

• SQLItemTrim=1

• Eliminates unused items from the SQL statement

• Enable Join trimming

• SQLJoinTrim=1

• Eliminates joins which are not needed based on the selection.


Tuning the Network

Network Performance

• Network performance is often neglected.

• Networks with poor latency or low bandwidth impacts user response time.

• Can also impact concurrent programs if the network between the application tier and data server tier is not optimal.

• A direct switch should be used between the application tier and data server tier

• Latency should ideally be ~1-2ms per round-trip.

Network Performance

• Use the ping command to determine the latency of requests. Vary the packet size to determine the average latency. Do not use the default ping packet size of 32 bytes.

D:\>ping aptierhost.us.oracle.com -l 2048

Pinging aptierhost.us.oracle.com [139.185.128.27] with 2048 bytes of data:

Reply from 139.185.128.27: bytes=2048 time=371ms TTL=252




D:\>ping aptierhost.us.oracle.com -l 512

Pinging aptierhost.us.oracle.com [139.185.128.27] with 512 bytes of data:





Network Performance

• Use the ping command to determine the latency between the application server and the data server. Use a packet size of 2K since this the default size for SQL*Net traffic.

aptierhost{applmgr}-2-> ping -s dbtierhost 2048

PING dbtierhost.us.oracle.com: 2048 data bytes

2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=0. time=1. ms





2056 bytes from dbtierhost.us.oracle.com (144.25.76.250): icmp_seq=5. time=1. Ms

---- dbtierhost.us.oracle.com PING Statistics----

6 packets transmitted, 6 packets received, 0% packet loss

round-trip (ms) min/avg/max = 1/1/1

Network Performance

• Measure the sustained bandwidth on both the client network (i.e. desktops) as well as the application-tiers.

• Use traceroute (or tracert) utility to determine number of hops and impact on latency.

• Monitor network statistics including packet rates, error rates, collisions, etc.. using tools such as netstat –i.

Network Performance

• Application Server Web Cache can be used to reduce network traffic between the client and middle-tiers.

• Compression

• Caching of images, style sheets, and Java script

• Web Cache 10.1.2 is certified with 11i


Network Performance

• Application Server Web Cache

• Compression reduces E-Business Suite page sizes considerably

253,771

65,592

0

50000

100000

150000

200000

250000

300000

Bytes

iAS

iAS +WebCache

Network Performance


• Compression reduces E-Business Suite page sizes considerably

104

29

0

20

40

60

80

100

120

TCP Turns

iAS

iAS +WebCache

Network Performance


R e s p o n s e T im e

5 3 .3

5 8 .7

1 3 .2

9 .01 1 .7

2 2 . 02 3 .1

8 . 5

0 .0

1 0 .0

2 0 .0

3 0 .0

4 0 .0

5 0 .0

6 0 .0

7 0 .0

L a n -1 0 0

M b p s

D S L - 5 1 2

K b p s

IS D N - 6 4

K b p s

D ia lu p - 5 6

K b p s

Se

co

nd

s

iA S

W e b C a c h e


Tuning the Database Tier

Data Server Tier

• Refer to the MetaLink document 216205.1 “Database InitializationParameters and Configuration for Oracle Applications 11i”.• Ensure mandatory parameters are set correctly.

• Optimally configure the buffer cache and shared pool as per the workload and the number of users:• [typical configuration for 2,000 Apps users]

• db_block_buffers=800,000

• shared_pool_size=2000M

• A poorly sized buffer cache results in excessive buffer gets and physical I/O.

• A poorly sized shared pool results in library cache and shared pool latch contention due to reloads and lack of space.

Data Server Tier

• Enable asynchronous I/O

• Consider using Quick I/O or equivalent if using file system based files for the DB files.• Quick I/O improves performance and scalability by simulating raw

devices, and avoiding file system cache traffic for the DB files.

• Use locally managed temp files (uniform) for the temporary tablespace.

• 9iR2: • Install Stats Pack (spcreate.sql) and use spauto.sql to automate hourly

snapshots.

• Stats Pack reports can be used to trend transaction rates and business flows.

• 10g – Use the AWR Reports

Data Server Tier

• Stats Pack Report (Instance Summary)

STATSPACK report for

DB Name DB Id Instance Inst Num Release Cluster Host

------------ ----------- ------------ -------- ----------- ------- ------------

PROD 317772662 crmprod 1 9.2.0.7.0 YES aproddbs1

Snap Id Snap Time Sessions Curs/Sess Comment

------- ------------------ -------- --------- -------------------

Begin Snap: 523410 19-Oct-06 11:00:01 ####### .0

End Snap: 523505 19-Oct-06 16:00:04 ####### .0

Elapsed: 300.05 (mins)

Cache Sizes (end)

~~~~~~~~~~~~~~~~~

Buffer Cache: 5,313M Std Block Size: 8K

Shared Pool Size: 2,864M Log Buffer: 1,024K

. . . . . . . . . . . .

Data Server Tier


Load Profile

~~~~~~~~~~~~ Per Second Per Transaction

--------------- ---------------

Redo size: 462,767.56 26,266.16

Logical reads: 54,537.81 3,095.50

Block changes: 3,237.12 183.74

Physical reads: 2,403.22 136.40

Physical writes: 272.19 15.45

User calls: 518.96 29.46

Parses: 223.29 12.67

Hard parses: 1.26 0.07

Sorts: 199.15 11.30

Logons: 1.85 0.11

Executes: 1,566.48 88.91

Transactions: 17.62

% Blocks changed per Read: 5.94 Recursive Call %: 88.46

Rollback per transaction %: 3.18 Rows per Sort: 102.78. . . . . . . . . . . .

Data Server Tier


Instance Efficiency Percentages (Target 100%)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Buffer Nowait %: 99.90 Redo NoWait %: 100.00

Buffer Hit %: 96.03 In-memory Sort %: 99.96

Library Hit %: 99.87 Soft Parse %: 99.44

Execute to Parse %: 85.75 Latch Hit %: 99.82

Parse CPU to Parse Elapsd %: 69.57 % Non-Parse CPU: 97.34

Shared Pool Statistics Begin End

------ ------

Memory Usage %: 90.55 76.14

% SQL with executions>1: 66.70 75.37

% Memory for SQL w/exec>1: 60.86 71.08

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total

Event Waits Time (s) Ela Time

-------------------------------------------- ------------ ----------- --------

db file sequential read 16,187,270 134,274 31.12

CPU time 110,764 25.67

global cache cr request 21,410,811 66,129 15.33

db file scattered read 3,184,765 23,571 5.46

KJC: Wait for msg sends to complete 10,722,532 20,446 4.74

Data Server Tier

• Stats Pack Report (Wait Events)

Avg

Total Wait wait Waits

Event Waits Timeouts Time (s) (ms) /txn

---------------------------- ------------ ---------- ---------- ------ --------

db file sequential read 16,187,270 0 134,274 8 51.0

global cache cr request 21,410,811 56,241 66,129 3 67.5

db file scattered read 3,184,765 0 23,571 7 10.0

KJC: Wait for msg sends to c 10,722,532 194,069 20,446 2 33.8

row cache lock 74,583 5,842 17,796 239 0.2

wait for unread message on b 73,977 4,624 13,054 176 0.2

SQL*Net more data to client 6,021,483 0 8,873 1 19.0

enqueue 114,204 19,898 7,427 65 0.4

buffer busy global CR 516,382 18 4,735 9 1.6

buffer busy waits 434,054 192 3,857 9 1.4

direct path read 1,239,648 0 3,226 3 3.9

inactive session 1,774 1,774 1,784 1006 0.0

latch free 242,171 19,203 1,694 7 0.8

db file parallel read 121,670 0 1,280 11 0.4

direct path write 1,270,008 0 1,110 1 4.0

log file sync 604,002 244 1,097 2 1.9

SQL*Net message from dblink 20,365 0 1,080 53 0.1

Data Server Tier

• Review Top 5 Timed Events section in the Stats Pack reports to determine the overall instance utilization.

• For latch free waits, latch contention is often a symptom due to a legitimate problem such as non-sharable SQL, sub-optimal SQL which performs full table or full index scans, dynamic object creation/removal, etc..

• Review the latch Statistics section of the Stats Pack report to determine the hot latches.

• Enable tracing for a few of the sessions waiting on the latch aswell as the holder to determine the actual cause.

Data Server Tier

• Stats Pack Report (Expensive SQL)

CPU Elapsd

Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value

--------------- ------------ -------------- ------ -------- --------- ----------

3,413,015 1 3,413,015.0 7.9 1970.29 6716.03 976039043

Module: ALECDC

SELECT distinct ooha.order_number , fu.user_name , rac.cu

stomer_name , to_char(wctc.last_update_date, 'DD-MON-YY hh24:mi

') , DECODE(SUBSTR(wctc.comments, 1, 21),

'NOTE: This Order-Line', wctc.process_flag || ': manually scre

ened', DECODE(wctc.process_flag ,'101','101: host u

2,229,632 1 2,229,632.0 5.2 2115.49 35448.22 1280227044

Module: ALECDC

select distinct pv.segment1 , nvl(pv.vendor_name_a

lt, pv.vendor_name) , ai.invoice_num , hr.name ,

'[email protected]' from po_vendors pv ,

ap_invoices_all ai , ap_holds_all ah , hr_organi

zation_units hr where ai.payment_status_flag||''='Y' and

1,990,002 1 1,990,002.0 4.6 2321.34 23934.41 4063073269

Module: OKILOADDR

BEGIN OKI_REFRESH_PUB.REFRESH_ADDRS( :errbuf, :rc ); END;

Data Server Tier

• Monitor CPU and Memory Utilization

• Monitor paging

• Monitor I/O statistics, including service times.

• Review Stats Pack reports and correlate top SQL to the business flows.

• Review expensive custom SQL

Data Server Tier

• Database Resource Manager Integration

• Define Resource Consumer Groups and associate specific concurrent programs with a resource group.

• Allows an administrator to constrain the amount of CPU resources used by a concurrent program.

• Resource Groups can also be defined for online users and concurrent requests using the profile:

• “FND: Resource Consumer Group”.

• Useful for Discoverer users or ad-hoc reporting users.

Data Server Tier

• Gathering Statistics• Do not gather statistics excessively on entire schemas or the

entire database such as nightly or weekly.

• Do not gather statistics on permanent objects during peak intervals.

• Gathering statistics invalidates cursors

• Gathering statistics requires dictionary and object level locks.

• Plans are not likely to change if the data distribution has not changed.

• Use the Gather Auto Option (11.5.10)

Data Server Tier

• Gathering Statistics• For tables which are growing at a rapid rate, gather statistics

only on those tables.

• Use only FND_STATS or the Gather Schema and Gather Table Statistics Concurrent Programs

• Do NOT USE the analyze nor dbms_stats command directly. It is not supported, and results in sub-optimal plans.

• Review the table and index statistics for the objects which appear in the top SQL section of Stats Pack.

Data Server Tier

• Gathering Statistics Enhancements (11i10)• Gather Schema Statistics

• Auto Gather option

• Gather Statistics only on tables which have changed

• Change threshold is user definable.

• Utilizes the Table Monitoring feature. To enable schema monitoring:

• SQL> exec fnd_stats.ENABLE_SCHEMA_MONITORING()

• Auto List option

• Lists the objects which have changed.

• Maintain history of Statistics Collection

• No Invalidate Option

• Does not invalidate cursors.

Data Server Tier

• Gathering Statistics Enhancements (11i10)

Data Server Tier

• Verifying the Statistics

SQL> set serveroutput on

SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL');===================================================================================================

Table OE_ORDER_LINES_ALL

===================================================================================================

last analyzed sample_size num_rows blocks

12-03-2004 22:59 3726829 37268290 3527197

Index name last analyzed num_rows LB DK LB/key DB/key CF

----------------------------------------------------------------------------------------------------

OE_ORDER_LINES_N1 12-03-2004 22:12 36018080 87310 2097282 1 5 11719150

OE_ORDER_LINES_N10 12-03-2004 22:12 26519270 68610 2230418 1 4 10949180

OE_ORDER_LINES_N11 12-03-2004 22:12 16310880 96200 11196380 1 1 7668050

OE_ORDER_LINES_N12 12-03-2004 22:12 50 1 20 1 1 19

OE_ORDER_LINES_N13 12-03-2004 22:12 1363149 3132 138681 1 3 547938

. . .

. . .

OE_ORDER_LINES_N5 12-03-2004 22:12 0 0 0 0 0 0

OE_ORDER_LINES_N6 12-03-2004 22:12 158583 410 158670 1 1 85032

OE_ORDER_LINES_N7 12-03-2004 22:12 103459 238 18873 1 2 48075

OE_ORDER_LINES_N9 12-03-2004 22:12 4276030 11060 3101971 1 1 2444110

OE_ORDER_LINES_U1 12-03-2004 22:12 37510470 91790 37510470 1 1 17574490

----------------------------------------------------------------------------------------------------

Histogram Stats

Schema Table Name Status last analyzed Column Name

----------------------------------------------------------------------------------------------------

ONT OE_ORDER_LINES_ALL present 03-12-2004 22:59 OPEN_FLAG

Data Server Tier

• Verifying the Statistics – Column Statistics

SQL> set serveroutput on

SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL',0,TRUE);===================================================================================================

Table OE_ORDER_LINES_ALL

===================================================================================================

last analyzed sample_size num_rows blocks

12-03-2004 22:59 3726829 37268290 3527197

----------------------------------------------------------------------------------------------------

Column Stats

Column name sample_size num_distinct num_nulls density last analyzed

----------------------------------------------------------------------------------------------------

ACCOUNTING_RULE_DURATION 0 37219060 0 12-03-2004 22:51

ACCOUNTING_RULE_ID 1882057 72 18398490 .013888888 12-03-2004 22:34

. . .

BOOKED_FLAG 3724692 2 0 .5 12-03-2004 22:51

. . .

CANCELLED_FLAG 3724692 2 0 .5 12-03-2004 22:51

. . .

LINE_ID 3721906 37219060 0 .000000026 12-03-2004 22:34

LINE_NUMBER 3721906 5649 0 .000177022 12-03-2004 22:34

LINE_SET_ID 10420 18873 37114860 .000052985 12-03-2004 22:34

OPTION_NUMBER 1350665 1176 23740270 .00085034 12-03-2004 22:51

ORDERED_ITEM_ID 3210194 50265 5117120 .000019894 12-03-2004 22:34

UPGRADED_FLAG 3125945 2 5987470 .5 12-03-2004 22:51

VEH_CUS_ITEM_CUM_KEY_ID 0 37219060 0 12-03-2004 22:34

VISIBLE_DEMAND_FLAG 668876 2 30558160 .5 12-03-2004 22:51

Data Server Tier

• Enable the Auto memory manager

• Pga_aggregate_target can be set to tune the PGA for the entire instance.

• Automatically tunes hash area and sort area sizes

• Returns unused memory to the OS.

• Enable System Managed Undo

• No more ORA-1555s!!!

• Set undo_retention to the length of the longest running request.

• Utilize Auto Segment Tablespace Management

• No need to manually set Freelist/freelist groups. A bitmap is used to automatically scale the segment related parameters.

• Enable PL/SQL Native Compilation

• Improves PL/SQL execution performance.

• Improves scalability by removing a lot of stress off of the shared pool.

Data Server Tier

• New Tablespace Model for the E-Business Suite:

• Consolidates the number of Apps tablespaces from 400 tablespacesto 10.

• Optimizes performance via the wide stripe model.

• Ideal for customers migrating to RAC or migrating to raw devices.

• Uniform extents, locally managed

• No more fragmentation!

• Migration script can be used to migrate a schema at a time.

Data Server Tier

• New Tablespace Model for the E-Business Suite:• Objects are classified by object type and access patterns:

• Transaction tables

• Transaction indexes

• Reference/SEED

• Interface

• Summary Management

• Archive

• NoLogging

• Queue

• Media

• Temporary

• Undo

• System

• Total = 12 Tablespaces

Data Server Tier

• Applications Tablespace Migration Utility• MetaLink note 248857.1

• ARU # is 3942506

• Migration timings: (complete migration)• 1.4 Terabyte customer DB

• 24 hours

• New size of DB after migration (850 GB).

• 180 GB customer DB

• 4 hours

• New size of DB after migration (120 GB).


Tuning the Application

Application Tuning

• Review the MetaLink note “Recommended Performance Patches for the Oracle E-Business Suite”, 244040.1.

• Recommended performance patches for all the modules and tech. stack components are consolidated in this note.

Application Tuning

• Workflow• When starting background engines via Concurrent Manager,

set the Process Stuck parameter to No.

• Start a separate background engine to handle stuck processes with a very low frequency such as once or twice a day.

• Use deferred activities to improve online response times and facilitate asynchronous processing for flows such as Scheduling, PO Document Approval, etc..

• Establish a regular frequency for purging Workflow runtime data (FNDWFPR - Purge Obsolete Workflow Runtime Data).

Application Tuning

• Workflow

• Run the script ($FND_TOP/sql/wffngen.sql) to translate the activity function calls into static calls. This improves performance of Workflow processes by 25%.

• Set init.ora parameter aq_tm_processes=1

• Disable retention on Workflow Queues• SQL> select owner,name,queue_table,retention

from dba_queues

where owner='APPLSYS' and retention != 0;

• SQL> exec

DBMS_AQADM.ALTER_QUEUE(queue_name=>:b1,retention_time=>0);

• Avoid dequeuing by correlation

• When starting the Workflow Background Process, avoid specifying the Item Type parameter.

• When starting the Workflow Agent Listener, do not specify the Correlation ID.

• For 11.5.9 or OWF G, apply patch 3940679.

• Apply RDBMS patch 4519477

Application Tuning

• Order Management

• OM Pack J contains numerous performance fixes including Pricing performance fixes.

• Ensure that debugging is disabled:

• OM: Debug Level, set to 0 for OFF

• QP: Debug Mode, set to Request Viewer Off

• WSH: Debug Enabled, set to No

• WSH: Debug Level, set to 0 for OFF

• Use the streamlined workflows.

• Removes unnecessary activities and sub-processes.

• Improves performance and amount of WF data significantly.

• New streamlined flow in 11.5.10

• Line Flow – Generic: Performance


Application Tuning

• Order Management

• For online order entry, use the new Quick Sales Order Form

• Provides a checkbox “deferred pricing” which can be used to defer price calculation until the order is saved.

• Also improves overall speed of order entry from a functional / navigational perspective.

• Enable AutoScheduling

• Use deferred activities

• Tax Calculation

• Credit Check

Application Tuning

• Payroll• Tune the number of threads, chunk size and the buffer sizes

(pay_action_parameters):

• THREADS=25

• CHUNK_SIZE = 20

• BAL BUFFER SIZE=550

• EE BUFFER SIZE =550

• RR BUFFER SIZE =550

• COST BUFFER SIZE=550• Set threads to ~1.5 times the number of data server processors.

• Set LOW_VOLUME=N to enable optimal plans.

Application Tuning

• OA Framework Applications• Refer to MetaLink note 123456.1 (Recommended Patches for

Applications) and MetaLink note 275880.1 (Framework Roadmap).

• JDBC Connection Pool• Upgrade to the latest 9iR2 JDBC Drivers

• MetaLink note 164317.1

• ARU 4201222 (9.2.0.6 JDBC Drivers)

Application Tuning

• Pool Monitor can be used to monitor the Framework Applications as well as the JVM utilization

• Application Module Pool

• Memory Utilization

• JVM Properties

• Pre 11i10: https://<hostname>/servlet/OAAppModPoolMonitor

• 11i10: Login->Diagnostics->Show Pool Monitor.

Application Tuning

• OA Framework Applications (Pool Monitor)

Application Tuning


Application Tuning


Application Tuning



Pool Monitor

Application Tuning (OA)

• About this Page


• About this Page


• About this Page


• Web Applications• Ensure the users are trained to use the Logout or Home

global buttons when completing their transactions.

• Do not train them to use the browser close (“x”) link.

• Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc..)

• Avoids memory leaks and timeout based invalidation.

Application Tuning

• Logging• FND: Debug Log Enabled

• Prior to RUP3: Set it to No.

• RUP3 and beyond: Set it to Yes.

• FND: Debug Log Level

• Should be set to Unexpected (Level=6)

• Query fnd_log_messages to determine the logging traffic.

• If the profiles are set as recommended above, you should not see many messages logged.

• If the query shows a large number of rows being created, there could be exceptions and errors which need to be investigated.

Application Tuning

• Profiles• Set the frequently referenced profiles at the User level to

avoid additional level checks. The check hierarchy is as follows:

• Session

• User

• Organization

• Server

• Responsibility

• Application

• Site

• Example: Set the profile “INV: Debug Trace” to “No” at the user level.


• Tracing Framework Applications

• Set profile FND : Diagnostics to Yes at user level

• Login to Self Service as the above user

• Click on Diagnostics icon at the top of page

• Select ‘Set Trace Level’ and click Go

• Select the desired trace level and click Save

• Perform the activity that you want to trace

• Exit application


� Tracing Framework Applications

Application Tuning

• AOLJ Connection Pool (dbc configuration)• Disable sanity checks in production environments

• FND_JDBC_USABLE_CHECK=false

• FND_JDBC_PLSQL_RESET=false

• Reduce the number of OXTA JDBC connections

• In jserv.properties file, add the following entries

• wrapper.bin.parameters=-DOXTAOutThreads=1

• wrapper.bin.parameters=-DOXTAInPoolSize=1

• If running multiple JVMs, the OXTA connections can be a significant portion of the total JDBC connections.

• Automatically set in 11i10

• Start with a reasonable default such as 100 or 200.

• FND_JDBC_MAX_CONNECTIONS=200

• Tune FND_JDBC_MAX_CONNECTIONS as per the amount of user concurrency (per JVM).

• AOLJ automatically decays idle connections and resizes the pool

Application Tuning

• AOLJ Connection Pool (dbc configuration)• AOLJ Supports the use of Services for RAC load balancing or

failover.

• dbc parameter APPS_JDBC_URL

• Used in place of DB_HOST, DB_NAME, DB_PORT

• Useful for automatic connection load balancing

APPS_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)

(ADDRESS=(PROTOCOL=TCP)(HOST=ap201ops)(PORT=1526))(ADDRESS=(PROTOCOL=TCP)(HO

ST=ap202ops)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=opsperf)))

Application Tuning

• AOLJ Connection Pool Monitoring

• Use the JSP (AoljDbcPoolStatus.jsp) to monitor the connection pool status.

• Lists DBC configuration parameter settings.

• Connection Statistics

• Locked Connections

• Available Connections

• Closed Connections

• Allows drill-downs to determine source of the connection.• http://apptier.oracle.com:1200/OA_HTML/jsp/fnd/AoljDbcPoolStatus.jsp

Application Tuning

• Connection Pool Status (AoljDbcPoolStatus.jsp)

Application Tuning

• Purging• Implement a regular purge process including transient data as

well as obsolete transactional data.

• Archive system related entities prior to purge such as

• FND_CONCURRENT_REQUESTS

• ICX_SESSIONS

• Purge old data before performing upgrades.

Application Tuning

Mass Additions Purge Report

Purge accounting tables

Assets

PurgePayables

Archive and PurgeReceivables

Order PurgeOrder Management

GL Archive and PurgeGeneral Ledger

Purge Concurrent Request and/or

Manager Data

Purge Workflow Runtime Data

AOL

Concurrent Program NameProduct

• Purging


11i10 Enhancements

• Ability to enable all levels of SQL trace as well as PL/SQL profiler from Forms, Framework based pages, and Concurrent Programs.

• Help->About Oracle Applications shows performance information including time spent in the Form and time spent in the DB.

• For concurrent programs, tracing can be enabled at the program definition level and at the request level.

• Trace file identifier is automatically set to the Applications user name • prod11i_ora_2738_BASMITH.trc

• New Report (PL/SQL Profiler)• Generates an HTML report of the PL/SQL profiler data.

11i10 Tracing Improvements

• PL/SQL Profiler Report


• PL/SQL Profiler Report


• Performance Info


• FND_STATS Improvements:• No invalidate option can be used to avoid invalidating cursors while

gathering statistics.

• Auto Gather has been added to allow statistics to be gathered only on those objects which have changed.

• Change threshold is configurable.

• Default is 10%.

• Significantly reduces the time to gather statistics.

• Improves manageability as FND_STATS automatically determines changed object list.

• Maintain history of all gather stats requests.

• Automatically determines degree of parallelism.

11i10 Performance Improvements

• Significant reduction in patching downtime

• Parallelized large table updates.

• FNDLOAD Improvements

• 15% for small LDTs

• Orders of magnitude for large LDTs.

• Improvements to distributed AD.

• Large indexes are now created in parallel.

• APPS_MRC schema has been eliminated.

• Optimized PL/SQL recompilation order for invalid objects.

11i10 Performance Improvements

For More Information

Using Discoverer 10.1.2 with Oracle E-Business Suite 11i313418.1

Installing and Configuring Web Cache 10.1.2 and Oracle E-Business Suite 11i306653.1

E-Business Suite Recommended Patch List 123456.1

Upgrading Oracle JDBC Drivers with Oracle E-Business Suite 11i164317.1

Oracle Applications Framework Release 11i Documentation Road Map275880.1

Using J2SE Version 5.0 with Oracle E-Business Suite 11i, Release 11.5.10304099.1

Recommended Performance Patches for Oracle E-Business Suite244040.1

Database Initialization Parameters and Configuration for Oracle Applications 11i216205.1

Implementing and Using the JSP Precompiler215268.1

Canceling Long Running Queries in Oracle Applications 11i138159.1

Upgrading Developer 6i with Oracle Applications 11i125767.1

Description:Note #:

Relevant OpenWorld Sessions

• S282216 Oracle E-Business Suite Applications Database Optimization Panel, Wednesday 10/25/2006 2:30 PM - 3:30 PM, Moscone West 2005

• S281654 Partitioning and Purging Best Practices for Oracle E-Business Suite, Thursday 10/26/2006 9:30 AM - 10:30 AM, Moscone West 2010

• S281384 Leveraging Oracle Database 10g Features with Oracle E-Business Suite, Thursday 10/26/2006 2:00 PM - 3:00 PM, Moscone West 2009 – 2011

Q U E S T I O N SQ U E S T I O N S

A N S W E R SA N S W E R S

Documents

- Oracle...• Sign-on to Apps with System Administrator responsibility • Navigate to Concurrent -> Program -> Define • Query the concurrent program