54151230 EBS Concurrent Manager Performance Best Practices

1© 2009 Oracle Corporation – Proprietary and Confidential

2

Day, Date, 2004

time p.m. ET

Teleconference Access:

North America: xxxx

International: xxxx

Password: Advisor

Thursday, Feb 10, 2011

06:00 PM CET (GMT +1)

Teleconference Access:

North America:

1866 230 1938

International (UK):

+44 (0) 1452 562 665

Conference ID :

37014291

Upcoming Webcasts in ATG area :

o 16 / 17-Feb-2011 : EBS - Workflow Overview & Best Practices

o March 2011 : E-Business Suite - APPS-Tier Loadbalancing

o April 2011 : EBS - Workflow Builder - Best Practices

o April 2011 : E-Business Suite - Workflow Mailer

Do you have any requests for future EBS - ATG Webcast

Events?

Please email your suggestions to

[email protected], subject: Topics of Interest.

EBS – Concurrent Manager Performance

© 2011 Oracle Corporation – Proprietary and Confidential

3

AGENDA

• Presentation and Demo – approximately 60 minutes

• Q&A Session – maximum 20 minutes


Please be aware that we got much more Registrations for this Webcast than expected and therefore we have decided following :

The question via the Q&A Panel are answered immediately by experts available on this session.

For the Q&A Session after the Presentation we are only taking questions from the phone – for max. 20 minutes

For any question, which have not been answered during the session,

please go to the MOS Communitiy „E-Business Suite – Utilities“

and use the Q&A - EBS – CCM Performance Webcast.

There we are also going to answer questions left over in the Q&A Panel.

https://communities.oracle.com/portal/server.pt?open=514&objID=224&mode=2&threadid=172074&aggregatorResults=T172074T165800T169451T169273T162458T167298T48006T160889T164392T163235&sourceCommunityId=259&sourcePortletId=268&doPagination=true&pagedAggregatorPag







4

Q&A panel

type your question here1

Send your question3

Who to ask?2


5

your question pop-up here



ATTENTION – AUDIO INFORMATION

Voice streaming/Audio broadcast is available.

For full audio access join teleconference.

Teleconference Connect details:

1. Conference ID: 37014291

2. International dial in: +44 (0) 1452 562 665 (UK)

3. International Toll Free numbers are available in Note 1282434.1

Note:

You can view this info anytime using WebEx menu:

Communicate > Join Teleconference

from your WebEx session

7

The following is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into

any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be

relied upon in making purchasing decision. The

development, release, and timing of any features

or functionality described for Oracle‟s products

remains at the sole discretion of Oracle.

Safe Harbor Statement


<Insert Picture Here>

EBS Concurrent Manager Performance - Best Practices

Presenter : Kandasamy Nainamalai (India) and Amira Abdelkader (Egypt)

Q&A Panel Experts :

Martin Fritz (Germany), Kalaivani Murugesan (India), Rakesh Chithuluri (India)

Maya Atmaram (US), Michael Costa (US), Raina Yancey (US), David Heisler (US)

9

<Insert Picture Here>

Agenda

• Overview of Concurrent Manager

• Enhancing Performance - Best Practices

Tuning Concurrent Queue Parameters

Specialized Concurrent manager

Output Post Processor – OPP

Tuning PCP/RAC

Transaction Manager (TM)

• Common / Typical issues and resolutionPurging Concurrent records

Running cmclean.sql

OPP and TM related issues.

Resolving concurrent requests

• Trouble-shooting Concurrent manager


10

R12 Architecture


11

Concurrent Manager - Introduction

• Batch Processing Sub-System

• Programs are run as operating system background processes.

• These programs may be written using a variety of Oracle tools, programming languages for executables, or OS scripts.

The Internal Concurrent Manager (ICM) controls all other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift, monitors for process failure, and cleans up if a failure occurs.

The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests do not run in the same conflict domain

The Standard Manager as shipped with Oracle Applications will accept and run any concurrent requests,


12

Concurrent Manager – Introduction

Concurrent > Manager > Administer

Are the Managers

up and running?


13

Concurrent Manager - Introduction


14

Concurrent Requests, Programs, and

Processes

Concurrentmanager

User requestsConcurrent

program

Run program ...

Run program X

Run program ...

Queue Table

System maintainsList of requests to

Start concurrent programs

Concurrent ManagersRead applicable

Requests and startConcurrent programs

Program XStarted


15

Concurrent Manager - Tables

Table Content

FND_CONCURRENT_REQUESTS Details of user requests, including

status, start date, and completion

date

FND_CONCURRENT_PROGRAMS Details of concurrent programs,

including execution method, whether

the program is constrained, and

whether it must be run alone.

FND_CONCURRENT_PROCESSES Cross-references between

concurrent requests and queues,

and a history of concurrent manager

processes

FND_CONCURRENT_QUEUES Information about each of the

concurrent manager queues


16

Concurrent Request Life Cycle

Phase Activity

Pending / Normal The request is waiting to be run

Pending / Standby The request is waiting at CRM.

Running / Normal The request is running

Completed / Normal The request has finished

Inactive / No Manager The request cannot be run; No

manager available to process it.


17

Enhancing Performance - Best Practices


Sleep Seconds

Cache Size

Purging

Gather Schema Statistics

Workload Management

Specialized Concurrent managers

Output Post Processor – OPP

Java Heap Size

Tuning PCP/RAC

Transaction Managers


18


• Sleep Seconds - is the number of seconds your

Concurrent manager waits between checking the list of

pending concurrent requests (concurrent requests waiting to

be started).

• Tip: Set the sleep time to be very brief during periods when

the number of requests submitted is expected to be high.

Otherwise set the sleep time to a high number (e.g. 2

minutes) . This avoids constant polls to check for new

requests.

• Concurrent Manager will not sleep at all if there are pending

requests.


19

Tunning Parameters-SleepConcurrent manager > define > Workshifts

Reduce the Sleep

Time


20


Cache Size - Increase it (number of requests cached) to at

least twice the number of target processes.

• For example, if a manager's work shift has 1 target process

and a cache value of 3, it will read three requests in one go,

and will run those three requests before reading any new

requests.

• If it is not configured, then concurrent manager will need to

scan the queue table unnecessarily which is an overhead.


21

Tunning Parameters- Cache(Cont)Concurrent > Manager > Define

Cache Size


22


• Process – Increase it if the pending/normal request are

high.

• For example, if the Standard Manager has high number of

pending request in the queue, then you can increase the

process , provided H/W resource available.

• Tip: Need to increase after assessing the server capacity. If

the server is not capable of handing such number of

concurrent requests in parallel, then it can degrade the

performance , instead of improving it.


23

Tunning-Processes (cont)Concurrent > Manager > Define > Workshifts

Increase the Number of Processes


24

Purging Log Files and Tables

Use the Purge Concurrent Request and/or the Manager Data program to purge Request Log files, Concurrent Manager Log Files, and report output files.

• To conserve space you should periodically delete

log and output files.


25

Purging Concurrent Requests

At regular interval, Purge backend tables that are holding

Concurrent Request related information and delete log/out

files from concurrent manager tier.

To purge, It is recommended to run the request the "Purge

Concurrent Request and/or Manager Data“ (FNDCPPUR)

with below Parameters at regular interval.

• Entity =All

• Mode=Age

• Mode Value=10 (Number of days to keep. To be set based

on the retention policy of the organization.)


26


Purging Log/out files

• Ensure that the log/out files are removed from the below

directory as you run "Purge Concurrent Request and/or

Manager Data program".

$APPLCSF/$APPLLOG

$APPLCSF/$APPLOUT

• Applying Patch 7834670 (on top of 11.5.10.2) to improve

performance & removes files from OS.

• Truncate the reports.log ($APPLLOG) file in log directory.

Refer Note.844976.1 for more details.


27


Defragment the tables to improve performance

• As you run FNDCPPUR, It deletes the rows from the

FND_CONCURRENT tables . Hence they get fragmented.

• It degrades the performance when full table scan is performed

• De-fragment the below tables periodically to reclaim unused

space. It improve performance. (Refer How to defragment)

• FND_CONCURRENT_REQUESTS

• FND_CONCURRENT_PROCESSES

• FND_CRM_HISTORY

• FND_ENV_CONTEXT

• FND_TEMP_FILES


28

Gather Schema Statistics

Gather Schema Statistics to improve performance

overall.

• At regular interval, run “Gather Schema Statistics”

• Use below parameter to speed up the request.

Schema Name ALL

Estimate Percent 10-40

Backup Flag NOBACKUP

History Mode NONE

Gather Options GATHER AUTO

• Run the request "Analyze All Index Column Statistics" on the

indexes of these tables (FND_CONCURRENT_PROCESSES

,FND_CONCURRENT_PROGRAMS,FND_CONCURRENT_REQUESTS,

FND_CONCURRENT_QUEUES)


29


Specialized / Dedicated Concurrent managers

• Create specialized concurrent managers which is dedicated

to some specific concurrent requests which need to be

processed in a different way.

Benefits

• Speed up Critical Short Running Requests which has long

wait time

• To avoid Long Running Request during Peak time

Tips

• Avoid creating an excessive specialized managers. It can

degrade the performance due polling on queue tables. You

need to create specialized managers only if there is a real

need.


30


Speed up Critical Short Running Requests

• Assume you have Business Critical "Short Running" ( ex.

Invoice Printing ) Concurrent request, But it is waiting at

Standard Manager for long time before getting picked up for

processing.

• You can then define a new Specialized Concurrent Manager.

And include the specific critical short run concurrent request

to the new manager that you created. This will ensure all your

critical short run concurrent requests are not waiting in the

queue for longer time.


31

Specialized / Dedicated Concurrent Managers



33


• Avoid Long Running Request during Peak time

This helps you to prevents users from executing such „Long

Running‟ Request during Peak time. This improves overall

performance of the system.

• Create a specialized manager to process the specific 'Long

Running' requests.

• Assign the specialized manager with a work shift with Off

peak time.

• Assume, The users submit the concurrent requests during

peak time, then it will get assigned to the specialized

manager.

• But specialized manager will not process it during peak time.


34

Specialized / Dedicated Concurrent Managers


35


Output Post Processor - OPP

Concurrent processing uses the Output Post Processor (OPP) to

enforce post-processing actions for concurrent requests.

Hints:

Ensure enough number of OPP exists to serve the incoming

request load. Otherwise increase “process”.

Thread Vs Process

• You can either increase process (or) threads. It depends on the load .

• Concurrent -> Manager -> Define > "Output Post Processor“

• oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5

• Each PCP node should have one OPP defined. Refer Note

563233.1 for more details.


36


Increase Number of OPP

processes

Increase number Threads

OR


37


• Heap Size

This setting prevents the error "java.lang.OutOfMemoryError: Java heap

space“

SELECT service_id, service_handle, developer_parameters

FROM fnd_cp_services

WHERE service_id = (SELECT manager_type

FROM fnd_concurrent_queues

WHERE concurrent_queue_name = 'FNDCPOPP');

Increase the maximum Java heap size for the OPP to 1024MB (1GB):

UPDATE fnd_cp_services

SET developer_parameters =

'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'

WHERE service_id = (SELECT manager_type

FROM fnd_concurrent_queues

WHERE concurrent_queue_name = 'FNDCPOPP');


38


Bi Publisher Enhancement parameters

1.Use XML Publisher's XSLT processor set to True

2.Enable Scalable feature of XSLT processor set to False.

Small Report - It is still recommended this remain false at Site level.

This has the advantage of faster performance by not using

temporary files for XSLT processing for small to medium reports.

Large Reports - with Patch 7599031,can be safely set to True at the

data definition level, enabling improved JAVA heap utilization along

with ScalableFlag parameter and 'Options' field with the value to -

Xmx1024M (or higher).

3.Enable XSLT runtime optimization set to True

Refer Note 737311.1 for more details



40


Bi Publisher Enhancement parameters

Large Reports - with Patch 7599031, „Enable Scalable feature of

XSLT processor„ can be safely set to True at the data definition

level, enabling improved JAVA heap utilization along

with ScalableFlag parameter and 'Options' field with the value to -

Xmx1024M (or higher).

Refer Note 737311.1 for more details




43

PCP / RAC

• Set profile option 'Concurrent: PCP Instance Check' to 'OFF' if

instance-sensitive failover is not required

• Run a concurrent program against a specific RAC instance with

PCP/RAC setup

Note 1129203.1 - How to run a concurrent program against a

specific RAC instance with PCP/RAC setup?

• If the failover of managers is taking too long, then refer

Note:551895.1 Failover Of Concurrent Manager Processes

Takes More than 30 Minutes


https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=551895.1

44

PCP / RAC


45

Transaction Manager (TM)

• Ensure enough number of TMs exist to serve the incoming

request load.

• Profile “Concurrent: Wait for Available TM” - Total time to wait

for a TM before switchover to next available TM.

• To improve performance , try setting this to 1- 2.

• In case all TM are busy, then increase number of TM.

• Use Advance Queue(AQ) with ATG RUP3 by setting System

Profile „Concurrent: TM Transport Type' to „QUEUE'

• Pipes Vs Queue

• Pipes are more efficient than Queue. But it require a

Transaction Manager to be running on each DB Instance

(RAC). However you might want to use "Queue" for easy

maintenance..


46

Common / Typical issues and

resolution

• Running cmclean.sql

• Purging Log Files and Tables are not deleted on the

OS level

• Resolving concurrent requests stucked


47

cmclean.sql Script

Why to run CMCLEAN.sql ?

Cleaning out the tables is a useful method of making sure

that there are no invalid statuses that can prevent the

managers from starting

Run the script, cmclean.sql, article note 134007.1

CMCLEAN.SQL - Non Destructive Script to Clean Concurrent

Manager Tables It will make sure the relevant status codes

are valid without deleting any information.

Hint:

What are the Meaning of the Codes in the STATUS_CODE and

PHASE_CODE Columns of FND_CONCURRENT_REQUESTS

Table? (Doc ID 152209.1)

How to Clean Out Concurrent Manager Tables? (Doc ID

155925.1)


48


Typical issues during Purging Concurrent requests and

resolution

Log/out files are not removed from OS file system

Performance issue

Apply Patch to fix above issues.

Purge Concurrent Request Hangs post cloning.

Update FND Table.


49


• Release 11i

• Apply Patch 7834670 (on top of 11.5.10.2)

• Ensure afppur.lpc is 115.36

• Set the system profile "Concurrent: Force Local Output File

Mode

Refer Note 822368.1 for more details.

Note :- Most of your issue should get fixed with patch 7834670. To set the

profile "Concurrent: Force Local Output File Mode" is set to "Yes", ALL

Concurrent Manager nodes must be able to access the output file location via

the local filesystem

• Release 12

• Ensure afppur.lpc is higher than 120.4.12000000.4

$strings -a $FND_TOP/bin/FNDCPPUR|grep 'Header'|egrep

'afppur.lpc„


50

Purge Request Hang after Cloning

Cloning will not update the host and instance specific

information used by Oracle Workflow

Review the following tables of the Target Instance:

FND_CONCURRENT_REQUESTS

LOGFILE_NAME Update with the correct path to the logfile

directory


OUTFILE_NAME Update with the new directory path on the

Target System

For more info Cloning Oracle Applications Release 11i with Rapid Clone

(Doc ID 230672.1)


51

Pending Requests

Pending / Normal

• The request is waiting for next available concurrent manager to

be picked and it will get processed.

Action

Increase “process” of the Concurrent Manager.

Pending / Standby

• The request is waiting at CRM due to conflict request which is

already being run.

Action

• Check the “Incompatibilities “ defined for the request.

• Check any request is running with “Run Alone “ flag.


52



53

Incompatible Programs:

N.B Conflict Domains is to prevent two programs from

concurrently accessing or updating the same data

Logicaldatabase

Program A Program B

Logicaldatabase

Program A

Domain 1 Domain 2

Program B


54



55


Running / Normal

Action

If the request is taking longer time, enable Database Trace and

tkprof and find out expensive DML. Steps given below.

Completed / Error

Action Check the request log file and take action accordingly

Inactive / No Manager

Action If “Standard Manager” has “Specialization Rules” defined

which has rows with “Include”. You must only define exclude. If

you add any program with “Include” options, other programs get

excluded automatically.


56



57

Trouble-shooting Concurrent manager

• How To Enable Tracing ( Request Running/Normal)

• Output Post Processor -OPP Timeout

• How to Report Performance Issues


58

How to Enable the Tracing

1- Enable Tracing For The Concurrent Program

System Adminstrator Responsibility

• Navigation: Concurrent -> Program -> Define

• Click the “Enable Trace” button

2- Turn On Tracing

• Responsibility: System Administrator

• Navigate: Profiles > System

• Query Profile Option Concurrent: Allow Debugging

• Set profile to Yes

3- Run Concurrent Program With Tracing Turned On

• Logon to the Responsibility that runs the Concurrent Program

• In the Submit Request Screen click on Debug Options (B)

• Select the Checkbox for SQL Trace

• Database Oracle Home: admin/udump/prod_ora_30806_APPS.trc


59

Enabling Trace at the program Level


60

Enabling Trace at the Request Level


61

How to enable the Tracing(cont)

• Once you have obtained the Raw trace file you need to format

the file using TKPROF.

$tkprof raw_trace_file.trc output_file explain=apps/<passwd>

sort=(exeela,fchela, prsela) sys=no

Note If you do not know the path for those directories, you can run

a command like the following query in SQL,

SQL> select name, value from v$parameter where name =

'user_dump_dest';


62


To improve BI Publisher system Set the below profiles

Error

The Output Post-processor is running but has not picked up this request.

No further attempts will be made to post-process this request, and the

request will be marked with Warning status.

Action

• Setting the profile option Concurrent: OPP Response Timeout to

a higher value may be necessary.

Tip If the value of the Concurrent:OPP Response Timeout profile option is 120

then increase it to 240. (Value * 2).

• Increase the number of processes or threads (or both)


63


To improve BI Publisher system Set the below profiles

Error

The concurrent manager has timed out waiting for the Output

Post-processor to finish this request.

Check that there are enough Output Post-processor service

processes running.More information may be found in the

service process logfile.

Action

• Check "Concurrent:OPP Process Timeout" profile option.

Tip Increase the setting to a higher value. For example: Set the value

to 10800 sec (3 hours). After this change, the concurrent request will

wait maximum 3 hours, for post processor to finish the request.


64

Reporting Concurrent Manager Issue

Logging an SR with support.

• Was there any recent change

• State if any request stay in Pending/Running long time

• State whether all/specific concurrent requests are running

slow.

• State CMCLEAN.sql was run

• Run FNDCCMDiagnostic115.sh & upload the output

• Provide total number of records in the below tables


FND_CONCURRENT_PROCESSES

FND_CRM_HISTORY

FND_ENV_CONTEXT

FND_TEMP_FILES


65

Reporting Concurrent Manager Issue

• Upload Concurrent Request log , if its specific to a program

• Upload Database Trace and tkprof output

• ICM (or) Standard manager log relevant

• Backend queue concurrent manager details

select

rpad(user_concurrent_queue_name ,50,'-') "Queue

Name",TARGET_NODE,

MAX_PROCESSES,running_processes,

nvl(to_char(SLEEP_SECONDS),'Not Set')

Sleep_Second,nvl(to_char(cache_size),'Not Set')

Cache_size

from

FND_CONCURRENT_QUEUES_VL fcq

order by 1;


66

For More Information

The recorded version of this webcast will be available within the next

48 hours. It can be downloaded from the Advisor Webcast Archive

Note 740297.1.

Via the ATG Advisor Webcast Note 1186338.1, you will be able to

download the Presentation in PDF Format and there are also the links

listed to the recording. It also lists the Webcasts and Materials from 2010.

OCM Customer Viewlet – available via Note 1087336.1

Do you have any requests for future EBS - ATG Webcast Events?

Please email your suggestions to [email protected],

subject: Topics of Interest.


https://support.oracle.com/CSP/main/article?cmd=show&id=1087336.1&type=NOT

mailto:[email protected]

67

Upcoming Webcasts around ATG :

16 / 17-Feb-2011 : EBS - Workflow Overview & Best Practices

March 2011 : E-Business Suite - APPS-Tier Loadbalancing

April 2011 : EBS - Workflow Builder - Best Practices

April 2011 : E-Business Suite - Workflow Mailer

For complete details on all upcoming Oracle Advisor Webcast Events, please see Note 740966.1, Oracle Advisor Webcast Schedule.

For ATG Specific Webcasts please check Note 1186338.1.

Do you have any requests for future ATG Advisor Webcast Events ?

Please email your suggestions to me :

[email protected], subject: Topics of Interest.

Applications Technology Group Webcasts




68

ATG Communities in MOS available


Following Communities are available in My Oracle Support

Oracle E-Business Suite

...

BI Publisher - Business Intelligence Products (not only EBS)

Core Workflow - Any Workflow isse not only E-Business Suite

Diagnostic Tools - Anything around EBS Diagnostics

E-Business Customizations - Your Customizations

Installation - Fresh Install of the E-Business Suite

Patch Review EBS - Review of Patches around the E-Business Suite

Performance - EBS Performance

Upgrade - EBS Upgrade

User Produktivity Kit - User Productivity Kit (UPK) available for the E-Business Suite

Utilities - Utilities / generic EBS DBA issues

...

This is the current list for the E-Business Suite – Applications Technology Group

69

ATG PIC Note 1160285.1




71

THANK YOU


Documents

54151230 EBS Concurrent Manager Performance Best Practices