086 RFC Monitoring

© 2011 SAP AG

Best-Practice Document

RFC Monitoring

Dietmar-Hopp-Allee 16 D-69190 Walldorf

DATE

July.2011

SOLUTION MANAGEMENT PHASE SAP SOLUTION

Operations & Continuous Improvement Generic Best-Practice Document

TOPIC AREA SOLUTION MANAGER AREA

- Technical Monitoring


RFC Monitoring

© 2011 SAP AG page 2/124

Table of Content

1 Introduction 4

1.1 Applicability, Goals, and Requirements 4

1.2 Best-Practice Procedure 6

1.2.1 Motivation for Interface Monitoring 6

1.2.2 Preliminary Tasks 6

1.2.3 Interface Monitoring Concept 6

2 General Overview of RFC and RFC Types 8

2.1 Remote Function Call 8

2.2 General Overview of RFC Types 9

2.2.1 Synchronous RFC (sRFC) 9

2.2.2 Asynchronous RFC (aRFC) 10

2.2.3 Transactional RFC (tRFC) 13

2.2.4 Queued RFC (qRFC) 16

2.2.5 Background RFC (bgRFC) 22

2.3 Dependencies and Convergence of Technologies 34

2.3.1 Queue Dependencies for qRFC 34

2.3.2 Queue Dependencies for bgRFC 36

2.3.3 Mixed-Mode RFC 37

2.4 Automatic Switching from qRFC to tRFC 38

3 Aspect: Error Monitoring 39

3.1 Manual Error Monitoring 39

3.2 Automated Error Monitoring 46

3.3 Troubleshooting: Error Examples and Solutions 48

3.3.1 Error – CALL_FUNCTION_REMOTE_ERROR 48

3.3.2 External Program Not Registered on Gateway 48

3.3.3 External Program Timeout on Gateway 49

3.3.4 Timeout During Connection Setup 50

3.3.5 Active Connection Table is Full 51

3.3.6 SQL Error 60 When Accessing Table ARFCSSTATE 52

3.3.7 CPIC Timeouts During Connection to External Program 53

3.3.8 Socket Too Large 56

4 Aspect: Backlog Monitoring 58

4.1 Manual Backlog Monitoring 58

4.2 Automated Backlog Monitoring 59

5 Aspect: Performance Monitoring 62

5.1 Manual Performance Monitoring 62

5.1.1 Overview of Available Tools for RFC Performance Analysis 65

5.1.2 Activation of Performance Monitoring with Transaction ST03N 72

5.2 Automated Performance Monitoring 84

6 Aspect: Resource Monitoring 85

6.1 Manual Resource Monitoring 86

6.2 Automated Resource Monitoring 90

7 Aspect: Data Management 92

7.1 Classic qRFC 92

7.2 Background RFC 93


RFC Monitoring

© 2011 SAP AG page 3/124

7.3 Manual Data Volume Monitoring 94

7.4 Automated Data Volume Monitoring 95

8 Setup Procedure for Automated Interface Monitoring 96

8.1 Automated qRFC Monitoring Using BPMon 97

8.2 Automated tRFC Monitoring Using CCMS 98

8.3 Automated bgRFC Performance Monitoring Using BPMon 99

8.4 Automated RFC Performance Monitoring Using CCMS 100

8.5 Automated RFC Connection Monitoring Using BPMon 100

8.6 Automated RFC Performance Monitoring 101

8.7 Automated RFC Connection Monitoring Using BPMon 106

9 RFC Performance Monitoring 108

9.1 Introduction 108

9.2 Performance Monitoring Tools 108

9.2.1 Procedure 109

9.2.2 Transaction ST03N 110

9.2.3 Transaction STAD 112

9.2.4 Transaction ST05 116

9.2.5 Transaction SE30 116

9.2.6 Transaction ST12 117

9.3 General Monitoring Guidelines for an SAP System 119

10 Further Information 123


RFC Monitoring

© 2011 SAP AG page 4/124

1 Introduction

1.1 Applicability, Goals, and Requirements To ensure that this best-practice document is the one you need, consider the following goals and requirements.

Goal of Using this Service This best-practice document helps you set up an interface monitoring concept with the focus on RFC monitoring for your SAP solution. This document outlines possibilities on how best to monitor RFC-based interfaces manually, as well as automatically, by using SAP Solution Manager. Both monitoring approaches aim to detect any irregularities or deviations or to detect error situations at an early stage.

These procedures intend to ensure that the interface processing meets the requirements for stability, performance, and completeness, as well as a smooth and reliable flow of the core business processes so that your business requirements are met.

Alternative Practices You can have SAP experts deliver this best-practice document on site by ordering a Solution Management Optimization (SMO) service for SAP Interface Management or SAP Business Process Management. These services are exclusively available within a support engagement, that is, SAP MaxAttention, SAP Safeguarding, or SAP Premium Support.

Staff and Skills Requirements To implement this best-practice document, you require the following teams:

Application Management Team

This team provides the information on the business background of the interfaces used and knows the necessary business requirements for the interfaces:

Business department

Solution support organization (for example, Basis support or application support)

Implementation project team

Business Process Operations Team

The Business Process Operations team will be responsible for applying the resulting procedures, derived from implementing this best-practice document. This team includes the following groups:

Persons designated to perform business-process-oriented monitoring and ensure that the process runs smoothly (for example, the business process champion for each business process)

All parties in your solution support organization and IT department involved in monitoring focused on the application aspects (application support, development support, job scheduling management)

SAP Technology Operations Team

All parties in your solution support organization and IT department who are involved in monitoring focused on the system administration side (program scheduling management, Software Monitoring team, and System Administration team, including the system administrator)

Business Process Champion

The business process champion is the person in the business department who is responsible for the successful execution of the business process. He or she coordinates all activities necessary for the business process. Therefore, he or she is usually responsible for the escalation paths in case of problems. The business process champion is often a second level in the escalation procedure, if the application monitoring team needs to escalate an issue.


RFC Monitoring

© 2011 SAP AG page 5/124

Necessary or Useful Training

BIT300 - Interface Technology Overview

E2E300 - Business Process Integration and Automation Management

ADM105 - Advanced R/3 System Administration

ADM315 - Workload Analysis

System Requirements

To set up automated RFC monitoring on SAP Solution Manager, the add-ons ST-PI (any release) and ST-A/PI (as of 01H) have to be installed on the connected managed systems. For certain Business Process Monitoring data collectors, a higher ST-A/PI release is required.

RFC connection monitor: As of ST-A/PI 01H (minimum SAP R/3 / Basis 4.6C)

qRFC monitor: As of ST-A/PI 01K (minimum SAP R/3 / Basis 4.6C)

tRFC monitor: As of ST-A/PI 01L (minimum SAP R/3 / Basis 4.6C)

bgRFC monitor: As of ST-A/PI 01M (minimum SAP NetWeaver Basis 7.0)

Nevertheless, it is always recommended to install the latest version of the ST-A/PI add-on in the managed systems to make use of the latest functionality and all key figures of the mentioned monitors.

Duration and Timing

Duration

Creating an interface monitoring concept depends very much on the complexity of your interface landscape and could take around one week for about 10 interfaces.

Timing

The best time to apply this best-practice document is during the planning phase or during the implementation phase of your SAP solution. The automated monitoring can also be applied later, during the operations phase.

How to use this Best-Practice Document

This document gives you an overview of interface monitoring tools and procedures for the different RFC types. The aim is not to name all available monitors and tools, but to focus on the more monitors and tools for each of the aspects discussed.

Once you have a general idea of the monitoring possibilities, you should evaluate the critical error situations in your system landscape and decide which monitoring procedure is suitable in your case. We do not recommend setting up full-blown monitoring, but suggest concentrating on selective alert situations that will inform you in exceptional cases.

In the next step, you can follow the setup procedure and implement the monitoring on your systems. In the case of business-process-related monitoring objects, you have the possibility to setup a monitoring concept focused on business process monitoring and involve your application teams in monitoring and error resolution activities.

For a better understanding of what should be monitored, it is essential to understand the RFC techniques and their dependencies. The first section will give you a general overview of the different RFCs. The second one will guide you through the setup of the automated RFC monitoring possibilities within SAP Solution Manager.

Sections 3 to 8 list monitoring objects, both for manual and automated monitoring procedures, concentrating on the five major areas: Resource Monitoring, Error Monitoring, Backlog Monitoring, Performance Monitoring, and Data Management.

The monitoring object tables list the following information:


RFC Monitoring

© 2011 SAP AG page 6/124

o Monitoring object

o Monitoring transaction or tool

o Monitoring frequency

o Indicator for an issue or error

o Monitoring activity or error handling procedure

o Responsibility

o Escalation Procedure

The last Section, RFC Performance Monitoring, includes information on scenario-independent tools for performance monitoring.

1.2 Best-Practice Procedure

1.2.1 Motivation for Interface Monitoring

Today‘s system landscapes are often decentralized and can consist of various interfaces. Different SAP systems, legacy environments, and business entities are integrated using different interface technologies. All those interfaces need to be monitored in terms of resource availability, processing errors, backlog situations, and performance.

IT operation relies completely on 100% data consistency. With an increased complexity of system landscapes, we face an increased risk of data inconsistencies. For early detection of inconsistencies, specific data consistency reports have to be executed on a regular basis and detailed error handling and disaster recovery procedures need to be defined. Proactive interface monitoring helps to identify and avoid inconsistencies.

Many RFC types (like tRFC, qRFC, and bgRFC) work asynchronously. This means there is a delay between sending the data out of the source system and receiving/posting the data into the target system. In this time lag, there are temporary differences between the two systems. They are not yet inconsistencies, but may become real inconsistencies if the messaging does not finish in a given time. This kind of backlog must be monitored as well to keep the amount and delay of temporary differences as small as possible.

1.2.2 Preliminary Tasks

Before performing this best-practice document, ensure that you perform the following preliminary tasks or checks in the system:

Complete all installation and post-installation actions and procedures, including customizing.

Ensure that the initial download has been successfully executed.

Apply all SAP recommendations from SAP Services sessions and any SAP recommendations resulting from customer problem messages.

Implement all current Support Packages upon availability.

1.2.3 Interface Monitoring Concept

For a successful and efficient interface monitoring concept, you have to define a process for the execution of the monitoring concept. This includes definition of the roles and responsibilities involved. You need to define who is supposed to carry out which activity and how the communication between the different roles within the support organization is supposed to take place.

An interface monitoring concept must be tightly integrated with the support organization. This includes integration with the incident/problem management process and the change management process. These processes have to be adjusted so that communication and escalation procedures contained within these processes are adequate to support the interface monitoring concept. This includes the final communication of open alerts to SAP.

Wherever communication connected with interface monitoring happens outside these support processes, separate communication and escalation paths and procedures have to be defined.


RFC Monitoring

© 2011 SAP AG page 7/124

For more details, see the separate best-practice document General Business Process Management

by navigating to http://service.sap.com/solutionmanagerbp →Topic area → Business process

operations.

Legend

This symbol indicates a paragraph from which you can navigate to another section or other documents for more detailed information on a topic.

This symbol indicates a paragraph from which you can navigate to another document within the SAP Service Marketplace for more detailed information.

http://service.sap.com/solutionmanagerbp


RFC Monitoring

© 2011 SAP AG page 8/124

2 General Overview of RFC and RFC Types

2.1 Remote Function Call

The RFC interface system is the standard SAP interface for communication between applications of different systems in the SAP environment, which includes connections between SAP systems and connections between SAP systems and non-SAP systems.

A remote function call (RFC) is the call of a remote-enabled function module (RFM) that runs in a partner system. Although it is also possible to call a function module in the same system as an RFC, RFCs are normally used when the caller and the called function module run in different systems. The calling system is the RFC client and the called partner system is the RFC server. RFC is based on the well-known remote procedure call (RPC) model from the UNIX-TCP/IP environment. RFC in the SAP environment is based on a CPI-C interface implemented by SAP.

The RFC communication process begins with an application issuing the command CALL FUNCTION <function> DESTINATION <dest> (where <function> represents a remote-enabled function module in the target system that is defined by <dest>). This generates an RFC request within a work process in the RFC client (that is, a dialog or background work process initiated by the application). The RFC request is serialized in the work process for transmission over the network and is passed to the target system. The RFC request can include various parameters.

Connection parameters: These include the host name, port number, and logon information of the target system. This information is maintained via transaction SM59.

Runtime data: This is the name of the remote function module as well as any specified input parameters (that is, importing, exporting, or changing table parameters).

The local gateway on the RFC client opens a TCP/IP connection on the target gateway (RFC server) and transmits the request to it. The target gateway then allocates a task at the target dispatcher, which identifies an available dialog work process to process the request. The work process on the target system then de-serializes and executes the request and returns the result following the same sequence in reverse (dispatcher→ gateway → dispatcher, and so on). RFC session context data is rolled in and out during the process (when the client is waiting for a response and after the server has issued its response), thus freeing up resources (memory and work processes).


RFC Monitoring

© 2011 SAP AG page 9/124

Figure 1 - RFC Process Communication Cycle

2.2 General Overview of RFC Types

There are five basic types of Remote Function Calls:

o Synchronous RFC (sRFC) o Asynchronous RFC (aRFC) o Transactional RFC (tRFC) o Queued RFC (qRFC) o Background RFC (bgRFC), with the variant of the local data queue (LDQ)

In the following sections, we will explain the unique roles of each RFC type and how each RFC type is created.

2.2.1 Synchronous RFC (sRFC)

Synchronous RFC involves executing function calls based on synchronous communication, which means that the systems involved (RFC client and RFC server) both must be available at the time the RFC call is made. An sRFC is made by calling a function via the CALL FUNCTION statement with extension DESTINATION, which identifies the target system.

http://help.sap.com/saphelp_nw2004s/helpdata/en/47/8f18e9aeba11d6b29500508b6b8a93/content.htm


RFC Monitoring

© 2011 SAP AG page 10/124

Figure 2 - Process Flow Synchronous RFC

When the RFC client makes the call to the RFC server, processing stops in the calling program. While waiting for the connection to the RFC server, no CPU is consumed and the work process is not available for other requests. After establishing the connection, and while the RFC is running on the RFC server, the sender program context rolls out of its dialog work process. The work process is then free for other requests. When the RFC server program returns results to the sender program on the RFC client, the sender program context is rolled in again and the RFC client calling program continues processing.

Synchronous scenarios have higher requirements for system availability and performance, and they also require complex error handling, that is, if the RFC client is using synchronous interfaces to manipulate data in the RFC server system, you are forced to implement your own transactional ID handling to guarantee that the data is not sent and processed twice. Therefore, the receiving application has to check if the same data has already been received (and processed). The programming on the sender and the receiver is quite complex and requires thorough programming with regard to error-proof processing.

For these reasons, synchronous RFC calls are preferably used for read-only accesses to remote systems, for example, for an availability check for material.

2.2.2 Asynchronous RFC (aRFC)

Asynchronous RFC is an asynchronous communication method that executes the called function module in the RFC server. You can use asynchronous remote function calls whenever you need to establish communication with a remote system, but do not want to wait for the function‘s result before continuing processing the calling program.

The ABAP code to enable asynchronous communication differs from synchronous only in the addition of the keywords STARTING NEW TASK <taskName>, whereby <taskName> can be any unique character string.



RFC Monitoring

© 2011 SAP AG page 11/124

Figure 3 - Process Flow Asynchronous RFC

In an asynchronous RFC, the called remote function is started immediately in the calling program and then continues processing on its own, independent of the calling program. With aRFC you have the option to make the aRFC not handle (aRFC without response) or handle (aRFC with response) the remote function output.

2.2.2.1 Asynchronous RFC Without Response

In the case of ―RFC without response,‖ the RFC connection closes immediately after the successful check for availability of the RFM. This means that if any errors occur after the connection closes, for example, if the RFC authority check that verifies if the RFC user is able to execute the RFM fails, the RFC client calling program is not notified. From a performance point of view, closing the connection so quickly is not optimal, especially if it is required to make multiple RFC calls to the same destination. It is important to consider this when implementing aRFC.

Figure 4 – aRFC Without Response

2.2.2.2 Asynchronous RFC with Response

With aRFC with response, you can receive results from the remote function module. To implement aRFC with response, the ABAP statement PERFORMING <subroutine> ON END OF TASK is used, where the specified subroutine must exist in the calling program (RFC client). The ABAP code to specify a subroutine takes the format:

FORM <subroutineName> USING <taskName>.

RECEIVE RESULTS FROM FUNCTION <remoteFunctionName>


RFC Monitoring

© 2011 SAP AG page 12/124

The RFC client calling program starts the remote function and it continues processing on its own, independent of the calling program. The calling program continues to a synchronization point, that is, a WAIT UNTIL statement is reached, which uses a logical expression to check the conditions specified are fulfilled. If so, the program processing continues directly after the WAIT UNTIL statement. Otherwise, the RFC client waits in rollout status for the output from the remote function modules.

Using the special syntax PERFORMING <form> ON END OF TASK you can specify the name of a subroutine, which is called automatically after the remote function module finishes processing. The remote function module output is retrieved using the RECEIVE RESULTS FROM FUNCTION statement. The keyword USING <taskname> is important when the subroutine is called. <taskname> receives the task name of the calling program (specified by STARTING NEW TASK in the calling program). This parameter lets you see which aRFC call just ended. When the subroutine has been executed, RFC connections are closed and control is sent back to the WAIT UNTIL statement in the calling program. This check/wait procedure repeats until the WAIT conditions are fulfilled, or until there are no more open RFC connections.

Figure 5 – aRFC with Response

The addition of KEEPING TASK to the RECEIVE statement causes the function group context that was loaded remotely to wait until the calling program has ended. This lets you use the same remote context for later aRFCs with the same task name and destination.


RFC Monitoring

© 2011 SAP AG page 13/124

Figure 6 – aRFC with Response (Addition of KEEPING TASK)

2.2.2.3 Asynchronous RFC Load Distribution Asynchronous RFC is designed for parallel processing, that is, the RFC client makes calls to the RFC server and starts the remote function module. Control is passed back to the RFC client, which can then make additional aRFCs. Asynchronous RFC does not assess the load on the RFC server. When making a large number of aRFC calls, the SAP server can get overloaded because aRFC involves asking the system to initiate and manage RFC calls while continuing to manage the main program and perhaps queuing up other sessions to execute at the same time. It is possible that the RFC server may run out of resources, that is, memory, CPU, or work processes. If the aRFC calls are made within the same logical SAP system, there is a way to avert this overloading of the RFC server. When calling a function module, you can specify a predefined RFC group (instead of a single server) using the addition DESTINATION IN GROUP <RFC group> (maintained in transaction RZ12 or SAPRFC). This addition selects the next free server in the group to execute the function module. This is achieved by specifying the addition DESTINATION IN GROUP after STARTING NEW TASK. You can only use this addition within the same logical system. You cannot specify a remote DESTINATION. Instead of specifying a specific RFC group, you can also enter the word DEFAULT. The server is selected from all the application servers of the SAP R/3 system. If all the servers of the specified group are overloaded, the exception RESOURCE_FAILURE is triggered; this can be handled by the RFC client calling program.

Refer to Maintenance of RFC server groups for more information.

2.2.3 Transactional RFC (tRFC)

Transactional RFC is an asynchronous communication method that executes the called function module in the RFC server only once (exactly once execution [EO]). The receiving system need not be available when the RFC client program is executing a tRFC. If the receiving system is not available or if there are communication errors, system errors, or application errors, the tRFC component stores the called RFC function, together with the corresponding data, in the SAP database under a unique transaction ID (TID). Then, a background job, RSARFCEX, can be scheduled to restart the tRFC. In addition, in the case of an error, the involved systems can check for duplicate delivery using the TID. This is in contrast to sRFC and aRFC, where you have to handle errors explicitly and you have to build your own mechanism for reissuing calls at a later time.


http://help.sap.com/saphelp_nw2004s/helpdata/en/bc/6ebb134f43ae42a792570d040e1945/content.htm


RFC Monitoring

© 2011 SAP AG page 14/124

You can use tRFC if you insert the addition IN BACKGROUND TASK before the DESTINATION statement. If you specify a COMMIT WORK statement, all RFCs in the calling program are bundled into a LUW and are executed on the RFC server. If the specified functions have the same destination, they form one LUW; if the specified destinations differ, each is executed in a separate LUW.

Figure 7 - Process Flow Transactional RFC


RFC Monitoring

© 2011 SAP AG page 15/124

On a technical level, the process works as illustrated below:

Figure 8 – tRFC Mechanism

The tRFC queue consists of tables ARFCSSTATE and ARFCSDATA on the sender side and ARFCRSTATE on the receiver side. The STATE tables contain status information about the tRFC call on the respective sides, while the DATA tables contain the list of function modules to be called, together with their parameter values.

Step 1: The RFC function to be called, together with its parameter values, is copied into the tRFC queue (ARFC tables). This is triggered by the COMMIT statement in the sender LUW.

Step 2: The data (function module and its parameters) is transferred by a synchronous RFC of the internal function ARFC_DEST_SHIP. On the receiver side, the data is stored in the tRFC queue first. This step is not triggered until the V1 update associated with the COMMIT statement on the sender side has successfully completed.

Step 3: The function module is executed in a dialog work process on the receiver side.

Step 4: ARFC_DEST_SHIP returns the status (success, system failure, communication failure) of the execution of the function module to the sender.

Step 5: If the status returned by ARFC_DEST_SHIP is ―success‖ or ―system failure,‖ the sender calls internal function ARFC_DEST_CONFIRM via sRFC. If the status returned by ARFC_DEST_SHIP is ―communication failure,‖ the sender does not know the processing status on the receiver side. In this case, the sender can repeat the tRFC call at a later time.

Step 6: The execution of ARFC_DEST_CONFIRM deletes the corresponding entries from the tRFC queue on the receiver side.

Step 7: On return of ARFC_DEST_CONFIRM, the sender deletes the tRFC entries from the sender tRFC queue.

tRFC is always used if a function is executed as a logical unit of work (LUW). Within an LUW, all calls are:

1. Executed in the order in which they are called

2. Executed in the same program context in the target system

3. Run as a single transaction; they are either committed or rolled back as a unit

Implementation of tRFC is recommended if you want to guarantee that the transactional order of the calls is preserved.

http://help.sap.com/saphelp_nw2004s/helpdata/en/35/26b221afab52b9e10000009b38f974/content.htm


RFC Monitoring

© 2011 SAP AG page 16/124

In the source system, you can use the administration transaction SM58 to display, execute, debug, and delete tRFC LUWs that are hanging in the communication layer. Entries that remain hanging in the tRFC queue should be investigated. The column status text details the reason for a message hanging in the tRFC communication layer.

Figure 9 – Administration and Monitoring of tRFC

2.2.4 Queued RFC (qRFC) With tRFC, generated LUWs are processed independently of each other. Therefore, the order in which they are processed is not always the order in which they are generated. To guarantee that multiple LUWs are processed in the order specified by the application, tRFC can be serialized using queues (inbound and outbound queues). This type of RFC is called queued RFC (qRFC). qRFC is therefore an extension of tRFC. It transfers an LUW (transaction) only if it has no predecessors (in reference to the sequence defined in different application programs) in the participating queues. Implementation of qRFC is recommended if you want to guarantee that several transactions are processed in a predefined order. This quality of service is called ―exactly once in order‖ (EOIO).

Figure 10 – qRFC Processing

Called

Function

Modules

Error

Message

TID Target

system


RFC Monitoring

© 2011 SAP AG page 17/124

You can use qRFC if you insert the additional ABAP function module calls TRFC_SET_QUEUE_NAME (to use outbound queues) or TRFC_SET_QIN_PROPERTIES (to use inbound queues and outbound queues) before the regular tRFC style CALL FUNCTION statement. The call to function TRFC_SET_QUEUE_NAME/TRFC_SET_QIN_PROPERTIES tells the system into which queues to insert the next RFC call.

Figure 11 – qRFC Using Outbound Queues

Figure 12 – qRFC Using Inbound Queues


RFC Monitoring

© 2011 SAP AG page 18/124

2.2.4.1 Queue Monitoring/Administration

In the SAP environment, inbound and outbound queues are monitored/administered using the queue administration transactions SMQ1 (outbound queues) and SMQ2 (inbound). The following functions are available:

List queues. Display queues with a list of accompanying LUWs and function modules, including input data. Start/stop transmitting the queues. Delete queues or queue entries.

Attention: Deleting qRFC queue entries (or even the whole queue) can create inconsistencies! The sender system relies on the transport to the receiver system. When the data is deleted at the tRFC/qRFC layer, this update will never arrive. Before deleting a queue (entry), clarify with the business responsible about the application content. Only delete a queue (entry) after you have verified that the content is really obsolete and/or that it can be recreated/resent by repeating certain application steps. For example in PI it would mean the application layer does not get informed the RFC entry was deleted and will logically still refer to it. The PI message belonging to it has an inconsistent state and needs special handling.

Figure 13 – SMQ1 qRFC Monitor for Outbound Queues


RFC Monitoring

© 2011 SAP AG page 19/124

Figure 14 – SMQ2 qRFC Monitor for Inbound Queues

2.2.4.2 Outbound and Inbound Schedulers

qRFC makes use of two schedulers: an outbound scheduler and an inbound scheduler. The schedulers are intended to distribute the load over the available system resources; both schedulers come as SAP standard.

2.2.4.3 Outbound Scheduler

On the outbound side, the load imposed on the receiving system is limited by the number of queues. If a large number of queues can be filled simultaneously, queues do not sufficiently control resources on the receiving system. The outbound scheduler (transaction SMQS) provides more exact control over resources on the receiving system. You can configure the Outbound Scheduler to limit the number of requests sent at the same time to an RFC destination either via qRFC or via tRFC.


RFC Monitoring

© 2011 SAP AG page 20/124

Figure 15 – Transaction SMQS – Outbound scheduler configuration

Normally, the outbound scheduler does not work continuously; therefore, its status on queue level is usually shown as INACTIVE. It only starts working if new entries are stored in an outbound queue or if a tRFC is sent to a destination. The outbound scheduler is monitored and configured using transaction SMQS. You can monitor the scheduler status and register new destinations. To register a destination, choose the Registration button.

Some important parameters that can be set using the Outbound Scheduler are listed below:

MAXCONN: Max. connections for a destination (max. DIALOG work processes in sending and receiving system). Default is 1 for destination of type T (non-SAP) and 10 for destination of type 3 (SAP system).

MAXTIME: Max. runtime for a destination. Default is 60 seconds (should not be set lower). Defines the time how long the entries of this specific queue are processed, until the scheduler switches to the next pending queues and entries of the other queues are processed.

Maintenance of RFC Server Groups:

A group of application servers can be defined via RZ12 and assigned within SMQS to a destination. To assign an RFC server group for the outbound scheduler that will process the requests to be sent, choose Edit → Change AS group. This allows all requests to the destination to be processed in this server group only. The maximum number of dialog work processes used by the outbound scheduler can also be defined via RZ12. (See SAP Note 74141: Resources Management for aRFC/tRFC/qRFC.) In addition, a destination can also be excluded from being handled through the outbound scheduler; you can do this by registering the destination in SMQS and then selecting the destination and choosing Edit → Exclude. The destination then appears as type N, like destination PRD in the example above. If a destination is excluded from the Scheduler, the transmission is executed as before (immediately and asynchronously to the application). Bear in mind that this may cause resource bottleneck problems in connection with updates. (See SAP Note 484753, regarding a description of the different types of outbound scheduler registration.) The outbound scheduler runs with the server group DEFAULT as a presetting. This means that all application servers of the local SAP system are used to activate all the registered queues. The server group DEFAULT does not need to be created and it is not visible in transaction RZ12. The server group default should not be used in high-volume systems because it can lead to system resource overload.


RFC Monitoring

© 2011 SAP AG page 21/124

Figure 16 – Transaction RZ12 - RFC Server Group Maintenance

Figure 17 – Assignment of an RFC Server Group to the Outbound Scheduler

The QOUT scheduler is especially advantageous when many tRFCs are created within a short time or when many different queues to a single destination are used. In these cases, you can balance the workload by limiting the number of parallel tasks per destination. Only tRFCs/qRFCs using destinations that are registered in SMQS are processed by the outbound scheduler.

2.2.4.4 Inbound Scheduler

When inbound queues are used, all LUWs are transferred to the inbound queue before being processed. Normally, none of the LUWs written in the inbound queue are automatically executed. Each application can activate these queues using APIs (via calling FM TRFC_QIN_ACTIVATE or QIWK_REGISTER). To avoid each application having to write a specific inbound scheduler, qRFC supports a general inbound scheduler (transaction SMQR). The inbound scheduler is configured on the basis of queue names. If new queue names are used, they must be registered using transaction SMQR Registration or via FM QIWK_REGISTER otherwise their entries will not be processed.

Like the outbound scheduler, the inbound scheduler does not work continuously. Its status is usually shown as INACTIVE. It only starts working if new entries are stored in an inbound queue. The inbound scheduler does not process the LUWs but triggers their processing in asynchronously started work processes. For the inbound scheduler, it is also possible to create (transaction RZ12) and to assign (SMQR → EDIT → Change AS Group) an RFC server group to limit the resources available for processing. The inbound scheduler runs with the server group DEFAULT as a presetting and, in high-volume systems, the DEFAULT server group should not be used due to potential for system resource overload.


RFC Monitoring

© 2011 SAP AG page 22/124

Figure 18 – Transaction SMQR – Inbound Scheduler Configuration

Some important parameters that can be set while using the inbound scheduler are listed below:

EXEMODE: D for dialog, B for background, depending on the type of the work process that should be used for processing queue entries

MAXTIME: Time spent by the inbound scheduler for working on the queue. If this time limit is exceeded, but an LUW of a queue is still running, the scheduler completes this LUW. Afterward, the LUWs of other registered queues are distributed to work processes by the inbound scheduler.

USERDEST: Logical destination (defined in transaction SM59) for processing LUWs in this queue. This enables you to change the client, user, and language for all qRFC LUWs.

NRETRY: Number of retries

TDELAY: Delay between retries

See also SAP Note 369007 for configuration of the inbound scheduler.

2.2.5 Background RFC (bgRFC)

2.2.5.1 Introduction to the bgRFC

The background RFC (bgRFC) is offered as a replacement for the classic tRFC and qRFC. It is available with SAP NetWeaver 2004s (SAP Basis 7.00). With SAP NetWeaver 7.1 (SAP Basis 7.10), there are some changes regarding the configuration and monitoring. This document is based on the 7.10 version.

bgRFC is a superordinate term for the new version of tRFC and qRFC. A parallel run of classic tRFC/qRFC and bgRFC is possible.

The background RFC works on the basis of units and performs better compared to the classic tRFC and qRFC versions. It comes with a new API and data model.


RFC Monitoring

© 2011 SAP AG page 23/124

Terminology:

The term unit for the bgRFC can be compared with the term LUW for classic tRFC/qRFC.

o A unit is a recorded sequence of collected function calls to be performed remotely.

o The recorded unit data is persisted in the database at COMMIT WORK.

The term destination for a bgRFC unit defines either a remote system for outbound scenarios or a distinct name for an application in an inbound scenario.

o Outbound destinations are maintained via the standard transaction SM59.

o Inbound destinations are maintained via the specific transaction SBGRFCCONF.

The bgRFC is based on a scheduler-driven queuing framework. This means that remote function calls are recorded, and execution—which is controlled automatically by a scheduler process—takes place at a later point in time. Several schedulers can be started to process bgRFC units.

bgRFC supports the following scenarios:

Processing on a remote system (outbound scenario)

o Use case: Asynchronous transactional processing of function calls in a remote system; processing is controlled by the caller system (inter-system communication for SAP to SAP and SAP to non-SAP)

Processing remotely by the inbound scheduler (outbound-inbound scenario)

o Use case: Asynchronous transactional processing of function calls in a remote system; processing is controlled by the receiver system (inter-system communication for SAP to SAP)

Processing on the same system (inbound scenario)

o Use case: Asynchronous transactional processing of function calls in the same system (intra-system communication = same system and same client)

Figure 19 – Overview of bgRFC Scenarios

In addition, there is a successor to the qRFC's "No-Send" scenario, where outbound calls are recorded but not sent by the outbound scheduler. Instead, the receiving system is supposed to fetch its RFC records itself (pull principle). In the bgRFC context, this is called local data queue (LDQ).

Outbound Scenario

Outbound - Inbound

Scenario

Inbound

Scenario

Outbound Scheduler

Inbound Scheduler


RFC Monitoring

© 2011 SAP AG page 24/124

The bgRFC provides two qualities of service (QoS) for remote function calls (asynchronous transactional system-to-system communication):

Exactly once (EO) => transactional units (like the classic tRFC)

o Each unit is an independent thread.

Exactly once in order (EOIO) => queue units (like the classic qRFC)

o Units have sequence dependencies.

Valid Combinations of QoS and Scenario:

QoS Inbound Outbound Outbound to Inbound

(No-Send)

tRFC (EO) X X - -

qRFC (EOIO) X X X X

2.2.5.2 Advantages of the bgRFC

The primary goal of the bgRFC project is to improve the runtime behavior of qRFC while remaining compatible with the qRFC‘s existing protocol and avoiding downtime in affected systems when customers upgrade to the new version. The improved design processes large amounts of highly sequential data, such as the data found in SCM scenarios, much more efficiently. The new bgRFC is available alongside qRFC. It is not possible to mix the new procedure with the classic functions. It is possible, however, to use both procedures in parallel. Since there is no overlap between the two procedures, and in an effort to prevent errors, you specify which procedure is to be used for outbound processing at the destination. That means all transactions that define dependencies between their qRFC calls have to either switch to the new bgRFC procedure or stick to the existing procedure. Synchronous and asynchronous RFC are not affected.

The classic qRFC model detects the dependencies among the individual units only when the data is processed by the RFC scheduler. For each destination, the outbound scheduler starts a destination scheduler that processes the data for a specific destination. The destination schedulers run on each destination only for a limited period of time, to balance the load between all destinations. Before each destination is processed, the order of the destinations must be determined again by the destination scheduler.

The new bgRFC design takes care of this problem by detecting the dependencies at the time the data is stored. In doing so, the RFC scheduler can find all units that can be executed instantly with minimum effort, and all dependencies are detected only once. The additional effort when storing the data is compensated to a large extent by efficient algorithms and optimizations in the database design.

For each client, a number of outbound schedulers are started that share the workload. The new RFC schedulers react more sensitively to the load in the target systems. In the previous model, the load balancing was based on the concept of the logon groups. This means that the information about the load in the destination systems available at runtime could be up to five minutes old. This information is updated at much shorter intervals in the improved bgRFC design.

The gateways, another potential source of bottlenecks, have also been optimized. The new concept regulates the maximum number of outbound schedulers that are allowed to run at the same time on an application server, and the maximum number of connections that all RFC schedulers are allowed to use. This limitation prevents the local gateway from becoming overloaded. The gateways of the destinations are also protected from overload by making the number of parallel outbound schedulers for each sending system and their maximum number of connections configurable.

Important difference:

For the bgRFC, there is no registration of queue names. In contrast to qRFC inbound, you cannot register or deregister queues to control whether requests should be processed automatically by the scheduler or not.


RFC Monitoring

© 2011 SAP AG page 25/124

The property that determines whether a bgRFC unit shall be processed automatically or manually via the monitors must be set for each particular unit at the time of creation.

2.2.5.3 Basic Syntax for the bgRFC

The new syntax for calling a function module for background processing (previously done using syntax IN BACKGROUND TASK) is as follows:

CALL FUNCTION 'function_name' IN BACKGROUND UNIT unit EXPORTING … TABLES …

The object reference unit points to an object of the type IF_BGRFC_UNIT. This object contains all information that is needed to run the function module in the background. This information is comprised of, for example, the execution destination, the distinction between qRFC and tRFC, and probably the number of queues across which the unit is spread. If the same unit object is used with multiple function module calls, then all these function modules are executed in one unit. It is possible to use multiple unit objects, in parallel, within a single application LUW. Consequently, multiple background units can be created in parallel within an application LUW. In the previous version, this was possible using the AS SEPARATE UNIT syntax addition. The advantage of the new solution is that multiple background units can accept multiple function modules. In the event of an invalid unit object, the exception CX_BGRFC_INVALID_UNIT is raised.

A destination object is required to send a unit. The destination object represents one execution destination or a set of execution destinations. Destination objects can be requested using the class methods from the class CL_BGRFC_DESTINATION_OUTBOUND for the outbound queue and the class CL_BGRFC_DESTINATION_INBOUND for the inbound queue. The destination for the outbound queue must be maintained in transaction SM59 and represents an execution destination for the outbound bgRFC type t and type q. The destination for the inbound queue must be maintained in transaction SBGRFCCONF. If the destination is not valid, the exception CX_BGRFC_INVALID_DESTINATION is produced.

Note: In the actual remote function call (CALL FUNCTION … IN BACKGROUND UNIT…), there is no DESTINATION keyword anymore.

Programming example of a transactional outbound bgRFC:

DATA: my_dest TYPE REF TO if_bgrfc_destination_outbound, my_unit TYPE REF TO if_trfc_unit_outbound. my_dest = cl_bgrfc_destination_outbound=>create( 'MY_DEST' ). my_unit = my_dest->create_trfc_unit( ). CALL FUNCTION 'MY_RFC_FUNCTION' IN BACKGROUND UNIT my_unit. COMMIT WORK.

This would create a transactional bgRFC call of function module MY_RFC_FUNCTION towards the outbound destination MY_DEST.

2.2.5.4 Configuration of the bgRFC

Basic settings in transaction SBGRFCCONF:

Creation of a supervisor destination in transaction SM59 to define the user under which the scheduler is running. Without this supervisor destination, the schedulers cannot be started. The created supervisor destination is entered into transaction SBGRFCCONF and is, afterward, locked in SM59.


RFC Monitoring

© 2011 SAP AG page 26/124

Figure 20 – Transaction SBGRFCCONF

Creation of separate inbound destinations for each application to distribute the load.

o Assign existing queue prefixes for a ―classic‖ qRFC to an inbound destination.

o Maintain logon server groups to an inbound destination for load distribution. They must be defined in transaction RZ12 or SMLG.

Figure 21 – Transaction SBGRFCCONF: Load Distribution

Customizing of the bgRFC scheduler in transaction SBGRFCCONF:

To optimize the bgRFC function in terms of system performance, you can make various settings for the bgRFC schedulers. This can be done on three levels:

System

Application servers

Destinations

This allows a specific configuration for different servers in a heterogeneous system landscape, to reach optimum throughput. There is independent customizing for the outbound and inbound schedulers.

Transaction

SBGRFCCONF

Transaction

SM59


RFC Monitoring

© 2011 SAP AG page 27/124

System-specific settings define parameters like number of messages per log and lifetime of log entries, compression settings, and deletion time for units (for the asynchronous physical deletion).

Application-server-specific settings define parameters like number of outbound or inbound schedulers per application server, maximum number of connections and destinations per server, percentage of available gateway resources, scheduler idle time, and maximum lifetime of load balancing entries


o Settings for "Scheduler Count" (Number of running bgRFC schedulers): -1: The number of schedulers is determined methodically. 0: Application server/destination is locked for bgRFC. >0: Maximum number of running schedulers for application server/destination.


RFC Monitoring

© 2011 SAP AG page 28/124

Destination-specific settings define parameters like number of schedulers for destination, maximum number of automatic retries per unit, wait time per unit and destination, maximum processing time for a destination, and number of open connections per destination.


2.2.5.5 bgRFC Monitoring

Features of the monitoring transaction SBGRFCMON:

The selection screen allows filtering on inbound and/or outbound units of type transactional and/or queued, as well as additional criteria like status, destination name, queue name, user name, or program/transaction name.

Hierarchical representation of destinations, queues and units:

o Details for each unit, for example, administrative data, queue name, or list of function module names

Set or remove locks on destinations, queues, or units

Deleting queues or units

Direct access to customizing, destination maintenance, and supportability tools

Access to supportability tools:

o Debugging of a bgRFC unit

o Runtime analysis for a unit

o Tracing a unit

o Showing unit history → calling transaction SBGRFCHIST

o bgRFC Performance Monitor → transaction SBGRFCPERFMON


RFC Monitoring

© 2011 SAP AG page 29/124

Monitoring transactional units in transaction SBGRFCMON:

Monitoring queued units in transaction SBGRFCMON:

Context menu for destination/queue:


RFC Monitoring

© 2011 SAP AG page 30/124

Context menu for destination/queue:

2.2.5.6 Troubleshooting for bgRFC

Showing unit history in transaction SBGRFCHIST:

The unit history shows the chronological order of status changes during the processing lifetime of a bgRFC unit.

The unit history can also be called per transaction code SBGRFCHIST directly. In this case, you first get an overview about all bgRFC units per selection criteria. Then you can choose an individual unit to see its status history.


RFC Monitoring

© 2011 SAP AG page 31/124

Display runtime analysis for a bgRFC unit:

The processing of a bgRFC unit can be executed with an SE30 runtime measurement. It can be displayed directly after processing from within the monitoring transaction SBGRFCMON.


RFC Monitoring

© 2011 SAP AG page 32/124

Display bgRFC unit trace:

The processing of a bgRFC unit can be executed with a trace. The trace can be displayed automatically after processing from within the monitoring transaction SBGRFCMON, or directly per transaction ST11 (filter column file name on string dev_bg*).

Application log for bgRFC:

The bgRFC writes various log entries into the application log. Call transaction SLG1 for log object BGRFC (there are several sub-objects available).

Example application log for customizing changes in transaction SBGRFCCONF:


RFC Monitoring

© 2011 SAP AG page 33/124

Example application log for manual actions in monitoring transaction SBGRFCMON:

bgRFC performance monitor (transaction SBGRFCPERFMON):

2.2.5.7 LDQ Monitoring

The monitor for the local data queue (LDQ) is called with transaction SLDQMON. It provides a selection screen for filtering on LDQ application and/or queue name. The left-hand tree control shows all LDQ applications with their corresponding queue names. On the right-hand side of the screen, for each queue, there is a list of LDQ units within a queue. Choose the Contents button to show a dialog box with the unit's character or binary format content.


RFC Monitoring

© 2011 SAP AG page 34/124

2.3 Dependencies and Convergence of Technologies

2.3.1 Queue Dependencies for qRFC When qRFC is used for serialization, dependencies are established between function modules. It is possible for multiple function modules of the same LUW to be processed in multiple queues. This is achieved using the following syntax:

Figure 24 – Programming LUW with Entries in Multiple Queues

In the example above, the two function modules ‗ABC‘ and ‗XYZ‘ make up one LUW. Each function module is placed in a different queue, that is, My Depending Q1 and My Depending Q2. All function modules of the different queues that are part of the same LUW are processed at once and are ended with one database commit. To ensure that these function modules are processed in the correct order on the receiving system, the qRFC manager sets a counter (counter is created automatically on commit work by the qRFC Manager to serialize the processing of those LUWs that extend over multiple queues. For the changes to be committed together, the function modules of the different queues that are part of this LUW must be at the front of all these queues. In the figure below, queue 2 is currently not processed further because its first LUW extends across queue 1, and this LUW in queue 1 is preceded by another unrelated LUW.


RFC Monitoring

© 2011 SAP AG page 35/124

Figure 25 – LUWs Spanning Across Multiple Queues

In the SAP system, these dependencies can be monitored using the qRFC monitor. The figure below shows three queues containing the function module entries of two LUWs. The first LUW extends over the queues My Depending Q1 and My Depending Q2 and the second LUW extends over My Depending Q2 and My Depending Q3. After selecting all queues and displaying them, you can see further details. In the following example, the inbound qRFC monitor shows Q3 waiting for Q2. It is waiting because the first LUW is not yet processed.

Figure 26 – Monitoring Dependencies Across Different Queues


RFC Monitoring

© 2011 SAP AG page 36/124

It is also possible to display all queues that are involved in the LUWs of one queue. This can be done by double-clicking on that queue in the queue monitor (see figure below). Double-clicking on the queue My Depending Q2 shows that one LUW has function modules in My Depending Q1 and a second LUW has function modules in My Depending Q3.

Figure 27 – Monitoring Dependencies Across Different Queues (2)

2.3.2 Queue Dependencies for bgRFC

Also for the bgRFC, a queued unit (LUW) can be stored across several queues. This is reached by handing over a list of queue names (instead of one queue name only) to the unit object.

The example above shows a queued bgRFC unit for function module RFC_PING. It is spread over two separate queues called Z_TEST_QUEUE_1 and Z_TEST_QUEUE_2. The dialog box shown on the screenshot can be called by choosing Display Corresponding Queues in the context menu of the bgRFC unit. Clicking in column Queue Name focuses the cursor in the left-hand side tree control.


RFC Monitoring

© 2011 SAP AG page 37/124

2.3.3 Mixed-Mode RFC There is a special feature called mixed mode that allows qRFC calls with send queue and normal tRFC calls with the same destination to be processed together within one LUW. If, within an LUW in mixed mode, the first call is a normal tRFC call, the function module TRFC_QUEUE_INITIALIZE is called first (see figure below). This ensures that the current LUW is processed via qRFC with send queue. If, within an LUW in mixed mode, the first call is a qRFC call, this initialization is done by function module TRFC_SET_QUEUE_NAME.

Figure 28 – Syntax for Implementing Mixed Mode

When mixed mode is implemented, any tRFCs are processed as qRFCs. Monitoring can be performed in SMQ1 and also in transaction SM58. Transaction SM58 (tRFC monitor for displaying and editing the tRFC transactions) now also allows processing of qRFC transactions. If you delete a tRFC entry with this transaction, the system automatically deletes the corresponding entries in the send queue table, if they exist. If you start an LUW, the system does not immediately transfer this LUW, but first checks whether this LUW needs serialization or whether it must wait due to predecessors in the queue.


RFC Monitoring

© 2011 SAP AG page 38/124

2.4 Automatic Switching from qRFC to tRFC

If the execution of a qRFC LUW is terminated by the error SYSTEM_FAILURE, the queue is blocked. This can result in backlogs in the queues. If you want to use qRFC instead of tRFC for better system performance, and not for serialization, it is possible to set the optional parameter TRFC_IF_SYSFAIL to X while you call the function module TRFC_SET_QUEUE_NAME. This results in all LUWs with the error message SYSTEM_FAILURE being converted to tRFC LUWs, enabling the qRFC manager to execute the next LUW in this queue. The LUWs that are converted to tRFC can then be administered/monitored using transaction SM58.

Figure 29 - Syntax for Implementing Automatic Switching in Case of SYSFAIL


RFC Monitoring

© 2011 SAP AG page 39/124

3 Aspect: Error Monitoring

From the previous sections, you know the general RFC processing mechanism. In the next sections, individual aspects of interface monitoring are discussed. The possibilities for setting up a selective and automated RFC monitoring within SAP Solution Manager are outlined in the section Setup Procedure for Automated Interface Monitoring.

RFC error monitoring intends to detect critical situations and exceptional cases during interface processing. As these incidents can endanger the entire data exchange between the systems involved, their identification can be extremely important and time-critical. Monitoring procedures should therefore ensure a clear visualization and prioritization of incidents to support their fast and effective resolution. Error monitoring includes the monitoring of all error situations documented within the system. Error situation can be identified by the occurrence of error messages or error statuses.

Monitoring Requirements:

As previously discussed, RFC communication provides five different techniques that can be used to meet different business requirements. In a typical solution landscape, a combination of the various RFC technologies is used. To monitor the different RFC technologies, SAP provides various tools. The graphic below illustrates the various technologies and the SAP tools that can be used to monitor RFC scenarios manually in a solution landscape.

Figure 30 – Standard RFC Monitoring Tools

3.1 Manual Error Monitoring

RFC communication can fail on either the sender system or the receiving system, so the sender and receiver will represent our two monitoring perspectives.


RFC Monitoring

© 2011 SAP AG page 40/124

The following table documents the handling of errors on the sender and receiver systems for the five types of RFC.

Monitoring Object

Monitor TA/Tool

Monitoring Freq.

Indicator or Error

Monitoring Activity or Error Handling Procedure

Respon-sibility

Escalation Procedure

RFC not being processed (sender system)

SM50/SM66

As required by business and when there are perform-ance problems

RFCs not processed. Multiple work processes remain in status STOPPED/HOLD with reason CPIC/RFC.

Run transaction SM50/SM66 and monitor work processes.

If many work processes in status STOPPED/HOLD and reason CPIC/RFC, check work process trace files for errors relating to CPIC/RFC by choosing the relevant work process and using transaction SM50 → [Menu] Process → Trace → Display File.

For further analysis, double-click on the work process in transaction SM50 when it is in status STOPPED/HOLD reason CPIC/RFC to display conversation ID. Next step is to run transaction SMGW to display a list of active connections on the gateway. Look for corresponding conversation ID and double-click on the conversation ID to get details about connection. Run transaction ST11 and search in the dev_rd and dev_rfc<n> trace files for conversation ID – examine reason for error.

Increase the individual trace levels if not enough detail is available. See SAP Note 532918 for further details.

This problem can be the result of insufficient resources in receiver system, so you should also check resource configuration in target system using transaction SARFC.

SAP Technology Operations team


Communication SM59 When there Commu Run transaction SM59. SAP SAP


RFC Monitoring

© 2011 SAP AG page 41/124

/connection problem (sender system)

are connection/communicat-ion problems

nication error- CPI-C errors

Choose RFC destination → [MENU] Test → Connection (to test if the connection to target system is working).

It is also possible to make an authorization check, i.e., SM59. Choose RFC destination → [MENU] Test → Authorization (to check if the user is authorized in the target system).

If an error occurs, double-click on the error message to get further details.

In addition, RFC Trace can be displayed, i.e., SM59 [Menu] RFC Display Trace. This will provide greater detail of any error.

Technology Operations team

Technology Operations team

Communication/connection problem (sender system)

SM21 Daily/As required by business

Communication errors, i.e., CPI-C errors

Run transaction SM21. Look for occurrence of communication CPI-C errors. Double click on error message to get details. Use SAP Note 63347 to translate the SAP and CPI-C return codes.



Communication/connection error (sender system)

ST11 and AL11 → work directory

When there are communication/connection problems

Communication error- CPI-C errors

Run transaction ST11 to display error log files. Communication errors are logged for the following components: RFC layer (dev_rfc<n>), Gateway (dev_rd), and work processes dev_w<n> (where <n> is the work process that handled the RFC call).

The above files can also be found using the following methods:

Using transaction AL11 → navigate to ‗work‘ directory.




RFC Monitoring

© 2011 SAP AG page 42/124

RFC Trace can also be displayed using

SM59 → [Menu] RFC →

Display Trace.

Gateway trace can also be displayed using

SMGW → [Menu] Goto

→ Trace → Gateway →

Display file.

Work process Trace can

also be displayed with

SM50 → Select a

process [Menu] Process → Trace → Display file.

In error situations, you can increase the trace levels to get more detail. (See Note 532918.)

(Trace levels should be changed back to default levels after solving error situation; otherwise, problems with file system space can arise).

Communication/connection error involving communication with external programs

SMGW and SM59

When there are connection/communication problems

Communication error- CPI-C errors

Run transaction SMGW → [Menu] Go To → Trace → External Programs → Activate. This will generate RFC (dev_rfc) and CPI-C (CPICTRC<pid>) trace files on all external programs which are called.

Alternatively, RFC and CPIC trace files can be generated for a specific external program. This is done using transaction SM59 → Special Options TAB, Set the RFC trace flag. This is propagated to the external program and the CPIC and RFC trace files are generated. This can also




RFC Monitoring

© 2011 SAP AG page 43/124

be achieved by registering the destination in transaction SM54.

For further information, including location of generated trace files, review SAP Note 47682.

Communication/connection error (communication is started from the external client program non-SAP System)

N/A When there are connection/communication problems

Communication errors

Activate CPI-C trace by including the parameter CPIC_TRACE=<tracelevel> in the sideinfo file or by

Setting the CPIC_TRACE environment variable in the external client system. The generated CPI-C trace file (CPICTRC<pid>) can be found in the directory in which the external program was started. For further information, review SAP Note 47682.



Communication/connection error (communication is started from the external client program non-SAP System)

N/A When there are connection/communication problem

Communication errors

Activate RFC trace by including the parameter RFC_TRACE=<tracelevel> in the sideinfo file or by

setting the RFC_TRACE environment variable in the external client system. For further information, including location of generated trace files, review SAP Note 532918.



Runtime error in interface (sender/Receiver system)

ST22 Daily/As required by business

Runtime errors

Run transaction ST22 and select time period. Look out for errors of type RFC*, CALL_FUNCTION*. Double-click on dump to obtain greater details.



Transactional RFC‘s stuck (sender system)

SM58 Daily/As required by business

Messages ―Hanging‘‘ or in status ‗Error‘

Run transaction SM58 to check state of RFC communication layer.

Choose a time frame to display a list of tRFCs that are ―hanging‖ or are in status error. To get more details for each tRFC, double-click.



qRFC not processed (sender system)

SMQ1 Daily/As required by business

Messages buildup in queues

Run transaction SMQ1 and choose various selection criteria. Look out for queues with many entries and/or queues that are static




RFC Monitoring

© 2011 SAP AG page 44/124

and queue in status error

(entries not getting processed). Select queue by double-clicking to get queue status (see SAP Note No. 378903 for description of status codes). Double-click again to get status of individual LUW that is responsible for error status of queue.

qRFC not processed (receiver system)

SMQ2 Daily/As required by business

Messages buildup in queue and queue in status error

Run transaction SMQ2 and choose various selection criteria. Look out for queues with many entries and/or queues that are static (entries not getting processed). Select queue by double-clicking to get queue status (see SAP Note No. 378903 for description of status codes). Double-click again to get status of individual LUW that is responsible for error status of queue.



tRFC/qRFC not processed (sender system)

SMQS As required by business

Outbound scheduler in status CPICEROR SYSFAIL

Run transaction SMQS and check status of ‗outbound scheduler‘. Double click on the status for detailed error information. (May be necessary to combine this analysis with analysis of system log (SM21) and the error logs i.e. dev_rfc*, dev_rd, dev_w*).




SMQR As required by business

Inbound scheduler in status CPICEROR SYSFAIL

Run transaction SMQR and check status of inbound scheduler. Double-click on the status for detailed error information. (May be necessary to combine this analysis with analysis of system log (SM21) and the error logs, i.e., dev_rfc*, dev_rd, dev_w*).



qRFC not processed (sender/receiver system)

SMQE When there are communication problems. For error analysis only

Error in log file

Activate qRFC log and or qRFC trace by running transaction SMQE → [Menu] QRFC → Trace → Activate Trace and transaction SMQE → [Menu] QRFC → Log → Activate Log (this should only be performed when analyzing errors because activating this has an




RFC Monitoring

© 2011 SAP AG page 45/124

overhead). For further details, see http://help.sap.com/s

aphelp_sm32/helpdata/en/4f/e05f3ca2de8974e10000000a114084/frameset.htm

RFC not processed

SM04 In case communication errors for a specific user

Error in log files

Run transaction SM04 and select user → [MENU] USER → Technical Information. This provides detailed user information. Take note of Conversation ID.

Next step is to run transaction SMGW to display a list of active connections on the gateway. Look for corresponding conversation ID, double-click on the conversation ID to get details about connection. Run transaction ST11 and search in the dev_rd and dev_rfc<n> trace files for conversation ID – examine reason for error.

Increase the individual trace levels if not enough detail is available. See SAP Note 532918 for further details.

RFC not processed (sender/receiver system)

SM04 → Trace

When there are error situations for a specific user

Error in log file

Activate the RFC trace generation for a logged-on R/3 user by selecting the user in question in Transaction SM04 and

choosing Edit → Trace →Trace on (in older releases) or

User → Trace → Activate. You can specify the R/3 components that

should write trace information via the display components in Edit → Trace → 'Display trace' (older releases) or in

User→Trace → Display. For the RFC trace, at least the task handler and the ABAP processor should always be



http://help.sap.com/saphelp_sm32/helpdata/en/4f/e05f3ca2de8974e10000000a114084/frameset.htm





RFC Monitoring

© 2011 SAP AG page 46/124

marked here. This activation has the effect that all the RFC links of the user in question are documented in the corresponding trace files of the application server where the activation via SM04 took place. If such a user-specific trace generation occurs, the user is highlighted in yellow in SM04.

bgRFC not processed

SBGRFCCONF

On demand Communication errors, queue backlog

Use transaction SBGRFCCONF to check configuration of inbound schedulers and outbound destinations.



bgRFC not processed or processing failed

SBGRFCMON and SBGRFCHIST

Daily / As required by business

Runtime errors, queue backlog

Use monitoring transaction SBGRFCMON. Check status of destinations and queues. Check unit history (transaction SBGRFCHIST) and traces (transaction ST11 for files "dev_bgRFC*"). Check application log (transaction SLG1) for log object BGRFC.



**When activating traces, ensure that the following dynamic profile parameters are set to the value "0" on all application servers that are not relevant for trace generation while the problem analysis is being carried out: gw/accept_remote_trace_level and rdisp/accept_remote_trace_level. For further information, see SAP Notes 573800 and 558254.

3.2 Automated Error Monitoring

Different types of errors can occur during RFC processing, depending on the program or function called and its exception handling. It could be errors written into the system log (/nSM21), ABAP dumps (/nST22), or application log entries (/nSLG1) that are important to be monitored.

For an automated error monitoring, the recommendation would be to set up a selective automated monitoring for errors that occur often.

Automated Monitoring Objects with SAP CCMS or SAP BPMon and SAP Solution Manager

Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


All RFC types (including bgRFC)

System Log (/nSM21) or ABAP Dumps (/nST22) Monitoring in SAP CCMS / SAP BPMon / System

Hourly Entries for the monitored object and sub-object (application errors in called programs)

Analyze the reason starting from transactions /nSM21 and/or /nST22.


Application Management team / business process champion


RFC Monitoring

© 2011 SAP AG page 47/124

Monitoring in SAP Solution Manager

All RFC types Application Log Monitoring (/nSLG1) in Business Process Monitoring in SAP Solution Manager

Hourly Entries for the monitored object and sub-object (application errors in called programs)

Analyze the reason starting in transaction /nSLG1.

Business Process Operations Team


bgRFC Application Log Monitoring (/nSLG1) and special bgRFC Monitor in Business Process Monitoring in SAP Solution Manager

Hourly Specific entries for the bgRFC processing with log object BGRFC (configuration, monitoring log, scheduler log)

Analyze the reason starting in transaction /nSLG1.


Application Management team

All RFC types RFC connection Monitoring in SAP Solution Manager as outlined in section ―Automated RFC Connection Monitoring‖

Hourly Connection problems due to system unavailability

Analyze the reason in transactions SM59.



qRFC Number of specified queues in status ―blocked‖ monitored in SAP CCMS or SAP Solution Manager BPMon (refer to section ―Automated qRFC Monitoring‖)

Hourly Queues in status ―blocked‖

Analyze the reason in transactions /nSMQ1 or /nSMQ2.

Business Process Operations team / SAP Technology Operations team


tRFC Errors in tRFC processing monitored in SAP CCMS / SAP BPMon in SAP Solution Manager (refer to section ―Automated

Hourly CPICERR, SYSFAIL, SYSLOAD errors

Analyze the reason in transaction /nSM58.




RFC Monitoring

© 2011 SAP AG page 48/124

tRFC Monitoring‖)

- For details on how to set up the automated monitoring of the application log, refer to the general Business Process Monitoring setup guide: http://service.sap.com/~sapidb/011000358700006137532006E

- For details on how to setup the automated monitoring of the system log or the ABAP dumps, refer to http://service.sap.com/solutionmanagerbp → Topic Area → System Monitoring within SAP Solution Manager.

- For details on how to set up the automated qRFC or tRFC error monitoring, refer to section Setup Procedure for an automated Interface Monitoring.

- For more specific details on how to set up the BPMon data collectors for qRFC, tRFC, and bgRFC monitoring, refer to the BPMon/IFMon setup guide: http://service.sap./com/bpm → SAP Business Process Management → Media Library → Technical Information → Setup Guide - Interface Monitoring https://websmp208.sap-ag.de/~sapdownload/011000358700000258142008E/Setup_Guide_IFMon.pdf

3.3 Troubleshooting: Error Examples and Solutions

3.3.1 Error – CALL_FUNCTION_REMOTE_ERROR

If you find the ABAP runtime error (ST22) CALL_FUNCTION_REMOTE_ERROR occurring with your custom programs, check if the default exceptions COMMUNICATION_FAILURE and SYSTEM_FAILURE are handled by the custom program. If these exceptions are not handled, this is an application problem, which must be analyzed and processed by the responsible application department. Refer to SAP Note 97522 for further details.

Figure 31 – ST22 CALL_FUNCTION_REMOTE_ERROR (ABAP Runtime Error ST22)

3.3.2 External Program Not Registered on Gateway

Problem:

After the upgrade to SAP R/3 Enterprise 4.70, the connection to the archiving system was often lost. Analysis showed the following:

http://service.sap.com/~sapidb/011000358700006137532006E

http://service.sap.com/solutionmanagerbp

http://service.sap./com/bpm

https://websmp208.sap-ag.de/~sapdownload/011000358700000258142008E/Setup_Guide_IFMon.pdf



RFC Monitoring

© 2011 SAP AG page 49/124

Figure 32 – External Program Not Registered in Gateway (SM59, SM21, and Developer Rraces)

Cause:

The external program is not registered on the gateway.

Solution:

Register the program on the gateway.

See SAP Notes 63930 and 353597.

3.3.3 External Program Timeout on Gateway Problem:

IDocs could not be sent from SAP R/3 to the third-party product, though the incoming process to SAP R/3 has no problem. Analysis showed the following:

Problem

during

initiation of

connection

Problem

during

initiation of

connection

Problem

during

initiation of

connection


RFC Monitoring

© 2011 SAP AG page 50/124

Figure 33 – External Program Timeout in Gateway (SM59 and Developer Traces)

Cause:

The program is registered on the gateway and is currently being used by a process. Another process attempts to access the registered program but cannot because it is already in use. The system waits a certain period of time (gw/reg_timeout); if it still cannot access the registered gateway program after this specified time period, it terminates with the error message ―timeout during allocate‖.

Solution:

Check if you can register the program on the gateway more than once, referring to SAP Note 353597.

Otherwise, increase the parameter gw/reg_timeout in transaction RZ11.

Review SAP Note 581509.

3.3.4 Timeout During Connection Setup Problem:

After the migration to another operating system, the RFC connections failed.

Timeout

when

establishing

connection

Timeout

when

establishing

connection

Timeout

when

establishing

connection


RFC Monitoring

© 2011 SAP AG page 51/124

Figure 34 – Timeout During Connection Setup (Developer Trace)

Cause:

The target system DEF was not reached.

Solution:

Test the RFC connection with SM59 and check if the user is authorized in the target system.

To make sure the target system can be reached you need to check the entries in the \etc\hosts file. All the listed hosts need to be reachable. Check vor valid IP adresses and if they are resolvable – ping the hostname, run nslookup to the <IP>, it should give back the correct IP adress and hostname (full qualified domain name).

If this is properly working you have to check all entries in the used property files, e.g. profile files, configuration files. Also check the environment variables and the \etc\<services> file.

3.3.5 Active Connection Table is Full Problem:

The central instance communication table is full, and de-allocated connections cannot be cleared out.

Figure 35 – Active Connection Table Full (Developer Trace)


RFC Monitoring

© 2011 SAP AG page 52/124

Cause:

Many old connections remain in the communication table.

Solution:

Display the communication table to check number of entries. (The size of the table or the number of entries is set in the parameter rdisp/max_comm_entries. The value should be at least as big as the value of parameter gw/max_conn.):

SM51 → Select a server → Goto → Server information → Communication table

Delete the old connections:

Logged on client: SMGW → Goto → Logged on Clients → Delete Client.

Normal gateway connection: SMGW → Goto → Active Connections → Delete connection

Adjust the gateway parameters, referring to the help document.

SMGW → Goto → Parameters → Display (to make them active, maintain the instance profile)

gw/cpic_timeout

gw/keepalive

gw/cpic_timeout

gw/gw_disconnect

3.3.6 SQL Error 60 When Accessing Table ARFCSSTATE Problem:

Many short dumps were generated related to table ARFCSSTATE and ARFCSDATA in the productive system.

http://help.sap.com/saphelp_nw04/helpdata/en/21/c6a73860bf5f2ce10000009b38f8cf/content.htm

http://help.sap.com/saphelp_47x200/helpdata/en/21/c6a73860bf5f2ce10000009b38f8cf/content.htm


RFC Monitoring

© 2011 SAP AG page 53/124

Figure 36 – SQL Error 60 When Accessing Table ARFCSSTATE (ABAP Runtime Error - ST22)

Cause:

One user is temporarily blocking a table that another user wants to access.

Solution:

Make sure that you have applied the latest RFC environment. You can use transaction SMQS – Information – Version, to do so. It should read: Current qRFC Version: 6.30.060; Supplement Number: 11

In case the RFC tables ARFCSSTATE and ARFCSDATA are growing too big, refer to SAP Notes 375566 and 324545 to maintain them in order to prevent deadlock situations.

3.3.7 CPIC Timeouts During Connection to External Program Problem:

Reports could not be generated successfully due to communication failures to the third-party product.


RFC Monitoring

© 2011 SAP AG page 54/124

Figure 37 – CPIC Timeouts During Connection to External Program (Developer Traces)

Conversation ID


RFC Monitoring

© 2011 SAP AG page 55/124

Conversation ID


RFC Monitoring

© 2011 SAP AG page 56/124

Figure 38 – CPIC Timeouts During Connection to External Program (CPIC Trace)

Cause:

The connection setup to an external program does not succeed within the time period determined by gw/cpic_timeout. The SAP Gateway then terminates the connection setup and reports an error. See SAP Note 516073.

Solution:

Contact the third-party product‘s support to investigate why the external program did not respond within 20 seconds.

Adjust the parameter gw/cpic_timeout.

3.3.8 Socket Too Large

Problem:

The standalone gateway that handles the RFC calls to the third-party product stopped.


RFC Monitoring

© 2011 SAP AG page 57/124

Figure 39 – Socket to Large (Developer Traces)

Analysis: Using transaction SMGW, type NEWG in the command field (to go to another gateway for monitoring, that is, standalone gateway instance). It was observed that the number of active connections for the standalone gateway was at its maximum, that is, 2048.

Cause:

Up to SAP Basis release 6.20, the gateway is only able to do 2048 connections simultaneously. Above that number, the system has reached its limit.

The RFC connection is not closed in the customer program after the external program completes its task and there is no RFC timeout. Therefore, more and more RFC connections are established until they reach the limit of 2048 connections.

Solution:

Redesign the external program so that it will close the RFC connections.


RFC Monitoring

© 2011 SAP AG page 58/124

4 Aspect: Backlog Monitoring

Backlog monitoring enables you to proactively monitor the number of messages that are processed or have not been processed, in a defined time window. Based on this information, a reporting on the interface throughput can be set up. This might serve as an indicator of delays in business-critical data flows, as well as an indicator for interacting applications if the data volume processed exceeds or falls below defined threshold values. This gives the corresponding organizational units the time necessary to adapt their operations to the upcoming increase or decrease of unfilled work items.

Backlog situations during RFC processing can negatively affect the overall system performance and delay the underlying business processes.

4.1 Manual Backlog Monitoring

Monitoring Objects w/o SAP Solution Manager

Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


Transactional RFCs stuck (sender system)

SM58 Daily/as required by business

Slow processing

Call transaction SM58 and specify time frame. Choose execute and look at number of entries in table. Investigate reason for tRFCs being stuck in the tRFC queue. Status text ―Transaction recorded‖ indicates lack of resources in the target system.

Ensure that background job

RSARFCEX is scheduled and running successfully: restarts messages that fail due to the following reasons/status:

Communication errors (CPIC), Recorded, System error (SYSFAIL), Terminated due to overload (LOAD), Temporary application error (RETRY), or Serious application error (NORETRY).



qRFC not processed (sender system)

SMQ1 Daily/as required by business

Message buildup in queue and queue in status error

Run transaction SMQ1 and choose various selection criteria. Look for queues with many entries and or queues




RFC Monitoring

© 2011 SAP AG page 59/124

that are static (entries not getting processed). Select queue by double-clicking to get queue status (see SAP Note 378903 for description of status codes). Double-click again to get status of individual LUW that is responsible for queue backlog.


SMQ2 Daily/as required by business

Message buildup in queue and queue status error

Run transaction SMQ2 and choose various selection criteria. Look for queues with many entries and or queues that are static (entries not getting processed). Select queue by double-clicking to get queue status (see SAP Note 378903 for description of status codes). Double-click again to get status of individual LUW that is responsible for queue backlog.



bgRFC not processed (sender and receiver system)

SBGRFCMON / SBGRFCCONF

Daily/as required by business

Message buildup in queue and queue status error

Run transaction SBGRFCMON. Look for destinations and queues with high number of unprocessed units. Run transaction SBGRFCCONF to check the status of inbound schedulers and outbound destinations.



4.2 Automated Backlog Monitoring

The CCMS and Business Process Monitoring within SAP Solution Manager provide the possibility for backlog monitoring of the qRFC processing. You can bundle queues that logically belong together to queue groups, and activate the monitoring on the number of entries within the group. If the number of entries of a queue would exceed a certain value, it might be an indicator for a backlog situation.

In certain cases you might need to monitor a queue on the age of the oldest entry. If there are entries older than a specified number of days, this can also indicate a backlog situation, even if the group consists of only a few entries.


RFC Monitoring

© 2011 SAP AG page 60/124

Automated Monitoring Objects with SAP Solution Manager

Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


tRFC backlog monitoring – by the number of entries

BPMon in SAP Solution Manager

Hourly Too many entries per specified destination might indicate a backlog situation.




tRFC backlog monitoring – by the age of entries


Hourly Old entries per specified destination can indicate a backlog situation.




qRFC backlog monitoring – by the number of entries

CCMS/ BPMon in SAP Solution Manager

Hourly Too many entries in the specified queue group might indicate a backlog situation.




qRFC backlog monitoring – by the age of entries

CCMS/ BPMon in SAP Solution Manager

Hourly Old entries in the queue group can indicate a backlog situation.




bgRFC backlog monitoring – by the number of entries


Hourly Too many entries in the specified queue group or destination might indicate a backlog situation.

Analyze the reason in transaction /nSBGRFCMON.



bgRFC backlog monitoring – by the age of entries


Hourly Old entries in the queue group or destination can indicate a backlog situation.

Analyze the reason in transaction /nSBGRFCMON.




RFC Monitoring

© 2011 SAP AG page 61/124

For details on how to setup the automated tRFC/qRFC/bgRFC backlog monitoring, refer to the section Automated tRFC/qRFC/bgRFC monitoring.


RFC Monitoring

© 2011 SAP AG page 62/124

5 Aspect: Performance Monitoring

This section will give you some suggestions of how to monitor your system‘s performance with regard to RFC. The first step is to understand that we actually need to look at two systems. Depending on the landscape and solution these can be SAP systems or third-party systems. Therefore, optimization steps may need to be done in two systems. The performance of RFC depends on many factors (type of business process, number of messages, activities running on the distributed systems, hardware, and so on).

Performance monitoring enables you to monitor interface performance and compare it with predefined key performance indicators. Proactively applied performance monitoring notifies in situations of reduced interface throughput, which in turn can avoid backlog situations for performance-critical interfaces. Reactive performance monitoring allows for documentation of the service level delivered.


The first step in performance monitoring is to have a baseline to measure against. There are different ways of using RFC; there is really no simple way to define what is a good performance or bad performance. Therefore, the first step is defining KPIs (key performance indicators) for RFC interfaces. By doing so, you define what minimum performance you require. This depends on a number of factors and is based on your business needs.

All parties in the process need to agree to these KPIs and to the actions to be taken if these limits are exceeded. One way to find these KPIs is to use your quality assurance system (or the future productive system) and measure it. These numbers might be useful in defining the KPIs. Another method is to analyze your business process. For example, if you need to run 3600 RFCs per hour (each processing the business data for one sales order) you need to be able to process one RFC per second. In the following sections we are going to describe some techniques that can be used to measure the performance of the RFC processing in an SAP system. There is also an SAP course called ADM315 Workload Analysis, which covers this and many more topics of performance monitoring in greater detail.

5.1 Manual Performance Monitoring


Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


Performance of the RFC processing – General Overview

(receiver system)

ST03N → Workload Overview

As req-uired by bus-iness

High RFC processing times

For information on using the ‗Workload Overview,‖ see section ST03n – Workload Overview below for details.

Review section Performance




RFC Monitoring

© 2011 SAP AG page 63/124

Monitoring of this document for information about how to further investigate any observed bottlenecks.

Slow processing of RFCs (sender system)

ST03N → RFC Client Profile

As req-uired by bus-iness.


For information on using RFC Client Profile, see section ST03n – RFC Client Profile below for details.



Slow processing of RFCs (receiver system)

ST03N → RFC Server Profile



For Information on using RFC Server Profile, see section ST03n – RFC Server Profile below.



Slow processing of RFCs (Sender system)

ST03N → RFC Client Destination



For information on using the RFC Client Destination tool to analyze performance, see section RFC Client and Server Destinations below.



Slow processing of RFCs (receiver system)

ST03N → RFC Server Destination

As re-quired by bus-iness


For information on using the RFC Server Destination tool to analyze




RFC Monitoring

© 2011 SAP AG page 64/124

performance, see section RFC Client and Server Destinations in this document.

Slow processing of RFCs (sender/receiver system)

STAD As req-uired by bus-iness


For information on using Single Statistical Records (STAD), see section Single Statistical Records (STAD) in this document.



Slow processing of bgRFC units (inbound or outbound)

SBGRFCMON / SBGRFCCONF / SBGRFCPERFMON


Backlog of units in queue

Run transaction SBGRFCMON. Look for destinations and queues with high number of unprocessed units. Run transaction SBGRFCCONF to check the status of inbound schedulers and outbound destinations.

Run transaction SBGRFCPERFMON to check key figures for inbound and outbound scenario.



Also see section backlog monitoring.


RFC Monitoring

© 2011 SAP AG page 65/124

5.1.1 Overview of Available Tools for RFC Performance Analysis

5.1.1.1 ST03N – Workload Overview

Once the RFC Statistics have been activated, they can be used to analyze RFC communication performance. For details on general usage of ST03N, refer to section 8.2.2 of this document.

For a general overview of how a specific server is performing, choose the relevant server that you want to analyze and the time period, and then go to Analysis Views → Workload Overview. Look at task type RFC for an overview of performance of RFC (Note: this is when this server acted as the RFC server).

The following calculation can be used to find out how many CPUs are used by RFC on each particular server:

(avg. CPU + avg. DB + avg. Load + Gen + avg. Roll-in & Roll-out time)/1000 ) * Number of Steps / seconds of the viewed period = number of CPUs used by RFC

If multiple instances are in use and if the result of the above calculation is high for the central instance but low for other application servers, some improvement might be gained by distributing the RFC load more evenly across the servers.

Figure 40 – Workload Overview with ST03N

For a more detailed analysis of RFC performance use the RFC profiles, select a specific server of interest, or select all servers (Total – as below), and choose a time period, for example, day, week, or month.


RFC Monitoring

© 2011 SAP AG page 66/124

Figure 41– RFC Client Profile – List of Called Function Modules

5.1.1.2 ST03N - RFC Client Profile

Using the RFC client profile (choose Analysis views → RFC profiles → Choose RFC profile: “RFC Client Profile”), as illustrated above, allows analysis of RFC communication when this particular server acts as the RFC client. Choosing the Function Module tab displays a list of the called function modules for the selected server/servers over the selected time period. As this is the RFC client, we should be looking out for long connection times – this can be achieved by sorting by ―Call Time‖ (average or total (In SAP NetWeaver 2004 and NetWeaver 2004s, these columns are called T Time and Ø Time/RFC.) to get those function modules with the longest processing times at the top of the list. Detailed analysis ( RFC trace) should be performed on the function modules where, as a rule-of-thumb, the following is true: call time – execution time > 20% (that is, >20% processing time is spent on establishing a connection). Possible reasons for long connection times include:

Establishment of the connection takes a long time, for example, RFC server is overloaded or insufficient number of registrations by the RFC Server programs.

Data transfer takes too long, for example, network bandwidth or amount of data to be transferred too high.

It is also possible to double-click on each function module to access greater detail about the called function module; information available includes local destination, remote destination, user who called function module, bytes sent, and bytes received. See figure below.

Choose a

specific server

or all servers

(Total).

Choose a

particular RFC

profile, for

example, Client,

Server.


RFC Monitoring

© 2011 SAP AG page 67/124

Figure 42 – Additional Information by Double-Clicking on Function Module

Choosing the Transaction tab allows you to view a list of all RFC-spawning transactions. Like above, double-clicking on the transaction provides further information.

Figure 43– List of RFC-Spawning Transactions


RFC Monitoring

© 2011 SAP AG page 68/124

Choosing the User tab allows you to view a list of all RFC-spawning transactions. Similar to above, double-clicking on the transaction provides further information.

Figure 44 – List of RFC Users

5.1.1.3 ST03N - RFC Server Profile

Using the RFC Server profile (Analysis views → RFC profiles → Choose RFC profile: “RFC Server Profile”), as illustrated below, allows analysis of RFC communication when this particular server acts as the RFC server. Choosing the tab Function Module displays a list of the called function modules for the selected server/servers over the selected time period. As this is the RFC server, you should be looking for high-load function modules, that is, function modules that are executed frequently and long-running function modules. This can be achieved by sorting by Number of calls and Total execution time to get those function modules that impose the most load on the RFC server.

Sort by Number of RFC Calls:

If, for example, the number of calls for a function module within a certain time frame (for example, 24 hours) indicates that it is called every second or with an even higher frequency, it should be analyzed why it is called so often.

Sort by Total execution time:

To get those function modules that cause the highest RFC load on the system, they should be analyzed in detail using an RFC trace.


RFC Monitoring

© 2011 SAP AG page 69/124

Figure 45 – RFC Server Profile – List of Called Function Modules

It is also possible to double-click on each function module to access greater detail about the called function module. Information available includes local destination, remote destination, user who called function module, bytes sent, and bytes received (see figure below).

Figure 46 – Further Information on RFC Server Function Module


RFC Monitoring

© 2011 SAP AG page 70/124

In addition to the above, the RFC Server Profile also provides tabs that display RFC-spawning transactions and also the RFC users for the specified period.

5.1.1.4 ST03N - RFC Client Destinations and Server Destinations

The RFC profiles also provide two other useful analysis tools: RFC Client destination profile and the RFC Server destination profile:

The RFC Client destination profile lists the targets (RFC destinations) that were called by the local system (RFC Client). As this is the RFC client, we should be looking for long connection times – this can be achieved by sorting by Call Time (average or total) (in SAP NetWeaver 2004 and NetWeaver 2004s, these columns are called T Time and Ø Time/RFC) to get those destinations with the longest processing times at the top of the list.

Detailed analysis should be performed on the destinations where, as a rule-of-thumb, the following is true: call time – execution time > 20% (that is, >20% processing time is spent on establishing a connection). Possible reasons for long connection times include:

Establishment of the connection takes too long, for example, RFC server is overloaded or insufficient number of registrations by the RFC Server programs.

Data transfer takes too long, for example, network bandwidth or amount of data to be transferred too high.

Figure 47 – RFC Client Destinations

It is also possible to double-click on each destination to access greater detail about the called destination. Information available includes transaction/report called, local destination, remote destination, user, bytes sent, and bytes received (see figure below).


RFC Monitoring

© 2011 SAP AG page 71/124

Figure 48 – RFC Client Destination, Additional Information by Double-Clicking on Function Module

RFC Server destination lists those targets (RFC destinations) that called the local system (RFC Server). Look for those that are executed frequently and have a high execution time. This can be achieved by sorting by Number of calls and Total execution time to get those destinations that impose most load on the RFC server.

Figure 49 – RFC Server Destinations

It is then possible to double-click on the problematic destination to get more details about which transactions/reports and which users are causing the high loads (see figure below). Further investigation should be carried out based on this information.


RFC Monitoring

© 2011 SAP AG page 72/124

Figure 50 – RFC Server Destinations – Further Details

5.1.2 Activation of Performance Monitoring with Transaction ST03N

In the workload monitor ST03N, you can display the workload caused by RFCs. By displaying the RFC profiles, you can answer the following questions:

Which transaction, which function module, and which user caused what workload through RFC calls as an RFC client or an RFC server?

What workload do the transactions, function modules, or users cause in which local or remote RFC destinations?

For transaction steps with RFC, the kernel writes sub-records with additional information about the processing time, the destination, and the function module used. The parameter stat/rfcrec (default: stat/rfcrec = 5) specifies the maximum number of sub-records of each type (RFC client, RFC server, RFC client destination, and RFC server destination) that the kernel writes. If more RFCs are performed during a transaction step, only the five calls with the longest execution time are logged, which is sufficient for performance analyses. The restriction to a maximum of five records represents a compromise between the required accuracy on one side and the workload created by the performance collector on the other side. A larger value for stat/rfcrec can lead to performance problems in the collector. The RFC client and RFC server records contain data such as execution time and called function for individual RFCs. The RFC destination records contain the total of all RFC calls per destination and, therefore, no additional information about the called function modules.

5.1.2.1 Activating the Statistical Collectors

It is possible that no RFC-statistics are stored, as it is possible to turn them off in the system. The steps to reactivate the monitoring are different for SAP R/3 4.x, SAP NetWeaver 2004, and SAP NetWeaver 2004s.


RFC Monitoring

© 2011 SAP AG page 73/124

Here we will describe the different methods of activation. For each system version you need to enter transaction ST03N → Expert Mode.

a) For SAP R/3 4.x:

In ST03N, you need to go into the collector branch → workload collector → Collector and reorg.

Figure 51 – Activation of RFC Statistics with ST03N

Choose the Modify parameters button.


In this screen, we suggest that you activate all the check boxes and options. At the least, the Cumulate RFC profiles entry under Statistical data to be cumulate & Controls for the collector has to be active. Then save the choices. In addition, it is highlighted above how you can modify the residence times for statistical data.


RFC Monitoring

© 2011 SAP AG page 74/124

b) For SAP NetWeaver 2004:

Call transaction ST03N → then Collector and Performance DB.


In all the mentioned sub-screens, you have the choice to use the SAP Defaults button. This option will set the parameters to values that are suggested by SAP. Make sure to note which parameters you modified in the system, as everything you have set manually will be erased by the defaults.

Choose Performance Database → Reorganization. Here you can modify all the settings relating to the retention times of performance data.

Figure 54 – ST03n Retention Time for Performance Data


RFC Monitoring

© 2011 SAP AG page 75/124

Next, choose Workload Collector → Control Data. In this screen, you can choose how much data is stored on the local file system. Be aware that there is no exact information on how large the files can get, so please make sure that you have enough disk space available for these logs.

Figure 55 – ST03N Retention of Statistic Files

Next, choose Workload Collector Database → Statistics to Be Created. Here, you can choose what data is stored in the system. We suggest activating all the checks, but the RFC statistics is the minimum that should be activated.

Figure 56 – ST03N – Selection of Data to Cumulate


RFC Monitoring

© 2011 SAP AG page 76/124

c) For SAP NetWeaver 2004s:

Call transaction ST03N → then Collector and Performance DB.


In all the mentioned sub-screens, you have the choice to use the SAP Defaults button. This option will set the parameters to values suggested by SAP. Make sure to note which parameters you modified in the system, as everything you set manually will be erased by the defaults.

Choose Performance Database → Reorganization. Here you can modify all the settings relating to the retention times of performance data.


RFC Monitoring

© 2011 SAP AG page 77/124

Figure 58 – ST03N Retention Time for Performance Data

Choose Workload Collector Database → Reorganization → Control. Here, you can modify all the settings relating to the retention times of aggregated performance data.

Figure 59 – ST03N Maintenance of Collector Parameters (Retention Periods Aggregated Data)

Choose Workload Collector → Instance Collector → Control. Here you activate collection of aggregated performance data at instance level.


RFC Monitoring

© 2011 SAP AG page 78/124

Figure 60 – ST03N Maintenance of Collector Parameters (Instance Statistics Aggregation)

Choose Workload Collector →Total Collector → Control. Here you activate collection of aggregated performance data at overall system level.

Figure 61 – ST03N Maintenance of Collector Parameters (Total Statistics Aggregation)

The parameter stat/rfcrec, which controls the number of RFC sub-records to be generated, is configured using ST03n (Expert mode) → Collector and Performance DB → Statistics Records and File → Online Parameters → Dialog Step Statistics. (This is the same procedure for all versions of SAP.)


RFC Monitoring

© 2011 SAP AG page 79/124

Figure 62 – Configuring the Number of RFC Sub-Records

5.1.2.2 Single Statistical Records (STAD)

For details on general usage of STAD, refer to section 8.2.3 of this document. From the initial STAD selection screen, limit the selection to retrieve relevant records and press Enter to display the relevant statistical records. Identify records with the longest response time and double-click on the record to

display further details (see figure below).

Figure 63 – Single Statistical Records - Details


RFC Monitoring

© 2011 SAP AG page 80/124

The figure below shows a statistical record with RFC. RFC+CPIC time is the time spent on RFC/CPIC handling. It includes communication time, processing time on target system, and roll wait time. If RFC+CPIC time is significantly greater than roll wait time (as it is above, that is, approximately 1.5 seconds), this indicates a problem during the communication between the systems. In this case, check whether the reason is due to:

A slow network connection

No work process available in the receiver system

If RFC+CPIC time is equal to roll wait time, this indicates that the communication between the systems is fast. If this is the case and you have identified an RFC call to be slow, you may need to perform further performance analysis of the RFC function module in the receiver system.

To analyze RFC performance further, you can use the RFC sub-records as shown below. Use the RFC button or scroll down to get to this particular analysis tool.

Figure 64 – STAD – RFC Sub-Records

The RFC sub-records show the five most expensive RFC calls to the particular destination. You should use this information to carry out an in-depth analysis of RFC performance and identify where time is being spent. SAP Note 552845 explains each of the statistics available in the RFC sub-records. Depending on findings, you may need to execute an RFC, SQL, or ABAP trace for detailed analysis of problem.


RFC Monitoring

© 2011 SAP AG page 81/124

Performance Trace (ST05)

You can also get information on RFC using an ST05 trace. The information is similar to Client and Server sub-records. For details on general usage of the performance trace ST05, refer to section 8.2.4 of this document. On the initial selection screen of ST05, choose the RFC trace and activate the trace accordingly. After you have captured your analysis period, deactivate the trace and display it. It should look something like the figure below (the figure represents an RFC trace for RFC Client). When you go to the detailed statement (double-click on RFC entry) of an RFC entry, the IP addresses of client and server are visible. In this case, the names of the server contain the SAP system name: Client and server below are both U68. The name of the RFC destination is not shown; therefore, the correct destination displayed in SM59 has to be determined by trial and error method, using the server names and IP addresses. The transferred data volumes for sending and receiving are given below. The duration is the total processing time of the RFC for sending until receiving the data. The RFC time in the detailed statement documents the processing time in microseconds on the client machine (normally the contribution of this value is small). The name of the function module is also documented (here: RFC_PING).

Figure 65 – ST05 RFC Client Performance Trace

Figure 66 – ST05 RFC Client Trace Statement Detailed


RFC Monitoring

© 2011 SAP AG page 82/124

On server side, incoming RFCs are also recorded by ST05. The RFC entry in ST05 has the operation mode = SERVER. The duration time is the time needed for execution of RFC on the server side.

In the details of an RFC entry (double-click on the RFC entry), an RFC time is given, which indicates the internal processing in microseconds. This time should not be used for performance analysis, since it describes only a small part of the complete RFC processing. The data transfer for sending and receiving is indicated in each row.

Figure 67 – ST05 RFC Server Performance Trace

Figure 68 – ST05 RFC Server Trace Statement Detailed

RFC trace using ST05 can be combined with an SQL trace. This can provide very useful information about the root cause of long RFC processing times. This is done using the selection criteria on the ST05 initial screen – by choosing both the RFC and SQL traces activate the trace accordingly. After you have captured your analysis period, deactivate the trace and display it. It should look something like the figure below.


RFC Monitoring

© 2011 SAP AG page 83/124

Figure 69 – ST05 Trace with Active SQL and RFC Trace Options

The above ST05 trace is shown with active SQL and RFC trace options. The SQL trace on the server side shows a full table scan on table EINA (RED). The incoming RFC (type Server) shows how long the execution of the RFC took. The time stamp of the RFC defines the start time where the execution starts. The first SQL statements occur shortly after this time stamp. From the time stamp of the first SQL statement until the last statement has finished it took 1.597 msec. This is nearly the complete processing time of the RFC on the server side. Hence, the performance problem on the server side is an expensive statement on table EINA due to wrong index.

ST12 can also be used to perform the above performance traces. See section 8.2.6 regarding usage of ST12.

5.1.2.3 Using SM59 to Test Connection Performance

The RFC connection can be tested in SM59. This tool is useful to test if the connection exists and works properly. For performance tests, it can only be qualitative, as described below.

The test is performed in several steps: First, only a connection to the server system is established. The time for the logon process is displayed. In the next steps, data packages of different sizes are sent and received. The sizes of the packages are NOT 10, 20, 30 kb, as stated in the connection test. The data is compressed and data type conversion can take place, which depends on the operating systems used on sender and receiver side. The logon time depends on the number of authorizations the RFC user has. A user with fewer authorization rights can logon faster than a user with many rights. The processing time of the logon process depends on the actual system load on the server side.

Since several table entries of user authorizations will be buffered in table buffers, the response can be significantly slower the first time than the following time. If there is a connection problem, an error message is displayed.


RFC Monitoring

© 2011 SAP AG page 84/124

Figure 70 SM59 – Testing Connection Performance

5.2 Automated Performance Monitoring

Using SAP Solution Manager and its Business Process Monitoring functionality, you can automatically monitor the average response time of selected programs or functions.

For details on RFC Performance Monitoring within SAP Solution Manager, refer to the section Automated RFC Performance Monitoring.


Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


Total response time of the RFC processing for specific programs and / or function modules

Business Process Monitoring within SAP Solution Manager (Refer to the section Automated RFC Performance Monitoring.)

As needed

Long response times

Start the analysis using transaction /nSTAD or /nST03N.


SAP Technology Operations team / Application Management team


RFC Monitoring

© 2011 SAP AG page 85/124

6 Aspect: Resource Monitoring

Resource monitoring includes the monitoring of the availability of involved components, as well as their resource consumption. This section will show you how resource monitoring for interface monitoring with RFC scenarios can be done. In the first subsection, you find possibilities on how to perform manual monitoring and error handling. The second subsection outlines possibilities for automated monitoring.


To enable the successful execution of interfaces, it is important that sufficient resources are available. It is important that the following resources are monitored for availability to ensure optimal interface performance.

Monitoring Objects:

Work processes

Queues

CPU

Memory

Buffers

Database

Figure 71 – Resource Overview in ALE Processing


RFC Monitoring

© 2011 SAP AG page 86/124

6.1 Manual Resource Monitoring


Monitoring Object

Monitor TA/Tool

Monitor Freq.

Indicator or Error


Responsibility Escalation

Procedure

Slow processing on sender/receiver systems

SM50 Hourly during peak hours and when there are performance problems or error messages

WP status, WP utilization

Monitor current state of individual work processes. Ensure that enough work process capacity is available at peak times.

If multiple work processes on sender system are in status Stopped, reason CPIC, and Action/Reason CMINIT,‘ this represents communication initialization and should last only a few milliseconds. If a high number of these types of entries are visible it indicates that the receiver system is overloaded, in which case target system resources should be checked.

The referenced error can also occur if multiple users want to use functionality of an external RFC program that is only started and registered once in gateway. The gateway queues all user request to be processes first in - first out. If the waiting time exceeds the value of gw/reg_timeout parameter (default: 60 seconds), the caller receives a communication error (timeout error appears in error logs). Possible overload should be addressed with external software provider. If possible, external program should be registered multiple




RFC Monitoring

© 2011 SAP AG page 87/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Indicator or Error



Procedure

times.

Slow processing

sender/ receiver systems

SM66 Hourly during peak hours and when there are performance problems or error messages

WP status, long running jobs

Similar to SM50 but for system-wide statistics



Slow processing

sender/ receiver systems

SARFC Hourly during peak hours and when there are performance problems or error messages

Number of aRFC resources currently available for asynchronous RFC calls

Run transaction SARFC and monitor current state of individual work processes. Ensure that enough work process capacity is available for aRFC communication at peak times.

If an application uses a lot of transactional or asynchronous RFC, it is possible that this can overload the participating application servers. It is important that enough work processes are available for both dialog users and for RFC communication.

Resource usage by RFC can be controlled using various profile parameters. See SAP Note 74141 for tuning hints regarding resource configuration for RFC.



Slow processing sender system

SM58 Hourly when there are performance problems or error messages

Status text shows

―Transaction recorded‖

Run transaction SM58 look out for status text ―Transaction recorded.‖

This indicates lack of resources in the target system. Check resources in the target system.



qRFC not processed

SMQS Hourly when

Status shows ―Waiting‖

Run transaction SMQS and check for status


RFC Monitoring

© 2011 SAP AG page 88/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Indicator or Error



Procedure

on sender system

there are performance problems or error messages

‗Waiting.‘ If status ‗waiting‘ is found, it indicates that sender system is experiencing a work process shortage. Check further in SMQS → Goto → QRFC resources (see SAP Note 527481).

Slow processing in sender system

SMQS Hourly when there are performance problems or error messages

Status of the destination shows ―WAITCONN‖ indicates lack of RFC resources in sending system assigned to this RFC destination

Check the number of work processes configured for this particular destination using transaction SMQS → check the value specified in column Max. Conn for the destination.

Check the configured number of resources dedicated to RFC against the number of available work processes on sender system by using transactions SARFC and SM50.

Decide if enough work processes are configured for destination.



qRFC not processed on receiver system

SMQR Status ‖WAITING‖ in SMQR

Run transaction SMQR and check for status ‗WAITING‘. If status ‗WAITING‘ is found it indicates that receiving system is experiencing a work process shortage. Check further in SMQR → Go to →QRFC resources. (See SAP Notes 527481 and 369007 for further details.)

qRFCs not processed sender/receiver system

SMQ1/SMQ2

Hourly when there are performance problems or error

Status of queues, if entries in a queue are not processed and queue remains in a certain

Run transaction SMQ1/SMQ2. Check whether there are entries in a queue that are not processed and the queue remains in a certain status for more




RFC Monitoring

© 2011 SAP AG page 89/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Indicator or Error



Procedure

messages status for more than 30 mins

than 30 mins. Refer to SAP Note 378903.

bgRFCs not processed

SBGRFCCONF & SBGRFCMON

On demand

Status of bgRFC queues and bgRFC schedulers

In case bgRFC units do not get processed, check status of the bgRFC schedulers.



System parameters for a high interface load

SMGW & ST11

During periods of very high ALE/RFC load and or performance problems

Error messages in

the gateway trace or other developer traces regarding terminations, i.e,.

SAP-RC=672, R3_LOGIN_FAILED, No wp_ca block received, No free block found in the WP Communication Area, MAX_CPIC_CLIENTS, CONN_EXCEEDED

Review recommendations as per SAP Note 384971



Operating system monitor

ST06 Hourly (depending on the business process) and when there are performance problems

High paging rate and CPU utilization

Run transaction ST06 and monitor the CPU and memory consumption. A hardware bottleneck can have a negative impact on the overall response time as well as the response time of an individual business transaction.

Especially monitor hardware load during high RFC transmission times.



R/3 System buffer monitor sender/receiver system

ST02 Hourly when there are performance problems or error messages

Occurrence of swaps

Monitor memory resource usage for specific R/3 application servers.

To ensure optimal performance, check that the R/3 parameters are set correctly with




RFC Monitoring

© 2011 SAP AG page 90/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Indicator or Error



Procedure

transaction ST02.

Incorrectly sized R/3 buffers or memory allocation can result in poor performance. One such example would be when a work process enters PRIV mode.

Database perform-ance monitor sender/receiver system

DB02 Weekly and when there are performance problems

Table sizes, table indices for tRFC and aRFC tables

Ensure that the data quality is sufficient, that there are no missing indexes, and that there is sufficient space available.

In general, if table sizes are larger than 500MB, reorganize the table and decrease its size.

See SAP Note 375566.

Monitor the growing of tables and indices, especially on tRFC and qRFC tables (ARFCSSTATE

ARFCSDATA

ARFCRSTATE). Since the tRFC and qRFC tables shrink and expand constantly, the storage quality of the indexes for these tables might be inadequate. This has a negative impact on the performance.



6.2 Automated Resource Monitoring

Many of the manual monitoring objects for the general system availability, performance, and stability are available within SAP CCMS and SAP Solution Manager.

The complete technical system resource monitoring can be performed automatically within the System Monitoring part of SAP Solution Manager individually per system and instance.


Monitoring Object Monitor TA/Tool

Monit Freq.

Indicator or Error

Monitoring Activity or Error Handling

Respon-sibility



RFC Monitoring

© 2011 SAP AG page 91/124

Procedure

SAP system resources - Number of free work processes, - Dialog steps per minute - CPU utilization - Memory usage - DB key figures – Etc.

CCMS / SAP Solution Manager‘s System Monitoring part

15 minutes

Resource shortage

Analyze the system stability and performance.

SAP technology operations team


For further details on System Monitoring within SAP Solution Manager, refer to http://service.sap.com/solutionmanager → Functions in detail → Solution Monitoring → System Monitoring.

http://service.sap.com/solutionmanager


RFC Monitoring

© 2011 SAP AG page 92/124

7 Aspect: Data Management


The RFC-relevant tables listed below have a direct impact on performance; these tables grow very quickly, and if not managed correctly cause interface(s) to execute slowly.

Monitoring Objects:

A data management strategy should be set up for the following RFC-relevant tables.

7.1 Classic qRFC

RFC-related tables

DB Table Name Table Description

ARFCSSTATE ARFC Call Status (Sender)

ARFCSDATA ARFC Call Data (Sender)

TRFCQOUT tRFC Queue (outbound)

ARFCRSTATE Status of ARFC Calls on Receiver Side

TRFCQDATA qRFC (inbound queue)

TRFCQSTATE qRFC Call Conditions (Inbound Queue)

TRFCQIN tRFC Queue (Inbound Queue)

For RFC tables, such as, ARFCSSTATE, ARFCRSTATE, ARFCSDATA, TRFCQOUT, TRFCQDATA, TRFCQSTATE, and TRFCQIN, performance problems often occur as a result of a large number of blocks allocated for these tables, outdated statistics, and poor storage quality.

Recommendation:

If you find the tables to be large/have poor storage quality, you can improve performance by reorganizing these tables. Once that is complete, follow the regular maintenance steps below.

The following reports are used for deleting entries from the RFC tables:

- RSARFCDL: Deletes tRFC Entries from Log File. This report deletes the error log of the

ARFC in the background. - RSARFC01: tRFC Reorganization. - RSTRFCES: Deletes ARFCSSTATE, ARFCSDATA, and TRFCQOUT entries. - RSARFCER: Deletes tRFC in reference to the status

Refer to SAP Notes 324545 and 375566 for additional details.

These tables are considered volatile, and statistics should not be collected. Also, due to the high volume of data and frequent changes, the indexes for these tables should be reorganized on a regular basis.


RFC Monitoring

© 2011 SAP AG page 93/124

In the following best-practice document you will find standard recommendations on the topic data management and specific recommendations for different DB tables, including the RFC interface-related DB tables: http://service.sap.com/~sapidb/011000358700005044382000E or visit quicklink

http://service.sap.com/ilm.

7.2 Background RFC

Overview of bgRFC-related database tables

Direction Qual. of service

Table name Table Description HV

all all BGRFC_UNIT_TIME Timestamp when bgRFCs Completed x

all all BGRFC_LOCK_TIME Timestamp when bgRFC Locks Removed o

all all BGRFC_UTASK_KEY Assignment of Update Keys to Unit Locks

inbound all BGRFC_IUNIT_HIST History of Inbound bgRFC Unit

outbound all BGRFC_OUNIT_HIST History of Outbound bgRFC Unit

inbound all BGRFC_I_DESTLOCK Inbound tRFC/qRFC: Destination Locks

inbound all BGRFC_I_RUNNABLE Inbound tRFC/qRFC: Executable Units

outbound all BGRFC_O_DESTLOCK Outbound tRFC/qRFC: Destination Locks

outbound all BGRFC_O_RUNNABLE Outbound tRFC/qRFC: Executable Units

inbound transact. TRFC_I_DEST Inbound tRFC: Client and Destination x

inbound transact. TRFC_I_SDATA Inbound tRFC: Unit Payload x

inbound transact. TRFC_I_UNIT Inbound tRFC: Unit Header x

inbound transact. TRFC_I_UNIT_LOCK Inbound tRFC: Unit Locks o

inbound transact. TRFC_I_ERR_STATE Inbound tRFC: Unit Error State

inbound transact. TRFC_I_EXE_STATE Inbound tRFC: Unit Execution State x

outbound transact. TRFC_O_DEST Outbound tRFC: Client and Destination x

outbound transact. TRFC_O_SDATA Outbound tRFC: Unit Payload x

outbound transact. TRFC_O_UNIT Outbound tRFC: Unit Header x

outbound transact. TRFC_O_UNIT_LOCK Outbound tRFC: Unit Locks o

outbound transact. TRFC_O_ERR_STATE Outbound tRFC: Unit Error State

outbound transact. TRFC_O_EXE_STATE Outbound tRFC: Unit Execution State x

inbound queued QRFC_I_QIN Inbound qRFC: Queue Order x

inbound queued QRFC_I_QIN_TOP Inbound qRFC: Queue (Top Unit) o

inbound queued QRFC_I_SDATA Inbound qRFC: Unit Payload x

inbound queued QRFC_I_UNIT Inbound qRFC: Unit Header x

inbound queued QRFC_I_UNIT_LOCK Inbound qRFC: Unit Locks o

inbound queued QRFC_I_ERR_STATE Inbound qRFC: Unit Error State

inbound queued QRFC_I_EXE_STATE Inbound qRFC: Unit Execution State x


http://service.sap.com/ilm


RFC Monitoring

© 2011 SAP AG page 94/124

outbound queued QRFC_O_QOUT Outbound qRFC: Queue Order x

outbound queued QRFC_O_QOUT_TOP Outbound qRFC: Queue (Top Unit) o

outbound queued QRFC_O_SDATA Outbound qRFC: Unit Payload x

outbound queued QRFC_O_UNIT Outbound qRFC: Unit Header x

outbound queued QRFC_O_UNIT_LOCK Outbound qRFC: Unit Locks o

outbound queued QRFC_O_REF_UNIT Outbound qRFC: Unit Reference

outbound queued QRFC_O_ERR_STATE Outbound qRFC: Unit Error State

outbound queued QRFC_O_EXE_STATE Outbound qRFC: Unit Execution State x

out-inb. queued QRFC_O_QIN Outbound/inbound qRFC: Queue Order x

no-send queued LDQ_DATA LDQ: Application Data of Unit X

no-send queued LDQ_PROGRESS LDQ: Pointer Table (Start, Progress, End) x

no-send queued LDQ_STATE LDQ: Status Values X

Column "HV" tries to show the potential risk of high data volume in the corresponding database table. Note that this is just an estimate, because at the moment, there is little practical experience.

X Potential high risk if the Local Data Queue scenario is used

x Potential medium risk

o Potential medium risk if the locking or delay functionality is used

Recommendations:

If you find that the above-mentioned tables are large and/or have a poor storage quality, you can improve performance by reorganizing the tables.

These tables are considered volatile, and therefore statistics should not be collected.

Instead, run report RS_QRFC_STATISTIC once (for ORACLE databases only). This report simulates a temporary load situation by filling the bgRFC tables with dummy units. Then suitable statistics are created. Afterward, the queue entries are deleted again. Table DBSTATC is modified to prevent the automatic update of statistics.

Also, due to the potential high volume of data, and frequent changes, the indexes for these tables should be reorganized regularly.

7.3 Manual Data Volume Monitoring


Monitoring Object

Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


Table Growth DB02, DB15

Daily Table sizes and free DB space

Analyze the reasons for the DB growth in transactions DB02 and DB15. Specifically look




RFC Monitoring

© 2011 SAP AG page 95/124

for RFC tables listed above.

7.4 Automated Data Volume Monitoring


Monitoring Object Monitor TA/Tool

Monit Freq.

Indicator or Error


Respon-sibility


DB growth (RZ20 → DB administration → Data archiving) or CCMS / SAP Solution Manager‘s System Monitoring

Daily Free DB space

Analyze the reasons for the DB growth in transaction DB15, DB01


SAP Technology Operations team / Application Management team


RFC Monitoring

© 2011 SAP AG page 96/124

8 Setup Procedure for Automated Interface Monitoring

The aim of this section is to outline the setup procedure for selective interface monitoring, which can be done in the context of the underlying business processes. The setup procedure is different for the different RFC types and monitoring aspects. The following sections outline the full scope for automated RFC monitoring, which is possible for error monitoring of s/a/t/q/bgRFCs, the backlog monitoring of t/q/bgRFCs, and specific performance monitoring based on the total response time of selective RFCs.

For some monitoring activities, it is possible to use a tool for automated monitoring. This is also possible with interface monitoring. Such automated monitoring is optimally implemented using Business Process Monitoring or System Monitoring in SAP Solution Manager.

Business Process Monitoring (BPMon) within SAP Solution Manager is the proactive and process-oriented monitoring of the core business processes of your company. It includes the observation of all technical and business-application-specific functions that are required for a smooth and reliable flow of the business processes.

Business Process Monitoring (BPMon) reveals even slight deviations from a predefined, ideal business process state, which would otherwise remain undetected until the flow of the process would be seriously impacted. It gives automated alerts, including the possibility to notify these via various communication means like e-mail, SMS, and so on. Types of errors that can be monitored are, for example, errors from logs (system log, application log, and so on), throughput and backlog KPIs for various applications, dialog performance, update errors, and so on. The possibility to keep the alerts for a defined time allows you to evaluate a kind of history and to identify trends in the alert occurrence at an early stage.

As a result, BPMon offers you the possibility to better visualize interface monitoring with the context of the underlying business processes. In the example below, you see an SAP CRM Outbound Telesales business process that communicates with the SAP ECC system using qRFC technology. From the business process graphic, you could drill down to the alerts of the underlying business process steps or interfaces and perform further alert handling.

- For further details on Business Process Monitoring within SAP Solution Manager, refer to http://service.sap.com/bpm.

- For further details on System Monitoring within SAP Solution Manager, refer to http://service.sap.com/solutionmanager → Functions in detail → Solution Monitoring → System Monitoring.

http://service.sap.com/bpm

http://service.sap.com/solutionmanager


RFC Monitoring

© 2011 SAP AG page 97/124

Figure 72 – Business Process Monitoring Within SAP Solution Manager

8.1 Automated qRFC Monitoring Using BPMon The BPMon qRFC monitor allows monitoring the qRFC processing and its individual queues for status (errors in processing) and backlog (delays in processing). This way you can be notified if the number of queue entries is higher than expected or the age of the oldest entry of the queue is too high, as well as the combinations.

qRFC queue


RFC Monitoring

© 2011 SAP AG page 98/124

The following BPMon key figures are available in the monitor IMQRFCMO with the add-on ST-A/PI 01K:

Type Description

Backlog Number of individual queues in group

Backlog Total number of entries in all queues in group

Backlog Average number of entries per queue in group

Backlog Maximum number of entries per queue in group

Backlog Age (in minutes) of oldest entry in group

Backlog Combination of Total entries and Oldest age

Status Number of entries with critical state in group

Status Age (in minutes) of oldest critical state entry in group

Status Combination of Entries and Age in critical state

Status Number of entries with interim state in group

Status Age (minutes) of oldest interim state entry in group

Status Combination of Entries and Age in interim state

Age Age of qRFC data collection

Find detailed information on the above key figures and the setup in the Interface Monitoring Setup Guide at http://service.sap./com/bpm → SAP Business Process Management → Media Library → Technical Information → Setup Guide - Interface Monitoring https://websmp208.sap-

ag.de/~sapdownload/011000358700000258142008E/Setup_Guide_IFMon.pdf

8.2 Automated tRFC Monitoring Using CCMS

For tRFC Interfaces, it is possible to monitor CPICERR, SYSLOAD, and SYSFAIL errors generically. It is not possible to distinguish between different RFC destinations. The required MTEs exist within the CCMS by default.

Figure 73 – tRFC Monitoring Tree Elements in SAP CCMS





RFC Monitoring

© 2011 SAP AG page 99/124

To integrate the tRFC monitors available in the CCMS of the satellite system into SAP Solution Manager, you have to navigate to transaction DSWP and then choose Operations Setup → Solution Monitoring → Business Process Monitoring → Setup Business Process Monitoring.

If an interface is selected for monitoring in SAP Solution Manager with the interface technique tRFC, a sub-node appears: Interface “<Name of the Interface>“(tRFC). Under this sub-node another node, Alert Monitors, appears. There, choose the Reload CCMS: tRFC Alerts button. All three default tRFC monitoring objects on the satellite systems (outbound objects on the sender side and inbound objects on the receiver side) are loaded into the session. To transfer all the automatically loaded threshold values at once, select Copy All. It is also possible to change the threshold values for the alerts manually.

Figure 74 – Configure tRFC Monitoring in SAP Solution Manager

After defining the monitoring customizing, you have to generate and activate the monitoring. As soon as the programs and/or functions that are measured have been executed and the data collector has run, you should see alerts if the threshold values have been met.

For details on BPMon, refer to http://service.sap.com/bpm or navigate directly to the setup guide at http://service.sap.com/~sapidb/011000358700006137532006E.

8.3 Automated bgRFC Performance Monitoring Using BPMon The BPMon tRFC monitor allows monitoring the tRFC processing to the individual receiver destinations (within the sender system) for status and backlog. This way you can be notified if the number of unprocessed tRFC LUWs is higher than expected or the age of the oldest entry (per destination) is too high, as well as the combinations.

The following BPMon key figures are available in the monitor IMTRFCMO with the add-on ST-A/PI 01L:

Severity* Description

Critical Number of tRFC entries in critical state

Critical Age of oldest entry incritical state

Critical Combination of Entries and Age in critical state

Interim Number of tRFC entries in interim state

Interim Age of oldest entry interim state

Interim Combination of Entries and Age in interim state




RFC Monitoring

© 2011 SAP AG page 100/124

*The severity refers to different tRFC status codes. Several status codes are grouped in either critical or interim states. A critical state indicates an immediate error situation. An interim state means a temporary error or backlog situation, typically subject to automatic (re-)processing.

Find detailed information on the above key figures and the setup in the Interface Monitoring Setup Guide at http://service.sap./com/bpm → SAP Business Process Management → Media Library → Technical Information → Setup Guide - Interface Monitoring. https://websmp208.sap-


8.4 Automated RFC Performance Monitoring Using CCMS

CCMS provides the functionality for automated bgRFC monitoring. The required MTEs exist within the CCMS by default.

The following monitoring key figures are available:

bgRFC inbound and bgRFC outbound per destination 1. Number of Tasks Being Used 2. Communication Error Rate 3. Average Data Rate

bgRFC inbound scheduler and bgRFC outbound scheduler 1. Memory Allocation 2. Current Throughput 3. Average Throughput

Figure 75 – bgRFC Performance Monitoring in CCMS

8.5 Automated RFC Connection Monitoring Using BPMon

The BPMon bgRFC monitor allows monitoring of the bgRFC processing per individual receiver destinations or queue group for status and backlog. This way you can be notified if the backlog number of unprocessed or erroneous bgRFC units is higher than expected or the age of the oldest entry (per destination/queue) is too high, as well as the combinations.

The following BPMon key figures are available in the monitor IMBGRFCM with the add-on ST-A/PI 01M:





RFC Monitoring

© 2011 SAP AG page 101/124

Category* Description

Backlog Monitoring Number of individual queues/destinations

Backlog Monitoring Total number of units in all queues/destinations

Backlog Monitoring Age of oldest unit

Status Monitoring for Errors Number of erroneous queues/destinations

Status Monitoring for Errors Total number of erroneous units

Status Monitoring for Errors Age of oldest unit in error state

Status Monitoring for Locks Number of locked queues/destinations

Status Monitoring for Locks Total number of locked units

Status Monitoring for Locks Age of oldest unit with a lock

* Each group of key figures (backlog, errors, locks) offers three available key figures: The number of affected queues/destinations, the total number of affected units, and the age of the oldest affected unit. It is recommended to configure at least one key figure from each group, but it may be not required to configure all of them. For example, you might only be interested in the age of the oldest unit, but not in the number of units, because temporary backlog peaks are normal in your scenario. Or, as another example, you could be interested in an immediate alert for erroneous units, no matter how old they are.

Find detailed information on the above key figures and the setup in the Interface Monitoring Setup Guide at http://service.sap./com/bpm → SAP Business Process Management → Media Library → Technical Information → Setup Guide - Interface Monitoring https://websmp208.sap-


8.6 Automated RFC Performance Monitoring

As of the add-on release ST-A/PI01H, it is possible to monitor dialog performance on program name and/or function module level. Therefore, the statistical records that have been written since the last collector run can be evaluated and an alert can be triggered if the measured valued exceeds the set threshold value. This monitoring is only possible for the server RFCs that have been executed on the monitored system, and not the ones called from the monitored system.

Find here an example for statistical records of type RFC in transaction /nSTAD:

Figure 76 – RFC Performance Monitoring – Find the Statistical Records in STAD





RFC Monitoring

© 2011 SAP AG page 102/124

By double-clicking on one of the above entries, you can find the program and function name that have been called.

Figure 77 – RFC Performance Monitoring – Find the Statistical Records in STAD


RFC Monitoring

© 2011 SAP AG page 103/124

Setup Procedure in SAP Solution Manager If you already know the name of the program and/or function module that needs to be monitored, you can log on directly to your SAP Solution Manager and perform the customizing there. Navigate to transaction /nDSWP and then choose Operations Setup → Solution Monitoring → Business Process Monitoring. Choose Setup Business Process Monitoring. Select the business process step to which you want to map the monitoring key figure. To set up the Dialog Performance Monitor, you need to select the Application Monitor.

Figure 78 – RFC Performance Monitoring – How to Include It in Business Process Monitoring

A sub-node is now created with the possibility to load available application monitoring objects from the involved satellite system via the <SID>: Reload App Mon Objects button. Afterward, all available monitoring objects are displayed in table Application Monitoring Object.


RFC Monitoring

© 2011 SAP AG page 104/124

Figure 79 – RFC Performance Monitoring – How to Include It in Business Process Monitoring (2)

Select Dialog Performance Monitor and save. As a result, a sub-node will be created.

In the Dialog Performance Monitor node, you have two different tab strips. Within the first one, Key Figures, you choose the alert types you want to monitor. According to the alert types you selected, new nodes are created when saving your settings—one for each alert type.


For Dialog Performance Monitoring of RFCs, only the Total Response Time can be evaluated.


RFC Monitoring

© 2011 SAP AG page 105/124


In the second tab strip, Detail Information, you specify the data collector period in column DC Period, that is, how often the data collector is supposed to run. This is the time window in which the corresponding statistical records are summarized and for which an average value is calculated. The smallest possible value is 5 minutes, which is usually most appropriate for the Dialog Performance Monitor. The default value is 60 minutes.

If you want the data collector to run only once a day at a specific time, you can maintain the column DC Start Time.

If the data collector should only run and evaluate certain days during the week, you can flag the corresponding weekday. In this example, no alerts would be raised on Saturday and Sunday.

Once you save your entries, you can customize the different alert types individually.

The program name can be maintained in column Value1 and the function module in column Value2.

It is possible to maintain either combinations of program name and function module, or just a program name or function module.

In our example, it is program SAPLIPC_V01 with the function module EXTRACT_DATA and the call type ―R‖ for RFC.


For Remote Function Calls, only the total response time can be monitored and only server RFCs are considered.

For each of the alert types, you can specify the following combinations:

- If the program name is maintained in column Value1 and a wildcard (*) is maintained for the function module name in column Value2, then the dialog performance for the selected program will be monitored for all function modules with the specified collector period.

- If the program name is maintained in column Value1 and the column Value2 is left blank, all statistics will be evaluated where the program name matches the customized value and the field for the function module name is blank. All function modules that are not blank are not evaluated.

The entries made in columns Value 1 and Value2 should be written in capital letters. Otherwise, it might happen that the corresponding Monitoring Object cannot be found.


RFC Monitoring

© 2011 SAP AG page 106/124

After defining the monitoring customizing, you have to generate and activate the monitoring. As soon as the programs and/or functions that are measured have been executed and the data collector has run, you should see alerts if the threshold values have been met.

For details on BPMon, refer to http://service.sap.com/bpm or navigate directly to the setup guide at http://service.sap.com/~sapidb/011000358700006137522006E.

8.7 Automated RFC Connection Monitoring Using BPMon Within the Business Process Monitoring functionality of SAP Solution Manager, an automated RFC connection test is possible. This way, an automated monitoring for any incidents regarding the system connection and availability can be established.

To set up this functionality, you have to log on to the SAP Solution Manager system and navigate to transaction /nDSWP. Choose Operations Setup → Solution Monitoring → Business Process Monitoring and then choose Setup Business Process Monitoring.

a. Within the so-called setup session, you have to select a business process and a business process step on a system for which you want to set up the RFC connection monitoring first.

Figure 83 – Automated RFC Connection Monitoring Within Business Process Monitoring

b. Select the Application Monitor entry and navigate to the next level on the left navigation menu.

Figure 84 – Automated RFC Connection Monitoring Within Business Process Monitoring (2)

c. Choose Load App.Monitors from SID to retrieve a list of monitoring objects available on the connected satellite system that are part of the installed ST-A/PI version.

d. Choose the F4 help in column Application Monitor and the entry BORFCCON from the next dialog box.




RFC Monitoring

© 2011 SAP AG page 107/124


e. On the Key Figures tab, select the key figure Availability of RFC connection and choose the scheduling mode for the data collector on the Detail Information tab, following the setup procedure description on the top of the screen.


f. On the next level, you have to select the RFC destination that needs to be monitored using the F4 help of

the column RFC Destination. Choose a Max. wait time [s], which is the time for which the connection test should be performed until an alert (defined in column YELLOW or RED?) would be raised.


After generating and activating the monitoring customizing, the RFC connection monitoring is scheduled. For details on BPMon, refer to http://service.sap.com/bpm or navigate directly to the setup

guide at http://service.sap.com/~sapidb/011000358700006137532006E.




RFC Monitoring

© 2011 SAP AG page 108/124

9 RFC Performance Monitoring

9.1 Introduction

In the Generic Part of this RFC monitoring best-practice document, you will find information about common monitoring activities that are not specific to special business process steps. The following topics are covered:

Performance Monitoring

General Monitoring Guidelines for an SAP System

In section Performance Monitoring Tools, you will find the description of a general procedure for performance monitoring of your SAP system. The different transactions that are mentioned in the procedure are described in detail below. Keep in mind that this is general information.

You should also consider the SAP training BC315 Workload Analysis. This course covers topics like workload monitor, statistical records, expensive SQL statements, memory management, database monitor, and buffer analysis, among others.

9.2 Performance Monitoring Tools

As part of the daily monitoring activities of your SAP system, you should not only watch system availability, system resource consumption, and middleware queues (among others), but also the performance of your SAP system. There are several transactions available with which you can monitor the average performance of the system and analyze the cause of bad performance. Bad performance of a system is generally equal to extremely high response times. This may be related to long-lasting transactions steps (dialog steps), background jobs running too long, and so on.

The response time of a transaction step is the difference in time between the point when the request arrives in the SAP system and the point when the SAP system completes the processing. In addition to this measurement of time, other different times, such as waiting times or database times, are still measured in SAP. If you subtract these components from the response time, this leaves the processing time, which cannot be directly measured. ABAP programming code is processed during the processing time.

Definitions of the most important components of the response time:

Time in workprocess Time actually spent in the work process (after the dispatcher has found a free work process)

Wait time The dispatcher is waiting for a free work process (dispatcher queue)

CPU time It is only given as a sum per transaction step in the workload statistics

DB time Time for accessing and waiting for the database interface, and therefore for the underlying database

Load/Generating time Time for loading/generating screens, ABAP programs, and CUA elements (not for presentation).

Roll-in time Time for rolling in the roll area context of a dialog step

Enqueue time Time for setting a logical SAP enqueue

Processing time This time is calculated as follows:


RFC Monitoring

© 2011 SAP AG page 109/124

Processing time = Response time - Wait time - Load/Generating time - DB time - roll-in time - Enqueue time

Important:

The CPU time should not be added to the above lines, as it is a total, consisting of parts of the times listed above that cannot be calculated. CPU time is returned by the operating system. The accuracy of the reported value depends on the timer of the operating system (under UNIX, for example, the timer works with 100 Hz so that the CPU time is reported as a multiple of 10 ms). Due to this rough resolution of the CPU time, it is possible that a slightly longer CPU time is displayed for a dialog step than for the response time.

The Rollout time is not part of the response time, as it only occurs after the data is transferred to the presentation server.

The Roll wait time is the time in which a work process is waiting for the response of an RFC. The roll wait time is processed on a client, whereas the user context is in the roll area. Therefore, neither resource is used up, nor does the bottleneck occur here for high roll wait time. High roll wait times occur frequently. This may be caused by calls of Windows or other external applications from within the SAP R/3 system. The application of the RFC is started here. The user context enters the roll areas and roll wait time is processed until the application is exited, that is, the RFC connection is cancelled. The transaction step is only exited at this point. Roll times also occur when the SAP R/3 system communicates with the frontend controls. While the data on the front end is being processed, the SAP R/3 context is rolled out, thus releasing the work process. This can occur several times during a transaction step. If front-end Microsoft Office applications such as Microsoft Word are started and closed only after a long time (several minutes), a very long roll wait time occurs here also. Roll wait is not part of the response time for transaction steps of task type RFC; the reason for this is that no resource use occurs for the application server.

9.2.1 Procedure

The following table gives you an overview of which transaction you can use for what purpose for performance monitoring and a first analysis of performance problems.

Transaction Trans. Code Use for … Use when?

Workload monitor

ST03N General performance overview for the different task types and analysis of system workload

Daily (e.g., three times) and upon performance problems

Workload / statistical records

STAD Detailed performance monitoring of a short time frame, a single user, or a specific transaction or program

Upon performance problems

Performance trace

ST05 SQL trace of a specific step, a short sequence of steps, or part of a long running job

Upon performance problems with high DB times

ABAP runtime analysis

SE30

Getting an overview of the duration and performance of the source code, from individual statements up to complete transactions

Upon performance problems with high CPU times

Single transaction analysis

ST12

Get an overview on the time distribution (DB, CPU, which function module or SQL statement, etc.)

Upon performance problems with high DB or CPU times


RFC Monitoring

© 2011 SAP AG page 110/124

Key Performance Indicators

Performance monitoring is most useful if you have previously defined key performance indicators. Generally speaking, key performance indicators are quantifiable measurements, agreed to beforehand, that reflect the critical success factors of an organization. In this case, we refer to the system response time for single dialog steps of your core business process(es), for transactions or background jobs. The key performance indicators should be agreed on by the business departments and the systems administrators (happy medium between is needed for business reasons and what is achievable from system side). It is also important to use the right granularity for KPIs. There is no need to define a KPI for the response time of each and every step in the process, but it important that they are not defined too generally (for example, less than 2 seconds for each dialog step), as this might be too tight or too wide for specific needs. Furthermore, performance monitoring should not only refer generally to a task type, but should instead go one or two steps further, to single transactions or even steps.

For example: Company ABC is using the call center scenario for inbound calling and is also using the marketing scenario to create target groups via InfoSet or BW queries. As the call center agents need very good response times for business partner searches, in order to answer a call and the queries for large target groups may take several seconds, it is not recommended that company ABC just monitors the average response time for dialog transactions. They should instead do the monitoring on transaction or even on dialog-step level.

Daily monitoring of system performance and the compliance of the response times to the established KPIs should be done using the workload monitor (transaction ST03N). This monitor provides an overview of the response times and their components (CPU time, DB time, wait time, and so on). If you experience performance problems, you should use this and other transactions for a deeper analysis:

If a single user reports problems, use transaction STAD to find what part of the response time is particularly high in his or her case. Compare it with other users to see if it is an isolated problem. Continue the analysis depending on the result.

If a single dialog step (one step within a transaction) has a bad performance, use transaction STAD to find what part of the response time is particularly high in this case. Continue the analysis depending on the result.

If you have a general performance problem, use the system monitors to check resource consumption. Also use the time profile of transaction ST03N to check if performance has been decreasing over time or if there are peaks. Use the user profile for a comparison among different users and user groups, if you follow a useful naming convention.

Get a performance trace (transaction ST05) if you find that the performance problems are related to high DB times. Get an ABAP runtime analysis (transaction SE30) if you find that the performance is related to high CPU or high processing times.

If you cannot find the cause of the performance problem or need further assistance for the analysis, contact the next support level or open an OSS message on component XX-SER-TCC.

9.2.2 Transaction ST03N

Transaction ST03N is a very powerful transaction with several functions. Therefore, we can only mention a few here, concentrating on what can be important for performance monitoring. The workload monitor is used to analyze statistical data from the SAP kernel. When analyzing the performance of a system, you should start by analyzing the workload overview. For example, you can display the totals for all instances and then compare the performances of individual instances over specific periods of time. You can quickly determine the source of possible performance problems using the large number of analysis views and the determined data.

You can use the workload monitor to display the:

Number of configured instances for each SAP R/3 system

Number of users working on the different instances

Distribution of response times


RFC Monitoring

© 2011 SAP AG page 111/124

Distribution of workload by transaction steps, transactions, packages, sub-applications, and applications

Transactions with the highest response time and database time

Memory usage for each transaction or each user per dialog step

Workload through RFC, listed by transactions, function modules, and destinations

Number and volume of spool requests

Statistics about response time distribution, with or without the GUI time

Workload and transactions used: listed by users, payroll number, and client

Workload generated by requests from external systems

For all of this data, you can display the data for a particular instance (not only the one to which you are logged on) or, optionally, totaled for all instances. Depending on your user mode, you can choose the time period for which you want to display the data between day, week, and month (or determine the length of time yourself using the Last Minutes‘ Load function). For most analysis views, you can display all or only certain task types.

The workload monitor has an interface that is divided into two parts. Use the tree structures on the left of the screen to make the following settings:

Select the user mode.

Select the time period for which you want to display the workload.

Select various functions and analysis views (which data you want to display).

The system then displays the result on the right of the screen in a standardized ALV grid control.

For general performance monitoring, you can use the following options:

User mode → choose expert mode

Workload → choose the instance and time frame you are interested in. Selection between the last days, the last weeks, and the last months is possible. If you are interested in a particular time within the last 24 hours, choose Detailed Analysis, then Last Minute’s Load, and then the required time frame on the appropriate instance.

In the first workload overview you get, you find the response times and its components for the different task types (background, dialog, HTTP, RFC, update, and so on). You can now choose Analysis Views → Transaction Profiles → Standard. Here you can get a more detailed view, selecting, for instance, the task type dialog and the aggregation per transaction.

The following screen shot shows transaction ST03N using the following options: Expert mode, workload for server us0306_Q4C_02, time frame from 03.01.2005 – 09.01.2005, standard transaction profile, showing dialog transactions sorted by average response time.


RFC Monitoring

© 2011 SAP AG page 112/124

Figure 88 –Transaction ST03N

9.2.3 Transaction STAD

In the STAD records, every step on SAP NetWeaver Application Server is recorded. The records for all instances of a SAP system are displayed. Business transaction analysis (transaction STAD) calculates the system resource usage of individual transactions for SAP R/3 systems and provides a detailed analysis of a transaction and the dialog steps. The selection criteria include user, transaction, program, task type, start date, and start time.

The analyzed time period can be larger than the interval defined by Read Time, as the system analyzes complete transactions as far as possible. However, it is not always possible to perform a complete analysis for long-running transactions. As the business transaction analysis is time-consuming, you should use as short an interval as possible (around 10 minutes).

The aim of this monitor is to analyze in detail how long the response times for particular steps in a process (or transaction) and are these response times distributed among their components (DB time, CPU time, Wait time, GUI time, etc) have been.


RFC Monitoring

© 2011 SAP AG page 113/124

If you want to use this transaction to analyze the performance of the steps performed by a particular user, proceed as follows:

Tell the user to execute the relevant steps once, slowly, one after the other, and write down the exact time at which he or she executed them.

Go to STAD and display the statistical records related to this single execution (use the user name and the appropriate time frame to display the correct records).

Format the output list (Sel. Fields button), including the fields GUI time, CUA reference program, and CUA internal command. The last two fields can help you to identify the single steps executed.

Use the time stamps of the statistical records and the execution times you wrote down to assign STAD records to the dialog steps that were performed.

Check which steps are actually having a bad response time and what part is contributing most to it (DB, CPU, GUI, and so on).

Depending on the result, proceed with a deeper analysis of the worst steps using, for example, transaction ST05 (for long DB times), transaction SE30 (for long CPU times), or a network analysis (not described in this document) in case of high GUI times.

Depending on what you are analyzing, it might be wise to let the user execute all steps twice and use the second execution for the analysis. This way you make sure that all buffers (for, example, program buffers) are filled.

You can also use this transaction to see, in detail, what was going on in the system during specific hours with bad performance. With this analysis, you can see if there were a few users having particularly high response times or if there was a general performance problem.

Procedure:

Go to transaction ST03N and search, using the time profile, for the hour with the worst performance or the highest workload.

Call transaction STAD with the following selection criteria:

Show all statistic records, sorted by start time → Checked

Start date → Today‘s date

Read time → 1 hour

Start time → start time of high workload (from ST03N)

User → *

Resp. time → >= 5000 ms

Task type → D (dialog) or B (batch) or R (RFC) or * (all)

Be aware that this query may take several minutes.

The following screen shots show the initial screen of STAD with the options: Show all statistic records, sorted by start time, reading of the records for 10 minutes starting from 16:50 hours, for all users* that executed dialog transactions lasting longer than 1000 ms.


RFC Monitoring

© 2011 SAP AG page 114/124

Figure 89 – Transaction STAD

The two screen shots below show the result of this query. For each time stamp, you see what program was executed within which transaction on which server. For each, the user, response time, time in work process, CPU time, DB time, GUI time, CUA ref. program, and CUA internal command are displayed. These columns are not all part of the first standard output. They were adjusted using the Select output fields button (F9).


RFC Monitoring

© 2011 SAP AG page 115/124

To make the following steps easier, you may want to download the list in spreadsheet format. Make sure you have displayed all columns you need before downloading. You also need to be aware that the STAD records are only kept in the system for a limited period (normally 24 or 48 hours).

Go through the list of long-lasting dialog steps and search for those dialog steps that are lasting long on average (those appearing several times). You may identify which steps belong together by comparing the transaction, the program, the CUA reference program, and the CUA internal command. The commands can be different in your system depending on customizing settings, user exits, particular system usage, and so on.

Execute the typical steps of your core business processes once. Execute the steps slowly, one after the other, and write down the exact time at which you executed them. The steps should be performed by a key user.

Go to STAD and display the statistical records related to this single execution (use the user name and the appropriate time frame to display the correct records).

Format the output list (Sel. Fields button), including the fields GUI time, CUA reference program, and CUA internal command.

Use the time stamps of the statistical records and the execution times you wrote down to assign the CUA internal commands or the CUA reference program to the dialog steps that were performed.

Use this information to identify which dialog steps have extremely high response times during your peak load time.

Use this information to analyze in detail what is harming the performance of the corresponding transaction or dialog step. Also use the information on the distribution of the response time (DB time, CPU time, GUI time) given in the list to identify the cause of the problem. Depending on the result, proceed with a deeper analysis of the worst steps using, for example, transaction ST05 (for long DB times), transaction SE30 (for long CPU times), or a network analysis (not described in this document) in case of high GUI times.

Additionally, search for those steps taking particularly long (for example, more than 10 seconds). Contact the responsible user to find out what he or she was doing at that time, if he performed some unusual steps, or if he noticed a bad response time. If you cannot identify the cause, you may use the procedure described for transaction ST05 for a deeper analysis.

For details on performance monitoring on function code level and the monitoring setup procedures, go to http://service.sap.com/~sapidb/011000358700003081072005E and navigate to section 2.1.



RFC Monitoring

© 2011 SAP AG page 116/124

9.2.4 Transaction ST05

The performance trace tool contains a range of trace functions that you can use to monitor and analyze the performance of the system in database accesses, locking, and remote calls of reports and transactions. It allows you to record database access, locking activities, and remote calls of reports and transactions in a trace file and to display the performance log as a list. It also provides extensive support for analyzing individual trace records.

To use the performance trace, you need the authorization to start transaction ST05 and the system authorizations Change trace switches (authorization STOM for authorization object S_ADMI_FCD) and Analyze traces (authorization STOR, also for authorization object S_ADMI_FCD).

The performance trace contains the following tracing options:

SQL trace: This allows you to monitor the database access of reports and transactions.

Enqueue trace: This allows you to monitor the locking system.

RFC trace: This provides information about Remote Function Calls (RFCs) between instances.

The aim of the procedure described below is to find the SQL statements that are causing high response times in the SAP system. If you identify a transaction or a process step that is taking too long, you may use transaction ST05 to find out if one or more SQL statements are causing the performance problem.

Procedure:

Call transaction ST05 and make sure you are on the same instance as the user you want to trace:

Check the option SQL trace (also enqueue and RFC trace if needed).

Activate the trace with filter.

Enter the user name of the user that will perform the steps.

Confirm.

Ask the user to perform the relevant steps (and nothing else).

Deactivate the trace.

Display the trace. Make sure the user name is correct. Write down the displayed time stamps (trace period) so you can find the same trace in the system later. Use the extended trace list to display the time stamps for each SQL statement.

Depending on what you are analyzing, it might be wise to let the user execute all steps twice and use the second execution for the analysis. This way you make sure that all buffers (for example, program buffers) are filled.

Go through the trace and search for the statements with a high duration (marked in red). Use the explain function to see if the correct index is being used for that statement.

If you cannot find the cause of the performance problem or need further assistance for the analysis, contact the next support level or open an SAP message on component SV-BO. Attach the trace, a detailed description of the performed steps, and the related STAD records to the message.

9.2.5 Transaction SE30

The runtime analysis tool allows you to examine the performance of any ABAP programs, such as reports, subroutines, function modules, or classes. The tool saves its results in performance data files, which you can display as lists. You can use these results to identify runtime-intensive statements, combine table accesses, and show the hierarchy of program calls. From the results of the runtime analysis, you can identify:

Excessive or unnecessary use of modularization units

CPU-intensive program functions


RFC Monitoring

© 2011 SAP AG page 117/124

User-specific functions that could be replaced with ABAP statements

Inefficient or redundant database access

Depending on the size of the program, considerable volumes of data can be generated during the runtime analysis. For this reason, this tool is defaulted to Full aggregation. This means that only the standard hit list is generated with all calls. The standard hit list does not include Group hit list, hit list of database tables, Class hit list, Instance hit list, Method hit list, Event hit list, hit list of internal tables, call hierarchy, or statistics. To cancel this restriction, switch off aggregation by replacing the standard variant in the initial screen with a temporary variant, for example. In this variant, you can then configure the measurement restrictions according to the selected objects to be measured.

The runtime analysis consists of two parts:

Recording performance data

Analyzing the performance data

In the first part, the system measures the transaction, program, or procedure, and writes the result to a performance data file. You can restrict the measurement to certain objects. You can also measure up to 10 external processes. In the second part, the performance data is analyzed, and the system displays the results in list form. For more information on this transaction and its use, see the application help.

The runtime analysis tool measures the CPU time required by the measurable components of a transaction, a program, or a procedure. The information is stored in a performance data file, which you can analyze either immediately or at a later date. The CPU time required to measure the runtime is subtracted from the total CPU usage. If you measure the runtime of a program more than once, you are unlikely to get the same result on each occasion. There are various reasons why it is difficult to obtain identical runtimes. For example, in the first measurement, the system might read data records from the table buffer on the application server, but in a second run, it may have to read them directly from the database. Runtimes also depend on the overall system or network load (for example, the number of jobs or systems currently active in your SAP system).

If you cannot find the cause of the performance problem or need further assistance for the analysis, contact the next support level or open an SAP message on component SV-BO. Attach the runtime analyses, a detailed description of the performed steps, and the related STAD records to the message.

For more information on automated performance monitoring via CCMS and SAP Solution Manager, read chapter 2.1 of the document Application Monitoring with SAP Solution Manager.

9.2.6 Transaction ST12

The Single Transaction Analysis tool allows you to examine the performance of transaction or ABAP programs, such as reports, subroutines, function modules, or classes, as well on the coding side tracing the ABAP code as on the database side (SQL trace / performance trace). The tool saves its results in performance data files, which you can display as lists.

You can use these results to identify runtime-intensive statements, to combine table accesses, and show the hierarchy of program calls. From the results of the runtime analysis, you can identify:

Excessive or unnecessary use of modularization units

CPU-intensive program functions

User-specific functions that could be replaced with ABAP statements

Inefficient or redundant database access



RFC Monitoring

© 2011 SAP AG page 118/124

ST12 allows you to aggregate the ABAP trace data per modularization unit (subroutine, loop, function call) and offers the possibility to analyze the ABAP trace in different ways with a graphical aid to identify the call tree structure more easily from different perspectives. The performance trace and the ABAP trace, if done from within one ST12 trace execution, are kept in one named version and can be analyzed together.

ST12 comes with the support tools add-on ST-A/PI (see SAP Note 69455 for details).

Note that the ST12 ABAP trace transaction is not officially documented and is only released for use by SAP or certified service consultants during SAP Services sessions (for example, SAP GoingLive Check or Solution Management Optimization services). ST12 is only available in two languages (English and German).

ST-A/PI is small and does not interfere with productive coding. It can be implemented at any time and there is no necessity to log off users.

For details about the use of transaction ST12 see SAP Note 755977.

Depending on the size of the program, considerable volumes of data can be generated during the runtime analysis. For this reason, this tool is defaulted to aggregation per calling position. This reduces the volume of records for a loop to the number occurrences in the coding of a loop – instead of separate records for each loop cycle. SAP Note 755977 gives you more details.

The runtime analysis consists of two parts:

Recording performance data

Analyzing the performance data

In the first part, the system measures the transaction, program, or procedure, and writes the result to a performance data file. You can restrict the measurement to certain objects. You can also measure up to 10 external processes. In the second part, the performance data is analyzed and the system displays the results in list form. For more information on this transaction and its use, see the application help.

The runtime analysis tool measures the CPU time required by the measurable components of a transaction, a program, or a procedure. The information is stored in a performance data file, which you can analyze either immediately or at a later date. The CPU time required to measure the runtime is subtracted from the total CPU usage. If you measure the runtime of a program more than once, you are unlikely to get the same result on each occasion. There are various reasons why it is difficult to obtain identical runtimes. For example, in the first measurement, the system might read data records from the table buffer on the application server, but in a second run, it may have to read them directly from the database. Runtimes also depend on the overall system or network load (for example, the number of jobs or systems currently active in your SAP system).

If you cannot find the cause of the performance problem or need further assistance for the analysis, contact the next support level or open an SAP message on component SV-BO. Attach the runtime analyses, a detailed description of the performed steps, and the related STAD records to the message.

For more information on automated performance monitoring via CCMS and SAP Solution Manager, read chapter 2.1 of the document Application Monitoring with SAP Solution Manager.



RFC Monitoring

© 2011 SAP AG page 119/124

9.3 General Monitoring Guidelines for an SAP System

For the administration of an SAP R/3 system, there are a number of monitoring activities we strongly recommend scheduling and supervising on a regular basis. The following list is not complete; jobs and tasks for the database administration (such as backups, archiving transaction logs, update statistics for cost base optimizer and so on), especially, are not mentioned. However, this list gives you an impression of what you have to do to keep a system running. If you have further questions on these subjects, please refer to the Solution Management Optimization service System Administration.

The table below lists transactions used for general system checks:


RFC Monitoring

© 2011 SAP AG page 120/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Monitor Time

Indicator or Error


Responsibility

SAP R/3 system workload analysis

ST03N Daily tbd Average dialog response time < 1000 ms

Review response times in comparison to system load and performance-influencing changes within the system of the analyzed period.

Identify and improve perfor-mance of any transaction whose response time exceeds the times defined in the service level agreement (SLA), if existing.

Monitor RFC response time statistics and Dialog response times for online transactions.

Software Monitoring team

Database perform-ance analysis

ST04 Weekly and when there are of performance problem

tbd Increased database response time / Expensive SQL statements

Monitor database statistics.

Monitor the buffer cache hit ratio

Check for missing indices; ensure that the data quality is sufficient and that there is sufficient space available.


Operating system monitor

ST06 Hourly (depending on the business process) and when there are performance problem

tbd High paging rate and CPU utilization

Monitor the CPU and memory consumption.

A hardware bottleneck can have a negative impact on the overall response time, as well as the response time of an individual business transaction.

Especially monitor hardware load during high RFC transmission times.


SAP R/3 system buffer monitor

ST02 Weekly and when there are performance problem

tbd Swaps Monitor memory resource usage for specific SAP R/3 application servers.

To ensure optimal performance, check that the SAP R/3 parameters are set correctly with transaction ST02.

Incorrectly sized SAP R/3 buffers or memory allocation can result in poor performance. One such example would be when a work process enters PRIV mode.


Local work process overview

SM50 Hourly during peak hours and when there are performance problem

tbd WP status, WP utilization

Monitor current state of individual work processes.

Ensure that enough work process capacity is available at peak times.



RFC Monitoring

© 2011 SAP AG page 121/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Monitor Time

Indicator or Error


Responsibility

s or error messages

System-wide work process overview

SM66 Hourly during peak hours

tbd WP status, long running jobs

Similar to SM50, but for system-wide statistics.


Database perform-ance monitor

DB02 Daily and when there are performance problems

tbd Table space sizes, table indices for tRFC and aRFC

Ensure that the data quality is sufficient, that there are no missing indexes, and that there is sufficient space available.

In general, if table sizes are larger than 500MB, reorganize the table and decrease its size.

See SAP Note 375566.

Monitor the growth of tables and indices, especially on tRFC and qRFC tables (ARFCSSTATE

ARFCSDATA

ARFCRSTATE)


ABAP dump analysis

ST22 Daily and when there are errors

tbd Dumps Check short dumps and analyze aborted programs.

Select a period and choose the List button. Select an error, and then choose Display to see the short dump.

For SAP-developed programs that experience frequent terminations, investigate SAP Corporate Portal for a possible resolution and, if necessary, open a customer message. For customer-developed programs, contact the appropriate member of the development team.


System log SM21 Daily and when there are errors

tbd Entries Check for general system program errors and table space errors or transactions that encounter errors.

Maintain date/time and select Reread system log. Position cursor on a time, and then choose Display.


Update tasks

SM13 Daily and when there are errors

tbd Status Check for entries that have status ERR.

Update records should be monitored regularly and the appropriate user should be contacted immediately to resolve any outstanding updates. Together with the user, determine whether the update request can be re-started or has to be deleted.



RFC Monitoring

© 2011 SAP AG page 122/124

Monitoring Object

Monitor TA/Tool

Monitor Freq.

Monitor Time

Indicator or Error


Responsibility

Display statistical record

Transaction STAD

When there is bad performance

tbd Status Determine the performance of single statistical records



RFC Monitoring

© 2011 SAP AG page 123/124

10 Further Information

Troubleshooting

If executing this best-practice document did not produce the desired results:

Search for related notes in SAP Corporate Portal.

Open an SAP customer message describing your problem.

Literature

For detailed information on how to administer an SAP R/3 system, see the book: Liane Will, SAP R/3 System Administration, 2003.

For information on how to monitor and tune the general system performance, see the book: Thomas Schneider, SAP Performance Optimization Guide, 2006.

SDN Blogs

New Business Process Monitoring functionalities in SAP Solution Manager – qRFC Monitoring http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/9861

http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/9861


RFC Monitoring

© 2011 SAP AG page 124/124

© 2011 SAP AG. All rights reserved.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies (―SAP Group‖) for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

Documents

086 RFC Monitoring