SAP basis typical problem

1

Typical Problems

Active Global Support

Application Platforms - Performance CoE

May 2011

2

© 2011 SAP AG. All rights reserved. 2

Typical Problems

SQL – Expensive DB accesses

1. Many Records

2. Many Executions

3. Identical SELECTs

4. High AvgTime/R. or Time/Exec.

ABAP – Inefficient Internal Table Processing and others

1. Nested Loops

2. Reads in Loops

3. COLLECT

4. Explicit Sort in Loop

5. Long running ABAP statements invisible in trace

RFC

1. Long running synchronous RFC

Update

1. Long running synchronous Update

Slide

3

9

14

21

28

37

45

51

59

62

69

3

SQL – Many Records

Expensive DB accesses

• Valid for all statements: SELECT, INSERT, UPDATE, DELETE

• The AvgTime/Rec can be higher than 1 – 10 milliseconds (10 – 500

microseconds when cached) for changing statements, depending on the

number of indexes

• Examples:• SE16, read all DD02L records

• Z_RFC_CALLS, parameter empty FAE = ‘X’

4



Many Records

Symptoms

Long duration, good value for AvgTime/R., many records (high Rec/Exec)

Document your findings

5



Many Records – What to check 1

Double click the statement in

ST05 in order to capture the

SQL statement that was sent to

the DB

Navigate to the source code of

the SELECT in order to capture

the OPEN SQL statement and

the ABAP call stack

Check whether the statement is

a FOR ALL ENTRIES (FAE)

Document your findings

6



Many Records – What to check 2

Check if table buffering is activated for the table

– Column BfTp contains one of (FUL, GEN)

– SQL statement text contains ORDER BY ... (FUL, GEN)

Check if the OPEN SQL matches the SQL statement in ST05

7



Many Records – What to do and where to go

Save the SQL trace in order to prevent it from being overwritten (see ST05 slide deck)

Collect your analysis‘s results from the previous slides (your screen shots and descriptions)

together with the name of the trace file (or the variant name for the new ST05) together

with the application server

– If the statement is driven by or occurs within SAP standard code and the table is not buffered

Application Expert

– If the statement is driven by or occurs within SAP standard code and the table is buffered

ABAP Expert

– If the call stack includes customer code

ABAP Expert

o If the table is buffered a technical analysis regarding table buffer needs to be done

o If the table is not buffered the ABAP code needs to be analyzed

8



Many Records – Hints for Optimization

Check how the result set is used in the application

– Use the where-used list for the target (table, work area) of the SELECT statement to understand how

the data is later used

o Deletions

o Filters (IF, CASE, CHECK, ...)

– Move filter criteria to the WHERE clause if possible („filter as early as possible“)

FOR ALL ENTRIES

– Make sure the driver table is not empty (should already have been checked in ABAP)

– Remove duplicate entries from the driver table

9

SQL – Many Executions



• The AvgTime/Rec can be higher for changing statements, depending on the

number of indexes (since they need additional I/Os for maintenance).

• Example report: Z_NESTED_SELECTS_OPT

10



Many Executions

Symptoms

Long duration, many executions and no identicals combined with a good value for

AvgTime/R.

11



Many Executions – What to check 1


the SELECT in order to

understand the position of the

call stack

Use the scope and/or forward

navigation or the bottom-up

analysis to find out the driver of

the statement (values passed to

the SELECT e.g. in a loop or

nested SELECT)

(see ABAP Basics slide deck)


driven by or executed within

customer or SAP standard code

(use column ‘changed by’ in

combination with the bottom-up

analysis

(see ST12 slide deck)

12



Many Executions – What to do and where to go

Save the SQL trace in order to prevent it from being overwritten (see ST05)

Collect your analysis results from the previous slides (your screen shots and descriptions)

together with the name of the trace file (or the variant name for the new ST05) together

with the application server

ABAP Expert

If the statement is driven by or occurs within SAP standard code

Application Expert

• 1st example: note it’s a JOIN

• 2nd example: VIEW

• For changing statements (INSERT, UPDATE, DELETE) try to use the array

interface (INTO / FROM TABLE or other bundling techniques)

13



Many Executions – Hints for Optimization

Bundle the SELECT statements

– In case the statement is driven by a loop over an internal table, rewrite the statement using a SELECT

... FOR ALL ENTRIES (FAE)

– In case the statement is driven by SELECT ... ENDSELECT loops, rewrite the statement using a JOIN

o For an inner join, alternatively consider creating a view in DDIC (can be reused by other programs

and be buffered)

– In case the selection criteria is known in advance, reading all relevant records with an array fetch and

processing them with READ statements later could be an option too. Make sure that only the required

records are read.

Examples

14

SQL – Identical SELECTs




number of indexes

• Example report: Z_NESTED_SELECTS

15



Identical SELECTs

Symptoms

Many value identical SELECT statements are executed

In old ST05, identical statements can be displayed from the basic list through

‘trace list -> display identical selects’

If MinTime/R =-1 bug in ST05

16



Identical SELECTs – What to check 1

Double click on % Identical (new ST05) to see which values are selected and how often

and how many records are returned

Check if the table is set to be buffered

17



Identical SELECTs – What to check 2


the SELECT in order to

understand the position of the

call stack

Use the scope and/or forward

navigation or the bottom-up

analysis to find out the driver of

the statement (values passed to

the SELECT e.g. in a loop or

nested SELECT)


driven by or executed within

customer or SAP standard code

(use column ‘changed by’ in

combination with the bottom-up

analysis

18



Identical SELECTs – What to do and where to go

Save the SQL trace in order to prevent it from being overwritten

Collect your analysis results from the previous slides (your screen shots and descriptions) together with

the name of the trace file (or the variant name for the new ST05) together with the application server

If the statement is driven by or occurs within SAP standard code

Application Expert

If the buffer type (BfTp) is FUL, GEN, SNG, CUST or DDIC

ABAP Expert

If the buffer type (BfTp) is DEFUL, DEGEN, DESNG or DEACT

Application Expert

If the table is not buffered

ABAP Expert

19



Identical SELECTs – Hints for Optimization

Duplicate Analysis

– Avoid identical SELECTs caused by passing duplicate values to the statement

Use reading module (example MARA_SINGLE_READ)

– Reuse existing reading module, if available (search function modules with <table>*SINGLE* or

<table>*ARRAY*)

– Negative buffer (statements which return 0 records to be added with flag notfound =‚X‘)

– Memory management, delete opportunity

Activate buffering

Rewrite statement to use the buffer

20

SQL – High AvgTime/R. or

Time/Exec.




number of indexes

• Example: Z_ST05_HIGH_AVG_TIME_PER_REC

21



High AvgTime/R. or Time/Exec.

Symptoms

Statement consumes significant amount of time

AvgTime/R. or Time/Exec is high

– Where high is significantly higher than 10 ms (10.000 microseconds, all times in ST05 are in

microseconds)

– Please note if the Rec/Exec is 0 <value <1, we use the Time/Exec instead of AvgTime/R.

– Please note that the Avg. Time / R. refers to physical I/O times. In case the program accesses data

from the db cache (logical I/O) the Avg. Time / R. can be 1ms or a few hundred microseconds. Even

with such times we may have an access path problem. Therefore the top statements (sorted

descending by duration) should be analyzed in any case!

22



High AvgTime/R. or Time/Exec. – What to check 1

Check the table type

– In case it is a View the avg. Time / R. might be higher while the access is still ok. This depends on

several things like e.g. number of tables involved in a view. In general we can say that up to 10 ms per

record per table are ok.

– In case it is a Pool or a Cluster table we might see more than 10 ms for Avg. Time / R. per record as

well, however if the time is significantly higher we have to forward the problem to the DB CoE anyway.

– In case it is a transparent table the avg. Time / R. should be < 10 ms

– You may want to use SE12 to see the object in more detail. (see ABAP Basics)

23



High AvgTime/R. or Time/Exec. – What to check 2

Check the statement text by double clicking on the statement

– Check whether the statement is a simple one. That is if the statement does not contain any of these

key words:

o DISTINCT, ORDER BY, SUM, COUNT, AVG, MIN, MAX, JOIN, GROUP BY ...

24



High AvgTime/R. or Time/Exec. – Check available indexes

In SE12 click on Utilities – Database Oject – Display in order to compare the available

indexes with the where clause

Alternatively use the DDIC information of ST05

25



High AvgTime/R. or Time/Exec. – What to do and where to go

Save the SQL trace in order to prevent it from being overwritten

Collect your analysis results from the previous slides (your screen

shots and descriptions) together with the name of the trace file (or

the variant name for the new ST05) together with the application

server to the DB Expert

Click on the menu System – Status in order to determine the

database

If the statement does not contain one of the key wordsDISTINCT, ORDER BY, SUM, COUNT, AVG, MIN, MAX, JOIN, GROUP BY

...

we most likely have an inefficient execution plan that needs to be

analyzed by the DB CoE

DB EXPERT

If the statement is a complex one (see slide 23) the technical

execution may still be fine

ABAP EXPERT / APPLICATION EXPERT

26



High AvgTime/R. or Time/Exec. – Hints for Optimization

The execution plan of the statement needs to be analyzed

– All available indexes must be analyzed whether they could support this statement

– If there is an appropriate index in place but not used, a root cause analysis (why the index is not used)

needs to be done on database level

– If there is no appropriate index in place, the statement needs to be checked whether it could be

rewritten (sometimes an ABAP or APPLICATION expert needs to be involved)

– If the statement can not be rewritten, an index creation has to be considered by the database expert

considering the circumstances of the application / system

SQL statements solution database

– http://service.sap.com/~sapidb/011000358700001413351999E

Is the right table used here

– For some application tables there also exist secondary tables which could be used depending on the

where-clause. See notes

o 185530 Performance: Customer developments in SD

o 187906 Performance: Customer developments in PP and PM

o 191492 Performance: Customer developments in MM/WM

If the technical executions is ok, the ABAP context needs be analyzed

27

ABAP – Nested Loops

Inefficient internal table processing

28


ABAP Inefficient Internal Table Processing

Nested Loops

Symptoms

The statement consumes a significant amount of time

The statement is a nested LOOP, DELETE or MODIFY

Net time is high (similar to gross time)

29



Nested Loops – What to check 1

Determine the position of

the statement

Verify the LOOP

(DELETE, MODIFY)

statement occurs within

an outer LOOP

– Please note that if there is

no direct loop visible the

statement might be in a

modularization unit which

is called in a loop

– Use forward navigation

(doubleclick on the form

routine and the displayed

hits) to navigate

Place the cursor on the inner LOOP statement (or the modularization unit which

contains the inner LOOP). The status line at the bottom of the window shows

the scope of this statement.

30




The new editor also shows the scope of a

statement

– If a statement is directly in a loop (same

hierarchy level) which starts (and ends) many

lines away from the source of the expensive

statement this option is very helpful

• Click on the expensive statement and then activate the bottom-up hierarchy

(1)

• For readability consider filtering everything that is not part of the call

hierarchy (2)

• Note significant changes in the number of executions of a call this is

usually where the loop starts. It is also possible that the call hierarchy splits

into several branches, for which we would see several swim lanes.

31




Determine the calling position and hierarchy

– Instead of the forward navigation the bottom up analysis can be very helpful

– Follow the arrows (see ST12) and evaluate column ‚number of calls‘ to see where loops might start to

increase the number of calls (note that LOOP AT statements are not part of the call hierarchy and are

not displayed in the bottom up analysis)

12

3

• See (3) on previous slide on how to get this view

32





– The per mod unit can be helpful as well

– Evaluate the number of calls column to see where loops starts increasing the number of calls

– Double click on the name of the internal table and if necessary follow the forward

navigation to the declaration (<name> LIKE / TYPE ...)

33




Determine the table type

Typical table definitions

TYPE [STANDARD/ SORTED/HASHED] TABLE OF

(default standard, if not explicitly

specified)

OCCURS <n>

TYPE <table_type> (DDIC or ABAP type)

(See ABAP Basics)

1

34



Nested Loops – What to do and where to go

For ST12: Note the analysis GUID

For SE30 / SAT: Note the application server and the trace details (user, date, time)

Document the detailed problem description

– table type (declared where, how)

– position of read stmt

– call stack, loop

Forward this information to an

ABAP Expert (not optimized statements)

Application Expert (optimized statements, e.g. sorted internal table,...)

Examples:

• Check if duplicate entries in internal table can be avoided – SORT <itab> and DELETE ADJACENT DUPLICATES FROM <itab>

• Sort standard internal tables wherever possible and use

READ ... BINARY SEARCH• If possible, change the table type into sorted or hashed

Perform a where-used analysis for the internal table

In the source code identify coding pieces where different sort criteria might have an

impact on the application logic

If the application logic does not depend on specific sorting criteria, change the table

type into sorted or hashed

If dependencies are found, consider copying the internal table into a temporary

internal table, process and refresh the internal table afterwards (buy performance with

memory)

• Loop at sorted standard table– Use READ ... BINARY SEARCH TRANSPORTING NO FIELDS to find the right index

and then LOOP FROM <index>. A t every loop check the loop condition is still valid,

otherwise exit the loop.

• Nested loops– Inner LOOP ... WHERE should be replaced by a sorted internal table and LOOP ...

FROM INDEX. Make sure the join condition between inner and outer internal tables is

still valid within each inner loop, otherwise exit from inner loop and continue with the

next value of the outer loop.

35



Nested Loops – Hints for Optimization

If the internal table is a sorted or hashed table we have to compare the key of the internal

table with the key used in the statement (WHERE for LOOP, DELETE, MODIFY)

For sorted tables we need at least the leading part of the key, for hashed tables we need

the full key of the table

For unsorted tables, consider sorting the internal table by the where condition

– Define as type sorted internal table, or

– Sort at runtime using the SORT statement (make sure this does not affect other calls of the same

internal table within the program)

36

ABAP – Reads in Loops


• READs in LOOPs typically consume a significant amount of time when the

READ is called many times.

• Net time is high for the READ statement

• Note that the bottom-up analysis only shows the READ and its call stack.

LOOP statements are not part of the call stack.

• Check avg time/exec (net time / no. of calls). Should be in the single-digit

microseconds range.

37



Reads in Loops

Symptoms

Statement consumes a significant amount of time

Statement is a READ

Net time is high

High number of statement calls

• When using the bottom-up analysis for the READ statement, the LOOP

containing the READ will not be displayed.

• Other LOOP-like statements: DO … ENDDO, WHILE … ENDWHILE,

SELECT … ENDSELECT.

38



Reads in Loops – What to check 1

Note the statement

position and definition of

the READ

– The new editor also

shows the scope of the

statement

– Document the LOOP(s)

which drive the READ

using forward navigation if

needed

Confirm that the READ

statement is in a LOOP

where the key is filled by

the LOOP


(1)


hierarchy (2)




39





– Can be done with forward navigation or bottom-up analysis

– Follow the arrows (see ST12) and evaluate the number of calls column to see where loops might start

to increase the number of calls (note that LOOP AT statements are not part of the call hierarchy and

are not displayed in the bottom-up analysis)

1 23

• See previous slide on how to get to this view

40





– Can be done with per mod unit as well.

– Evaluate the number of calls column to see where loops start increasing the number of calls

– Double click on the name of the internal table and if necessary follow the forward

navigation to the declaration (<name> LIKE / TYPE ...)

– For sorted tables also determine the key fields and if the key is defined as unique or

non-unique

41




Determine the table type

Typical table definitions:

TYPE [STANDARD/ SORTED/HASHED] TABLE OF

(default standard, if not explicitly

specified)

OCCURS <n>

TYPE <table_type> (DDIC or ABAP type)

(See ABAP Basics)

1

42



Reads in Loops – What to do and where to go



Detailed problem description (table type (declared where, how), position of read stmt, call

stack, loop)


ABAP Expert (not optimized statements)

Application Expert (optimized statements, e.g. sorted internal tables,...)

Examples:

• Check if duplicate entries in internal table can be avoided • SORT <itab> and DELETE ADJACENT DUPLICATES FROM <itab>

• Sort standard internal tables wherever possible and use READ ... BINARY

SEARCH

• Loop at sorted standard table• Use READ ... BINARY SEARCH TRANSPORTING NO FIELDS to find the right index

and then LOOP FROM <index>. A t every loop check the loop condition is still valid,

otherwise exit the loop.

• Leading part of the key: Leading key fields which are used in the where

condition without gaps

43



Reads in Loops – Hints for Optimization

If the READ is on a standard table and the expensive statement does not use the BINARY

SEARCH option, consider sorting the internal table and use READ ... BINARY SEARCH

If the internal table is a sorted or hashed table we have to compare the key of the internal

table with the key used in the statement (WITH KEY for READ)

For sorted tables we need at least the leading part of the key, for hashed tables we need

the full key of the table

44

ABAP – COLLECT


• COLLECTs in LOOPs typically consume a significant amount of time when

the COLLECT is called many times and the COLLECT cannot be optimized

by the SAP kernel

• Net time is high for the COLLECT statement

• Note that the bottom-up analysis only shows the COLLECT and its call stack.


• Check avg time/exec (net time / no. of calls). Should be in the single-digit

microseconds range.

45



Collect

Symptoms


Statement is a COLLECT

Net time is high


46



Collect – What to check 1

Note the statement


the COLLECT



statement

• (1) Generate a where-used list by placing the cursor on the internal table

that is used with the COLLECT and

(2) pressing the where-used list button. Confirm the pop-up dialog.

• Double clicking on the found locations for changing statements s(such as

APPEND, INSERT, MODIFY, SORT, ASSIGNING) will lead to the source

code

47




Generate a where-used list

Look for statements which

change the internal table of

the COLLECT

Document their source code

positions1

2

48

Local Search: <Ctrl-F> within ABAP editor screen searches local (part of)

program




1 To perform a full text search on the

main program, use buttons

(Find)

(Find next)

Searching globally within the ‚main

program‘ considers used

• ABAP Includes

• Classes (of used methods)

• Function Groups (of used

function modules)

2

2

49



Collect – What to do and where to go



Detailed problem description

– Position of COLLECT statement

– Position of statements which change the internal table


ABAP Expert (if driven by customer programs)

APPLICATION Expert (if driven by SAP standard programs)

– For standard tables it is important to understand that the temporary hash administration of

the target table must not be destroyed

– Using a hashed table instead of a standard table is the better and more stable choice.

–COLLECT on standard tables is obsolete and should no longer be used

50



Collect – Hints for Optimization

Analyze which statement (other than COLLECT) changes the internal table

Check whether the changes are necessary or can be omitted

Target table should not be changed

Result of the COLLECT is unique

– Hashed table should be used for COLLECT

COLLECT on a standard table is obsolete

51

ABAP – Explicit Sort in Loop


• SORTs in LOOPs typically consume a significant amount of time when the

SORT is called many times.

• Net time is high for the SORT statement

• Note that the bottom-up analysis only shows the SORT and its call stack.


52



Explicit Sort in Loop

Symptoms


Statement is a SORT

Net time is high






53



Explicit Sort in Loop – What to check 1

Note the statement


the SORT

– Use the forward

navigation to find the

driving statements

Confirm that the SORT






54




Note the statement


the SORT

– Use the forward

navigation to find the

driving statements



statement

Confirm that the SORT



(1)


hierarchy (2)




55





– Can be done with forward navigation or bottom-up analysis

– Follow the arrows (see ST12) and evaluate the number of calls column to see where loops might start

to increase the number of calls (note that LOOP AT statements are not part of the call hierarchy and

are not displayed in the bottom-up analysis)

1 23

See (3) on previous slide on how to activate the per-mod. unit display

56





– Can be done with per mod. Unit

– Evaluate the number of calls column to see where loops might start to increase the number of calls

57



Explicit Sort in Loop – What to do and where to go



Detailed problem description (table type (declared where, how), position of read stmt, call

stack, loop)




– Analyze the program data flow in order to understand where the SORT is really necessary

58



Explicit Sort in Loop – Hints for Optimization

Avoid the sort by inserting lines without destroying the sort order

SORT only when necessary

Consider moving the SORT outside of the LOOP

59

ABAP – Long running ABAP

statements invisible in trace

Others

60


ABAP – Others

Long running ABAP statements invisible in trace

Symptoms

Net time spent in mod. unit cannot be explained by recorded statements (when traced with

internal tables)

61


ABAP – Others

Long running ABAP statements invisible in trace – what to do



Detailed problem description

– Document call stack and the observation that time cannot be explained with any of the recorded

statements




62

Long running synchronous RFC

RFC

• This is valid for both scenarios:• many short RFC calls

• Few long RFC calls

63


RFC

Long running synchronous RFC - Symptoms

Symptoms

An ABAP trace shows a high ABAP time and statements where the top time consumers

start with RFC ...

64


RFC

Long running synchronous RFC - What to check 1

Check the source code

and verify that it is a

CALL FUNCTION ...

DESTINATION

65


RFC


The time for the RFC is not spent in the process we just traced

the process just waits for the RFC to be finished (ROLL WAIT time if executed in DIA,

PROCESSING time if executed in BGD)

It is spent in the RFC module or in the network

Analyze remote execution time

Check the STAD record and compare the Calling time with the remote execution time

If the remote execution time is a large part of the calling time we have to trace the remote

execution (see next slide)

If the remote execution time is a small part of the calling time the time is spent in the

network, an ABAP or Application expert has to analyze if the no. of calls can be reduced

66


RFC


In order to analyze the remote execution time in more detail we have to run and trace the

application again and

– execute a context trace (see ST12)

– Collect the external trace file into ST12 (choose the one with the name of the function group of the

called RFC module

– Alternatively you may want to use the TASKS & HTTP trace for any workprocess on all server s in

order to trace the main program and the RFC program

Now we can see the trace of the code in the RFC function module

67


RFC

Long running synchronous RFC - What to do and where to go

Collect the ST12 trace and the STAD record with your findings

Depending on the finings in the trace of the RFC module forward the relevant data (ST05,

ST12, ... Traces and descriptions)

ABAP, APPLICATION or DB EXPERT

The RFC type can be checked by

APPLICATION EXPERT

68


RFC

Long running synchronous RFC - Hints for Optimization

Analyze the traces of the RFC function module and proceed accordingly

It might be possible to change the synchronous RFC to an asynchronous RFC

The type of the calls (synchronous/ asynchronous), the number of the calls (bundling) and

the code executed in the calls might be an issue that could be optimized

69

Long running synchronous Update

Update

70


Update

Long running synchronous Update - Symptoms

Symptoms

An ABAP trace shows a high DB time where the COMMIT WORK statement consumes a

significant part of the time

71


Update

Long running synchronous Update - What to check 1

Verify that it is a

synchronous update

Check the source code

and verify it is a

COMMIT WORK AND

WAIT that is executed

72


Update

Long running synchronous Update - What to check 2

The time for the COMMIT WORK AND WAIT is not spent in the process we just traced

It is spent in the update module execute in the update process

In order to analyze that time in more detail we have to run and trace the application again

and

– execute a context trace (see ST12)

– Collect the external trace file into ST12 (choose the one with program RSM13000)

– Alternatively you may want to use the TASKS & HTTP trace for any workprocess on all server s in

order to trace the main program and the update program

Now we can see the trace of the code in the update function module

73


Update

Long running synchronous Update - What to do and where to go

Depending on the findings in the trace of the update module forward the relevant data

(ST05, ST12, ... Traces and descriptions) to

ABAP, APPLICATION or DB EXPERT

The update type can be checked by

APPLICATION EXPERT (e.g. regarding transactional correctness)

74


Update

Long running synchronous Update - Hints for Optimization

Analyze the traces of the update function module and proceed accordingly

It might be possible to change the synchronous update to an asynchronous update

The type of the calls, the number of the calls and the code executed in the calls might be

an issue that could be optimized

Documents

SAP basis typical problem