IMS Performance - General analysis

Richard [email protected] PerformanceApril 25th, 2016

IMS Performance: General Analysis

Agenda

Performance Evaluation Process

Resource Measurement Facility for z/OS (RMF)

IMS Performance Analyzer for z/OS (PA)

2



§ External Throughput Rate (ETR)

§ Internal Throughput Rate (ITR)– A measure of CPU efficiency or path length– ITR = Transactions per second / CPU % Busy

§ Transaction rate projection when CPU is 100% busy

§ Higher ITR is GOOD and indicates– workload is more efficient– Overall service time per transaction is lower

4

Stress/Measure

Analysis

Create/Customize Environment

IMS PerformanceEvaluation Cycle

Performance Evaluations

§ New IMS versions– New functions, enhancements

§ Languages– COBOL, Java

§ z Systems– z13, zEC12

§ DASD– DS8870, DS8800

§ z/OS Operating Systems– z/OS 2.2, 2.1

5

Current Hardware Environment

§ Processors– IBM System zSystems z13 Model 2964-7B1 (NE1)– IBM System zEnterprise EC12TM Model 2827-785 (HA1)

§ Storage– IBM System Storage® DS8000® series latest model:

§ DS8870 (16 FICON® channel paths) § DS8800 (8 FICON® channel paths)

§ Coupling Facilities– Internal Coupling Facilities (ICFs) with Coupling Facility

Control Code (CFCC) level 20.

6

IBM z13 Model 2964-7B1 (NE1)

IBM zEC12 Model 2827-785 (HA1)

IMS Performance Measurement Procedure

1. Start environment

2. Start workload and ramp up to steady state

3. Clear SMF datasets

4. Start RMF I with an interval set

5. Start RMF III with the same interval as RMF I

6. Wait for one or two RMF intervals

7. Switch SMF dataset

8. IMS - /SWITCH OLDS

9. IMS - /CHECKPOINT STATISTICS

10. IMS - /TRACE SET ON MONITOR ALL

11. Wait for measurement interval

12. IMS - /TRACE SET OFF MONITOR ALL

13. IMS - /CHECKPOINT STATISTICS

14. IMS - /SWITCH OLDS

15. Switch SMF dataset

16. Dump SMF dataset

17. Run RMF post processor report

18. Run IMS PA against OLDS used between the IMS /SWITCH OLDS period

19. Run IMS PA against MONITOR dataset

7

Resource Measurement Facility for z/OS (RMF)

What is RMF?

§ Performance management tool that measures system activity and generates reports based on data collected in System Management Facility (SMF) records

§ Provides performance analysis and problem determination by reporting:– LPAR CPU busy %– DASD device activity– Storage summary

§ RMF data gatherers:– Monitor I (historical)– Monitor II (snap shot monitoring)– Monitor III (sysplex analysis)

§ RMF Postprocessor: reporting tool for SMF records

9

RMF Address Spaces

10

RMF Control

S RMF

Monitor I (zz session)

F RMF,S ZZ

RMF Monitor III Gatherer

F RMF,S III

SYS1.PROCLIB(RMF) SYS1.PROCLIB(RMFGAT)

SYS1.PARMLIB(ERBRMF00)


Monitor I and III Data Gathering

11

1. MeasurementsCACHE SMF 74.5CHANNEL SMF 73CPU SMF 70.1CRYPTO SMF 70.2DEVICE SMF 74.1ENQ SMF 77IOQ SMF 78.3FCD SMF 74.7ESS SMF 74.8PAGESP SMF 75PAGING SMF 71TRACE SMF 76VSTOR SMF 78.2WKLD SMF 72.3IRLM SMF 79.152. TimingCYCLE(1000)NOSTOPSYNC(SMF)3. Reporting / RecordingRECORDREPORT(REALTIME)SYSOUT(A)4. User ExitsNOEXITS

Report Classes

§ Report classes are split by main components:– IMS Control region– IMS DL/I region– IMS DBRC region– IMS Dependent regions (e.g. MPP, IFP, BMP, JMP JBP, etc.)– IMS Connect– DB2 Master– DB2 DDF– IRLM– …

12

RMF Monitor I Overview

RMF Monitor I Parameters

§ Start RMF Monitor I using command: S RMF,,,MEMBER(xx)

14

SYS1.PARMLIB(ERBRMFxx)

RMF Monitor I CPU Activity Report

§ Number of processors

§ LPAR busy % (Total/Average), used to calculate ITR– ITR = ETR / LPAR busy %

§ Machine type and model – z13 Model 2964-7B1 (NE1)

15

RMF Monitor I Partition Data Report

§ DED – processors are dedicated to this partition (for repeatability)

§ DMX – processors are mixed dedicated and shared

16

RMF Monitor I Channel Path Activity Report

§ To display channel path types, use command D M=CHP

§ zHPF (z high performance FICON)

17

RMF Monitor I DASD Activity Report

§ AVG RESP TIME – Response time (ms): sum of IOSQ, PEND, DISC and CONN time. If RESP Time is high, look into these components

§ AVG PEND TIME – Average time an I/O request is waiting for hardware

§ Monitor the volumes on which IMS OLDS and WADS are defined to check their response times

18

RMF Monitor I Virtual Storage Activity Report

§ Most volatile is SQA and CSA both above and below the 16MB line

§ System CSA Usage for the Period

§ Key 7 IMS CSA Usage for the Period

19

RMF Monitor III Overview

RMF Monitor III Parameters

§ Start RMF monitor III via modify command: F RMF,START III, options… – e.g. F RMF,START III, NOOPTIONS, CYCLE(250), MINTIME(60), SYNC(1M),

CFDETAIL, NOSTOP

§ Start with member parameter:– F RMF,START,MEMBER(xx)

§ CFDETAIL - Coupling Facility activity data will be collected

§ SYNC(xM) – Synchronizes the report to a specified minute within the hour

21


RMF Monitor III Parameters

§ The RMF Post Processor is used to generate the RMF Workload Activity Report from the SMF log data sets

22

RMF Monitor III Summary Report

§ RMF Post Processor control statement - SUMMARY(INT,TOT)

23

RMF Monitor III Workload Activity Report

§ IMS Transaction Rate per second (ETR) – END / S

§ Service Unit (SU) – represents unit of work

§ Service Time (ST) – processor CPU time

§ SU / transaction – (SU/ sec) / ETR

§ ST / transaction – ST / ENDED

§ Set PROJECTCPU = YES in IEAOPTxx member to see how much processing can be offloaded onto zIIP or zAAP processors

24

RMF Monitor III Serialization Delay Report

§ Helps analyzing serialization related performance problems, provides contention information on system and address space level

§ REPORT(SDELAY): This report is only available in XML output format. You will need to specify XPRPTS DD name in postprocessor job

§ Generates both serialization delay summary and details. Summary provides information on system locks showing avg. contention time and GRS latch/enqueue statistics

25

RMF Monitor III Serialization Delay Report

§ Serialization delay details: provides information on lock details

§ For each lock type, shows max of 20 address spaces with the longest contention time reported

26

RMF Monitor III Coupling Facility Activity Report: Usage summary

27

RMF Monitor III Coupling Facility Activity Report: Usage summary

28

RMF Monitor III Coupling Facility Activity Report:Structure Activity§ Detailed information on each structure broken out by system and total

§ CHNGD and DELAYED REQUESTS, it could indicate shortage of subchannels

29

RMF Monitor III Coupling Facility Activity Report:Subchannel Activity§ Shows number and type of links

§ PTH BUSY: Number of CF requests rejected from this system because all paths were busy

30

IMS Performance Analyzer for z/OS (IMS PA)

What is IMS PA?

§ Key performance tool used to generate resource utilization and performance reports that can be used to analyze the performance of IMS systems.

§ We generally produce IMS PA log reports against the IMS log data sets, however IMS Monitor, IMS Connect Event Data, and OMEGAMON data and can be used as input to produce reports.

32

IMS PA Reports

§ Specific Reports– Transit Time Analysis– CPU Usage Report– DLI Call Statistics– Scheduling Statistics– Latch Statistics– Logger Statistics– Summary Report

33

IMS PA JCL for Batch Processing

§ Sample IMS PA JCL to generate a list of reports using IMS OLDS as input

§ IMS PA accepts IMS Monitor, IMS Connect Event Data, OMEGAMON ATF and TRF

34

IMS PA Log Information Report

§ Log information report shows statistics for various IMS log records

§ IMS x3730 record shows the transaction rate per sec (1,979 TPS)

§ IMS x5050 record shows the number of database updates

35

IMS PA DL/I Call Statistics

§ Similar information for transaction rate (ETR) can be found in DL/I statistics report under Transactions process by above Programs

§ A large value for Programs that reached Termination indicates no program reuse

36

IMS PA: Analyzing Full Function Transaction Response Time

End User vs. Host Response Time

§ End User Response Time

§ Host Response Time

38

Transit Time Analysis Report

§ The Total column under the Average Transit time category shows the average host response time

§ Transit Time Analysis By Transaction Code Report:

39

Transaction Flow

40

Input Queue Time

§ Time spent in the input queue or on the message queue buffers waiting for message region

§ Major part of response time in busy systems

41

Input Queue Time Tuning Considerations

§ Time waiting for scheduling

§ Processing time of scheduling

42

Analyzing Input Queue Time

§ Time waiting for scheduling:– Are there enough dependent regions to process the arriving transactions?

§ IMS PA: Internal Resource Usage, Dispatcher Statistics– Are the parameters concerning transaction scheduling properly tuned?

§ TRANSACT macro parameters§ EXEC parameters of MPP regions

§ Processing time of scheduling:– I/O to the message queue data set

§ IMS PA: Internal Resource Usage - Message Queue Pool Statistics– I/O to the ACBLIB data set

§ IMS PA: Total System IWAIT Detail – Check for scheduling failures, some useful commands

§ /DISPLAY commands§ IMS PA: Internal Resource Usage – Scheduling Statistics

43

Time Waiting for Scheduling

§ Are there enough dependent regions to process the arriving transactions?– What is the arriving transaction rate?– What is the average transaction processing time?– How many regions are available for scheduling?

44

Time Waiting for Scheduling

§ Are the parameters concerning transaction scheduling properly tuned?– TRANSACT macro parameters:

§ MSGTYPE= (CLASS=) Are there a sufficient number of regions defined with the proper transaction class?

§ PRTY= Used to determine scheduling priority of this transaction.§ PROCLIM= Number of messages of the transaction code a program can process in

a single scheduling.§ PARLIM= Threshold value of queued transactions used to schedule a new region.

– EXEC parameters of MPP regions:§ CLn=

45

Processing Time of Scheduling

§ I/O to the message queue data set– IMS PA: Internal Resource Usage - Message Queue Pool Statistics

§ High value of “Locate calls satisfied in the Pool” indicates Message Queue Pool is satisfying most requests without the need for I/O

46


§ I/O to the ACBLIB data set– IMS PA: Total System IWAIT Detail ACBLIB block loading IWAITs and ACBLIB

miscellaneous IWAITs.

§ Check for scheduling failures, some useful commands:– /DISPLAY PROGRAM – /DISPLAY TRANSACTION – /DISPLAY POOL (PSBW/DMBP/PSBP/EPCB) – /DISPLAY ACTIVE REGION

§ System definition considerations:– APPLCTL macro parameter, check SCHDTYPE=– DBPCB parameter, PROCOPT=

47


§ To check for scheduling failures, use the following reports:– Scheduling Statistics– Intent Failure Summary – Pool Space Failure Summary

48

Processing Time Events

§ Preparation of PCB

§ Initialization of the application program

§ Issuing the message prime GU call

§ Updating the database (if any)

§ Issuing the ISRT IOPCB

§ Sync point processing (The program updates the database physically)

§ Releasing the scheduled resources

49

Processing Time

§ This is a time from the first GU IOPCB call issued by the application to the completion of program processing after the syncpoint

§ The PGM EXEC column under Avg. Transit Time shows the average processing time per transaction code

50

Processing Time

§ Processing time of scheduling to first DL/I call

§ Application processing time

§ The Scd-DLI column of this report shows the average scheduling to first DL/I call time of each transaction code.

§ The Elapsed column of this report shows the average processing time of each PSB and transaction code.

51

Time to Schedule First DL/I Call

§ Processing time of scheduling to first DL/I call tuning considerations

§ Reducing the program loading time by:– Preloading the high use application and language modules

§ Reducing the I/O to PGMLIB– Using the LLA function of z/OS to reduce directory I/O– Using the VLF function of z/OS to reduce module I/O

§ Reconsider the concatenation sequence of STEPLIB on MPP execution JCL to optimize searching

§ Checking the usage of DASD containing the PGMLIB to optimize I/O time

52

Application Processing Time

§ Application processing time tuning considerations

§ Reducing the I/O IWAIT elapsed time by:– Reducing each I/O IWAIT elapsed time

§ Check the accessing rate to the DASD containing the databases and change the allocation if needed

§ If the I/O rate to the message queue data set is high, consider increasing the size of the message queue buffer (QBUF).

– Reducing the number of I/O IWAIT per CALL§ Reorganize the database to improve space utilization.§ Check the DBD and change the CI size or other parameters if needed.§ Check the logic of the randomizer if the database is HDAM or PHDAM.§ Increase the database buffers because the buffer shortage results in the re-reading

of the CIs or blocks which have already been read in the previous calls.

53

Output Queue Time

§ The time from the program termination to the completion of sending a message

§ The output column under Avg. Transit Time report shows the average output queue time for each transaction code

§ The output queue time should be close to 0 if the QPOOL is optimized

54

Summary Report

§ Summary Report for Full Function workload

§ Same fields as in the transit analysis report can be requested (Input queue, process time, etc.) but many other values are shown such as schedule time, syncpoint time etc.

§ Helps narrow down long input queue and processing times

55

IMS PA: Analyzing Fast Path Transaction Response Time

Transaction Flow

§ Transit time is the time spent before the scheduling of the FP router to the output message

57

Total Transit Time

§ The Total column under the Average Transit time category shows the average transit time

§ Fast Path (EMH) Transit Time Analysis By Transaction Code Report:

58

Input Queue Time

§ The time from scheduling the fast path input edit/routing exit routine to issuing of the GU call by the application program

§ The Input Queue column under the Average Transit Time category shows the average input queue time for each transaction code

59

Region Occupancy

§ Fast Path transactions are processed as wait for input, input queue time is equal to the time waiting for an IFP region

– Are there enough dependent regions to process the arriving transactions?§ How high is the transaction arriving rate?§ How long is the average transaction processing time?§ How many IFPs are available for scheduling?

– IMS PA IFP Region Occupancy report shows the region utilizations. High region utilization may cause long input queue times.

60

Processing Time

§ The time from GU call of the application program to the sync point

§ Database is updated, if the program is doing so, and sync point processing is done

§ The Pgm Exec column under the Average Transit Time category shows the average program execution time for each transaction code

61

Processing Time

§ Program execution time is mainly the time to access the database, pick the transactions to be analyzed

§ FP Resource Usage and Contention report shows DEDB calls, ADS I/O calls, common buffer usage and contention

§ Analyze transactions which access the ADS more than the number of DEDB calls.– (DEDB Calls Reads Avg + DEDB Calls Updates Avg < ADS I/O Reads Avg)

§ This indicates the situation when the segment is contained in the overflow area (dependent or independent) or the database has long synonyms chains. The reorganization of the database accessed by the transaction can improve the processing time.

62

Processing Time

§ Transactions which includes the process of CI contention. (Contentions CI/Sec > 0)– This indicates when the segment is contained in the overflow area (dependent or

independent). Or the multiple programs which access the same root segment can run simultaneously. The reorganization of the database accessed by the transaction can improve the processing time.

§ Transactions which includes the process of buffer steal. (Common Buffer Usage Stl > 0)– This indicates the invocation of buffer steal caused by the shortage of NBA buffer.

Recommendation is to increase the number of NBA.

§ Transactions which includes the process of buffer wait. (Common Buffer Usage Wts > 0)– This indicates the invocation of buffer steal caused by the shortage of whole Fast Path

buffers. Recommendation is to increase the allocation of DBFX.

§ Transactions which includes the process of buffer wait. (Contentions Tot OBA > 0)– This indicates the invocation of OBA wait caused by the shortage of NBA.

Recommendation is to increase the allocation of NBA.

63

Output Queue Time

§ Time from syncpoint to entry to the output buffer

§ The Output Queue column under the Average Transit Time category shows the average output queue time per each transaction code.

§ Output queue time is mainly the time needed to write the logs to disk, so it depends on the disk where the WADS and OLDS are located.

64

Output Queue Time

§ IMS Logging Rate = BLKSIZE (bytes) * OLDS Writes initiated per sec

§ Average WADS/OLDS I/O write times

65

Summary

§ Performance Evaluation Process

§ RMF– http://www-03.ibm.com/systems/z/os/zos/features/rmf/

§ IMS PA– http://www.ibm.com/support/knowledgecenter/SSAVHQ_4.4.0/ipi-product_landing.dita– http://www.redbooks.ibm.com/abstracts/sg248071.html?Open

66

Technology

IMS Performance - General analysis