Ims12 workbench data visualization - IMS UG May 2014 Sydney & Melbourne

© 2013 IBM Corporation ©2014 IBM Corporation

IBM Transaction Analysis Workbench Version 1.2 IMS-DB2 and more

© 2013 IBM Corporation ©2014 IBM Corporation ©2014 IBM Corporation

Agenda 1. Introduction: issues of problem solving today

2. Solution: IBM Transaction Analysis Workbench

3. DB2 exception reporting and extract

4. Interactive log analysis – signature feature

5. ISPF dialog – plus Eclipse GUI for first responder

6. Session Manager – collaborative problem solving

7. Applications development

8. IMS DB2 scenario

9. SMF reports

10.CICS-DBCTL

1


Applications have evolved

2

1980s application: in-house users only; simple data, single data store

Today: users are customers; data is complex, heterogeneous, often distributed

Transaction Manager: IMS DC or CICS

Users: company employees

Database: IMS DB or DB2

Users: employees, customers (public), automated systems

Transaction Manager: IMS DC or CICS

WebSphere MQ

IMS DB

DB2

non Z servers


DB2

MQ

DLI Where did the delay occur? A single transaction can

have activity across many subsystems

To quickly identify performance issues, you need to track the entire transaction

Subsystem-specific approaches and tools offer a limited perspective

Each subsystem has its own activity log and SMF records

3

IMS


Traditional approaches and tools based on “silo” model

4

There are many tools to help analyze individual transaction environments on System z:

One transaction, not four!

CICS Performance Analyzer

IMS Performance Analyzer

DB2 Performance Expert

Many others...

CICS IMS DB2

CICS

IMS

DB2

MQ

One transaction, not three!

IMS DB2

MQ

Each tool is well-suited to its environment, but SME uses their own tools


IBM Transaction Analysis Workbench for z/OS A tool for problems in the big picture:

– For “first responders” and subject-matter experts (SMEs)

– For SMEs in different areas

Provides a life cycle view of transaction activity across subsystems – Changes the way problem resolution is performed

– Ensures everyone is looking at the same transactional data

Goes in-depth. – Uses SMF, trace, and log records to follow transaction flow

Better assignment of problems to the correct group – Improved confidence in problems assigned to experts

5


Coverage: many subsystems, many logs

IBM Confidential

IMS CICS DB2 WebSphere MQ, App Server z/OS

IMS log and trace CMF performance class (SMF 110) DB2 log MQ log extract Various SMF

record types

IMS monitor CICS trace DB2 trace (IFCIDs)

MQ statistics (SMF 115-1, -2) OPERLOG

CQS log stream MQ accounting (SMF 116)

IMS Connect event data (collected by

IMS Connect Extensions)

WAS request activity

performance statistics

(SMF 120-9)

OMEGAMON ATF

IRLM long lock detection

(SMF 79-15)


Cross support is key with fewer staff Why is support so difficult?

– SME may be in silos • Cross training may be difficult • Takes too much time using current tools • Unable or unwilling to cooperate easily

What does good cross-platform tool achieve? – Conservation of SMEs’ time (a valuable, limited resource)

– Transparency of information (everyone using the correct data from the same time period)

SMEs need to see the big picture and the benefit of collaboration – Reduced time to resolution

– More SME time focused on problem resolution

– Cross-training of first responders and SMEs

7


What’s new in Workbench V1.2 CICS-DB2 and IMS-DB2 transaction exception processing

Was the problem in CICS or DB2? IMS or DB2? (CICS-DBCTL already supported in V1.1.)

Enhanced support for DB2 trace records Detailed field-by-field formatting for more than 60 IFCIDs.

Workflows and session templates Subject-matter experts (SMEs) can define a workflow (a sequence of analysis tasks) and save it in a session template. When creating a new session, users can select the session template that best matches the report problem.

Eclipse-based rich client platform (RCP) user interface Implements a subset of the ISPF dialog: create a session; run a workflow; assign to appropriate SME.

Automated SMF file selection

SMF 42.6 DASD Data Set I/O report

8


Problem: today’s instrumentation data overwhelms traditional tools Good performance monitoring should identify possible performance

issues before they become critical

Today's systems create so much instrumentation data that existing techniques cannot keep up: takes too long, costs too much!

9

Instrumentation data

Report

Traditional batch analysis/reporting tool

• Processes and reports on all records

• Processing time and cost grows with size of instrumentation data, beyond practical limits

• Reports can grow too long to be useful, and contain unwanted detail


Solution: Exception detection

10

Terabytes of instrumentation data

Workbench

Transaction index

Exception: a transaction that matches specific exception criteria, such as long response time or an abend

z/OS batch process that efficiently crawls data for exceptions

Reporting and interactive analysis on reduced data


Solution: Transaction Index

11

CICS performance records SMF 110

CICS Transaction Index

DB2 accounting records

SMF 101, IFCID 3

DB2 Transaction Index

IMS log

IMS Transaction

Index

WebSphere MQ accounting

records SMF 116

WebSphere MQ Transaction

Index

Workbench exception processing

1. Transaction indexes are created by the workbench (a session workflow will create them) 2. They are used to identify all the transaction and UOR workloads in IMS, DB2 and CICS 3. The transaction index is a special extract - one record per transaction in time sequence 4. Contain summarized performance and resource usage information 5. Can be filtered to include exception transactions only 6. Can be used for reporting and to identify problem transactions


DB2 exception extract and reporting (DB2X) The DB2 exception extract and reporting facility helps to identify

and analyze problems associated with DB2.

DB2 accounting records (SMF 101) are checked against user-specified exception criteria. DB2 threads that are identified as exceptions are reported and extracted for further analysis.

The detailed list report lists those DB2 accounting records that triggered an exception – with the reason

The extract writes those DB2 accounting records that triggered an exception to an extract data set. This data set can then be used for more specialized analysis, for example: o Tivoli OMEGAMON XE for DB2 Performance Expert o Transaction Analysis Workbench ISPF dialog interactive

analysis; merged with the DB2 log and other traces to examine the transaction lifecycle picture.

12


Response and CPU time: / Apportion rollup Response time . . 0.5 (0.000001-99 seconds) CPU class 1 . . . 0.1 In-DB2 elapsed . . 0.2 CPU class 2 . . . 0.05 Database I/O . . . 0.05 Lock suspend . . . 0.05 Stored Procedure: Elapsed . . . . . 0.1 (0.000001-99 seconds) CPU . . . . . . . 0.05 Row activity: Fetched . . . . . 1000 (0 to 999999) Inserted . . . . . 100 Updated . . . . . 100 Deleted . . . . . 100 Locking: Deadlocks . . . . 1 (0 to 999999) Suspends . . . . . 20 Timeouts . . . . . 1 Lock requests . . 50 Buffering: Get pages . . . . 50 (0 to 999999) Update pages . . . 30 Logging: Log records . . . 100 (0 to 999999) Abnormal conditions: Abort . . . . . . 1 (1=check for condition) Check pending . . 1

DB2X: Specify exception thresholds

13

DB2 threads that exceed one of more of these thresholds are considered exceptions


DB2X: List of DB2 exceptions

V1R2M0 2014-02-20 Thursday DB2 Exception List Page 1

SSID Correlation Connect Plan Auth id Time Exception Threshold LUWID

---- ------------ -------- -------- -------- --------------- ----------------------- --------- -----------------------------

DBA4 MQP1DB2SRV02 RRSAF CSQ5L710 STC@ZOSN 10:02:04.512982 Abort 1 1 FTS1/DBA4LU/CCBDD2D0C2CD/0002

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Response 1.402972 0.5 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Class 1 CPU 0.387519 0.1 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Rows fetched 6329 1000 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Lock requests 267 50 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Get pages 2264 50 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Update pages 483 30 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 AXSSIGNO DB2CALL PTS46 PROTEUS 10:02:07.263130 Log records 574 100 FTS1/DBP4LU/CCBDD2D20BE5/0005

DBP4 MXMSCHD DB2CALL DSNREXX MXM 10:08:44.907535 Get pages 2668 50 FTS1/DBP4LU/CCBDD44E91C7/0002

DBP4 MXMSCHD DB2CALL DSNREXX MXM 10:08:46.466276 Get pages 2668 50 FTS1/DBP4LU/CCBDD4500BE1/0002

DBA4 MQP1DB2SRV02 RRSAF CSQ5L710 STC@ZOSN 10:10:04.877879 Abort 1 1 FTS1/DBA4LU/CCBDD49ADF64/0002

DBP4 AXS#GENE DB2CALL PTS46 PROTEUS 10:10:05.766411 Response 1.338092 0.5 FTS1/DBP4LU/CCBDD49A71E6/0011

DBP4 AXS#GENE DB2CALL PTS46 PROTEUS 10:10:05.766411 Class 1 CPU 0.361788 0.1 FTS1/DBP4LU/CCBDD49A71E6/0011

DBP4 AXS#GENE DB2CALL PTS46 PROTEUS 10:10:05.766411 Lock requests 135 50 FTS1/DBP4LU/CCBDD49A71E6/0011

DBP4 AXS#GENE DB2CALL PTS46 PROTEUS 10:10:05.766411 Get pages 223 50 FTS1/DBP4LU/CCBDD49A71E6/0011

DBP4 AXS#GENE DB2CALL PTS46 PROTEUS 10:10:05.766411 Update pages 72 30 FTS1/DBP4LU/CCBDD49A71E6/0011

14

For each exception that was triggered, the thread and exception details are listed

SSID DB2 subsystem id Correlation Correlation id Connect Connection name. If the name is not available then the connecting system type is substituted Plan Plan name Auth id Authorization id Time End of accounting interval stamp. This is not the thread start time. This time matches the record timestamp displayed in the ISPF dialog. Exception The exception event that was triggered – its description and value Threshold The exception threshold that was exceeded LUWID Logical unit of work id, used to uniquely identify the DB2 accounting record.


Interactive Log analysis using the ISPF dialog Allows you to view transactions events across all the

subsystems

Merges data sources from IMS, DB2, CICS and MVS

Displays records with friendly descriptions and useful information

Analyze transaction event latencies with relative and elapsed times

Use a filter to show required records only

Use tracking to isolate all the records associated with an individual transaction

Select a record to see more detail (full mapping)

15


IMS-DB2 transaction Part 1

16

BROWSE FUW000.QADATA.FBOSPM4C.IMS.D130703.INDEX Record 00005529 More: < > Command ===> Scroll ===> CSR Navigate < 00.00.01.000000 > Date/Time 2013-07-03 16.33.53.849552 / ———— Tracking ———————————————————————— Wednesday 2013-07-03 Time (Relative) CA01 IMS Transaction TranCode=FBOICT42 Region=0002 16.40.55.075503 01 Input Message TranCode=FBOICT42 +0.000000 31 DLI GU TranCode=FBOICT42 Region=0002 +0.000266 5600 Sign-on to ESAF Region=0002 SSID=DBA6 +0.005517 112 Thread allocate FBOIAP42 DBA6 +0.005996 177 Package allocation FBOIAP42 DBA6 +0.006304 380 SP entry FBOSPM4C STMT=001044 DBA6 +0.006511 177 Package allocation FBOSPM4C DBA6 +0.006681 060 SQL SELECT STMT=000009 DBA6 +0.006796 058 SQL call completion SQLCODE=0 STMT=000009 DBA6 +0.016090 061 SQL DELETE STMT=000011 DBA6 +0.016387 0020 Begin UR +0.017584 325 Trigger entry USERDEL STMT=000011 DBA6 +0.017762 177 Package allocation USERDEL DBA6 +0.017836 061 SQL INSERT STMT=000002 DBA6 +0.017887 058 SQL call completion SQLCODE=0 STMT=000002 DBA6 +0.018139 Continued on next foil...

1. IMS transaction starts with an Index record (CA01) 2. IMS log (green), DB2 trace (yellow) and DB2 log (turquoise) events are shown 3. Transaction calls a DB2 stored procedure which issues SQL calls, call results are shown

IMS

DB2



17

... Continuation from previous foil 061 SQL DELETE STMT=000003 DBA6 +0.018179 0600 Delete from a data page . . . +0.019696 0600 Delete from a data page +1.082712 058 SQL call completion SQLCODE=0 STMT=000003 DBA6 +1.083240 325 Trigger exit SQLCODE=0 DBA6 +1.083254 058 SQL call completion SQLCODE=0 STMT=000011 DBA6 +1.083310 499 SP statement execution detail DBA6 +1.083370 380 SP exit FBOSPM4C SQLCODE=0 STMT=001044 DBA6 +1.083400 053 SQL request SQLCODE=0 STMT=001044 DBA6 +1.083436 5600 Commit Prepare starting Region=0002 SSID=DBA6 +1.084442 084 Prepare start DBA6 +1.084894 0020 End commit phase 1 +1.085008 085 Prepare end DBA6 +1.086509 03 Output Message Response LTerm=FUNTRM07 +1.086577 3730 Syncpoint End of Phase 1 Region=0002 +1.086598 074 Terminate thread start DBA6 +1.086969 0020 Begin commit phase 2 +1.087056 0020 End commit phase 2 +1.087728 003 Thread accounting DBA6 +1.089002 075 Terminate thread end DBA6 +1.089944 5600 Commit Continue completed Region=0002 SSID=DBA6 +1.089970 37 Syncpoint Message Transfer Region=0002 +1.089987 5612 Syncpoint End of Phase 2 Program=FBOIAP42 Region=0002 +1.090012 31 Communications GU LTerm=FUNTRM07 +1.090054 36 Output Message Dequeue LTerm=FUNTRM07 +1.093952 ******************************* Bottom of Data ********************************

IMS

SYNCPOINT

DB2

Delay

4. SQL DELETE causes trigger to delete many more table rows, causing the delay



18

BROWSE FUW000.QADATA.FBOSPM3C.IMS.D131011.INDEX Record 00006876 More: < > Command ===> Scroll ===> CSR Navigate < 00.00.01.000000 > Date/Time 2013-10-11 11.29.17.925791 / ———— Tracking ———————————————————————— Friday 2013-10-11 Time (LOCAL) CA01 IMS Transaction 11.33.58.759345 TranCode=FBOIAT42 Program=FBOIAP42 Userid=FUNTRM14 LTerm=FUNTRM14 Terminal=SC0TCP14 SSID=IDDG Region=0001 RecToken=IDDG/0000000E00000000 OrgUOWID=IDDG/CC17F12AB6B85204 InputQ=0.000332 Process=1.094017 OutputQ=0.000028 TotalTm=0.134377 CPU=0.111167 ———————————————————————————————————————————————————————————————————————————— . . . ———————————————————————————————————————————————————————————————————————————— 003 DB2 Thread accounting DBA6 11.33.58.893342 TranCode=FBOIAP42 Program=FBOIAP42 Userid=FUNTRM14 RecToken=IDDG/0000000E00000000 ClientID=IDDG Region=0001 LUWID=FTS3/DBA6LU/CC17F12AB853/0002 RESP=1.086945 CPU1=0.107330 CPU2=0.000688 I/O3=0 Source=IMS_MPP SEL=4 DEL=1 CAL=1 LogRecs=9777 GetPage=19777 UpdPage=9776 MaxLock=162 ———————————————————————————————————————————————————————————————————————————— ******************************* Bottom of Data ********************************

5. Scroll right to view the index record summary, CA01 is the IMS transaction 6. The DB2 accounting (IFCID 003) shows excessive logging, buffering and locking


+0120 QW0058ID... Scan information +0120 Scan type.... 'SEQD' Rows processed... +24069 +0130 Rows examined.... +24069 +0138 Rows qualified after stage 1... +24069 +0140 Rows qualified after stage 2... +1 +0148 Rows inserted.... +0 +0150 Rows updated... +0 +0158 Rows deleted... +0 +0160 Pages scanned.... +428 +0164 Pages scanned (RI)... +0 +0168 Rows deleted (RI).... +0 +0170 Pages scanned (LOB).... +0 +0174 Pages updated (LOB).... +0 +01A0 QW0058TY... Statement-level information +01A0 SQL statement type... 4000 +01A8 Statement ID... +28917 +01B0 Sync reads... +0 Getpages... +428 +01C0 Rows examined.... +24069 +01C8 Rows processed... +0 Sorts...... +0 +01D8 Index scans.... +0 +01E0 Table space scans.... +1 +01E8 Buffer writes.... +0 +01F0 Parallel groups.... +0 +01F8 In-DB2 elapsed... 0.008537

Enhanced support for DB2 trace (IFCID) records Detailed formatting of IFCID-specific fields

19

+0120 QW0058ID... 'SEQD' Scan type Off QW0058IX... 'INDX' Index On QW0058SD... 'SEQD' Sequential data Off QW0058SW... 'SEQW' Sequential data workfile


Enhanced support for DB2 trace records (cont.) Generating DB2 trace records can be expensive, and can

result in very large log files: you do not want to simply start all traces.

Workbench introduces the concept of trace “levels” (1 - 4) that categorize IFCIDs based on their usefulness (for transaction analysis) and cost overhead: – Program invocation

– SQL

– I/O

– All (caution: may result in high volumes of data)

In the ISPF dialog, enter the command: TRACE n (n: 1 - 4) to show progressively more detail. TRACE 4 shows all available trace records.

20


Workbench ISPF dialog 1. Sessions: collaborative approach to problem determination

2. Controls: common place for shared definitions

3. Systems: define IMS, DB2, CICS and MVS systems – allows workflows to perform automated file selection (locate the required log files)

4. Process: ad hoc log file processing 21

File Help —————————————————————————————————————————————————————————————————————————————— V1R2M0 Transaction Analysis Workbench - Primary Option Menu Option ===> 0 Profile Customize your dialog profile 1 Sessions Analyze problems using the session manager 2 Controls Define record filtering and formatting controls 3 Systems Define the systems where transactions are processed 4 Process Analyze ad hoc log files X Exit Quit the workbench Session Repository . . . 'FUW120.SESSIONS' +


Session Manager 1. Register your problems

2. Allows users to collaborate - see work already done so far

22

Session Manager Row 1 of 17 More: < > Command ===> NEW Scroll ===> CSR NEW Register a new Session / Key Status Description 00000001 OPEN DB2: CICS read via SP doing table space scan 00000002 DONE DB2: CICS update via SP doing table space scan 00000003 DONE DB2: IMS tran generating cascade deletes 00000004 DONE DB2: IMS tran generating trigger deletes 00000005 OPEN DB2: Java update via SP waiting for WLM 00000006 DONE DB2: IMS tran calling inefficient DB2 SP 00000007 OPEN DB2: Java update via SP waiting for CPU 00000008 OPEN DB2: z/OS WAS Java app gens RI cascade deletes 00000009 OPEN DB2: z/OS WAS Java app gens trigger deletes 00000010 OPEN DB2: z/OS WAS Java app calling inefficient SP 00000011 OPEN CICS-DBCTL 00000012 OPEN CICS-DBCTL deadlock 00000014 OPEN DB2: Contention ******************************* Bottom of data ********************************


The session – the place for collaboration 1. Register problem details

2. Workflow contains all the tasks to be performed

3. Files contains the list of log files for this problem

4. Investigate provides interactive problem determination

5. History contains written notes and jobs about the problem 23

File Help —————————————————————————————————————————————————————————————————————————————— Session 00000003 Option ===> Description : DB2: IMS tran generating cascade deletes 1 Register Update the problem registration details 2 Workflow Perform the diagnostic tasks 3 Files Locate and manage the log files required for diagnosis 4 Reporting Run batch reports 5 Investigate Perform interactive log file analysis 6 History Review the problem history


Session registration details 1. What is the problem? Including short and long descriptions

2. When and where did the problem occur? Important for log selection!

3. Who is going to fix it?

24

Session Details Command ===> Scroll ===> CSR Key . . . . : 00000003 Description . DB2: IMS tran generating cascade deletes Severity . . 3 Reference . . FUW-755 —— When problem occurred ——— Reported by . Tony YYYY-MM-DD HH.MM.SS.TH Assigned to . John From 2013-10-11 11.15.00.00 Status . . . OPEN To 2013-10-11 11.40.00.00 Template . . + Zone LOCAL Systems involved: / System + Type + ICDG IMS IDDG IMS DBA6 DB2 FTS3 IMAGE ******************************* Bottom of data ********************************


Workflows and session templates SMEs can use session templates to populate new sessions with the

tasks needed to prepare the problem for evaluation – Created sessions include: systems involved and a sequence of tasks (workflow)

for analyzing the problem

25

Session template

Systems

Workflow

Task

Task

…

Session

Systems

Workflow

Task

Task

…

When creating a session, users can select the session template that best matches the type of problem

Each task is either JCL for a batch job, or a note containing instructions to the user


Session: Workflow 1. Contains tasks – batch jobs and instructions

2. Batch jobs can locate log files, create extracts and transaction indexes, run reports

3. Can be predefined with a “template” set up by the expert

4. All jobs run under the session create a new task

5. Batch job output is saved and can be viewed here

26

Tasks Row 1 to 9 Command ===> Scroll ===> CSR NEW Create a new task AUTO Create file selection and extract tasks SCHED Schedule all the tasks (or select required tasks only) / Task Status Description 1 CC 0000 Select IMS logs for IMS system IDDG 2 CC 0000 Select SMF files for DB2 system DBA6 3 CC 0000 Select DB2 logs for DB2 system DBA6 4 CC 0000 Create log extract for IMS system IDDG 5 CC 0000 Create SMF extract for DB2 system DBA4 6 CC 0000 Create log extract for DB2 system DBA6 7 CC 0000 Create index for IMS systems 8 DONE Please assign the problem to John, our DB2 expert. ? 9 CC 0000 DB2 performance and accounting analysis for system DBA6 ******************************* Bottom of data ******************************** Task Output Row 1 to 5

/ DDname StepName ProcStep Rec-Cnt Jobname JobID Max-RC JESMSGLG JES2 32 JCH#TAWD JOB03728 CC 0000 JESJCL JES2 25 JESYSMSG JES2 91 SYSPRINT FUWBATCH 37 S DB2X FUWBATCH 418 *************************** Bottom of data *****************************


Session: Files 1. All log files and other data sets associated with the problem

are registered here

2. Files can be automatically located - IMS, DB2 and SMF

3. Manually enter other types of log files

4. Supports all data set types and log streams

27

Locate and Manage Log Files Row 1 of 8 More: < > Command ===> Scroll ===> CSR NEW Insert a new log file. AUTO Run automated file selection to locate log files. Log Files: ———— System ———— File / Exc Data Set Name Name Type Type S IMS.IDDG.INDEX IDDG IMS EXTRACT S IMS.IDDG.SLDS IDDG IMS LOG * IMS.IDDG.TRACE IDDG IMS LOG * IMS.IDDG.MONITOR IDDG IMS MON S DB2.DBA6.ARCLOG DBA6 DB2 LOG S SMF:IFASMF.FTS3.SMF FTS3 IMAGE SMF ******************************* Bottom of data ********************************

SMF log stream

IMS and DB2 logs


Session: Reporting 1. IMS Performance Analyzer – selected reports only

2. CICS Performance Analyzer – selected reports only

3. CICS-DBCTL – end-to-end from CICS into IMS

4. DB2 – exception reporting and extract

5. OPERLOG – MVS operations log (SYSLOG) 28

Reporting Option ===> Select a reporting option then press Enter. 1 IMS Transaction and system analysis using IMS PA 2 CICS Transaction and system analysis using CICS PA 3 CICS-DBCTL Combined CICS and IMS analysis of transactions 4 SMF z/OS and subsystem analysis 5 DB2 DB2 accounting exception analysis 6 OPERLOG Sysplex operations log (SYSLOG)


Session: Investigate 1. Session log files are merged in time sequence

2. Often log files are very large. Use time slicing to process required time period only – very quick!

3. Select all or some of the files only

29

Investigate Row 1 of 4 More: < > Command ===> Scroll ===> CSR —————————————— Time Slice (ON) ———————————————— Time Date Duration HH.MM.SS.thmiju YYYY-MM-DD HH.MM.SS Zone Filter + 16.39.36.351066 2013-07-03 00.00.00 LOCAL / Type Start Time Date Duration Coverage IMS 16.39.36.351066 2013-07-03 Wed 00.02.08 PARTIAL IMS 16.35.26.490921 2013-07-03 Wed 00.06.42 PARTIAL DB2 16.33.53.849552 2013-07-03 Wed 00.27.46 COMPLETE DTR 16.39.03.904776 2013-07-03 Wed 00.02.55 PARTIAL ******************************* Bottom of data ********************************

Type Data Set Name Coverage IMS IMS.IDDG.INDEX PARTIAL IMS IMS.IDDG.SLDS PARTIAL DB2 DB2.DBA6.ARCLOG COMPLETE DTR SMF:IFASMF.FTS3.SMF PARTIAL ******************************* Bottom of data ********************************


Eclipse-based rich client platform (RCP) UI

30

How Workbench can help application development teams

31


Application releases must work and perform when deployed Application teams perform validation testing during roll-out

– Is performance a part of validation testing?

– If performance validation is done, who does the validation?

– What criteria are used?

Does the evaluation occur at the transaction level?

What is the cost of performance validation testing?

What is the cost of a failed roll-out due to poor performance?

Does system programming have time to help?

32


Inhibitors to instrumentation data usage by application development teams

Value of instrumentation data not known – May not know what is available and how to use it

– Not a traditional development tool

Do not know how to obtain the data or data access not allowed – May not have access to system parts

Limited or no knowledge of tools that use instrumentation data

Limited access to system programmers’ time – Reluctant to bother system programmers to get help

33


Solution - Value of instrumentation data to application development teams

Workbench demystifies instrumentation data: – Allows application teams to evaluate

• Transaction response times • Application database update patterns

– Allows application teams to diagnose application errors and/or performance issues

– If SME help is needed, they can ask better questions of the right SME

34


How Workbench helps Automates the collection of instrumentation data

– Application development teams do not have to acquire those skills

Performs automated reporting of validation testing – Includes reporting via CICS PA and/or IMS PA, in addition to its

own reports

Analyses instrumentation data for performance exceptions – Provides easy recognition of validation testing against expected

results

Provides transaction life cycle views of transaction exceptions – Identify what part of transaction is causing problem

Saves results of each validation testing run

Facilitates collaboration with system programmers and/or DBAs for help with transaction exception diagnosis

35


Summary of application team benefits Automate tasks often unfamiliar to application teams

– Data acquisition – get the data needed for problem analysis

– Autonomics – automated transaction analysis (life cycle)

– Reporting – basic reporting without tool-specific knowledge

Enables collaboration with other experts – Shared data approach

• DBA, system programmer provide assistance when needed – Fewer SMEs may need to be involved

Analysis of applications performance testing – Exceptions process provides evaluation of validation runs

– Deeper transaction evaluation if exceptions reported

36

Scenario: IMS-DB2 problem

37


Scenario: IMS DB2 problem 1. On the following slides, we present an example

scenario: a user has reported a long transaction response time for an IMS transaction performing DB2 updates

The analysis is divided into two parts: 1. The first responder:

• Registers the problem in the Workbench session manager and collects the log files

• Follows a process orientated script to assign problem to initial expert – Based on what is found

2. The subject-matter expert performs a “deep dive” on the problem: reviewing the reports, and using interactive analysis to identify the specific log records for the cause of the problem

38


File Help —————————————————————————————————————————————————————————————————————————————— Session Details Row 1 to 3 of 3 Command ===> Scroll ===> CSR Key . . . . : 00000026 Description . IMS DB2 problem Severity . . 4 Reference . . TICKET-127 —— When problem occurred ——— Reported by . JOHN YYYY-MM-DD HH.MM.SS.TH Assigned to . JAMES From 2012-06-24 15.20.00.00 Status . . . OPEN To 2012-06-24 16.50.00.00 Template . . IMS+DB2 + Zone LOCAL Systems involved: / System + Type + IADG IMS DB3A DB2 FTS1 IMAGE ******************************* Bottom of data ********************************

39

First responder: Creating a session

The template (set up by the expert) populates the system list (where the problem occurred), as well as the workflow task list (preparatory jobs to select log files and create extracts)

Create a session (main menu ► option 1 Sessions ► NEW). Select the environment (template) where the problem occurred.

© 2013 IBM Corporation ©2014 IBM Corporation 40

Eclipse-based GUI

1. Register a new problem; work on an existing problem 2. Execute the workflow to locate the required diagnostic data 3. Run reports; view the output


File Mode Filter Time Labels Options Help —————————————————————————————————————————————————————————————————————————————— BROWSE IMPOT01.SESSION7.TRANIX + Record 00004609 More: < > Command ===> Scroll ===> CSR Navigate < 00.00.01.000000 > Date/Time 2013-10-11 11.15.00.000000 / ———— ————————————————————————————————— Friday 2013-10-11 Time (LOCAL) TX CA01 Transaction 16.33.33.575325 UTC=16.33.33.575316 TranCode=MQATREQ1 Program=MQATPGM Userid=FUNTRM15 LTerm=FUNTRM15 Terminal=SC0TCP15 Region=0004 OrgUOWID=IADG/C62D2CB467860940 IMSID=IADG IMSRel=101 RecToken=IADG/0000003600000000 CPU=0.041999 InputQ=0.000562 Process=0.497229 TotalTm=0.497791 RegTyp=MPP DBCalls=5 ---------------------------------------------------------------------------- CA01 Transaction 16.33.59.157812 UTC=16.33.59.157802 TranCode=MQATREQ1 Program=MQATPGM Userid=FUNTRM15 LTerm=FUNTRM15 Terminal=SC0TCP15 Region=0004 OrgUOWID=IADG/C62D2CCCCD3E6F81 IMSID=IADG IMSRel=101 RecToken=IADG/0000003A00000000 CPU=0.013980 InputQ=0.000543 Process=0.424378 TotalTm=0.424921 RegTyp=MPP ---------------------------------------------------------------------------- CA01 Transaction 16.34.30.389305

41

Subject-matter expert: Exception candidate investigation

This display has been filtered to show IMS transaction index (CA01) records with a process time of greater than 0.4 seconds. Enter TX to show records related to a transaction


File Mode Filter Time Labels Options Help —————————————————————————————————————————————————————————————————————————————— FUWPRBRF GXH.FUW.JCH1.FUW745.UPDATE.CICS.EXTRACT Record 00000001 More: < > Command ===> Scroll ===> CSR Navigate < 00.05.00.000000 > Date/Time 2013-05-31 16.27.24.275202 / ———— Tracking ———————————————————————— Friday 2013-05-31 Time (Relative) TX 6E13 CICS Transaction TranCode=FB66 Task=944 16.27.24.275202 086 Signon start DBA6 +0.003469 072 Create thread start DBA6 +0.003546 112 Thread allocate DBA6 +0.003805 073 Create thread end DBA6 +0.003830 053 SQL DESCRIBE/COMMIT/ROLLBAC SQLCODE=0 STMT=000158 DBA6 +0.004096 233 SP entry FBOSP006 STMT=000196 DBA6 +0.005104 015 Index scan begin DBA6 +0.005874 018 Scan end DBA6 +0.006097 055 SQL set current SQLID DBA6 +0.006188 053 SQL DESCRIBE/COMMIT/ROLLBAC SQLCODE=0 STMT=000281 DBA6 +0.006209 060 SQL SELECT STMT=000344 DBA6 +0.006365 017 Sequential scan begin DBA6 +0.006478 006 Read I/O begin DBA6 +0.006582 007 Read I/O end DBA6 +0.006950 018 Scan end DBA6 +1.609979 058 SQL call completion SQLCODE=0 STMT=000344 DBA6 +1.610035 061 SQL UPDATE STMT=000423 DBA6 +1.610336 017 Sequential scan begin DBA6 +1.610463 0020 DB2 Unit of Recovery Control - Begin UR +1.610733 0010 DB2 Savepoint +1.610733 0020 DB2 Update In-Place in a Data Page +1.610749 018 Scan end DBA6 +1.610771 058 SQL call completion SQLCODE=0 STMT=000423 DBA6 +1.611141 233 SP exit FBOSP006 SQLCODE=0 STMT=000196 DBA6 +1.611397 053 SQL DESCRIBE/COMMIT/ROLLBAC SQLCODE=0 STMT=000196 DBA6 +1.611448

42

Transaction life cycle investigation 1. Start tracking a transaction (here, a CICS transaction)

2. See the transaction life cycle events from the related logs (here, an SMF file and a DB2 log), merged together with no preparation required

3. Notice the jump in elapsed time

4. In this case, the problem was caused by a table scan in a DB2 stored procedure. A drill down of the DB2 trace was able to determine this.


Detail DB2 event data view using forms view ********************************* Top of data ********************************* +018C Code... 058 SQL Call completion RC=0000 STMT=002896 DBA6 +0198 Date... 2012-11-21 Wednesday Time... 17.40.04.013647.813 Package +0034 Location...... 'DB2ALOC' Collection ID........ 'CSQ5L710' +0056 Package name... 'CSQ5L710' Consistency token.... 193153A81425EA0D +0072 SQLCA...... SQL communication area (SQLCA) +0072 SQLCAID.... 'SQLCA ' SQLCABC.... +136 SQLCODE.... +0 +0082 SQLERRML... +0 SQLERRM.... ' ' +00CA SQLERRP.... 'DSN ' SQLERRD1... +0 SQLERRD2... +0 +00DA SQLERRD3... +0 SQLERRD4... FFFFFFFF SQLERRD5... +0 +00E6 SQLERRD6... +0 SQLWARN0... ' ' SQLWARN1... 'N' +00EC SQLWARN2... ' ' SQLWARN3... ' ' SQLWARN4... ' ' +00EF SQLWARN5... '1' SQLWARN6... ' ' SQLWARN7... ' ' +00F2 SQLWARN8... ' ' SQLWARN9... ' ' SQLSTATE... '00000' +00FC Statement number... +2896 +0106 Query command ID... 00000000 Query instance ID.... 00000000 +0118 QW0058ID... Scantype +0118 Data type.... 'INDX' Rows processed.. +234 Rows examined.... +12 +012C Rows qualified... +7 After stage 1... +4 After stage 2.... +3 +0140 Rows inserted.... +17 Rows updated.... +12 Rows deleted..... +24 +0158 Pages scanned.... +76 +015C Pages scanned (RI)... +0 Rows deleted (RI).... +0 +0160 Pages scanned (LOB).. +0 Pages updated (LOB).. +0 +0188 QWHS....... Product section standard header +0194 DB2 subsystem.... 'DBA1' +01BC QWHSLWID... LUWID +01BC Network ID... 'FTS1' LU name.... 'DBA1LU ' +01C4 Uniqueness value... CA80E6B51165 Commit count... +1 ******************************** Bottom of data ********************************

Program statement number 2896 caused an index scan that processed 234 rows in the table


+002C QW0065..... IFCID data Package +002C Location... 'DB2BLOC' Collection ID.... 'MQATPGM' +004E Package name... 'MQATPGM' +0060 Consistency token.... 189E34F81745545D Statement +006A Statement type... 91 Cursor name.... 'C1' Isolation.... 'U' +0080 Reoptimization... 0000 Statement number... +835 +0088 Cursor scrollability... 40 +0089 Cursor sensitivity... 40 +008A Result table type.... 40 Close commit... D5 +0094 Query command ID... 0 ·———————————————————————————————— Field Zoom —————————————————————————————————· | File Menu Help | | ——————————————————————————————————————————————————————————————————————————— | | BROWSE JCH.FUW.P0000003.D130625.T094351.EXTRACT + Line 00000000 | | Command ===> Scroll ===> PAGE | | ******************************* Top of data ******************************* | | +007F QW0065I.... 'U' Isolation level of the SQL statement. | | | | Off QW0065RR... 'R' RR (repeatable read) | | Off QW0065RS... 'T' RS (read stability) | | Off QW0065CS... 'S' CS (cursor stability) | | On QW0065UR... 'U' UR (uncommitted read) | | Off QW0065XR... 'X' XR (Repeatable read with X lock) | | Off QW0065XS... 'L' XS (Read stability with X lock) | | ******************************* End of data ******************************* | ·—————————————————————————————————————————————————————————————————————————————·

Zoom

Zoom to see more detail about log record fields


File Mode Filter Time Labels Options Help —————————————————————————————————————————————————————————————————————————————— BROWSE JCH.FUW.P0000003.D130625.T094351.EXTRACT Record 00003251 More: < > Command ===> Scroll ===> CSR Navigate < 00.00.01.000000 > Date/Time 2013-06-22 14.57.57.969312 / ———— Tracking ———————————————————————— Saturday 2013-06-22 Time (Elapsed) 380 SP entry FBOSPM3C DBA6 15.18.02.907449 TranCode=FBOIAP42 Userid=FUNTRM06 ClientID=ICDG LUWID=FTS3/DBA6LU/CB8C9439E347/0001 ———————————————————————————————————————————————————————————————————————————— 380 SP exit FBOSPM3C SQLCODE=0000 DBA6 0.444391 TranCode=FBOIAP42 Userid=FUNTRM06 ClientID=ICDG LUWID=FTS3/DBA6LU/CB8C9439E347/0001 ———————————————————————————————————————————————————————————————————————————— 003 Thread accounting DBA6 0.003521 TranCode=FBOIAP42 Program=FBOIAP42 Userid=FUNTRM06 Region=0001 RecToken=ICDG/0000000100000000 ClientID=ICDG RESP=0.448242 CPU1=0.324230 CPU2=0.000791 I/O3=0.003360 Source=IMS_MPP GtPgRq=284 SyPgUp=6 Suspnd=0 DeadLk=0 TimOut=0 MxPgLk=2 Sel=4 Ins=0 Upd=0 Del=1 LUWID=FTS3/DBA6LU/CB8C9439E347/0002 ———————————————————————————————————————————————————————————————————————————— ******************************* Bottom of Data ********************************

45

Life cycle events: expanded summary view

Scroll right to show the records in expanded view with elapsed or relative times: Elapsed – time between log record events Relative – time since start of transaction (or other selected event)


Identifying events for review or collaboration

Enter FIND LUWID on the command line. Enter G to “tag” (bookmark) this DB2 record.

File Mode Filter Time Labels Options Help —————————————————————————————————————————————————————————————————————————————— BROWSE IMPOT01.SESSION7.TRANIX + Record 00005399 More: < > Command ===> Scroll ===> CSR Navigate < 00.00.01.000000 > Date/Time 2013-06-22 14.57.57.969312 / ———— Tracking ———————————————————————— Saturday 2013-06-22 Time (Relative) CA01 Transaction 16.33.33.575325 UTC=16.33.33.575316 TranCode=MQATREQ1 Program=MQATPGM Userid=FUNTRM15 LTerm=FUNTRM15 Terminal=SC0TCP15 Region=0004 OrgUOWID=IADG/C62D2CB467860940 IMSID=IADG IMSRel=101 RecToken=IADG/0000003600000000 CPU=0.041999 InputQ=0.000562 Process=0.497229 TotalTm=0.497791 RegTyp=MPP DBCalls=5 ---------------------------------------------------------------------------- TAG IMS DB2 transaction with long response time +0.021122 ---------------------------------------------------------------------------- G 0020 DB2 Unit of Recovery Control - Begin UR +0.021122 Userid=FUNTRM15 IMSID=IADG URID=00002A4010EA LUWID=FTS3/DB3ALU/C62D2CB46A5A/0001 ---------------------------------------------------------------------------- 0020 DB2 Update In-Place in a Data Page +0.021138 DBID=0105 PSID=0002 URID=00002A4010EA ----------------------------------------------------------------------------

A DB2 expert can now use the DB2 Log Analysis Tool to investigate the associated DB2 table updates, based on the transaction’s URID

http://www-01.ibm.com/software/data/db2imstools/db2tools/db2lat/

http://www-01.ibm.com/software/data/db2imstools/db2tools/db2lat/


Problem resolution: end of scenario The cause of the IMS transaction problem has been

narrowed down to a slowdown in DB2

Sufficient information about the DB2 update activity has been collected and can be passed on to the DB2 DBA for further investigation

Automatically locates log files for the problem time range for supported subsystems – SMF

– IMS log

– DB2 log and trace

Enables a collaborative problem analysis: – Between first responders and subject-matter experts

– Between experts in different areas

47


SMF reports

System events or constraints can affect transaction processing

Workbench provides reports for selected SMF record types, specifically aimed at identifying performance-related issues

Subsystem-related: SMF 33-2: APPC conversations

SMF 88-1: System Logger

SMF 101: DB2 accounting

SMF 116: WebSphere MQ

System-related: SMF 30: Address Space activity;

including CICS, IMS, DB2

RMF 70-1: CPU usage

RMF 76: Page data sets

RMF 78-2: Virtual Storage

SMF 42-6: DASD data set I/O

SMF 64: VSAM data set I/O


SMF 30: Address Space activity ----------Interval------------- System ---------- CPU ---------- EXCPs Start Date/Time Duration Type Name Jobname Comp TCB SRB %CPU /Sec 2011-03-04 15:37:01 00:01 STE MVS1 IMSCTL1 0004 0.445357 0.023205 15.1 477 2011-03-04 15:37:01 00:01 STT MVS1 IMSCTL2 0004 0.445357 0.023205 15.0 476 2011-03-04 15:37:06 00:01 STE MVS1 IMSCTL3 0004 0.404175 0.011985 19.3 309 2011-03-04 15:37:06 00:01 STT MVS1 IMSCTL4 0004 0.404175 0.011985 19.2 309 2011-03-04 15:43:24 00:01 STE MVS1 IMSCTL5 0004 0.904357 0.046920 18.9 590 2011-03-04 15:43:24 00:01 STT MVS1 IMSVTL6 0004 0.904357 0.046920 18.9 589 2011-03-04 15:44:05 00:01 INT MVS2 CICSPR1 0000 7.966200 0.241357 15.2 140 2011-03-04 15:44:58 00:01 INT MVS2 CICSPR2 0000 0.141780 0.004335 11.2 100

---- System ---------- CPU ---------- EXCPs ----Storage----- -Paging/Sec- Type Name Jobname Comp TCB SRB %CPU /Sec <16M >16M 64bit In Out Swap

STE MVS1 IMSCTL1 0004 0.445357 0.023205 15.1 477 1M 11M 0M 0 0 0 STT MVS1 IMSCTL2 0004 0.445357 0.023205 15.0 476 1M 11M 0M 0 0 0 STE MVS1 IMSCTL3 0004 0.404175 0.011985 19.3 309 1M 11M 0M 0 0 0 STT MVS1 IMSCTL4 0004 0.404175 0.011985 19.2 309 1M 11M 0M 0 0 0 STE MVS1 IMSCTL5 0004 0.904357 0.046920 18.9 590 1M 11M 0M 0 0 0 STT MVS1 IMSVTL6 0004 0.904357 0.046920 18.9 589 1M 11M 0M 0 0 0 INT MVS2 CICSPR1 0000 7.966200 0.241357 15.2 140 4M 1366M 0M 0 0 0 INT MVS2 CICSPR2 0000 0.141780 0.004335 11.2 100 0M 12M 0M 0 0 0

At regular intervals, every address space can be monitored for unusual spikes (or lulls) in system-related resource consumption including CPU and I/O.


Local Partner ----------- Time --- Start Time LU Name Direction UserId Job Name SyncLvl InputQ Process 18:16:47.624543 MVSLU02 Outbound TWM#RBAT Syncpt .324737 ** Partner ** TPname=IADGEXP_PROFILE 18:16:47.796620 IADGAPPC Inbound IADGMPPA Syncpt .166232 .154551 *** Local *** TPname=IADGEXP_PROFILE

SMF 33-2: APPC/MVS Conversation List report

Partner ----------- Time ----------- ------ Bytes ------ ection UserId Job Name SyncLvl InputQ Process Total Received Sent

bound TWM#RBAT Syncpt .324737 .324737 68 83

TPname=IADGEXP_PROFILE

ound IADGMPPA Syncpt .166232 .154551 .320783 83 68 TPname=IADGEXP_PROFILE

APPC requests processed on z/OS are logged to SMF. These requests may end up being processed by an IMS or CICS transaction. A breakdown of processing inside MVS can identify bottlenecks and other performance related issues.


SMF 42-6 DASD Data Set I/O report

This example is for an IMS WADS (write-ahead data set):

51

---------- DASD response time breakdown (average) ------------ ---- Maximum ---- Response Queuing Pending Connect Disc Rd Disc Wrt DAO Response Service 0.000512 0.000000 0.000000 0.000384 0.000000 0.000000 0.000000 0.001024 0.001024 ate rate per second ------ --- Cache I/O per sec ---- Read Hits Write Hits Seq RLC ILC 0 0% 0 0% 120 0 0

DSN: IADJ.VA10.WADS9 -- I/O per sec -- ---------- DASD response time breakdown (average) ---------- Reads Writes Response Queuing Pending Connect Disc Rd Disc Wrt D 0 120 0.000512 0.000000 0.000000 0.000384 0.000000 0.000000 0.0000 ---- Cache candidate rate per second ------ --- Cache I/O per sec ---- Total Hits Read Hits Write Hits Seq RLC ILC 0 0% 0 0% 0 0% 120 0 0


--Splits-- -------Calls------- -RLS Activity- CA CI Ext EXCPs Get Upd Del Ins LSR CF DASD

VS.RKM2EDS3.DATA 5 17 1 3322 13 1 0 1314 0 0 0 VS.RKM2EDS3.INDEX 0 5 1 1796 0 259 0 0 0 0 0 VS.RKM2EDS3.DATA 5 18 1 3378 13 1 0 1340 0 0 0 VS.RKM2EDS3.INDEX 0 5 1 1850 0 261 0 0 0 0 0 VS.RKM2EDS3.DATA 5 19 1 3436 13 1 0 1353 0 0 0 VS.RKM2EDS3.INDEX 0 5 1 1902 0 275 0 0 0 0 0 DATA 0 0 1 3754685 23K 8658 4353 1602 0 0 0 INDEX 0 0 1 3739616 13 0 0 0 0 0 0

--Splits-- Close Date Time Data set name CA CI Ext EXCPs 2011-02-02 16:00:01 FUNDIP.OME.FTS1MVS.RKM2EDS3.DATA 5 17 1 3322 2011-02-02 16:00:01 FUNDIP.OME.FTS1MVS.RKM2EDS3.INDEX 0 5 1 1796 2011-02-02 16:00:01 FUNDIP.OME.FTS1MVS.RKM2EDS3.DATA 5 18 1 3378 2011-02-02 16:00:01 FUNDIP.OME.FTS1MVS.RKM2EDS3.INDEX 0 5 1 1850 2011-02-02 16:00:32 FUNDIP.OME.FTS1MVS.RKM2EDS3.DATA 5 19 1 3436 2011-02-02 16:00:32 FUNDIP.OME.FTS1MVS.RKM2EDS3.INDEX 0 5 1 1902 2011-02-02 16:00:55 FUNDIP.ANF.QUEUE.DATA 0 0 1 3754685 2011-02-02 16:00:55 FUNDIP.ANF.QUEUE.INDEX 0 0 1 3739616

SMF 64: VSAM Data Sets

VSAM data sets are commonly used as databases in IMS and CICS. As these data sets are re-opened (or extend), information about their I/O activity and general health (splits) is available.


----------- Number of Address Spaces ----------- ---- In --- -In Ready- -Out Ready- -Out Wait- Avg Max Avg Max Avg Max Avg Max 151 156 7 86 0 1 0 0 77 80 1 15 0 0 0 0 69 72 1 9 0 0 0 0

------------------------ Number - Interval Start -- System - %CPU Busy - IO ---- In --- -In Ready- -Out R Date Time Name LPAR MVS Rate Avg Max Avg Max Avg 2010-08-17 23:45:00 FTS1 68.75 87.42 2282.4 151 156 7 86 0 FTS2 4.07 4.50 9.4 77 80 1 15 0 FTS3 4.03 4.39 12.6 69 72 1 9 0 2010-08-18 00:00:00 FTS1 61.15 72.16 1934.8 150 155 5 76 0 FTS2 4.15 4.72 8.4 77 79 1 8 0 FTS3 3.88 4.41 11.7 69 73 1 13 0

SMF 70-1: RMF Processor activity CPU constraints are one of the most common causes of a slowdown in performance, and often has flow-on effects including contention.

Out Ready identifies the number of address spaces waiting for dispatching on the CPU

CPU Busy and IO Rate are the classical system performance indicators. Look for spikes that might indicate a slowdown.


SMF 75: RMF Page Data Set activity Date: 2010-08-17 Time: 23:45:00 SID: FTS1 Page -------- Slots Used ------- % Bad In Trans Number Pages Type Alloc Min Max Avg Full Slots Use Time I/O Req Xferd VIO Dat PLPA 44999 20078 20078 20078 45% 0 0 0 0 0 FUN Common 89999 3129 3129 3129 3% 0 0 0 0 0 FUN Local 1080K 101K 101K 101K 9% 0 0 0 10 10 Y FUN Local 1080K 102K 102K 102K 9% 0 0 0 10 10 Y FUN Local 1080K 103K 103K 103K 10% 0 0 0 6 6 Y FUN Local 1080K 109K 109K 109K 10% 0 0 0 13 13 Y FUN

Time: 23:45:00 SID: FTS1

lots Used ------- % Bad In Trans Number Pages Min Max Avg Full Slots Use Time I/O Req Xferd VIO Data Set Name

078 20078 20078 45% 0 0 0 0 0 FUNDI1.FTS1.PAGE.PLPA

129 3129 3129 3% 0 0 0 0 0 FUNDI1.FTS1.PAGE.COMMON 01K 101K 101K 9% 0 0 0 10 10 Y FUNDI1.FTS1.PAGE.LOCAL1 02K 102K 102K 9% 0 0 0 10 10 Y FUNDI1.FTS1.PAGE.LOCAL2 03K 103K 103K 10% 0 0 0 6 6 Y FUNDI1.FTS1.PAGE.LOCAL3 09K 109K 109K 10% 0 0 0 13 13 Y FUNDI1.FTS1.PAGE.LOCAL4

With the advent of large amounts of cheaper memory, Page data set performance is often less of a problem today, but none the less should be monitored occasionally for constraints.


val Start -- System ---------------------- Usage ----------------------- Time Name Type Size Min Time Max Time Avg Pct

-13 23:45:00 FTS1 CSA 3364K 612K 23:44:59.60 612K 23:44:59.60 612K 18.2

ECSA 384M 131M 23:44:59.60 131M 23:48:49.24 131M 34.1 SQA 1744K 444K 23:44:59.60 444K 23:44:59.60 444K 25.5 ESQA 47772K 22120K 23:47:19.06 22212K 23:44:59.60 22177K 46.4 FTS2 CSA 3364K 376K 23:44:59.60 376K 23:44:59.60 376K 11.2

SMF 78-2: RMF Virtual Storage activity

- Interval Start -- System ---------------------- Usage -------------- Date Time Name Type Size Min Time Max Time 2010-06-13 23:45:00 FTS1 CSA 3364K 612K 23:44:59.60 612K 23:44:59.60 6 ECSA 384M 131M 23:44:59.60 131M 23:48:49.24 1 SQA 1744K 444K 23:44:59.60 444K 23:44:59.60 4 ESQA 47772K 22120K 23:47:19.06 22212K 23:44:59.60 221 FTS2 CSA 3364K 376K 23:44:59.60 376K 23:44:59.60 3

IMS and CICS still use large amounts of CSA and ECSA for common storage. In the event that storage cannot be obtained, subsystems can stop or worse.


Reg Max CICS ancode PSBname PST Typ Duration Locks Recovery Token Resource Task

1CSAC3 PCM0F0 49 11.534336 0 CI1CSAC3/C5BF632F08B62783 HNMTRM01 00088603 1ESAE1 PCM0F0 127 111.149056 44 CI1ESAE1/C5BF62D0456F8085 00036462 1ESAE5 PCM0F0 102 11.534336 0 CI1ESAE5/C5BF63D077B36503 HNMTRM01 00088040 1FSAF3 PCM0F0 40 98.566144 44 CI1FSAF3/C5BF637DEF1A2001 00032398 1ASAA2 PRE0F0 90 11.534336 26 CI1ASAA2/C5BFBB316C472003 SHSECN08 00013029 1ASAA1 PSA0F0 60 11.534336 2 CI1ASAA1/C5BFBB3166E1F584 00048273

SMF 79-15: IRLM Long Lock Detection Cycle Entry Reg Max Time Number Type IMS ID Trancode PSBname PST Typ Duration Locks Reco 08:51:47.440 25853771 Wait ISA2 CI1CSAC3 PCM0F0 49 11.534336 0 CI1C 08:51:47.440 25853771 Block ISA3 CI1ESAE1 PCM0F0 127 111.149056 44 CI1E 08:54:36.250 25854107 Wait ISA3 CI1ESAE5 PCM0F0 102 11.534336 0 CI1E 08:54:36.250 25854107 Block ISA4 CI1FSAF3 PCM0F0 40 98.566144 44 CI1F 15:25:31.580 25900783 Wait ISA1 CI1ASAA2 PRE0F0 90 11.534336 26 CI1A 15:25:31.580 25900783 Block ISA1 CI1ASAA1 PSA0F0 60 11.534336 2 CI1A

IMS database locks that are held by transactions for an extended period (several seconds) are logged to SMF; and can be analyzed to determine if there is an application problem.


SMF 88-1: System Logger Log Stream Logstream name MVSID Structure name Group First interval start [email protected] FTS1 *DASDONLY* 14:30:00.00 3/04/2011 ----------- IXGWRITES ---------- ---------------- DELETIONS --- Bytes Count Count Byt Writn to With Without Aft Total Average Interim DASD DASD Offlo Count Bytes Bytes Storage Write Write w. DA -------- -------- -------- -------- -------- -------- ------ Total 29862 19177K 642 122692K 24950 4075 1025 Rate(/Sec) 0 5 35 0 0 Minimum 0 0 0 0 0 Maximum 1322 862741 5480448 1685 839 70328

tructure name Group First interval start Last interval stop Total Interval DASDONLY* 14:30:00.00 3/04/2011 16:00:00.00 4/14/2011 0985:30:00

------- ---------------- DELETIONS ----------------- Bytes Count Count Bytes Bytes Writn to With Without After Int Stor Average Interim DASD DASD Offload w/o DASD Bytes Storage Write Write w. DASD Write ------- -------- -------- -------- -------- --------

642 122692K 24950 4075 102547K 16691K 35 0 0 29 5

0 0 0 0 0 5480448 1685 839 7032832 3436544

CICS and IMS both rely on log streams for critical services; including message handling and journaling. Problems can be avoided by monitoring their I/O and offload activity.


SMF 101: DB2 Thread Accounting

DB2 Plan ---------- Connection ----------- Thread SSID Name Name Type Count DB3A CEXTPGM IADG IMS MPP 68

Start: 2010-06-24 15:27:39 End: 2010-06-24 16:44:00 Class1: Thread Time Avg: Elapsed=70.43305 CPU= .011006 Interval: 01:16:20 Max: Elapsed=2045.732 CPU= .013724 Rate/sec: < 1 Class2: In-DB2 Time Avg: Elapsed= .015108 CPU= .006035 Max: Elapsed= .033537 CPU= .008234 Class3: Suspend Time Avg: Total = .008709 I/O= .000000 Lock/Latch= .002404 Other= .006305 Max: Total = .017377 I/O= .000000 Lock/Latch= .007199 Other= .010178 Buffer Manager Summary Avg: GtPgRq= 7.0 SyPgUp= 3.0 Max: GtPgRq= 7 SyPgUp= 3 Locking Summary Avg: Suspnd= .0 DeadLk= .0 TmeOut= .0 MxPgLk= 1.0 Max: Suspnd= 0 DeadLk= 0 TmeOut= 0 MxPgLk= 1 SQL DML Query/Update Avg: Sel= .0 Ins= 1.0 Upd= 1.0 Del= 1.0 Max: Sel= 0 Ins= 1 Upd= 1 Del= 1 SQL DML 'Other' Avg: Des= .0 Pre= .0 Ope= 1.0 Fet= 9.0 Clo= 1.0 Max: Des= 0 Pre= 0 Ope= 1 Fet= 9 Clo= 1

All transactions that use DB2 cut accounting records that show how DB2 performed in the application and across into DB2.


SMF 116: WebSphere MQ Accounting reports MQACCT4 Printed at 10:50:30 2/03/2011 Data from 09:00:40 03/03/2010 to 09:59:52 03/03/2010 SSID: SYSB Type: CICS Name: CICSSYSP Tran: TRTI Threads: 2 Other Avg Count 6.0 Avg Elapsed 0.000116 Avg CPU 0.000112 In-MQ Time (Total) Elapsed: 0.000233 CPU: 0.000224 In-MQ Time (Average) Elapsed: 0.000116 CPU: 0.000112 SSID: SYSB Type: CICS Name: CICSSYSP Tran: TRTL Threads: 4 In-MQ Time (Total) Elapsed: 0 CPU: 0 In-MQ Time (Average) Elapsed: 0 CPU: 0 Queue: APPLICATION_A_REQUEST QType: LOCAL IType: NONE GDisp: Q_MGR QCount: 4 Count Elapsed CPU Susp Elp JnlWrt Elp PS Req's PS Rd Elp Expir _________ _________ _________ _________ __________ _________ _________ ______ OPEN 15.0 0.000019 0.000009 CLOSE 15.0 0.000002 0.000002 INQ 15.0 0.000009 0.000008 In-MQ Time (Total) Elapsed: 0.001861 CPU: 0.001222 In-MQ Time (Average) Elapsed: 0.000465 CPU: 0.000305

Detailed MQ accounting can be requested to show the impact of MQ on transaction performance.


File Edit Edit_Settings Menu Utilities Compilers Test Help ——————————————————————————————————————————————————————————————————————————————— VIEW FUW110.WTWM.REPORTS(OPERLOG) - 01.03 Columns 00001 00072 Command ===> Scroll ===> CSR ****** ***************************** Top of Data ****************************** 000001 //OPERLOG JOB ,CLASS=A,NOTIFY=&SYSUID 000002 /*JOBPARM SYSAFF=FTS1 000003 //FUWBATCH EXEC PGM=FUWBATCH 000004 //STEPLIB DD DISP=SHR,DSN=FUW.SFUWLINK 000005 //SYSPRINT DD SYSOUT=* 000006 //SYSIN DD * 000007 LOGSTREAM OPERLOG:SYSPLEX.OPERLOG 000008 START 2011-04-06-08.40.00.00 STOP 2011-04-06-09.00.00.00 000009 REPORT OPERLOG 000010 CODE(OPERLOG) 000011 COND TEXT(2) EQ 'DFS' 000012 COND TEXT(*) EQ 'BACKOUT' 000013 /* ****** **************************** Bottom of Data ****************************

OPERLOG report: JCL

Note the LOGSTREAM command in the SYSIN data set, identifying the input log stream. The COND statements filter the reported OPERLOG records.


OPERLOG report: output

FTS3 2011096 08.41.42.57 STC36951 DFS2484I JOBNAME=IBB1#ARC GENERATED BY LOG AUTOMATIC ARCHIVING IBB1 FTS2 2011096 08.41.48.71 STC37128 DFS058I 08:41:48 START COMMAND IN PROGRESS ICDZ FTS2 2011096 08.41.49.80 STC37128 DFS551I IFP REGION ICDZIFP1 STARTED ID=00001 TIME=0841 ICDZ FTS2 2011096 08.41.49.89 STC37128 DFS551I MESSAGE REGION ICDZMPP1 STARTED ID=00002 TIME=0841 CLASS=001,000,000,000 ICDZ FTS2 2011096 08.41.52.04 STC37128 DFS551I IFP REGION ICDZIFP3 STARTED ID=00003 TIME=0841 ICDZ FTS3 2011096 08.47.36.05 STC36951 DFS554A FUWTCIC 00002 FUWTCIC DFHTWM04(3) 000,0777 2011/096 8:47:36 RTKN=FUWTCIC C79459EA853EFB03 IBB1 FTS3 2011096 08.47.51.05 STC36951 DFS968I DBD=DI21PART WITHIN PSB=DFHTWM04 SUCCESSFULLY BACKED OUT IBB1 FTS3 2011096 08.47.51.05 STC36951 DFS980I BACKOUT PROCESSING HAS ENDED FOR DFHTWM04 IBB1

From the previous JCL request, it is simple to identify the IMS subsystem messages associated with the transaction failure.


CICS DBCTL For all CICS transaction you can analyze CMF (SMF 110) records.

However CMF records do not contain all of the details of the IMS events that are triggered by DL/I requests. Those details are in the IMS log.

Transaction Analysis Workbench enables you to analyze CICS DBCTL transactions using: – CMF records

– IMS log records

– A combined view of both

In particular Workbench enables you to analyze exceptions:

CICS DBCTL transactions that abended or had a long response time.

62


Example CICS-DBCTL summary report

63

ss

CICS

ss

IMS

2014-03-26 Wednesday CICS-DBCTL Summary Page 1 Tran APPLID CMF Count Response CPU Time IMS Reqs IMS Wait ABEND Rate/Sec ---- -------- --------- -------- -------- -------- -------- -------- -------- BANK CICSP1 60 11.12982 0.008967 35 4.256977 10 0 08 Count Elapsed CPU Time StaDelay Schedule IC Wait PS Wait --------- -------- -------- -------- -------- -------- -------- 42 10.94999 0.004092 0.011668 0.000183 0 0 07 Count DB call DB Gets DB Upds IO Count IO Time LockWait --------- -------- -------- -------- -------- -------- -------- 41 33 13 19 4 0.003438 3.980170 FP Count FP call FP Gets FP Upds FP Wait FP Fail --------- -------- -------- -------- -------- -------- 41 19 7 11 0 7 Synctime Phase 1 Phase 2 FP PH2 OTHREAD -------- -------- -------- -------- -------- 0.011938 0.006555 0.005383 0.002232 0.017659

Technology

Ims12 workbench data visualization - IMS UG May 2014 Sydney & Melbourne