Lunch n Learn - Solving CICS performance issues

© 2015 IBM Corporation

APA/CICS PA Webinar

August 04 2015

Solving CICS performance issues….if you’re looking for trouble, you’ve come to the right place.

Lamar Woods

Sr. Client Technical Professional, Z series Problem Determination Tools

[email protected]

Eric Higgins

Client Technical Specialist, CICS tools

[email protected]

mailto:[email protected]


The following terms are trademarks or registered trademarks of the

International Business Machines Corporation in the United States and/or

other countries:

– CICS, CICS Explorer, CICSPlex SM, DB2, QMF, IMS, MQSeries, WebSphere,

IBM, SupportPac, OS/390, z/OS, zSeries, RMF, Tivoli, Tivoli Enterprise,

OMEGAMON XE

Java and all Java-based trademarks and logos are trademarks or

registered trademarks of Sun Microsystems, Inc. in the United States

and/or other countries.

Microsoft, Windows, and the Windows logo are trademarks of Microsoft

Corporation in the United States, other countries, or both.

Other company, product, service names, and logos may be trademarks or

service marks of others.

Preface

2

© 2015 IBM Corporation3

Abstract

Identifying and solving CICS performance issues before they get out of

hand can be viewed as looking for trouble before it finds you.

IBM’s CICS Performance Analyzer is a historical performance analysis tool that provides

the detailed performance information required to understand the impact of application

performance. When paired with the performance measurement features of the IBM

Application Performance Analyzer, there’s no better combination of capabilities for

uncovering and solving CICS performance issues – right down to the source code level.

What’s the value of identifying performance issues before they have a chance to

escalate?

• How often do you go looking for tuning opportunities?

• What can a performance measurement tool do to help resolve bad application

behavior?

In this webcast we combine capabilities of CICS Performance Analyzer and Application

Performance Analyzer to:

• Identify performance tuning candidate applications at a CICS system level

• Perform a subsequent deep-dive performance analysis right down to the application

source code level


For our scenario, we are using a version of the supplied Catalog manager example (ECAT) that is supplied with CICS.

The example is constructed around a simple sales

catalog and order processing application, in which a user

can perform various tasks such as:

List items in a catalog.

Inquire on individual items in the catalog.

Order items from the catalog.

Administrative functions

We will use CICS Performance Analyzer and Application Performance Analyzer to look for performance issues with ECAT.

For details on the ECAT sample, see https://www-01.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.exampleapplication.doc/topics/dfhxa_t100.html

Let’s go looking for some trouble!

Our Application

https://www-01.ibm.com/support/knowledgecenter/SSGMCP_5.2.0/com.ibm.cics.ts.exampleapplication.doc/topics/dfhxa_t100.html


What is CICS PA? A Comprehensive Performance Reporting and

Analysis tool for CICS

Provides DB2 and MQ subsystem performance reporting related to CICS workload

Provides ongoing system management and measurement reports on all aspects of CICS application performance

Historical performance complimenting real-time Omegamon monitoring

How does CICS PA work? Uses SMF data as input

Easy to use interface for report generation (over 250 supplied report forms)

Performance and Statistical analysis

Graphical performance analysis via the explorer

CICS

PERFORMANCE ANALYSIS

CICS Performance Analyzer for z/OS

Why this tool is important• Reduce both time/resource required to analyze off-

line performance data (usually massive) for tuning

and capacity planning purposes.

• Enables deep-dive CICS performance analysis and

understanding of usage trends

• Aids capacity planning and tuning

• Helps quickly identify and eliminate trends leading to

online performance problems


CICS PA – Review Performance Alerts – Demo Backup Slide

6

Performance Alerts:

• Define alerts with specific threshold values

• Critical, warning, and informational levels

• Only report on activity exceeding the threshold

In this case we see several alerts on 2015-07-08 for Tran ECAT where the CPU and

Response time thresholds were exceeded

Next we’ll look at some of the performance intervals for ECAT


CICS PA – Review Summary data – Demo Backup Slide

7

Transaction Summary:

• Interactive view of performance data

• Filter & sort by column

• Customizable layouts

• The data is summarized into 5 minute intervals.

• 2 of the intervals shown have high average response times and high average CPU

utilization

Next we’ll look the response time data graphically for ECAT


CICS PA – Create charts to analyze data – Demo Backup Slide

8

Transaction Response Time Bar Chart:

• Visual presentation makes it easy to see anomalies

• Most intervals have a good response times, but a couple are way out of range

Right click on one of the poor response time bars to look into more performance

summary details


Transaction Response Summary Pie Chart:

• Shows dispatch time is 99.92% of Response time

• Shows CPU time is 90.75% of dispatch time

• Something is using a lot of CPU



Transaction CPU Summary Pie Chart:

• 77% of CPU is on L8

• Let’s look at our RMI usage to see what’s driving L8 CPU



RMI Usage Pie Chart:

• 100% of the RMI activity is related to DB2

Let’s look at one more thing with CICS PA



CICS PA – Review Statistic Alerts – Demo Backup Slide

12

• This alert shows that we exceeded out DB2 entry thread wait threshold.

• At this point we need to look deeper into ECAT to see what the programs are doing

to sometimes cause a lot more DB2 activity

• We will use Application Performance Analyzer to drill into ECAT’s performance at the

program level.

Statistic Alerts:

• Define alerts with specific threshold values for SMF 110 sub-type 2 (stats) records

• CICS PA includes hundreds of sample alerts

• Critical, warning, and informational levels

• Only report on activity exceeding the threshold



The example is constructed around a simple sales

catalog and order processing application, in which a user

can perform various tasks such as:

List items in a catalog.

Inquire on individual items in the catalog.

Order items from the catalog.

Administrative functions

Problem: ECAT transaction is taking too long and we will use Application Performance Analyzer to look for performance issues with ECAT


Some trouble has been found!

Our Application (Problem Detected)



What is APA?

A tool that helps you perform application

performance tuning for applications running on

z/OS systems

Measures and report on how system resources

are used by applications running in virtually any

z/OS address space such as TSO and batch

How does APA work? Automatically starts to monitor job performance

when the job or program becomes active

Analyzes transactions running under IMS, CICS, WAS, as well as DB2 Stored procedures

Displays overall system activity enabling you to check job execution online and select which active job to monitor

Provides multiple summary reports to assist in identifying key areas of performance

Online analysis and reports that can be created as PDF or XML files that can be viewed on workstations or transferred easily to other applications

Proven 3270-based interface and free graphical interface

APPLICATION PERFORMANCE

ANALYSIS

Application Performance Analyzer for z/OS

Why this tool is important• Helps maximize the performance of your

applications and improve the response time of your online applications and batch turnaround

• Identifies constraints and improves the entire

application’s performance no matter where the

problem resides (CICS, IMS, DB2, MQ, COBOL,

PLI, ASM, JAVA)

• Isolate performance problems in new and existing

applications

• Aids application design, development and

maintenance lifecycle


APA – Open the APA Perspective - Demo Backup Slide

15

Start an APA monitoring session

We will monitor activity in a specific CICS region where the ECAT transaction is invoked.

New Observation


APA – Specify Measurement Criteria - Demo Backup Slide

16

Specify the Number of Samples you want taken and the Duration.

In this example, 25,000 samples will be taken over a duration of 10 minutes.

Specify CICS Region pattern and select Get Active Jobs button


APA – Select CICS Region to Monitor - Demo Backup Slide

17

Select one or more active CICS Regions to monitor activity.

In this example, CICSAOR5 region will be monitored.

Select CICSAOR5


APA –Region Selected - Demo Backup Slide

18

Selected regions are added to the Selected Jobs List.

CICSAOR5 added to the selected jobs list and will be monitored during this observation session.


APA – Select Data Extractors and Submit - Demo Backup Slide

19

Data Extractors are selected in order for APA to collect additional information specific to a

subsystem environment.

CICS and DB2 data extractors are needed in order for APA to collect specific information.

Click Submit to start APA monitoring


APA – Observation Session Submitted - Demo Backup Slide

20

New request is added to the Observation List panel.

APA is now monitoring CICS Region CICSAOR5. Now let’s go to that region

to invoke the ECAT transaction that has the performance problem.


APA – Invoke the CICS Transaction - Demo Backup Slide

21

EnterEnter

Logon to the CICS Region being monitored.

Note: You must select the CICS data extractor in the Schedule New Measurement

panel, and enter the transaction name(s) or patterns you want to observe.

CICSAOR5 region activity is being monitored where the ECAT transaction is

being invoked.


APA – Test the Application- Demo Backup Slide

22

EnterEnter

Test the online application for performance issues while APA monitors the

region activity.

Select Option 1 to display the catalog items.


APA – Select an Item from the Catalog - Demo Backup Slide

23

EnterEnter

Select an item from the catalog to order.

There are two distinct types of data that Application Performance Analyzer

gathers when the CICS data extractor is active: Session statistics, and

Transaction measurement data.


APA – Problem Detected - Demo Backup Slide

24

EnterPF5

After pressing PF5 so that the application suggests other products, I noticed

the clock at the bottom of the screen and that it was taking longer than

expected to display the results.

A transaction is only sampled if it was specified when the Observation

Request was created.


APA – Back to APA for Analysis - Demo Backup Slide

25

Return to APA to see what could possibly be going on in the application.

APA’s observation session on CICSAOR5 has ended. The Analysis Reports can now be

accessed to determine what happened.


APA – Performance Analysis Reports - Demo Backup Slide

26

Performance Analysis Reports enables in-depth analysis of the performance data collected.

Start your Analysis with the S01 Measurement Profile Report.


APA – S01 Breakdown of CPU Activity - Demo Backup Slide

27

The S01 Measurement Profile Report provides an at-a-glance summary of various

aspects of the measurement data and helps you choose which other reports to

concentrate on. This Report shows an overall percentage of time in three states: CPU

time, Wait time, and Queued time.

Here, it collected 25,000 samples. With each sample, it determined which one of the

states it was in. Then drilled in to get the details about exactly what it was doing at that

moment. Over the duration, it collected a statistical sample that we can use to determine

where that time was actually spent.


APA – S09 Measurement Analysis Report - Demo Backup Slide

28

The S09 Measurement Analysis Report provides recommendations of where some

performance improvements might exist. It is also suggests other reports to obtain more

detail about the area in question.

In this case, the report shows that there is a High CPU percentage of activity

observed in a single load module…and that there is high average response time for

a terminal attached CICS transaction.


APA – CICS Measurement Reports - Demo Backup Slide

29

The CICS Reports show CICS performance related activity occurring in the selected

region during the Observation Session.


APA-E04 CICS Mean Service Time by TXN - Demo Backup Slide

30

The E04 Report shows a list of all the transactions that were observed during the

monitoring session.

Click module to expand it

You can see here that, out of the average 28 second response time, over 20

seconds of that was caused by SEQUEL processing…


APA – Expand DBRM Module ECP120- Demo Backup Slide

31

Use this report to see how time was spent by the CICS transactions that were

executing during the observation session.

Expanding that, you can see that an OPEN statement was the cause.


APA – Click on OPEN to get details- Demo Backup Slide

32

Right click an item for more options.

Select Details to see more detail information regarding the OPEN

SQL statement.


APA – SQL Statement Information Details- Demo Backup Slide

33

This panel shows the additional detail pertaining to the SQL statement to help you in

your analysis. This information can be provided to a developer or DBA to help you

tune this statement.

We can see here that the cause of the bulk of this response time was

this one SQL statement in program ECP120. Scroll down for further

information.


APA – Formatted SQL Statement - Demo Backup Slide

34

A formatted version of the SQL statement can be displayed.

We can also see the fully formatted SQL statement in program ECP120.


APA – DB2 Reports - Demo Backup Slide

35

There are other reports you can use for further analysis. The DB2 Performance

Reports displays measurement data directly related to SQL activity.


APA –F03 DB2 SQL Activity by DBRM- Demo Backup Slide

36

The DB2 SQL activity by DBRM report shows DBRM’s that were running

during the measurement.

That DBRM is expanded and we can see that same SQL OPEN statement

that was displayed in the CICS report we saw earlier. So you can reach the

same conclusions from different reports.


APA – Back to E04 CICS Report - Demo Backup Slide

37

Some reports, such as the E04 CICS Mean Service Time Report, also show

other information that you can drill down into for further analysis.

Besides the SQL statement, you can also see that program ECP110 is

responsible for just over 7 Seconds of the average response time.

Expanding that program, you see other programs and system modules that

program ECP110 called.


APA – Module ECP110 Analyzed - Demo Backup Slide

38

You can drill down into source statements of application programs.

Right click on module ECP110 and select “Source Program Mapping” to get

program source statement information for application programs.


APA – Drill Down into the Source - Demo Backup Slide

39

Source Program Mapping shows CPU time right down to the individual

statement level.

Statement 625 was the cause of most of the CPU time…The pink line

shows the CPU time of a system module that was executed because of

that statement. In other words, the CPU time of the system module was

attributed to that statement.



Problem: CICS PA Transaction Performance Alerts indicated that a threshold was exceeded by the ECAT Transaction

Action: A developer was notified to take a closer

look at the application.

Cause: Using APA, developer determined that an OPEN Sequel Statement in module ECP120 needs to be tuned and is the main cause of the slow response time for the ECAT transaction


Trouble is contained!

Our Application Conclusion



Summary

41

Use the combined capabilities of CICS Performance Analyzer and

Application Performance Analyzer for z/OS to:

• Identify performance tuning candidate applications at a CICS system

level

• Perform a subsequent deep-dive performance analysis right down to

the application source code level

Uncover and solve your CICS performance issues—right down to the source

code level TODAY!


Exploitation of GUIs made Simple!

Simplicity

All PD Tools GUIs talks to PD Tools Common Server (a quick to start and extensible!).

Additional capabilities added to the server by a simple configuration script (APA V13.1 exploits this feature)

PD Tools Studio: All PD Tools GUIs package together including z/OS Explorer

You can integrate into CICS Explorer or IMS Explorer for Development.

Flexibility

Select capabilities needed to create your own workbench.

Select capabilities you need from a single update site.


Download CICS Explorer from the IBM web site

www.ibm.com/software/htp/cics/explorer/


Download the PD Tools plug-ins or the PD Tools Studio at:

www.ibm.com/software/awdtools/deployment/pdtplugins


More information

Web pages

– http://ibm.com/cics

– http://ibm.com/cics/showcase

– http://ibm.com/cics/tools

– http://ibm.com/software/awdtools/deployment

Analyst reports

– IBM CICS Tools: Discovery and Optimization for the Next Generation link

– IBM PD Tools – leads the pack again link

Threadsafety & Consolidation - extracting optimum performance from CICS Prezi YouTube

Social media

– Like CICS on Facebook at CICS Hursley or WebSphere and CICS Support

– Watch videos on YouTube - CICS and CICS Tools and IBM System z PD Tools

– View presentations on Slideshare

– Follow IBM_CICS on Twitter

http://www.ibm.com/support/docview.wss?uid=swg21384915

– See multiple channels by using the CICS Social Media Aggregator.

Subscribe to

– CICSbuzz for the latest newshttp://www-01.ibm.com/software/htp/cics/enews/

– Debug Tool newsletter http://www-01.ibm.com/support/docview.wss?uid=swg21422089

http://ibm.com/cics

http://ibm.com/cics/showcase

http://ibm.com/cics/tools

http://ibm.com/software/awdtools/deployment

http://public.dhe.ibm.com/software/htp/cics/tools/IBM_System_z_CICS_Tools_Whitepaper_2014.pdf

https://www.ibm.com/services/forms/signup.do?lang=en_US&source=swg-rszswg

http://prezi.com/70hukvwqs0ma/threadsafety-consolidation-extracting-optimum-performance-from-cics/

https://www.youtube.com/watch?v=svmHxgkeoOI&list=UU3LNUaOa3YyEW8B6oBcf9dA

https://www.facebook.com/IBMCICS?ref=hl

https://www.facebook.com/WebSphereSupport

https://www.youtube.com/channel/UC3LNUaOa3YyEW8B6oBcf9dA

https://www.youtube.com/user/IBMPDTOOLS

http://www.slideshare.net/IBM_CICS

http://www.ibm.com/support/docview.wss?uid=swg21384915

https://www-304.ibm.com/social/aggregator/cics

http://www-01.ibm.com/software/htp/cics/enews/

http://www-01.ibm.com/support/docview.wss?uid=swg21422089



Software

Lunch n Learn - Solving CICS performance issues