Making it difficult for those IMS problems to hide

© Copyright IBM Corporation 2016. Materials may not be reproduced in whole or in part without the prior written permission of IBM.1

Making It Difficult for Those IMS problems to Hide

Jeff MaddixIMS Worldwide Technical [email protected]

408-463-4956

IMS Customer Internship – April 2016

IMS Customer Internship, April 2016, Silicon Valley Lab© Copyright IBM Corporation 2016. Materials may not be reproduced in

whole or in part without the prior written permission of IBM. 2

Making it Hard for those IMS problems to Hide - Agenda

� Section 1: MVS Setup Recommendations

� Section 2: IMS Setup Recommendations

� Section 3: Preservation of Standard Documentation

� Section 4: Manual Dump Intervention

� Section 5: Situation Specific Diagnostics



Section 1.

MVS Setup Recommendations



MVS Setup RecommendationsMaximize System Trace Table Size

�Maximize System Trace Table Size.

• Advantages:

• Extremely valuable for a large variety of problem types such at ABENDs, loops, and performance issues.

• Setup

� Set MVS System Trace table size to 999K or 3M in z/OS 1.9 or higher

–Default size is only 1M

–z/OS 1.9 max size is 9G

� The MVS command "TRACE ST,999K" or “TRACE ST,3M” can be specified in the MVS COMMNDxx SYS1.PARMLIB member.

� See the z/OS MVS Systems Commands Manual for further detail



�Maximize System Trace Table Size.

• Use the MVS Display command to ensure the system trace table is active:

– DISPLAY TRACE

IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479

SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION

ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K)

• Considerations:

� System Trace Table is page fixed storage.

� Supply a reasonable value to the nnnM or nG parameter after considering the available central storage and the actual storage required for system trace.

� Supplying a large value for nnnM or nG might cause a shortage of pageable storage in the system.

MVS Setup RecommendationsMaximize System Trace Table Size



MVS Setup RecommendationsIncrease Master Trace Table Size

� Increase master trace table size.

• Advantages:

� Master trace maintains a table of the most recently issued operator messages.

� Allows view of external events at the time of failure.

• Setup

� Set MVS Master Trace Table size to 1000K

–Default size is only 24K

� "TRACE MT,1000K" can be specified in the SCHEDxx member of SYS1.PARMLIB.

� See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning Guide/ Ref, and z/OS MVS System Commands manuals for complete details



MVS Setup RecommendationsIncrease Master Trace Table Size

� Increase master trace table size

• Use the MVS Display command to ensure the master trace table is active:

– DISPLAY TRACE

IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479

SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION

ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)

• Considerations:

� Ensure Master Trace Table is large enough to span most error time frames

� Uses Subpool 229 Key 0 High Private Pageable Storage of the master scheduler address space



MVS Setup RecommendationsUse the Common Storage Tracker

�Use the Common Storage Tracker

• Advantages:

• SVC dumps (or RMF reports) will provide CSA/ECSA ownership information with jobname, time, and requesting module information.

• Great for common storage outage (storage leaks) problems and fragmentation analysis.

• Setup

� Use the DIAGxx SYS1.PARMLIB member to contain the request. Specify DIAG=xx in the IPL system parameters or use the SET DIAG=xx operator command.

–Example: DIAGxx member: VSM TRACK CSA(ON) SQA(ON)

� See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning Guide/ Reference, and z/OS MVS System Commands manuals for complete details.



MVS Setup RecommendationsUse the Common Storage Tracker

�Use the Common Storage Tracker

• Display the Common Storage Tracker status by using the following MVS command:

• DISPLAY DIAG

• Considerations:

� Negligible Performance impact. Table stored in ESQA

IGV007I 11.44.05 DIAG DISPLAY 713

VSM TRACK CSA(ON) SQA(ON)

VSM TRACE GET(OFF) FREE(OFF)

VSM ALLOWUSERKEYCSA(YES)

VSM BESTFITCSA(NO)

VSM USEZOSV1R9RULES(YES)

TRAPS NAME()

CBLOC

VIRTUAL24()

VIRTUAL31()

REUSASID(NO)

AUTOIPL SADMP(NONE) MVS(NONE)



MVS Setup RecommendationsTune CHNGDUMP MAXSPACE

� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump.

• Advantages:

• Higher likelihood that SVC dumps will be captured in their entirety without worry of "Partial Dump".

• Setup


� Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate CHNGDUMP command at IPL time.

–Example: CD SET,SDUMP,MAXSPACE=5000M - Default size is 500M

�3000M is standard for large multi-address space SVC Dumps.

–See z/OS MVS Diagnosis: Tools and Service Aids, z/OS MVS System Commands, and z/OS Initialization and Tuning Guide for additional details.

• Continued on next page





Display the MAXSPACE setting using the following MVS Command:

D DUMP,OPTIONS

SDUMP- ADD OPTIONS

(ALLPSA,NUC,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP,

Q=YES,GRSQ,XESDATA),BUFFERS=00000000K,

MAXSPACE=00005000M,MSGTIME=99999 MINUTES,

MAXSNDSP=015 SECONDS,AUXMGMT=ON ,DEFERTND=NO

•Continued on next page





• Considerations:

• MAXSPACE is the total virtual storage used by SDUMP

• SDUMP data capture is serialized so only one dump can be written at a time. However, MVS releases the serialization after data capture, so another dump can be started before the first one gets written out (especially if something causes a delay in writing the first dump out to DASD). So MAXSPACE virtual storage might have to be temporarily "shared" by more than one dump.

� Ensure central storage and/or local page datasets are large enough to contain their normal peak load plus additional SVC dumps!

� See MVS Initialization and Tuning Guide for additional details.



MVS Setup RecommendationsAutomatic Dump Dataset Allocation

� Ensure automatic dump dataset allocation is in place.

• Advantages:

• SVC dumps will be allocated to the correct size without worry of "Partial Dump".

• Setup

� Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate DUMPDS commands to set this up.

–Example: DUMPDS NAME=, DUMPDS ADD, and DUMPDS ALLOC=ACTIVE

–See z/OS MVS Diagnosis: Tools and Service Aids and z/OS MVS Systems Commands manuals for complete details.

• Considerations:

� Ensure assigned storage class has enough space to ensure adequate SVC dump storage requirements



�Section 2.

IMS Setup Recommendations



IMS Setup RecommendationsUse FMTO Option D

� Specify IMS control region EXEC parm value: FMTO=D

• Produces an SDUMP for terminating and non-terminating errors

� Non-terminating errors include:

– IMS dynamic allocation failures

– Some ESAF (External Subsystem Attach Facility) failures

• Allows use of SYSMDUMP as a backup dump in case of SDUMP failures



IMS Setup RecommendationsSYSMDUMP DD for IMS Control Regions

� The SYSMDUMP specification will be used as a backup dump by IMS in the event

that SDUMP processing should fail.

� Specify SYSMDUMP DD in JCL of:

• IMS CTL Region

• IMS DLI/SAS Region

• IMS DBRC Region

� The following dump options should be specified in the SYS1.PARMLIB(IEADMR00)

member to ensure that adequate areas of MVS storage are dumped to diagnose the

problem under most circumstances:

• SDATA=(CSA,RGN,SQA,SUM,TRT,ALLNUC,LPA)

�The SYSMDUMP SDATA options can also be added via the CHNGDUMP command

• CD SET,SYSMDUMP=(CSA,RGN,SQA,SUM,TRT,ALLNUC,LPA)



IMS Setup RecommendationsSYSMDUMP DD, Continued

� SYSMDUMP DD Specification

• DISP=MOD should be specified because it is possible for multiple tasks to

generate dumps during the processing of an error. We probably will need to see

the first dump produced.

GDG Example

//SYSMDUMP DD DISP=(MOD,CATLG),DSN=IMSAP05.SYSAIMS.SYSMDUMP(+1),

// DCB=(LRECL=4160,RECFM=FB,BLKSIZE=29120),UNIT=3390,

// SPACE=(CYL,(750,100),RLSE),DATACLAS=dtcls,VOL=SER=volser

� Testing the SYSMDUMP function can be done on a test system by temporarily setting the SDUMP to NODUMP via the CHNGDUMP command. This forces a SYSMDUMP to be created.

• CD SET,SDUMP,NODUMP

• Then modify IMS out to test the SYSMDUMP function : F IMSTEST DUMP



IMS Setup RecommendationsSYSMDUMP DD, Continued

� Ensure that the SYSMDUMP SDATA options are properly set up.

Display the SYSMDUMP settings using the following MVS Command:

D DUMP,OPTIONS

IEE857I 16.01.50 DUMP OPTION

SYSMDUMP- ADD OPTIONS NUC,SQA,LSQA,SWA,TRT,RGN,LPA,CSA,SUM,ALLNUC)



IMS Setup RecommendationsSYSUDUMP DD

• The SYSUDUMP specification will be used by IMS dependent regions for failure events

• Application dump formatter tools rely on the SYSUDUMP DD statement to be present in the JCL

• Specify SYSUDUMP DD in JCL of IMS Dependent Regions

• The following dump options should be specified in the MVS SYS1.PARMLIB(IEADMP00) member to ensure that adequate areas of MVS storage are dumped

• SDATA=(CB,ERR,SUM) PDATA=(JPA,LPA,PSW,REGS,SA,SPLS)

• Display options using MVS D DUMP,O command

IEE857I 16.23.15 DUMP OPTION

SYSUDUMP- ADD OPTIONS SDATA=(CB,ERR,SUM), PDATA=(SA,REGS,LPA,JPA,PSW,SPLS)



IMS Setup RecommendationsIMS Table Traces Activated

� Turn on the IMS Dispatcher, Scheduler, DLI, and Lock traces to capture general system diagnostics and database access history

• Specify the following options in the IMS PROCLIB member: DFSVSMxx

• DISP=ON, SCHD=ON, DL/I=ON, LOCK=ON

• Or, use IMS command:

• TRA SET ON TABLE nnnn

Where nnnn is alternately = DISP, SCHD, DLI, and LOCK

• These traces have relatively low CPU overhead

• SCHD being the lowest and DLI being the highest

� For non-production environments, when recreating failures, run with table traces on for the functional areas being tested



IMS Setup RecommendationsIMS Table Traces Activated to External Trace

� IMS external tracing allows for IMS trace table output to be placed on IMS trace datasets rather than on the IMS OLDS when:

• DFSVSMxx PROCLIB member with nnnn=OUT is specified rather than nnnn=ON

• DISP=ON, SCHD=ON, DL/I=OUT, LOCK=OUT

• The IMS TRACE command specifies OPTION LOG

• /TRA SET ON TABLE nnnn OPTION LOG

� Increases IMS system throughput should external tracing be desired versus sending trace data to the IMS OLDS.

• External trace dataset logging can add 1% as measured at SVL. One large IMS user reported a 2-4% CPU increase using their IO subsystem

� External trace datasets are allocated in the following order:

1) DASD JCL - DFSTRA01 and DFSTRA02 DDNAMES

2) DASD MDA- DFSTRA01 and DFSTRA02 Dynamic Allocation Members

3) TAPE MDA - DFSTRA0T Dynamic Allocation Member

4) IMS OLDS - If none of the above are found



IMS Setup RecommendationsIMS Table Traces Display Status

� Display IMS Table Trace status

Command: /DIS TRACE TABLE

TABLE TRACE LOGGING ENTRIES VOLUME SYS3

DL/I ON 2268 SYS3

DL/I ON MEDIUM SYS3

LOCK ON MEDIUM SYS3

PI OFF OFF SYS3

LATC OFF OFF 3024 OFF SYS3

DISP ON OFF 1260 MEDIUM SYS3

SCHD ON OFF 630 MEDIUM SYS3

SUBS OFF OFF 504 OFF SYS3

DLOG ON OFF 756 MEDIUM SYS3



IMS Setup RecommendationsExternal Trace Environment

� IMS external tracing specifications

• DFSTRA01 and DFSTRA02DSORG SequentialRECFM VBLRECL 4004BLKSIZE 20024 (LRECLx n +4)

-Set up unique GDG with

- minimum – retained 24 hour limit for archive

- ideal – retained to last IMS restart or longer

hlq.imsid.TRACE.ARCHIVE(0)

-Use automation to archive filled trace dataset off to trace archive GDG datasets on external trace dataset switch

DFS2864I EXTERNAL TRACE DATA SET xxxxxxxx FULL - SWITCHING TO yyyyyyyy. imsid

- and for IMS ABEND Scenario –

DFS3896I ATTEMPTING TO PURGE OLDS BUFFERS



IMS Setup RecommendationsExternal Trace Environment

� Display IMS external tracing specification

Command: /DIS TRACE XTRC.

IMS ACTIVE TRACES

IMS EXTERNAL TRACE IS TRACING TO DISK

XTRC DDNAME ALLOC STATUS STATUS

DFSTRA01 DYNA ALLOCATED CLOSED

DFSTRA02 DYNA ALLOCATED ACTIVE

DFSTRA0T DYNA UNALLOCATED CLOSED



�Section 3.

Preservation of Standard Documentation



Preservation of Standard Documentation

�Consider implementing normal operating procedures to preserve the following documentation near the time of error

MVS Console (SYSLOG)

SYS1.LOGREC

IMS OLDS/SLDS

JES JOBLOG of jobs related to failure

Any Dumps Produced

For elusive problems, these items might be requested a month or more after the problem has been reported.



Ideal Time Periods for Syslog, Logrec, SLDS, and Joblog Preservation

MVS Console (SYSLOG), Logrec, SLDS and Joblogs

• The ideal time frame:

�Back to the last IMS restart

�MVS Console from the prior clean execution (For comparison)

• The moderate time frame:

�24 hours of MVS Console messages

• The minimum time frame:

�Two IMS system checkpoint intervals



Dump Preservation

� All dumps created near the time of IMS failure should be retained

SYS1.DUMP Datasets should be examined

� Multiple dumps may be created

� Keep all dumps at time of failure regardless of the subsystem

SYSMDUMP for the IMS Control, DLI/SAS, and DBRC regions need to be examined in case of primary SYS1.DUMP failures.

� Save these datasets if a dump was produced

� If GDG is used for copies, examine in ISPF for tracks used

SYSUDUMP should be saved for affected IMS dependent regions



�Section 4.

Manual Dump Intervention



Additional Manual Dump Intervention

�While IMS produces SDUMPs for most internal errors without human intervention, IMS Wait/Loop or partial loss of function conditions will require intervention to produce an SVC dump

�IMS wait/loops can be caused by interaction with many address spaces including:

CTL DLI/SAS DBRC IRLM CQS

OM RMSCIODBA

cctlMPP

(DBDC,

DBCTL,

DCCTL)

IFP BMP

IMS Connect

JMP JBP

APPC VTAMTCPIP

RRS

MQ

Series

Non-IBM

Tools

IBM Tools

IXGLOGR

DB2ODBM

WLM

Other?

**

******

*

*

ap thr ap thr ap thr ap thrap thr ap thr

thrap

Subsys *

Subsys *



How Much Data to Dump?

� Due to the complex interactions between the many address spaces, it is sometimes difficult to determine exactly where the source of the problem lies without a dump of the associated address spaces and their associated data areas

• Omission of any interrelated address space or data area adds to the possibility that the dump may not be sufficient to solve the problem

• MVS dumps have a maximum job total of 15 per dump



IMS Dump Techniques

IMS SVC Dumps can be requested using various techniques

• MVS SYS1.PARMLIB IEADMCxx

• DUMP Command Parmlib

• MVS SYS1.PARMLIB IEASLPxx

• SLIP Command Parmlib

• Manual MVS DUMP command

• Customized JCL can be built and submitted

• Others?....



IEADMCxx, MVS SYS1.PARMLIB

� DUMP Command Parmlib Member

• Create SYS1.PARMLIB members called IEADMCxx for each customized dump command

• Advantages

• Can be used to customize IMS dumps prior to error event.

• Simple operator interface

• Disadvantages

• All Jobs must be active or no dump is produced on the system on which the dump command is entered. Remote systems without a jobname match will dump the DUMPSRV address space

• Avoid issues:

• Use wildcards * (multiple chars) and ? (single char) and similar naming conventions can be used to specify the jobnames

• Edit the Parmlib member jobnames prior to issuing the command

• See z/OS MVS Initialization and Tuning Guide and z/OS System Commands



� Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters:

IEADMCxx Example for IMS

JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),

SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)

Where: Where: Where: Where:

j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.

j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.

j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2

j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.

Note: It is recommended to use wildcards if possible to ensure jobname matches to allow

for cases where one or more regions might already be down.

Example: Using the following naming conventions, the JOBNAME statement could be reduced: IMSPCTLA, IMSPDLIA, IMSPDRCA IMSPRLMA CICSP1A CICSP2A CICSP3A

JOBNAME=(IMSP*A,CICSP?A)



IEADMCxx DUMP Activation

� To request a dump from the IEADMCI1 parmlib member, enter the following MVS commands:

• DUMP TITLE=(DUMP OF IMS and CCTL Regions ),PARMLIB=I1

A dump dataset will be created on the MVS image from which the dump command is entered



IMS SYSPLEX Dump Considerations

�IMS SYSPLEX implementations need to consider the possibility that a hang or problem on one IMSplex member may be due to a problem originating from another member, particularly in the area of IRLM locking

�Problems such as IMS Wait/Loops or partial loss of function conditions which require intervention to produce an SVC dump, should include SVC dumps from other members of the IMSplex

�Challenge:

Ensure that a dump is taken for all necessary address spaces on each system.



� Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters:

Sysplex IEADMCxx Example

It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down

Note: The DSPNAME, XESDATA, COUPLE and REMOTE parms are added in SYSPLEX

environments

JOBNAME=(j1,j2,j3,j4,j5,j6,j7),

SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),

DSPNAME=('j4'.*,'XCF*'.*),

REMOTE=(SYSLIST=(*('j1','j2','j3','j4’,’j5’,’j6’,’j7’),

SDATA,DSPNAME))

Where:

j1 = IMS Control Region Jobname j2 = DBRC Region Jobname

j3 = IMS DLI Region Jobname j4 = IRLM Region Jobname

j5 = CCTL Region 1 J6 = CCTL Region 2

j7 = CCTL Region 3



Sysplex IEADMCxx DUMP Activation

� To request a dump from the IEADMCI1 parmlib member, enter the following MVS command:

• DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=I1

� One dump dataset will be created on each MVS image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs



Storage as a Moving Target

� Storage can be a moving target while being dumped, especially if multiple address spaces interact. Here are some considerations:

• If the MVS CHNGDUMP SET, SDUMP=(Q=Y) command , is specified, all address spaces that are not exempt from system-wide nondispatchabilityare set nondispatchable while global storage (CSA, SQA) is dumped.

–By quiescing most system activity during SVC dump global storage capture, the dump will contain a more consistent image however, your system activity will nearly cease during global storage capture

– The Q =Y option is generally not recommended because non-affected jobs will be temporarily quiesced.



Storage as a Moving Target, Continued

•In all cases (Q=Y or Q=N), dumped address spaces are made nondispatchable while global storage and the private region storage is being dumped for those address spaces

•There is no intentional dispatching coordination between requested address spaces in a dump.

• All private storage is dumped through scheduling of SRBs to the requested address spaces.

• This SRB activity is asynchronous.

• Multiple SRBs to multiple address spaces will run independently and loosely in parallel



�Section 5.

IMS Situation Specific Diagnostics



�Situation Specific Diagnostics:Control Region Wait/Hang

� Control Region Wait/Hang

• Take a SVC dump using one of the methods discussed earlier in the presentation.

• Consider dumping related regions in the dump

• If a partial wait occurs, consider taking two of the above mentioned SVC Dumps to identify normal activity versus abnormal activity.

• Note: Modify dumps of IMS are no longer the preferred method to obtain wait documentation.

�IMS Control Region

�IMS DLI/SAS Region

�DBRC Region

�IRLM Region

�CQS

�Operations Mgr

�Resource Mgr

�Structure Call Interface

�TCPIP

�Websphere

�ESAF - DB2, MQSeries, others

�Other Regions ???

�Other IMSplex members with

all their related regions

�ODBM

� = Can't do without

�Troublesome IMS Dep Regs

�CCTL Regions

�ODBA

�IXGLOGRC

�RRS

�APPC

�WLM

�VTAM



�Situation Specific Diagnostics:Control Region Wait/Hang, Continued

JOBNAME=(isa?ims,isa?dls,isa?irlm,ci1?sa??),


DSPNAME=('ISA?IRLM'.*,'XCF*'.*),

REMOTE=(SYSLIST=(

SP1A('isa?ims','isa?d*','isa?irlm','ci1?sa??'),

SP1C('isa?ims','isa?d*','isa?irlm','ci1?sa??'),

SP1E('isa?ims','isa?d*','isa?irlm','ci1?sa??'),

SP1F('isa?ims','isa?d*','isa?irlm','ci1?sa??'),

SDATA,DSPNAME))

� IEADMCxx Parmlib (IEADMCA1) example for a slow moving performance problem or hang in a DBCTL data sharing environment with IRLM

• Dump Command to activate the above dump:

• DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=A1

• One dump will be created on each of the specified sysplex members



�Situation Specific Diagnostics:Application System Timeout

� Application lockouts are detected by IRLM and MSG DXR162I is issued.

DXR162I irlmx CYCLE NUMBER nnnnnnnn PROCESSED FOR TIMEOUT

Explanation: A TIMEOUT candidate has been detected on at least one IRLM. If an

incompatible blocker was found, the IMS TIMEOUT EXIT has been driven to

present WAITER/BLOCKER information

Set the following SLIP to capture a dump for this event

SLIP SET,A=(SVCD),MSGID=DXR162I,


JL=(ISA*,CI1?SA??),ID=M162,DSPNAME=('ISA?IRLM'.*,'XCF*'.*),

REMOTE=(JOBLIST,DSPNAME,SDATA),

END



�Situation Specific Diagnostics:IMS Loop

� If the IMS Control, DLI, DBRC or Dependent Region enters a loop state:

• Turn on the MVS System Trace Table up to 5M - if possible

• TRACE ST,5M

– See MVS System Trace considerations regarding real memory earlier in this presentation

– Use IBM Application Performance Analyzer for z/OS or equivalent tool to capture module CPU utilization information for high CPU region into a report

– Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions

• Note: IMSplex partner dumps are probably not required for loop problems, unless they are also looping

– Reset the MVS system trace table to its original settings



�Situation Specific Diagnostics:DBRC Related Problems

� DBRC Related Problems

• Obtain a listing of the DBRC RECONs prior to repair attempts

• Use Recover Control Utility (DSPURX00) - LIST.RECON

command

• If the RECON dataset listing cannot be obtained due to size, then

alternatively obtain a subsystem listing

• Use Recover Control Utility (DSPURX00) - LIST.SUBSYS ALL

command

• LIST.RECON is preferred



�Situation Specific Diagnostics:DBRC Related Problems, Continued� DBRC Related Problems, Continued

• Use D GRS,CONTENTION command on each system sharing the RECON

to determine if the RECON is held at the exclusion of other waiters

D GRS,CONTENTION

S=SYSTEMS DSPURI01 RECON1.IMSBP1

SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS

BT21 IBA2FDBR 0124 009AF7F8 EXCLUSIVE OWN

BT23 IBA1FDBR 0090 009AF1B0 EXCLUSIVE WAIT

BT21 IBA4FDBR 05E0 009AF7F8 EXCLUSIVE WAIT

BT26 IBA4DBRC 05DE 009F15B8 EXCLUSIVE WAIT

BT21 IBA1DBRC 00D7 009FF118 EXCLUSIVE WAIT

BT23 IBA2DBRC 0600 009F15C8 EXCLUSIVE WAIT

� If so, dump the owning address space

DUMP COMM=(dump title)

R nn,JOBNAME=(j1),SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ),END



�Situation Specific Diagnostics:DBRC Related Problems, Continued

� Obtain before and after copies of the RECONs

• JCL Example for Recon dump to file using DFDSS

//JJJJJJJJ JOB ( )

//* DUMP RECON TO FILE

//STEP1 EXEC PGM=ADRDSSU,REGION=0M

//SYSPRINT DD SYSOUT=*

//DASD DD DSN=hhhhhhhh.RECON.BACKUP,DISP=(NEW,CATLG),

// UNIT=3390,SPACE=(CYL,(5,5),RLSE)

//SYSIN DD *

DUMP DS(INCLUDE(IMS.RECON*)) -

OUTDD(DASD)

/*



�Situation Specific Diagnostics:DBRC Related Problems, Continued

� JCL Example for Recon restore from file using DFDSS

//JJJJJJJJ JOB ( )

//* TO DOWNLOAD TAPE LOADED WITH DFSDSS

//* NEED TO USE RENUNC TO RENAME TO YOUR HLQ

//* TAPE MAP WILL NOT SHOW LRECL OR BLKSIZE....

//S01RSTR EXEC PGM=ADRDSSU


//IN DD DISP=SHR,DSN=hhhhhhhh.RECON.BACKUP

//DASD DD UNIT=3390,DISP=OLD,VOL=SER=SYS285

//SYSIN DD *

RESTORE DATASET(INCLUDE( ** )) -

INDDNAME(IN) -

OUTDDNAME(DASD) -

CANCELERROR -

CATALOG -

REPLACE -

IMPORT -

TGTALLOC(SOURCE) -

RENAMEUNCONDITIONAL(hhhhhhhh) -

WAIT(2,2)

/*



�Situation Specific Diagnostics:DBCTL Related Problems

� DBCTL Related Problems

• Issue IMS /DISPLAY ACTIVE and /DISPLAY CCTL commands, multiple times

• /DIS ACTIVE

• /DIS CCTL ALL

• /DIS CCTL cctlname

• Set the AP portion of the CICS trace to level 1-2

• Set the FC (FILE CONTROL) portion of the CICS trace to level 1-2

• Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions, especially the CCTL regions



�Situation Specific Diagnostics:Database Problems

�For problems in the database arena, obtain

�Damaged database dataset

• Database Image copy of broken database in state prior to damage

• Image Copy of logically related databases

• IMS SLDS from all data sharing IMS subsystems

• Save to last good database image copy of damaged database

• If possible, and not already set, use IMS command and save output:

• /TRA SET ON TABLE nnnn OPTION LOG

• Where nnnn alternately = DLI and LOCK with external trace dataset recommended

• Joblog and SYSOUT from the Pointer Checker broken database

• Joblog and SYSOUT from batch jobs that accessed the damaged database



�Situation Specific Diagnostics:Database Problems, Continued

�For problems in the database arena, obtain (Cont...)

• DBD for broken databases

• PSBs that access the broken DB

• List.RECON and LIST.HISTORY DBD(broken database)

• SMF 60 (VSAM Volume Data Set Updated), 62 (VSAM Component or Cluster Opened), and 64 (VSAM Component or Cluster Status) 79 (IRLM) records from all data sharing systems

-Back to last good image copy of broken database

• Any dumps produced related to the broken database

• MVS Syslog

• IMS CTL Joblog

• IMS DLI Joblog



�Situation Specific Diagnostics:Database Problems, Continued

�For problems in the database arena, obtain (continued)

• For VSAM datasets, close the broken KSDS and run IDCAMS commands:

� VERIFY

� EXAMINE ITEST NODTEST

� EXAMINE NOITEST DTEST

� LISTCAT ALL

� If the dataset cannot be closed, run IDCAMS commands:





� LISTCAT ALL

� If the data set is open and being updated then an EXAMINE may reflect what appears to be a problem because the

data set is in a state of flux. Therefore, if you run the EXAMINEs twice and the errors are the same, there is

probably a problem. If the errors are different, then there is no problem.



�Situation Specific Diagnostics:ESAF Interface Problems

� Problems involving the ESAF (External Subsystem Attach Facility)

• Turn on the IMS SUBS TRACE to the external trace dataset

• IMS command: /TRA SET ON TABLE SUBS OLDS

• Take a SVC dump of the related regions before and after the recreate attempt.

� Ensure that pertinent regions for the affected subsystem (db2mstr, db2dbm1, mqseries) are included with the other related IMS regions

� Note: IMSplex partner dumps are probably not required for ESAF problems, unless they are also experiencing the problem.



�Situation Specific Diagnostics:DC Related Problems

� Problems involving Data Communications

• Issue IMS DISPLAY NODE x command.

• /DIS NODE nodename

• Turn on IMS NODE trace. Data will be captured in the IMS TYPE6701 log record. Print records using IMS utility programs DFSERA10/DFSERA30.

• /TRA SET ON NODE nodename

• Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS NODE trace. Note: GTF must be active with the USR option.

• F NET,TRACE,TYPE=BUF,ID=nodename

• F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)

• Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM.



�Situation Specific Diagnostics:APPC Related Problems

� Problems involving APPC

• Turn on the IMS LUMI Trace to the external trace dataset

•

• /TRACE SET ON TABLE LUMI OPTION LOG

• /TRACE SET ON LUNAME XXXXXXX INPUT

• /TRACE SET ON LUNAME XXXXXXX OUTPUT

• (XXXXXXX is the partner LU)

• Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS LUMI trace. Note: GTF must be active with the USR option

•

• F NET,TRACE,TYPE=BUF,ID=nodename

• F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)

• Turn on the Transaction trace for TPPCB DL/1 calls.

• /TRACE SET ON TRAN tttttttt

• tttttttt is the transaction name

• IMS TYPE6701 log records will be produced



�Situation Specific Diagnostics:APPC Related Problems

� Problems involving APPC , (cont...)

• Turn on the MVS APPC component trace.

• TRACE CT,ON,200M,COMP=SYSAPPC

• nn,OPTIONS=(GLOBAL),END

• When the problem has been recreated, stop the CTRACE.

• TRACE CT,OFF,COMP=SYSAPPC

• APPC component trace sends its trace buffers to a SYS1.DUMP dataset.

• The following IPCS commands can be used to format the trace:

• CTRACE COMP(SYSAPPC) SHORT - One line summary entries

• CTRACE COMP(SYSAPPC) FULL - Detailed trace entry formatting

– Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM and the APPC regions.



�Situation Specific Diagnostics:ODBM Related Problems

� Problems involving ODBM

• Set the ODBM BPECFGxx PROCLIB member with the following table traces including enough space to span 1 hour

TRCLEV=(DISP,LOW,BPE,PAGES=2000) /* DISPATCHER TRACE */

TRCLEV=(AWE,HIGH,BPE,PAGES=3000) /* AWE SERVER TRACE */

TRCLEV=(HASH,HIGH,BPE,PAGES=1000) /* BPE HASH TRACE */

TRCKEV=(ERR,HIGH,ODBM,PAGES=300) /* ODBM ERR TRACE ON HIGH */

TRCLEV=(ODBM,HIGH,ODBM,PAGES=5000) /* ODBM TRACE ON HIGH */

TRCLEV=(PLEX,HIGH,ODBM,PAGES=32000) /* PLEX TRACE ON HIGH */

� Considerations: Ensure that enough virtual private storage is available for the ODBM region to support these internal traces via REGION= startup parm



�Situation Specific Diagnostics:ODBM Related Problems, Continued

� In a Sysplex environment with XCF SCI routing to the ODBM routing from IMS Connect

• Set the SCI BPECFGxx PROCLIB member with the following table traces including enough space to span 1 hour

TRCLEV=(DISP,LOW,BPE,PAGES=2000) /* DISPATCHER TRACE */

TRCLEV=(AWE,HIGH,BPE,PAGES=3000) /* AWE SERVER TRACE */

TRCLEV=(PLEX,LOW,SCI,PAGES=1000) /* PLEX SCI TRACE */

TRCLEV=(INTF,LOW,SCI,PAGES=1000) /* INTF SCI TRACE */

TRCLEV=(ERPL,HIGH,SCI,PAGES=2000) /* ERPL SCI TRACES TO HIGH */

TRCLEV=(INTP,HIGH,SCI,PAGES=2000) /* INTP SCI TRACES TO HIGH */

� Considerations: Ensure that enough virtual private storage is available for the SCI region to support these internal traces via REGION= startup parm



�Situation Specific Diagnostics:ODBM Related Problems, Continued

� Use IMS Connect related enhancement tools to capture event traces through IMS connect to their logs

– If these tools are not available, the IMS Recorder trace can be used

�Set up automation to monitor ODBM threads for hang conditions using the DISPLAY CCTL ALL and DISPLAY CCTL ODBMnnn command based on non-zero recovery tokens which do not change for a period of time

� Take a dump of the ODBM environment prior to issuing IMS STOP THREAD command against the hung threads

�ODBM

�IMS Connect

�IMS Control Region

�IMS DLI/SAS Region

�DBRC Region

�IRLM Region

�Structure Call Interface

�RRS

�Other IMSplex members

with all their related

regions



�Situation Specific Diagnostics:CQS Additional Manual Dump Intervention

� IMS/CQS environment additional dump considerations include:

• Structure dumps – Primary, Overflow, Lock entries

• CQS Region

• Other CQS Clients with their related CQS regions

• Above regions from the other IMSplex members

• MVS Logger



� Create two SYS1.PARMLIB members:

�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures

SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA –––– IMS PartnersIMS PartnersIMS PartnersIMS Partners

JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),

SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE),

DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=

(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),

SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME))

Where: Where: Where: Where:

j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname

j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname

j5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Region

j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM

j9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Region

It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down.



� SYS1.PARMLIB member for structures and MVS logger

�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures

SYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIB

JJJJOOOOBBBBNNNNAAAAMMMMEEEE====((((IIIIXXXXGGGGLLLLOOOOGGGGRRRR)))),,,,DDDDSSSSPPPPNNNNAAAAMMMMEEEE====((((''''IIIIXXXXGGGGLLLLOOOOGGGGRRRR''''....****)))),,,,

SSSSDDDDAAAATTTTAAAA====((((AAAALLLLLLLLNNNNUUUUCCCC,,,,LLLLPPPPAAAA,,,,PPPPSSSSAAAA,,,,RRRRGGGGNNNN,,,,SSSSQQQQAAAA,,,,TTTTRRRRTTTT,,,,CCCCSSSSAAAA,,,,GGGGRRRRSSSSQQQQ,,,,XXXXEEEESSSSDDDDAAAATTTTAAAA,,,,CCCCOOOOUUUUPPPPLLLLEEEE)))),,,,

STRLIST=STRLIST=STRLIST=STRLIST=

(STRNAME=(STRNAME=(STRNAME=(STRNAME=imsmsgq01imsmsgq01imsmsgq01imsmsgq01,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),

STRNAME=STRNAME=STRNAME=STRNAME=imsmsgq01oflwimsmsgq01oflwimsmsgq01oflwimsmsgq01oflw,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER)

, , , ,

STRNAME=STRNAME=STRNAME=STRNAME=mvslogqmsg01mvslogqmsg01mvslogqmsg01mvslogqmsg01,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,

ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))

Where:Where:Where:Where:

imsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure name

imsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure name

mvslogqmsg01..............is the associated logger structuremvslogqmsg01..............is the associated logger structuremvslogqmsg01..............is the associated logger structuremvslogqmsg01..............is the associated logger structure



�Situation Specific Diagnostics:CQS - IEADMCxx DUMP Activation

� To request a dump from the IEADMCIA and IEADDMCIB parmlib members, enter the following MVS command:

DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)

One combined dump dataset will be created on the MVS image from which the command is entered.

One dump dataset will be created on each image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs for PARMLIB member IA.

NOTE: MVS logger support only needs the MVS logger address space from the system which experiences the problem.



�Situation Specific Diagnostics:CQS - MVS Log Stream

�The merged MVS log stream can be used to examine CQS log

records.

• IEBGENER can be used along with the default log stream

subsystem exit routine, IXGSEXIT, to copy the log records at

time of failure for later analysis



�Situation Specific Diagnostics:CQS - MVS Log Stream, JCL Example

//CQSCPYLG JOB USERID,USERID,MSGLEVEL=1,CLASS=K

//*********************************************************************

//* This job copies a CQS log stream to a dataset (max 32K / record) *

//* *

//* - Replace the DSN on the SYSUT1 card with your CQS logstream *

//* name. *

//* *

//* - Replace the DSN on the SYSUT2 card with your desired output *

//* dataset name. You may also need to adjust the space *

//* allocations, depending on the size of your logstream. *

//* *

//* Changes: *

//* 02/24/97 ben Creation *

//*********************************************************************

//STEP1 EXEC PGM=IEBGENER,REGION=1024K


//SYSUDUMP DD SYSOUT=*

//SYSIN DD DUMMY

//SYSUT1 DD DSN=SYSLOG.MSGQ01.LOG,

// SUBSYS=(LOGR,IXGSEXIT),

// DCB=(BLKSIZE=32760)

//SYSUT2 DD DSN=CQS.LOG.COPY,

// DISP=(NEW,KEEP,DELETE),

// VOL=SER=USER05,

// SPACE=(CYL,(2,10)),

ƒ // UNIT=SYSDA



�Situation Specific Diagnostics:CQS - SRDS

� Save the most recent CQS SRDS (Structure Recovery Data Set) for each structure dumped

• Use IDCAMS repro if LRECL is acceptable (less than 32761)

• If LRECL greater that 32761, use DFSDSS Copy - DFSMS DFDSS ADRDSSU utility



�Situation Specific Diagnostics:MSC Related Problems

� If you use IMS MSC (Multiple Systems Coupling) and experience a related

problem

• Take a SVC dump of the coupled IMS regions (minimally the CTL

regions, but the problem may reside in any IMS related region as

discussed earlier)

• In addition, ensure that the VTAM and/or TCPIP address space is

included. Do this as close to the problem time as possible, prior to

attempting to fix the problem

• Save the IMS OLDS for both coupled systems from the time of the

message creation

• Issue DISPLAY LINK ALL and DISPLAY LINK ALL MODE commands to

show the status and queue counts for the logical link and physical link

information for partner



�Situation Specific Diagnostics:MSC Related Problems, Cont.

MSC related problems, continued

Issue the following SPOC commands at 1 minute intervals (5 times)

Example:

//RXSPOC3 JOB ('A=W005'),BATCH,CLASS=K,MSGLEVEL=(1,1),

// TIME=1440,USER=USRT001,PASSWORD=xxxxxxxx

/*ROUTE PRINT STLVM3/MADDIX

//SPOC EXEC PGM=CSLUSPOC,

// PARM=('IMSPLEX=PLEX1,ROUTE=IMS1,F=BYCOL')

//* PARM=('IMSPLEX=PLEX1,F=BYRSC')

//* PARM=('IMSPLEX=PLEX1,F=WRAP')

//STEPLIB DD DISP=SHR,DSN=IMSBLD.I12RTSMM.ARESLIB

//SYSPRINT DD SYSOUT=*,

// DCB=(RECFM=FB,LRECL=121,BLKSIZE=2420,DSORG=PS)

//SYSIN DD *

QUERY MSLINK NAME(LNK12V02) SHOW(ALL)

QUERY MSLINK NAME(LNK12V02) SHOW(STATISTICS)

UPDATE MSLINK NAME(LNK12V02) START(STATISTICS) OPTION(RESET)



�Situation Specific Diagnostics:MSC Related Problems, Cont.

MSC related problems, continued

If using VTAM, Turn on the VTAM Internal Trace and VTAM Buffer trace for both coupled systems.

Start GTF with BLOK=4M, and TRACE=USRP,USR=(FE1,FEF,FF1),END

using a 1000 Cylinder GTF Trace dataset.

– F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG),BFRNUM=8

– F NET,TRACE,TYPE=BUF,ID=nodename

• The MSC LINK trace should also be turned on for both coupled systems (link# or ALL is acceptable for the link specification)

– TRACE SET ON LINK link# LEVEL 4 MODULE ALL

� IMS TYPE6701 records containing TM control blocks for each message at key points

� Note: The options on this page will affect performance, so use only if the prior dump and Query command analysis indicate it is necessary



�Situation Specific Diagnostics:CTL/DLI Region “Out of Storage”

�IMS control region “out of storage” conditions (ABEND80A ABEND878) can result in ABEND40D (Recursive errors in MVS RTM)

• Problem: Unable to diagnose problem because the dump dataset doesn’t contain private storage for the failing region.

• Solution: Set the following four SLIP traps to capture the initial “out of storage” prior to RTM gaining control



�Situation Specific Diagnostics:CTL/DLI Region “Out of Storage”, Continued

SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname,

SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),

ID=xxxx,END ID=xxxx,END ID=xxxx,END ID=xxxx,END

SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname,

SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),

ID=xxxx,END ID=xxxx,END ID=xxxx,END ID=xxxx,END

WhereWhereWhereWhere

jobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMS

CTL region,CTL region,CTL region,CTL region,

= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region

Enter the following two SLIP traps for both the IMS

Control and IMS DLI/SAS regions (4 slips total)



Making Hard for those IMS Problems to Hide -Summary

� MVS Setup Recommendations

� IMS Setup Recommendations

� Preservation of Standard Documentation

� Manual Dump Intervention

� Situation Specific Diagnostics

Technology

Making it difficult for those IMS problems to hide