DougClifton_DB2 Utilities Update 2011-1

8/9/2019 DougClifton_DB2 Utilities Update 2011-1

1/55

1 2011 IBM Corporation

DB2 for z/OS Utilities Update: Now and Next

Doug CliftonExecutive IT Specialist

[email protected]


2/55


Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL

PURPOSES ONLY.

WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE

INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS, WITHOUT

WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.IN ADDITION, THIS INFORMATION IS BASED ON IBMS CURRENT PRODUCT PLANS AND STRATEGY,

WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.

IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR

OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS ORTHEIR SUPPLIERS AND/OR LICENSORS); OR

ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT

GOVERNING THE USE OF IBM SOFTWARE.


3/55


Agenda

Recent DB2 9 enhancements

Current DB2 9 work

DB2 10 utility enhancements

Summary

Based on presentations by:

Haakon Roberts

DB2 Development


4/55


Delivery of enhancements in maintenance stream

When prudent to do so

Risk vs. benefit

Resource Simplify utility processing

Reduce resource consumption

Improve performance

Reduce CPU consumption


5/55


SORTNUM Elimination CHECK INDEX, REBUILD INDEX, REORG, RUNSTATS

PK45916 (V8) & PK41899 (V9)

Better performance, more robust, simpler SORTNUM no longer required

Difficult to estimate: failure if too low, excessive sort work allocation if too high

New zparms UTSORTAL & IGNSORTN (online changeable)

UTSORTAL YES|NO Use RTS data to estimate number of rows to sort

DB2 will dynamically allocate sort work datasets

If SORTWK DD cards not hard coded

IGNSORTN YES|NO

Override utility job setting of SORTNUM

Recommendation

Turn on UTSORTAL, test it, then consider turning on IGNSORTN

DSNU3340I 168 08:13:52.66 DSNUGLSR - UTILITY PERFORMS DYNAMIC ALLOCATION OF SORT DISK SPACE


6/55


Other recent enhancements Permit use of ALIASes for LOAD, RUNSTATS and UNLOAD

PK77061 (V9)

New DSNACCOX stored procedure to gather statistics from catalog and make utility

recommendations See PK44133

DSNACCOR still supported

Better information for DPROPR/QRep or other IFI 306 readers

Write diag log record at utility termination so IFCID 306 readers can trigger refresh PK78558 (V9)

EAV dataset support

PK81151 (V8 & V9)

Improved LOAD/UNLOAD processing with NUMRECS parameter PK88970/PK88972/PK88974 (V8 & V9)

Replaces SORTKEYS at table space level with NUMRECS at table level

Simpler, eliminates risk of LOAD failure for load of multiple tables with skewed datadistribution


7/55


Other recent enhancements LOAD/UNLOAD LOB file reference variable performance

PK75216 (V9)

PDS only, not HFS

56% ET reduction on UNLOAD, 93% ET reduction on LOAD

LOAD and UNLOAD to/from virtual file

USS named pipe support with templates

PK70269 (V8 & V9)

PK96023 (V8 & V9)

LBI on UNLOAD 60% CPU reduction, 50% ET reduction

LOAD COPYDICTIONARY

PK63324/PK63325 (V9)

REORG avoidance prime empty partitions with compression dictionary Avoid FRVs for LOAD/UNLOAD of zero length LOBs

PM12286 (V9)

Performance


8/55


Performance utility CPU consumption

Focus on real CPU reduction & zIIP exploitation

DB2 utilities have been zIIP-enabled since 2006

Real CPU cost reduction in V9 10-20% for COPY & RECOVER

5-30% for LOAD, REORG, REBUILD INDEX

20-60% for CHECK INDEX

35% for LOAD partition

30-40% for RUNSTATS INDEX

40-50% for REORG INDEX

70% for LOAD REPLACE partition with dummy input

Flashcopy exploitation in DB2 10 dramatically reduces CPU consumptionfor COPY & reduces CPU for RECOVER & inline copies

More zIIP offload in DB2 10 with RUNSTATS


9/55


Performance zIIP exploitation for sort processing

In spite of CPU reduction in V9, there is continued focus on

CPU consumption for utilities

Sort can consume ~60% of total utility CPU time

DB2 in concert with DFSORT provides zIIP offload of DB2

utility memory-object fixed-length record sort processing

Requirements:

DB2 APAR PK85889 (V8 or V9)

DFSORT APAR PK85856

z/OS 1.10

PTFs can be applied independently of each other

Exploitation is automatic


10/55


DB2 Sort for z/OS v1.1 Announced Aug 10th, GA Sep 24th

Provides high speed utility sort processing for DB2 for z/OS

data Provides CPU & elapsed time reduction

Up to 30% reduction in elapsed time

Up to 50% reduction in CPU consumption

zIIP-enabled for further CPU cost reduction

Improved resilience, resource management & data availability

*Customer resul ts may vary. Resul ts based on analysis don e at SVL lab


11/55


New solutions for DB2 9

LOAD/UNLOAD FORMAT INTERNAL

Delivery early 2011

Unload and load data in true internal format

Avoid field processing 30% ET reduction, 50% CPU reduction measured for LOAD

50% ET reduction, 60% CPU reduction for UNLOAD

LOAD PRESORTED

PM19584 delivery in V9 & V10 post-GA

Avoid sort overhead

Up to 25% CPU reduction, 33% ET reduction depending on no of indexes

Can combine with Utility Enhancement Tool PRESORT option

Support REORG of multiple part ranges

PK87762 & PM13259 (V9)

E.g. REORG PART 1,45:71,500:503,4010

More efficient, improved availability, exploit parallelism

Allow CHECK SHRLEVEL CHANGE to use FASTREPLICATION(REQUIRED)

PM19034 (V9)

Fail CHECK utility rather than incur application outage


12/55


Reduced need for REORG INDEX

List prefetch of index leaf pages based on non-leaf information for range

scans

Improved performance for part-level REORG with NPIs & REORG INDEX

Index list prefetch results in up to 60% elapsed time reduction

Reduced need for REORG with compress on insert

New REORGCLUSTERSENS RTS column

If no clustering-sensitive queries then avoid REORG to restore clustering

DSNACCOX enhanced

REORG SHRLEVEL CHANGE for all cat/dir page sets

REORG SHRLEVEL REFERENCE|CHANGE to remove REORP

DB2 10 REORG improved availability & removed restrictions


13/55


REORG SHRLEVEL CHANGE for LOBs

Independent of whether LOBs are LOG NO or LOG YES

No mapping table required Base table space must be LOGGED

REORG FORCE option to cancel blocking threads

FORCE ALL or just READERS

Same process as CANCEL THREAD so requires thread to be active inDB2 for it to be cancelled

Threads cancelled on final drain

Reduced application outage for REORG with inline stats

Update catalog after dedrain REORG support for multiple part ranges

REORG support retrofitted to V9 in PK87762

LISTDEF support is not retrofitted

DB2 10 REORG improved availability & removed restrictions


14/55


New AUX keyword on REORG of partitioned base for improved LOB handling

Permit rows to flow between partitions

Allows REORG REBALANCE with LOB columns

Allows ALTER of LIMITKEY with LOB columns

Permits move of rows between parts on PBG REORG

Permits deletion of corresponding LOBs on REORG DISCARD

Default is AUX NO unless LOB objects required to complete REORG

No XML column support for classic partitioned or PBR

No mapping table change

Utilities improved availability & removed restrictions

LOB

LOBPart 1

Part 2


15/55


V10: LOAD/UNLOAD

Remove MAX_UTIL_PARTS zparm

Restriction removed for REORG in

V9 Improved performance for LOAD

REPLACE with LOB data

Up to 50% elapsed time reduction

Spanned record support for

LOB/XML data

LOBs & XML documents inlined inSYSREC with base data

Option in addition to FRVs

Performance & portability

0

100

200

300

400

500

600

700

5000 10000 20000 30000 40000

LOBs unloaded

S

e

c

o

n

d

s

e

la

p

s

e

d

PDSE

VBS


16/55


V10: COPY

Dataset-level Flashcopy support

COPY, RECOVER, REORG, LOAD, REBUILD INDEX, REORG INDEX

New zparms & utility parms to govern

Significant CPU & elapsed time reduction for large pagesets

Create transaction-consistent image copies from SHRLEVEL CHANGE

CPU time per object (z10)

0

0.5

11.5

2

2.5

3

0 25 50 75 100

Object size (MB)

S

ec

onds

IC

FC

Elapsed time per object (z10)

00.5

11.5

22.5

3

3.5

0 25 50 75 100

Object size (MB)

S

econds

IC

FC


17/55


V10: COPY

Post-GA Flashcopy enhancements

Support for REORG AUX YES to FlashCopy aux objects

Remove requirement for sequential inline copy for REORG

Additional zparms with APAR PM26762

Govern whether FlashCopy is preferred or required for CHECK utilities

Govern FlashCopy in PPRC environments

Govern FlashCopy use by RECOVER to avoid conflict with BACKUP SYSTEM

CPU time per object (z10)

0

0.5

11.5

2

2.5

3

0 25 50 75 100

Object size (MB)

S

ec

onds

IC

FC

Elapsed time per object (z10)

00.5

11.5

22.5

3

3.5

0 25 50 75 100

Object size (MB)

S

econds

IC

FC


18/55


V10: COPY

Improved dataset management & performance

CHANGELIMIT will not allocate copy dataset unless copy taken

&ICTYPE now matches actual image copy type

Use RTS for CHANGELIMIT performance

Incremental copy will not allocate copy dataset unless pages

changed

Delivery post-GA


19/55


V10: RECOVER

New BACKOUT YES option for point in time recovery

True rollback, not run of generated SQL undo statements

Requires COPY YES for indexes VERIFYSET option to fail PIT recovery if entire set not included

Base, LOB, XML, history not RI

ENFORCE option to avoid CHKP/ACHKP when subset of set recovered

Improved performance due to avoidance of set checking (RI, aux)


20/55


V10: Stats

RUNSTATS PROFILE support for simplification

RUNSTATS on views

Autonomic features through new stored procedures & catalogtables

All catalog statistics columns made updatable

RUNSTATS SHRLEVEL REFERENCE updates RTS

TOTALROWS & TOTALENTRIES columns

zIIP-enablement for RUNSTATS

Auto sampling rates & page sampling instead of row sampling

Significant CPU & ET savings

TABLESAMPLE SYSTEM AUTO


21/55


V10: CHECK

CHECK utilities will no longer set CHKP/ACHKP

CHECK SHRLEVEL CHANGE default changed to fail if Flashcopy not

available

ZPARM to govern

CHECK DATA enhanced for XML support

Document validation

Schema validation

Automated exception table processing for XML documents


22/55


V10: Other

Removed UTSERIAL lock for greater utility concurrency

SQL SELECT on SYSLGRNX

Expect delivery post-GA

LISTDEF & TEMPLATE enhancements

LISTDEF support for CHECK DATA

LISTDEF support for multiple part ranges on REORG

LISTDEF support for DEFINED YES|NO|ALL

Improved utility performance since unnecessary to build & then discardstructures for undefined objects

Default changed to DEFINED YES & empty lists result in RC4

REPORT RECOVERY support for SLBs

DSNACCOX enhancements Support hashed pagesets

New RTS columns for SSD, other


23/55


Summary

Eliminate outages

Improve performance

Reduce resource consumption

Reduce complexity & improve automation

Day 1 utility support for core DB2 10 function

Continued delivery of performance improvements & features ofreal value


24/55


25/55


Agenda

General recommendations

Backup

Recovery/QUIESCE

MODIFY RECOVERY

LOAD/UNLOAD

REORG

CHECK utilities

DSN1COPY

Summary


26/55


General recommendations


27/55


Summary

Newer releases are better than older releases

Performance

Function

But be mindful of part level REORG in V9

Newer maintenance levels are better than older maintenance

levels Performance

Function

Go to Utility Update talks at conferences to get the

latest information


28/55


Sort processing Improved utility sort processing

CHECK INDEX, REBUILD INDEX, REORG, RUNSTATS

PK45916 (V8) & PK41899 (V9)

Better performance, more robust, simpler SORTNUM no longer required

Correct value hard to determine, resulting in utility failure if too low or excessivesort work allocation if too high

New zparms UTSORTAL & IGNSORTN (online changeable)

UTSORTAL YES|NO Use RTS data to estimate number of rows to sort

DB2 will dynamically allocate sort work datasets

If SORTWK DD cards not hard coded

IGNSORTN YES|NO

Override utility job setting of SORTNUM Recommendation

Turn on UTSORTAL, test it, then consider turning on IGNSORTN

DSNU3340I 168 08:13:52.66 DSNUGLSR - UTILITY PERFORMS DYNAMIC ALLOCATION OF SORT DISK SPACE


29/55


Sort processing Sort work data set allocation

If assigning a data class though ACS routine, ensure its defined assingle volume only

Turn off space constraint relief in assigned data class Use large volumes for sort work data sets

>64K tracks supported since z/OS 1.7

DDs for ACS routines:

SORTWKnn, SWxxWKnn, DATAWKnn, DAxxWKnn, STATWKnn,STxxWKnn

See infoapars II14047 & II14213 for sort/DFSORT recommendations

Add //SORTDIAG DD DUMMY to utility job JCL

Negligible cost Adds 10 lines of DFSORT diagnostic output required in unlikely event

of problem


30/55


PBGs No LOAD or REORG parallelism No pruning of partitions

No LOAD at partition level

Rows can flow from one part to another withinpart range

LOAD / REORG at table space level will grow new parts asneeded

REORG cannot grow new parts if LOB column exists

REORG cannot move rows between parts if LOB column exists

REORG of single part or subset of parts will not grow newparts

Rows must fit back into part, but may not!

PCTFREE/FREEPAGE may cause REORG to fail

What to do if REORG fails because rows wont fit?

View as single table and REORG whole table space

If LOB columns exist then may need to UNLOAD/RELOAD

PK83397 new zparm to ignore PCTFREE/FREEPAGE

Solutions planned for Vnext

Part 1

Part 2UTRW

UTRW/UTRO

PBG

Part 3UTRW


31/55


RRF RRF row format introduced in V9 NFM to remove overhead of processing rowswith variable length columns

Some concern about conversion to RRF from BRF on REORG or LOAD in V9 NFM

Primarily concern is worse compression ratio for rows with many var length columns

Currently PK78958 disables RRF conversion for compressed page sets

Conversion is also blocked for tables with editprocs

UTS table spaces are expected to be in RRF format

Hidden zparm can govern conversion/creation in BRF/RRF format

Recommendation: Leave hidden zparm alone

unless cannot tolerate RRF for new page sets in V9 NFM

Compressed page sets will not get converted

New page sets, including UTS page sets, will be RRF

If hidden zparm is turned off, UTS page sets created in BRF format run a higher risk ofdefects

New developments

PK87348/PK85881

Externalise zparm to govern BRF/RRF conversion, provide external options onLOAD/REORG


32/55


Utilities on demand with supplied stored procedures

Run utilities only when necessary and not on fixed schedules

Information on the current status of all objects is contained in Real-

Time Statistics (RTS) tables DSNACCOR/DSNACCOX apply our suggested thresholds and

formulas against a list of objects and recommend utility actions

DSNACCOX in V9 NFM has improved RTS exploitation andrecommendations

Leverage the ability to invoke utilities programmatically via storedprocedures

DSNUTILS for EBCDIC parameters

DSNUTILU for UNICODE parameters

Rich application logic can control what is run and when Refer to the DB2 Utility Guide and Reference Appendix B and

samples

Use LISTDEFs and TEMPLATEs for further simplification andautomation of utilities


33/55


BACKUP


34/55


COPY

Exploit parallelism

Consider OPTIONS EVENT(ITEMERROR,SKIP)

Sets UTRW state only for duration of copy of individualpage set

But increases COPY overhead

Serialisation required for each pageset on the fly

Consider taking incremental copies and MERGECOPY

MERGECOPY marks relevant page set UTRW!

Copy indexes on large, critical tables

Particularly if rarely or never updated

Must include in MODIFY RECOVERY


35/55


Backup solutions

Multiple options

BACKUP SYSTEM

Significant DASD investment required Can be complex to set up & administer, but invocation simple

Must understand any limitations that currently exist

E.g. DASD mirroring issues, dataset movement issues, etc.

Sequential image copies Tried and trusted solution since V1.1

Other external backups, such as volume-level backups

Outside of DB2s control

Requires careful management and co-ordination Expect further options in future

Choice is dependent on environment and requirements, all optionswill continue to be supported


36/55


37/55


Recovery/QUIESCE

Maximise exploitation of parallel restore and Fast Log Apply

Recover multiple objects in a list in parallel but ideally


38/55


MODIFY RECOVERY


39/55


MODIFY RECOVERY

Base your MODIFY strategy on your backup strategyand not vice versa

Consider running every time a backup is taken or atleast weekly

REORG SYSLGRNX regularly for optimal performanceand minimal MODIFY impact on system

DB2 9 has RETAIN LAST n, GDGLIMIT and BSDSoptions

Careful with GDGLIMIT if you use multiple GDGs for asingle object

Will not clean up orphan entries in V9

Run MODIFY to delete recovery information from priorto a REORG that materialises row alterations

Makes subsequent REORGs more efficient


40/55


LOAD/UNLOAD


41/55


LOAD/UNLOAD

Run with LOG NO, REUSE, KEEPDICTIONARY ifpossible

Allocate inline copy data sets to DASD

Split up input dataset and drive LOAD partitionparallelism in a single LOAD

Use SORTNUM elimination

Specify SORTKEYS if input is on tape or variablelength

For LOAD REPLACE, consider loading into a clonethen renaming tables or datasets

Consider using USS named pipes

Refer to PK70269 for V8 and V9


42/55


43/55


LOAD/UNLOAD V10 considerations LOAD into a hashed table is expensive

Alternative process:

1 DDL to set up an indexed table

2 LOAD data into table

3 Alter table to organize by hash

1 No clustering index allowed

4 REORG with AUTOESTSPACE YES

Total elapsed time


44/55


REORG


45/55


REORG LOAD COPYDICTIONARY can avoid need to run REORG

If an NPI exists then concurrent REORGs of parts in the same table space is

not permitted in V9

Due to shadowing of NPIs, due to removal of BUILD2 phase Expect to provide ability to REORG disparate parts via APAR in V9

Be aware that REORG of a single part with an NPI can take longer in V9

Need to shadow NPI

If NPI is disorganized then it can take longer since unloading keys in order

But parallelism in V9 on unload/reload/logapply means multi-part REORG

should be faster and the log phase is much better at keeping up with

logging rates

So REORG multiple (adjacent) parts in the same REORG statement

Inline statistics can increase application outage duration

Catalog statistics updated after switch but whilst data is drained


46/55


REORG SHRLEVEL CHANGE main recommendations

Use DRAIN ALL to minimize application impact

(DRAIN_WAIT+MAXRO)


47/55


REORG INDEX vs. REBUILD INDEX REBUILD INDEX SHRLEVEL CHANGE provided in V9

Excellent for create of new non-unique indexes and for indexes

that are broken or already in RBDP

Does not operate against a shadow, so will set RBDP if not

already set

REORG INDEX operates against a shadow

Conclusion:

REBUILD INDEX SHRLEVEL CHANGE if index is broken or

already RBDP

REORG INDEX if index is accessible and want to maintainavailability


48/55


CHECK utilities


49/55


CHECK

If no applications are impacted and you just want to checkconsistency, then:

Either run CHECK SHRLEVEL CHANGE

Or run standard CHECK utility but be ready with REPAIR to reset

CHKP/ACHKP states

Or could consider running SQL ISO(UR) instead

If running SHRLEVEL CHANGE:

It will not reset CHKP or ACHKP states, nor will it set them

Make sure the page set is on FlashCopy-enabled DASD

If not then youll get a slow copy with the data in UTRO

If using DASD mirroring or BACKUP SYSTEM then use ZPARM

UTIL_TEMP_STORCLAS to prevent impacting either

Refer to PK41711


50/55


LOB integrity checking Aux index CHECK INDEX

Base table space

LOB table space

Row

Row

LOB

LOB

CHECK DATA

CHECK LOB

CHECK DATA on base tablespace

SCOPE AUXONLY

AUXERROR REPORT

AUXERROR INVALIDATE


51/55


Checking data integrity

NODEID

Index

Base table space

XML table space

Cols:

DOCIDNODEIDXML Record Value

XMLIndex

CHECK INDEX on DOCID, NODEID, XML indexes

CHECK DATA on base tablespace

SCOPE AUXONLY

AUXERROR REPORT

AUXERROR INVALIDATE

DOCID

Index

CHECK INDEX(2)

CHECK DATA(3)

CHECK INDEX(1)

Cols:DOCIDXMLCOL

CHECK INDEX(2)


52/55


DSN1COPY


53/55


DSN1COPY

DSN1COPY is an essential part of the utilities portfolio

DSN1COPY runs standalone and cannot ensure that data matchesdefinition at target

All target datasets must be pre-allocated for multi-piece table spaces

Areas to watch out for:

BRF-RRF mismatch

Tolerated by SQL, but not REORG

Convert page sets to ensure copy is RRF-RRF or BRF-BRF

If that isnt possible, e.g. if image copy is BRF, then UNLOAD from BRFimage copy and LOAD into RRF page set

Data definition changes, e.g. columns added

Use REPAIR VERSIONS at target site

For alterations prior to V8, REORG at source before DSN1COPY Different table space types or different segsizes

Not policed, abends will occur

DSN1COPY


54/55


DSN1COPY

Areas to watch out for:

XML

Data-dependent information kept in catalog table XMLSTRINGS

Cannot DSN1COPY XML table space from one subsystem/group toanother

DSN1COPY within a subsystem/group is fine

Solution is UNLOAD/LOAD/CROSSLOADER

DOCID is a sequence generated by DB2 DSN1COPY to a newtarget where the DOCID is lower will result in -803 on insert

because DB2 generates a value of n but n already exists in the

table. ALTER of the sequence isnt allowed for DB2-generated

sequences.

SELECT NEXT VALUE FOR can be used to increment

the number up to the max DOCID in the table


55/55


Summary

Stay reasonably current on versions and maintenance

Understand what this gives you in terms of utility capability Revisit your existing utility jobs to benefit from new options

Documents

DougClifton_DB2 Utilities Update 2011-1