DB2 I/O Performance: NCCMG ® August 1, 2006 Speaker: Jeff Berger Northern California CMG Enhanced I/O Performance On the System z9 with DB2, FICON and

DB2 I/O Performance: NCCMG

®

August 1, 2006

Speaker: Jeff Berger

Northern California CMG

Enhanced I/O Performance

On the System z9 with DB2, FICON and the DS8000

®

2

IBM Software Group | DB2 Information Management Software


The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

Trademarks

The following are trademarks or registered trademarks of other companies.

* Registered trademarks of IBM Corporation

* All other products may be trademarks or registered trademarks of their respective companies.

Linux is a registered trademark of Linus Torvalds

Penguin (Tux) compliments of Larry Ewing

Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries

UNIX is a registered trademark of The Open Group in the United States and other countries.

Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.

SET and Secure Electronic Transaction are trademarks owned by SET Secure Electronic Transaction LLC.

Notes:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

390ACF/VTAM*AIX*APPN*CICS*DB2*e-business logo*ESCON*GDPS*Geographically Dispersed Parallel Sysplex*

FICONHiperSocketsHPRIBM *IBM logo *IMSMagstar *MVS/ESANet.Data*Netfinity

OS/2 *OS/390 *Parallel Sysplex *pSeriesRACF *RMFRS/6000 *S/390 *S/390 Parallel Enterprise ServerSysplex Timer *

Virtual Image Facility *VM/ESA *VSE/ESAVTAM *WebSphere *xSeriesz/OSzSeriesz/VM

3



Disclaimers

This document contains performance informationPerformance is based on measurements done in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the numbers stated here.

4



Agenda

Introduction to sequential I/O performance

Review CKD architecture, track level CCWs, and MIDAWs

Introduce the z9, FICON Express 2 & 4, and the DS8000

Introduce some DB2 V8 and V9 features

The anatomy of a DB2 prefetch operation

Review of Media Manager, EF data sets, striping

DB2 logging and other DB2 functions

Performance of backup/restore, parallel queries

Using new hardware to solve your performance problems

5



Sequential I/O performance

This presentation is about sequential I/O, not random I/O

DB2 sequential I/O is important in the following applications:

daily full image copy cycles

processing of LOBs (large objects)

daily/weekly table unloads to feed data warehouse/marts

database recovery

Measurements of DB2 prefetch I/Os are sufficient to understand how hardware affects all of the above

6



0

20

40

60

80

100

120

Throughput (MB/sec)

3990-6 RVA ESS E20 ESS F20 ESS 800 DS8000 MIDAWs

EF Non-EF

DB2 Sequential Prefetch (4K pages)Exponential Growth

1993 1997 2000 2001 2002 2005 2005 ESCON FICON EXP1 FICON EXP2 200Mbit/sec 1Gbit/sec 2GBit/sec

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the numbers stated here.

7



0

20

40

60

80

100

120

140

Th

rou

gh

pu

t (

MB

/sec

)

FICON Express 2 FICON Express 4 DB2 V9

DB2 Sequential Prefetch (4K pages)Continuing to improve


8



Sequential “scenarios”

1. Single stream, data in cache Zero disconnect time

Sensitive to the speed of the channel, link and Host Adapters

2. Single stream, prestage data from disk Also may have zero disconnect time

Non-zero disconnect time indicates that the disk or Device Adapter(s) is slower than the channel

3. Highly parallel streams spread across Raid ranks Zero disconnect time

Sensitive to CU bus capacity as well as the number of channels and the channel speeds

9



DB2 Sequential Prefetch I/O Response Timesz9 EC, FICON Express 2, DS8000, 10K RPM 300GB drives

0

1

2

3

4

32x4K 64x4K

Prefetch quantity

I/O R

esp

on

se T

ime

(ms)

CPU Pend

Connect DisconnectIf data is cache resident,

disconnect goes away, but other components are unaffected

Disconnect time is not caused by latency

As the prefetch quantity increases…..

both connect time and disconnect time increase

the ratio of connect to disconnect decreases

10



DB2 Sequential Prefetch Throughputz9 EC, FICON Express 2, DS8000

0

20

40

60

80

100

120

140

160

Th

rou

gh

pu

t (M

B/s

ec)

From cache From 10K RPM 300GB disk

64x4K

8x32K

32x4K

4x32K

11



DB2 V8 Sequential vs Dynamic Prefetch (4K pages)z9 EC, FICON Express 2, DS8000Assuming that the data set is cache resident

0

50

100

150

200

250

Th

rou

gh

pu

t (M

B/s

ec)

Sequential Prefetch Dynamic Prefetch

12



Measurements

2002/2003

DB2 Version 8, z990, FICON Express 1, ESS 800, 2 Gbit/sec link, z/OS 1.3

2005

DB2 Version 8, z9, FICON Express 1 and 2, DS8000, 2 Gbit/sec link, z/OS 1.6 (pre-MIDAWs) and z/OS 1.7 (MIDAWs)

All z9 measurements done using z9 Enterprise Class (EC)

2006

DB2 Version 9

FICON Express 4, DS8000, 4 Gbit/sec link, z/OS 1.7

13



Review of CKD Architecture

14



FICON EF Performance PenaltyDB2 Table Scans

0

50

100

Thro

ughp

ut

(MB/

sec)

Non-EF EF

FICON Exp. 2, DS8000

0

10

20

30

40

50

Thro

ughp

ut

(MB

/sec

)

Non-EF EF

FICON Exp. 1, ESS 800

15



Scatter Reads and Gather WritesChannel programs use real addresses

Must be able to scatter reads or gather writes to/from discontiguous real storage

CKD architecture has always had two methods

Indirect Address Words (IDAW)

• An IDAW used to be 4 bytes. (Hence, a “word”)

• Now IDAWs are usually 8 bytes to accommodate 64-bit addresses

Data Chaining

A MIDAW is a Modified Indirect Address Word

A MIDAW is 16 bytes, including a two-byte length field

16



Channel Programs

A Channel Program consists of a series of 8-byte CCWs (Channel Command Word)

Each CCW contains opcode, flags, length field, and address pointer

CCWs are executed sequentially unless the opcode is a TIC which acts like a branch or “go to” statement

17



CKD Channel Program Architecture

4K page on 4K boundary

4K page on 4K boundary

CCW = Channel Command Word 0 8 16 32 63 Real AddressLengthFlagsOpcode

IDAW List (Indirect Address Words)

0 63

Real Address

Real Address

Real Address

Real Address

Must end on 4K boundary

Must start on 4K boundary

The operation continues until the Length is satisfied (or there is no more data to read)

18



CCW Chaining 0 8 16 32 63 Real AddressLengthFlagsOpcode

Chain command (CC)Chain data (CD)

Channel Program

0 63

If CD is set, continue the current CCW operation to the next CCW.

The channel will fetch the next CCW if CC is set.

If neither CD nor CC are set, the channel program terminates.

InDirect Address Word List (IDAW)

19



EF suffix

Write Count Key Data using data chaining

count field

user’s data

1DDC 8 real address

CCWs

4096 real address

32 real address

DC

real address

real address

real address

Write an 8 byte count field followed by a 4128 byte data field.

20



Chaining prior to Shark and FICON

Command chaining was the only way to transition from one record to the next record in the same channel program.

i.e. To read/write two records required two CCWs

IDAWs were used for “scatter reads” and “scatter writes”, when the record required more than one 4K frame.

Data chaining was rarely used. There were two cases where data chaining was used:

Format writes, to chain the 8-byte Count field to the Data field

EF datasets, to chain the user’s data to the 32-byte suffix

Data chaining performed well on ESCON.

21



Track-level CCW op-codesSupported by Shark (and other compatible control units)

Purpose: To reduce the number of CCWs (but actually it didn’t because we didn’t have MIDAWs)

Read Track Data and Write Track Data

Alternative to traditional Read Data and Update Write op-codes

One CCW can stream data across records within a track

CKD count fields are not read or written

Traditional Write Count-Key-Data still used for most formatting

These op-codes were introduced in 2000 and implemented by Media Manager.

This was good for ESCON, but did not help FICON.

22



Data chaining with Shark and FICON

FICON data chaining hurts performance when the data chunks are small

i.e. Data chaining hurts the performance of EF datasets, as well as the performance of format writes

With ESCON, data chaining was not a performance problem, and EF I/O was as efficient as non-EF I/O

Since the overhead of data chaining was “per block”, large blocks were largely unaffected by data chaining

Data chaining causes higher channel utilization than IDAWs.

Especially so with 32 byte chunks

Command chaining is also inefficient. i.e. The best way to optimize FICON performance is to minimize the number of CCWs.

23



MIDAWs

24



What is the MIDAW Facility?Modified Indirect Data Address Word

Improves the performance of sequential I/Os using 4K datasets, especially with Extended Format

Eliminates the EF performance penalty and shrinks the small DB2 page size performance penalty

MIDAWs implemented by Media Manager, and supported by EXCPVR

EXCP does not support MIDAWs

Lower utilization of the FICON channel, link, and CU host adapter

25



MIDAW Requirements

Requires the System z9 processor and z/OS® 1.7 or 1.6 + APARs

APARs OA10984, OA13324, OA13384

Currently not supported by non-IBM storage vendor products

If you want to be allow a multi-volume data set to span a mixture of device types that span CUs where some support MIDAWs and some don’t, you need….

UA25186 (HDZ11J0) or UA25187 (HDZ11K0)

26



MIDAW Concepts0 40 48 127

reserved flags

64

count data addresses (64 bits)

Flags: Bit 40 - last MIDAW in listBit 41 - skip transfer to main storage (like Skip in CCW)Bit 42 - data transfer interruption control (like PCI in CCW)

CCWs with data chaining can direct arbitrary portions of a data block to separate buffer areas, that is, scatter-read or gather-write

27



Write Count Key Data using MIDAWs

1D 4136 real addressCCW

8

4096

32L

real address

real address

real address

The sum of all MDAW counts must equal the

CCW count and the “Last” flag must be set.

count field

user’s data

EF suffix

IDA

28



MIDAWs reduce the number of CCWsExample: Read or write a full 3390 track

Type of dataset,With or without MIDAWs

Number of CCWs per track

Read or Update

Format Write

4K, Non-EF, Pre-MIDAWs 12 24

4K, Non-EF, MIDAWs 1 12

4K, EF, Pre-MIDAWs 24 36

4K, EF, MIDAWs 1 12

16K, Non-EF, Pre-MIDAWs 3 6

16K, Non-EF, MIDAWs 1 3

16K, EF, Pre-MIDAWs 6 9

16K, EF, MIDAWs 1 3

29



Media manager channel program limitations

If DB2 provides too many buffers to MM, the I/O request is split into two I/Os, which can effect throughput by 10 to 15%

Prior to today, MIDAWs and CCWs in same 4K real frame.

Impact: 256K format writes of 4K pages

UA25366 (HDZ11J0) and UA25367 (HDZ11K0) move MIDAWs to a separate frame, which is still not enough for 512K

Sufficient for DB2 V8, but not always sufficient for DB2 V9

Impact: 512K reads and writes of 4K pages, and 512K format writes of 8K pages.

Future: Allow MIDAWs to use more than frame

30



MIDAWs improves the performance in two ways

Simplex effects (i.e. single stream):

This benefit is exclusive to EF datasets

Multiplexing effects (i.e. parallel I/O):

This benefit affects both EF and non-EF datasets

Single stream and multiplexing benefits are additive

EF benefits are much greater than non-EF benefits

There is no non-EF benefit unless multiple streams share the channel

31



Channel Efficiency

0

40

80

120

160

200

0 50 100 150

Channel Utilization (%)

Thro

ughp

ut (M

B/s

ec)

LessEfficient

MoreEfficient

Channel efficiency is the relationship of throughput to channel utilization

32



DB2 V8 Table Scans using 4K pages, EF DatasetsFICON Express 2, DS8000

Channel Efficiency

0

50

100

150

200

0 20 40 60 80 100

Channel Utilization

Th

rou

gh

pu

t (M

B/s

ec)

Pre-MIDAWs MIDAWs

I/O Response Time

0

1

2

3

4

5

0 1 2 3 4

Number of streams

I/O

Res

po

nse

Tim

e (m

s)

Pre-MIDAWs MIDAWs

33



DB2 V8 Table Scans using 4K pages, Non-EF DatasetsFICON Express 2, DS8000

Channel Efficiency

0

50

100

150

200

0 20 40 60 80 100

Channel Utilization

Th

rou

gh

pu

t (M

B/s

ec)

Pre-MIDAWs MIDAWs

I/O Response Time

00.5

11.5

22.5

33.5

0 1 2 3 4

Number of streams

I/O

Res

po

nse

Tim

e (m

s)

Pre-MIDAWs MIDAWs

34



Benefits of MIDAWs for DB2 table scan & 4k pagesFICON Express 2, DS8000

MIDAWs increases the throughput of EF datasets by 67%

MIDAWs eliminates the performance gap between EF and non-EF datasets 0

20

40

60

80

100

120

Th

rou

gh

pu

t (M

B/s

ec)

Non-EF EF

Type of data set

Pre-MIDAWs MIDAWsConfiguration:

MIDAW : z/OS 1.7

Pre-MIDAW: z/OS 1.6

DB2 for z/OS Version 8

4000 byte row size

System z9 109

FICON® Express2

2 Gbit/sec link

DS8000 control unit

*This performance data was measured in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,

and the workload processed.

35



DB2 V8 Table Scans using 4K pages, EF DatasetsFICON Express 2 versus 4

Channel Efficiency

050

100150200250300

0 20 40 60 80 100

Channel Utilization

Th

rou

gh

pu

t (M

B/s

ec)

FICON Express 2

FICON Express 4

I/O Response Time

02468

1012

0 5 10 15

Number of streams

I/O

Res

po

nse

T

ime

(ms)

FICON Express 2

FICON Express 4

36



Small Block Performance Penalty

When prefetching 128K per I/O, 16K page size performs worse than 4K page size, even though the number of bytes per track is the same, and the I/O is not disk bound

Why? Because the channel program for 4K required more CCWs than for 16K.

If the number of CCWs were the same, the channel efficiency would be the same. In fact, with MIDAWs and non-EF datasets, the channel programs look identical. Therefore, the channel efficiency is the same. With EF, they are almost the same.

Yet, even if the channel efficiency is the same, the control unit may or may not perform better with larger blocks

37



Benefits of MIDAWs: 4K vs 16K DB2 page sizeFICON Express 2, DS800

MIDAWs shrinks the performance gap between 4K and 16K CIs

0

0.5

1

1.5

2

I/O

Res

po

nse

T

ime

(ms)

4K page 16K page

EF Data SetPrefetch I/Os

Pre-MIDAWs MIDAWs

Configuration:

MIDAW : z/OS 1.7

Pre-MIDAW: z/OS 1.6


4000 byte row size

System z9 109

FICON® Express2

2 Gbit/sec link

DS8000 control unit



38



When is MIDAWs not beneficial?

MIDAWs do not help large block sizes

MIDAWs do not help if the FICON channel utilization is low

If you FICON utilizations are always low, then maybe you could get better performance if your batch workload used more parallelism.

MIDAWs mitigates the effect of using parallelism (for DB2 queries and online utilities) on OLTP

39



New hardware

40



z9 I/O highlights

Improved System I/O capabilities

40% increase in the maximum number of FICON Express2 channels

• Up to 336 on z9-109 vs. 240 on z990

80% increase in peak I/O bandwidth per book

Up to 30% increase in effective Sap capacity*

• based on OLTP-T workload measurements (55K io/sec per sap @70% sap utilization)

Enterprise Class (EC) and Business Class (BC)

BC machines have lower MIPS than EC (except for zIIPs)

MIDAWs

Improved channel efficiency *This performance data was measured in a controlled environment on a z9-109 running the LSPR OLTP-T workload under z/OS 1.6. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the

user’s job stream, the I/O configuration, the storage configuration, and the workload processed.

41



New hardware components in 2005

FICON Express2 channels (based on 2gbit links)

DS8000 storage subsystem

System z9

New hardware components in 2006FICON Express4 channels (4gbit links)

DB2 Version 9 to become available in 2007

42



Introducing FICON Express 2

0

50100

150200

250300

MB

/sec

FICONExpress 1

FICONExpress 2

Bi-directional Throughput

Capacity

Available in 1Q/05

FICON Express 2 increased the channel bandwidth by 58%

FICON Express 2 exceeds the uni-directional bandwidth of a 2Gbit/sec link

Given a 2Gbit channel, a mixture of reads and writes is necessary to exceed 200 MB/sec

These measurements were made using DB2. Other sources of performance data are not based on DB2 workloads.

43



Introducing FICON Express 4

050

100

150200250300

MB

/sec

FICONExpress 2

FICONExpress 4

Uni-directional Throughput

Capacity

Availability?

Utilizing a 4gbit/sec link, FICON Express 4 increases the uni-direction channel bandwidth by 50%

These measurements were made using DB2. Other sources of performance data are not based on DB2 workloads.

44



DS8000 highlightsNew POWER5 SMP processors Up to 256 GB cache FC-AL to link DA pairs to DDMsScalability up to 192 TBUp to 128 2 Gb/sec FICON host ports (max 16 ports recommended for an

individual host) Performance

A DS8100 has the performance capacity of two ESS 800s with half the footprint

Combined with FICON Express 2, the performance capacity for DB2 queries and utilities can be as much as three times higher than the ESS 800

The DS8300 is recommended for LPAR configuration, or for workloads making extensive use of advanced copy services (i.e. PPRC and Flashcopy)

45



DB2 page sizes

DB2 index compression

Dasd space management

46



DB2 Page Size …..

DB2 supports page sizes of 4K, 8K, 16K and 32K Large pages accommodate larger rows

A row cannot space pages and a page is limited to 255 rows

Larger objects are stored in LOB table spaces, and a row may contain a row id that identifies a large object in a corresponding LOB table space.

Small pages are best for OLTP to maximize buffer hit ratio

Beginning with DB2 V8, the CI size can be equal to page size A large CI size may achieves better sequential performance, depending

on the I/O configuration

For data integrity reasons, DB2 does not allow striping unless the CI Size matches the page size

47



….. DB2 Page Size

Starting in V9, DB2 supports up to 32K page size for indexes

A small page size will maximize the buffer hit ratio

A large page size will reduce the number of index levels

48



DB2 Table Scan using an EF Dataset

0102030405060

Th

rou

gh

pu

t

(MB

/se

c)

4K 16K

Page Size

ESS 800, Pre-MIDAWs, FE1

Configuration:

z/OS 1.3

DB2 for z/OS V8

4000 byte row size

System z900

FICON® Express1

2 Gbit/sec link

ESS 800 control unit

Configuration:

z/OS 1.7

DB2 for z/OS V8

4000 byte row size

System z9-109

FICON® Express2

2 Gbit/sec link

DS8000 control unit

020406080

100120

Th

rou

gh

pu

t (M

B/S

ec)

4K 16K

Page Size

DS8000, MIDAWs, FE2

Large page sizes were good with FICON Express 1, but less beneficial with FICON Express 2

49



DB2 Index Compression…..

Index compression is new to V9

Page level compression

Unlike data row compression:

Buffers contain expanded pages

Pages are decompressed when read from dasd

Prefetch performs the decompression asynchronously

A buffer hit does not need to decompress

Pages are compressed by the deferred write engine

Like data row compression:

An I/O bound scan will run faster

DSN1COMP utility can be used to predict space savings

50



Index Compression Example: Suppose an index could compress 3-to-1

Compressed 4K

Decompressed 16K buffer

12K used

4K unused

Compressed 4K

Decompressed 8K buffer

8K Bufferpool50% dasd space reductionNo increase in virtual storage cost

16K Bufferpool67% dasd space reduction33% increase in virtual storage cost

51



…..DB2 Index Compression

The CI Size on dasd of a compressed index is always 4K

A 4K expands into a 8K or 16K buffer, which is the DBA’s choice. This choice determines the maximum compression ratio.

Compression of key prefix and Rid Lists

A Rid List decribes all of the rows for a particular index key

An index with a high level of non-uniqueness, producing long Rid Lists, achieves about 1.4-to-1 compression

Compression of unique keys depends on prefix commonality

52



Index Compression: Performance

CPU cost is mostly inconsequential. Most of the cost is asynchronous, the exception being a synchronous read. The worst case is an index with a poor buffer hit ratio.

Example: Suppose the index would compress 3-to-1. You have three options…..

1. Use 8K buffer pool. Save 50% of dasd. No change in buffer hit ratio or real storage usage.

2. Use 16K buffer pool and increase the buffer pool size by 33%. Save 67% of dasd, increase real storage usage by 33%.

3. Use 16K buffer pool, with no change in buffer pool size. Save 67% of dasd, no change in real storage used, decrease in buffer hit ratio, with a corresponding increase in synchronous CPU time.

53



>4K Page Size for Indexes

V9 supports 4K, 8K, 16K and 32K page sizes for indexes

A large page size is very good for reducing the frequency of CI splits, which is costly in data sharing environment.

The downside: As with large pages for table spaces, the buffer hit ratio could degrade.

54



Non-padded indexes

Non-padded indexes were introduced in DB2 V8 Useful when an index contains one or more VARCHAR columns

Facilitates index only access

Saves DASD space

The CPU cost is significant. It grows as a function of the number of VARCHAR columns.

Index compression and non-padded indexes are complementary

55



DB2-managed “sliding scale” Extent sizes when the secondary space quantity is not specified

0

50

100

150

200

250

1 21 41 61

Extent numberE

xte

nt

siz

e (

cy

l)

DSSIZE=4G, PRIQTY=200 Cyl

0

20

40

60

80

100

120

1 21 41 61 81 101

Extent number

Ex

ten

t s

ize

(c

yl)

DSSIZE=4G, Default PRIQTY

Extent size = Extent number Extent size = max (extent number, 10% of PRIQTY)

56



Sliding scale when DSSIZE = 64GB

0

100

200

300

400

500

6001

32

61

92

12

2

15

3

18

3

21

4

24

5

Extent number

Ext

ent

size

(cy

l)

DSSIZE=64G, Default PRIQTY

More details can be found in the DB2V8 SQL Reference Guide

57



SMS Classes

DB2 V9 supports the specification of SMS constructs on the DB2 Stogroup definition

58



Long term page fixing

DB2 V8 moved all buffers into 64-bit storage, which increases the cost of page fix/free (unless you were using data space buffer pools)

To compensate, DB2 introduced the PGFIX(YES|NO) long term page fix option. It is a buffer pool option.

Long term page fixing can save 6% of the CPU for an I/O intensive OLTP DB2 workload.

Actual savings may be higher as the amount of real storage increases

Some throughput increase associated with DB2 queries and utilities

PGFIX can be altered dynamically, but the buffer pool must be flushed to alter it

All measurements shown in this presentation used PGFIX(YES)

59



The anatomy of a DB2 prefetch operation

60



DB2 Index Scan z9 EC, FICON Express 2, DS8000 Assuming that the index is cache resident

0

50

100

150

200

250

Th

rou

gh

pu

t (M

B/s

ec)

DB2 V8 DB2 V9

61



DB2 Dynamic Prefetch vs Sequential Prefetch

At Bind time, the DB2 optimizer may select Sequential Prefetch

At execution time, Data/Index Manager may choose Dynamic Prefetch

Sequential Prefetch schedule one SRB at a time (except at the start of a query), but Dynamic Prefetch schedules two concurrent SRBs

This can double the throughput if the data is cache resident, if the channels are not busy, and if the scan has low CPU time

Table space scans always use sequential prefetch.

In Version 8, for indexes the DB2 optimizer made the choice based on index statistics

In Version 9, the optimizer always relies on dynamic prefetch for everything other than table space scans.

62



DB2 Prefetch

DB2 employs prefetch for table scans, index scans, and LOB reads

Prefetch runs under an “asynchronous SRB” while the user’s thread evaluates the predicate.

While the predicate evaluation and prefetch engines are being done, the control unit may be “prestaging” data into its cache

Suspension due to prefetch delay is reported in DB2PE/Omegamon as “Other Read I/O”

“Other Read I/O” is characteristic of queries that have low CPU time:

LOBs and large rows

Multi-row fetch (e.g. DSNTEP4 Selects)

Simple SQL predicates

Copy and Recover utilities

63



Prefetch Quantity

When migrating from V8 to V9, you should expect fewer I/Os and higher I/O response times

Prefetch quantity for compressed index is based on uncompressed page size. This may result in more prefetch I/Os than an uncompressed index. For example, a scan of a compressed 16K index with a 3-to-1 compression ratio will do 33% more I/Os than its equivalent uncompressed index.

Prefetch type DB2 V8 DB2 V9 BPSIZE *

VPSEQT < 160MB

DB2 V9 BPSIZE *

VPSEQT >= 160MB

DB2 V9 BPSIZE *

VPSEQT >= 320MB

Dynamic 128K 128K 128K 128K

Sequential 128K 128K 256K 256K

Utility 256K 256K 256K 512K

64



Prestaging effects for a table scan

Suppose that the control unit prestage rate is 150 MB/sec and the channel transfer rate is 100 MB/sec

If the CPU predicate evaluation is slower than 150 MB/sec, there will be no “Other Read I/O” delays, and no I/O disconnect time, and the elapsed time will equal the CPU time of the query

As the channel speeds up, disconnect time increases

A larger prefetch quantity causes higher connect time and higher disconnect time

Disks with 15K RPMs will achieve less disconnect time than 10K RPMs and increase throughput if the data is not cache resident

Suppose the data is cache resident

No I/O disconnect time

If the CPU predicate evaluation is slower than 100 MB/sec, there will be no “Other Read I/O” delays, and the elapsed time will be equal to the CPU time of the query

65



DB2 Sequential Prefetch Throughputz9 EC, FICON Express 2, DS8000

0

20

40

60

80

100

120

140

160

Th

rou

gh

pu

t (M

B/s

ec)

From cache From 10K RPM 300GB disk

64x4K

8x32K

32x4K

4x32K

66



0

0.5

1

1.5

I/O

Resp

on

se

Tim

e (

ms)

FICON Express 2 FICON Express 4 FICON Express 4

DB2 Sequential Prefetch (4K pages)Using MIDAWs and the DS8000 with 4 gbit/sec FICONCache resident table


DB2 Version 8: 32x4K prefetch

DB2 Version 9: 64x4K prefetch

67



DB2 V9 vs V8 Table Scan using FICON Express 4Cache resident data

Configuration:

z/OS 1.7

4000 byte row size

System z9 EC

FICON® Express4

4 Gbit/sec link

DS8000 control unit

Using V9 with FICON Express 4, the channel throughput is 193 MB/sec, but the effective throughput is only 139 MB/sec.

0

25

50

75

100

125

150

Th

rou

gh

pu

t(M

B/s

ec)

V8 V9

68



Parallel queries

69



DB2 Prefetch Throughput Capacity16 partitions, 4K CI Size2002 measurementsESS 800, 8 FEx1 channels, 2 Gbit/sec link, z990 host

The model 800 bus is capable of 550 MB/sec with half-track blocks, but DB2 table spaces are more limited

0

50

100

150

200

250

300

350

Th

rou

gh

pu

t (M

B/s

ec)

Non-EF EF

Type of dataset

Performance is based on measurements in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the numbers stated here.

70



DS8000: DB2 Prefetch Throughput Capacity as a function of the number of FICON Express 2 Channels 64 degrees of parallelism

0

400

800

1200

1600

Th

rou

gh

pu

t (M

B/s

ec)

2 4 8 16

Number of FEx2 Channels

The DS8000 bus is capable of 1800 MB/sec with 16 FICON, but DB2 table spaces are more limited.

16 channels achieves 35% higher throughput than 8 channels, but this is probably not true with FICON Express 4

Each LCU is limited to 8 channels, but the LCUs may be divided into two channel sets



Configuration:

Pre-MIDAWs: z/OS 1.6

System z9 109


4000 byte row size

System z9 EC

FICON® Express2

2 Gbit/sec link

DS8000 control unit

71



DS8000: DB2 Prefetch Throughput Capacity 8 FICON Express 2 Channels, 64 degrees of parallelism

0

200

400

600

800

1000

1200

Th

rou

gh

pu

t (M

B/s

ec)

Pre-MIDAWs MIDAWs

EF 4K Non-EF 4K Non-EF 16K

MIDAWs eliminates the EF performance penalty and shrinks the small CI penalty

Performance is still sensitive to the number of bytes per track



Configuration:

MIDAW : z/OS 1.7

Pre-MIDAW: z/OS 1.6


4000 byte row size

System z9 EC

FICON® Express2

2 Gbit/sec link

DS8000 control unit

72



Review of Extended Format Data Sets

DB2 Striping Considerations

73



What is an Extended Format Data Set?

Both VSAM and QSAM/BSAM data sets may be EF

Supported only on SMS-managed volumes

DSNTYPE=Extended

Uses a fixed length records (even if RECFM=VB)

A 32-byte suffix is appended to every physical record

The suffix contains a length field to logically describe “short blocks”

The suffix is also used to increase reliability

EF datasets may only be accessed via Media Manager

Media Manager manages the suffix so that the suffix is completely transparent to the caller

74



What is Media Manager?

Why do we have it?

To limit dependencies on changes to the I/O architecture

Implements the latest and greatest I/O channel program architecture, such as track level CCWs and MIDAWs

What is it?

It is an I/O driver that knows how to map a relative byte address to a CCHHR, using a fixed block architecture to translate the address. It also supports striping.

What isn’t it?

It is not a buffer manager. The caller supplies the buffers.

Who can use it?

Any supervisor state program accessing a fixed block data set.

75



Why use Extended Format?

Benefit from the performance advantages of Media Manager

VSAM Extended Addressability (>4GB data sets)

Applies to linear datasets, but there are dependencies on the caller of Media Manager (e.g. DB2) to support EA

Striping

BSAM, QSAM and KSDS Compression

QSAM

Instead of default BUFNO=5. the default BUFNO is 2 times the number of tracks times the number of stripes

Double Buffering (to facilitate CPU and I/O overlap)

76



DB2 striping restrictions

With DB2 Version 9, all DB2 data sets can be striped provided that the CI size matches the page size. (Version 8 was the first version to support large CIs.)

Version 8 restriction: Archive logs could not be striped unless you have the Archive Log Accelerator

77



DB2 striping versus partitioning

CURRENT DEGREE = ‘ANY’ causes a query to use parallelism if the table space is partitioned.

Partition parallelism uses more buffers than does striping

Partitioning is always a better method of achieving parallelism than striping, unless you are storage constrained, because…..

Partitioning uses more storage

Partitioning is unaffected by I/O variance

Partitioning uses CPU parallelism

Striping be combined with partitioning to improve the performance of partition-level operations (e.g. Backup or reorg one partition)

Striping can also be used for non-partitioned indexes

78



DB2 Table Space Striping versus DB2 Partitioning

0

50

100

150

200

Th

rou

gh

pu

t(M

B/s

ec)

1 2

Number of stripes

Configuration:

z/OS 1.7

System z9 EC


4000 byte row size

FICON® Express2

2 Gbit/sec link

DS8000 control unit



0

50

100

150

200

Thro

ughp

ut

(MB

/sec

)1 2

Number of partitions

79



When is striping advantageous? Striping is a good strategy for taking high I/O connect time and

spreading it across multiple channels.

If some control units are busier than others, then spreading stripes across control units is good. If this helps increase the cache hit ratio (for random access), disconnect time will go down. If this helps reduce control unit bus contention (for sequential access) , connect time will go down.

Striping isn’t necessary to avoid IOSQ time, because PAV avoids IOS queue time.

Striping does not help if your channels are saturated.

As your channel technology improves, you should reduce the number of stripes or stop using striping.

80



Recommended number of stripes, examples:

Configuration

Connect time(ms) Recommended

number of stripes32x4K prefetch

64x4K prefetch

ESS 800FICON Express 1

2.7 5.0 4 to 8

DS8000FICON Express 2

0.9 1.5 2 to 4

DS8000FICON Express 4

0.7 1.1 1 to 2

Rule of thumb: Try to reduce connect time to 1.0ms. For selected data sets, you may try to reduce to 0.5ms if this won’t cause high channel utilizations.

81



Why can too many stripes be bad?

If pend time is high relative to connect time, then too much of your channel time is consumed by I/O overhead

High channel utilizations or CU utilizations cause I/O response time variance which hurts striping

An imbalance in the number of blocks per stripe causes I/O response time variance which limits parallelism.

Example: CI Size is 32K. Prefetch quantity is 4x32K. A 5th stripes is useless. If you use 3 stripes, two stripes will be idle while the system is reading the 4th CI. This limits parallelism.

If the CI size is large, choose the number of stripes for a DB2 data set to be a power of 2.

Adding unnecessary stripes for DB2 data sets causes extra I/Os, and adding unnecessary stripes for non-DB2 data sets wastes real storage.

82



Logging, preformatting, Reorg, etc.

83



Striping the DB2 log datasets

Two stripes increases the maximum throughput of DB2 inserts by 30% to 50%, but on faster devices most insert streams are CPU bound.

Use the same connect time rule of thumb for logs and table spaces. For logs, consider mass inserts and Fast Log Apply (e.g online Reorg of a very large table space).

Version 9 increases the effectiveness of striping during Fast Log Apply, because the number of input I/O buffers was increased from 15 to 120.

Version 9 permits the archive log datasets on Dasd to be striped too. This prevents log offloading from becoming a bottleneck when the active logs are striped. Version 9 also permits the archive logs to use BSAM compression.

84



DB2 Inserts: Log throughput Capacity

0

10

20

30

40

Th

rou

gh

pu

t (M

B/s

ec)

Non-EF 2 stripes

Type of Log Dataset

ESS 800, Pre-MIDAWs, FE1

Configuration:

z/OS 1.3

DB2 for z/OS V8

4000 byte row size

System z900

FICON® Express1

2 Gbit/sec link

ESS 800 control unit

Configuration:

z/OS 1.7

DB2 for z/OS V8

4000 byte row size

System z9-109

FICON® Express2

2 Gbit/sec link

DS8000 control unit

0

20

40

60

80

100

120

Thro

ughp

ut

(MB

/Sec

)Non-EF 2 stripes

Type of Log Dataset

DS8000, MIDAWs, FE2

85



Preformatting (versus formatting)

Preformatting accomplishes two things:

Allow subsequent update writes to be done in parallel without regard to track boundaries.

Update HI-U-RBA (High Used RBA) in the catalog

OLTP uses preformatting, because datasets don’t get closed

DB2 Utilities can use normal format writes, because the data doesn’t need to be secure until the dataset is closed. Close will update the catalog.

Preformatting is asynchronous, except after allocating a new extent

The “preformat quantity” is the frequency that we update the catalog

If the quantity is too little, we update the catalog too often

If the quantity is too much, then synchronous preformatting will cause user’s to wait.

86



Mass Inserts

Mass inserts are affected by preformatting performance, especially with LOBs. LOBs don’t require logging and are not CPU bound.

DB2 Version 9 increased the preformat quantity from 2 cylinders to 16

This can yield >20% improvement in elapsed time for LOBs.

If the datasets are striped, the preformat quantity is in terms of Control Areas

E.g. The control area with 2 stripes is 8 tracks per stripes

The benefit of updating the catalog less often is higher for striped datasets and for faster devices

Cross Loader utility (used to copy one table space to another) also benefits

87



DB2 Load Utilityz9 EC, FICON Express 2, DS8000Cache resident input file

0

20

40

60

80

100

120

140

Th

rou

gh

pu

t (M

B/s

ec)

16x16K 32x16K

Table Space: #pages/IO x page size

I/O reduction produces up to 10% increase in throughput. Higher benefits expected with striping and with FICON Express 4.

88



Online LOB Reorg

Version 8 Reorg LOB

Online Reorg not supported

Used only to defragment each LOB, but could not reclaim unused space

Version 9 Reorg LOB

Online Reorg copies LOBs to a shadow dataset, just like Reorg for tables and indexes

Unused space is reclaimed

89



Online Reorg

Version 8 PTF: PK25742

At one shop this PTF reduced Online Reorg time from 2 hours to 45 minutes due to a significant reduction in the Sortbuild phase

90



Fast Log Apply

Fast Log Apply is the last phase of both Online Reorg and Recover

FLA consists of log reads and database updates.

DB2 Version 9 improves log read throughput, especially with the faster channels.

When applying a small percentage of log records, FLA is gated by log reads. When applying a large percentage of log records, FLA is gated by database I/Os. Ergo, V9 improves the case where the percentage is small.

DB2 Version 9 also permits the archive logs on dasd to be striped and compressed, yielding yet further FLA performance improvements as well as storage space savings.

91



Backup, Restore and Recover

92



Backup and Recovery

Version 8 introduced Backup/Restore utilities Uses SMS Copypools, new to z/OS 1.5, and requires HSM

BACKUP creates a “Point In Time” backup of databases and logs

• Uses volume-level Flashcopy

RESTORE restores all volumes using volume-level Flashcopy

Version 9 Recover utility will use dataset level Flashcopy using the volume-level backups created by BACKUP utility Requires z/OS 1.8

93



Online Check

DB2 Version 8

Check Index uses dataset level Flashcopy

DB2 Version 9

Check Data and LOB use dataset level Flashcopy

94



DB2 Utilities using eight FICON Express2 channelChannel utilizations ~10%

020406080

100120

Thro

ughp

ut

(MB

/sec

)

Non-EF EF

Type of data set

DB2 Copy Utility

Pre-MIDAWs MIDAWs

Performance is based on measurements in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the numbers stated here.

020406080

100120

Th

rou

gh

pu

t (M

B/s

ec)

Non-EF EF

Type of data set

DB2 Recover Utility *

Pre-MIDAWs MIDAWs

Configuration:

MIDAW : z/OS 1.7

Pre-MIDAW: z/OS 1.6


4000 byte row size

4K CI Size

System z9 EC

FICON® Express2

2 Gbit/sec link

DS8000 control unit

* Restore phase only

Recover impacted by Media Manager limitation which has since been lifted by apar

95



Conclusion

96



Creating a balanced configuration

A balanced I/O configuration is one where all components perform about equally.

FICON Express 1 and the ESS 800 are a good match

FICON Express 1 and the DS8000 are a good match

FICON Express 2 and the ESS 800 is a poor match

However, FICON Express 2 channels can be used to support several ESS 800s

FICON Express 2 or 4 and the DS8000 are an excellent match

z9-109 FEx2 FICON Link DS8000HA

97



Using hardware to address I/O response time components

High pend time: More channels or faster channels, MIDAWs

High connect time:

Faster FICON links/directors, more/faster channels and CU host adapters, more CU bus capacity

MIDAWs

Striping

High disconnect time

More real storage, more control unit cache, more disks, more control units

Faster disk RPMs (tradeoff between performance and capacity)

High IOSQ time is always a secondary symptom. It is better to attack the primary symptom first, but to reduce IOSQ time:

More UCBs, new PAV

98



DB2 Logging Issues (unique to synchronous writes)

If you are using remote copy, assume you have a network problem, unless you can see that you have high log connect times

If you aren’t using remote copy, and you have any disconnect time on your logs, then you need more NVS storage, or faster RPMs

If you have high connect time:

Upgrade your hardware as you would for table spaces

Use striping or add more stripes

Documents

DB2 I/O Performance: NCCMG ® August 1, 2006 Speaker: Jeff Berger Northern California CMG Enhanced I/O Performance On the System z9 with DB2, FICON and