SAP® TRBK Banking Benchmark using IBM DB2 pureScale® for ... · The project is referred to as the SAP® TRBK Benchmark using DB2 pureScale® on x86/Linux. 1.2 Document content The

© Copyright IBM Corporation All Rights Reserved 2011

Information Management Performance and

Benchmarks Toronto Lab Information Management Development Boeblingen

Products & Solutions Support Center Montpellier

SAP® TRBK Banking Benchmark

using IBM DB2 pureScale® for Linux/Intel

Benchmark report

Document information

Name DB2 PS SAP TRBK Benchmark Report

Version V1.3.2

Date 14/12/11

Authors Ref to chap 1.3

IBM DB2 Toronto Lab – IM Devlt lab BOE – PSSC Montpellier


DB2 PS SAP® TRBK

2/168

Table of content

1 Introduction ........................................................................................................................ 5

1.1 Document purpose ...................................................................................................... 5

1.2 Document content ....................................................................................................... 5

1.3 Document authors ....................................................................................................... 6

1.4 Reminders ................................................................................................................... 7

1.5 Trademarks .................................................................................................................. 7

2 Executive Summary ............................................................................................................. 9

3 Project Overview & Requirements ..................................................................................... 11

3.1 Objectives .................................................................................................................. 11

3.2 Tests scenario description .......................................................................................... 11

3.2.1 Day processing scenario ..................................................................................... 11

3.2.2 Night processing scenario ................................................................................... 11

3.2.3 Scalability of the architecture ............................................................................. 11

3.2.4 High Availability ................................................................................................. 11

3.2.5 Disaster Recovery ............................................................................................... 12

4 Architecture – system configuration .................................................................................. 13

4.1 Overall layout ............................................................................................................ 13

4.2 Servers overview........................................................................................................ 14

4.2.1 Application servers - HS22 .................................................................................. 14

4.2.2 Database servers - x3850 x5 ............................................................................... 15

4.3 Storage overview ....................................................................................................... 18

4.3.1 Assumptions for DB2 / GPFS / Storage design ..................................................... 21

4.3.2 LUN and file system for DB2 with 768 disks FC .................................................. 23

4.3.3 Detailed of the LUNs configuration ..................................................................... 24

4.4 Network fabric overview ............................................................................................ 25

4.4.1 Infiniband network ............................................................................................. 25

4.4.2 Fiber Channel network ....................................................................................... 26

4.4.3 Ethernet network ............................................................................................... 26

4.5 Operating Systems ..................................................................................................... 28

4.5.1 Suse Enterprise Linux 10 SP3 .............................................................................. 28

4.5.2 I/O device drivers ............................................................................................... 28

4.5.3 Settings tuned .................................................................................................... 30

4.6 DB2 pureScale ............................................................................................................ 30

4.6.1 DB2 concepts ..................................................................................................... 30

4.6.2 Integrated Technologies ..................................................................................... 31

4.7 SAP ............................................................................................................................ 33

4.7.1 TRBK overview ................................................................................................... 33

4.7.2 Benchmark Configuration ................................................................................... 34

4.7.3 System Installation ............................................................................................. 35

5 Test Results : Performance, Scalability & Availability .......................................................... 43

5.1 Overall test configuration .......................................................................................... 43

5.3 TRBK Day Run ............................................................................................................ 43

5.3.1 Day run settings + results achieved .................................................................... 43

5.3.2 Scalability ........................................................................................................... 44



DB2 PS SAP® TRBK

3/168

5.3.3 Specific Day Run Tuning ..................................................................................... 44

5.3.4 Conclusion........................................................................................................ 46

5.4 TRBK Night Run .......................................................................................................... 47

5.4.1 Night run settings + results achieved .................................................................. 47

5.5 Experiments with packagesize and number of work processes ................................... 49

5.5.1 Resource usage ............................................................................................... 49

5.5.2 Conclusion........................................................................................................ 50

5.6 High Availability Test .................................................................................................. 51

5.6.1 Results ............................................................................................................... 51

5.6.2 GPFS IO fencing .................................................................................................. 51

5.6.3 Measuring the outage time ................................................................................ 52

5.6.4 Methods to generate the failure ......................................................................... 52

5.6.5 Behaviors in different failure scenarios ............................................................... 52

5.6.6 Reduce the client reroute time ........................................................................... 53

5.6.7 Reduce the number of SAP® errors for read only transactions ........................... 53

6 Disaster Recovery tests ...................................................................................................... 54

6.1 Layout change............................................................................................................ 54

6.2 Results ....................................................................................................................... 54

6.3 Site Failure – Storage replication ................................................................................ 55

6.3.1 Site Failure Implementation ............................................................................... 56

6.3.2 Disaster recovery mechanism ............................................................................. 56

6.3.3 High level process of recovery. ........................................................................... 57

6.3.4 GPFS configuration ............................................................................................. 60

6.3.5 Recovery Procedure on DR Site .......................................................................... 63

6.3.6 Going back to the production configuration ....................................................... 74

6.4 Storage interconnect failure and catchup ................................................................... 83

6.4.1 Implementation ................................................................................................. 83

6.4.2 Storage Interconnect test mechanism ................................................................ 86

6.4.3 High level process .............................................................................................. 86

6.4.4 Storage Interconnect test results ........................................................................ 89

6.5 Queue Replication to logical copy .............................................................................. 94

6.5.1 Methodology ..................................................................................................... 95

6.5.2 Results ............................................................................................................... 96

6.6 Site failure with Storage Replication and QRep .......................................................... 98

6.6.1 QRep & Site Failure Implementation .................................................................. 98

6.6.2 Results ............................................................................................................... 98

6.7 Backup during on-line operations............................................................................... 98

6.7.1 On line backup Implementation ......................................................................... 98

6.7.2 On-line Backup mechanism .............................................................................. 100

6.7.3 Online Backup Procedure ................................................................................. 104

6.7.4 Create Online backup using Remote FlashCopy ................................................ 107

6.7.5 Refresh Online backup clone using Remote FlashCopy ..................................... 115

6.7.6 Accessing FlashCopy database Clone (from DR site) .......................................... 121

6.7.7 Going back to the production configuration ..................................................... 126

6.7.8 Conclusion ....................................................................................................... 127

7 SAP® TRBK Certification results ....................................................................................... 128



DB2 PS SAP® TRBK

4/168

7.1 Architecture – system configuration ........................................................................ 128

7.1.1 Overall layout ................................................................................................... 128

7.1.2 Application servers - HS22 ................................................................................ 129

7.1.3 Database servers - x3690 x5 ............................................................................. 129

7.1.4 Storage overview ............................................................................................. 132

7.1.5 Network fabric overview .................................................................................. 138

7.1.6 Operating Systems ........................................................................................... 140

7.2 Certification results .................................................................................................. 141

7.3 Tips & recommendations ........................................................................................ 142

7.3.1 Database Configuration .................................................................................... 142

7.3.2 SAP® Application Server .................................................................................. 143

7.3.3 Operating System ............................................................................................. 143

7.3.4 Network Topology ............................................................................................ 144

8 Best Practices for DB2 pureScale® and SAP® TRBK ......................................................... 145

8.1 Tablespace Definitions ............................................................................................. 145

8.1.1 Extent Size........................................................................................................ 145

8.1.2 Increase Size .................................................................................................... 145

8.2 Table Compression .................................................................................................. 146

8.3 Data and Index Percent Free .................................................................................... 146

8.4 Data Partitioning ...................................................................................................... 146

9 Conclusions ..................................................................................................................... 147

10 Appendix ..................................................................................................................... 148

10.1 db2dsdriver.cfg ........................................................................................................ 148

10.2 Malloc Bomb............................................................................................................ 150

10.3 DB2 Registry Settings ............................................................................................... 151

10.4 DB Manager Configuration Settings ......................................................................... 151

10.5 DB2 Configuration Settings ...................................................................................... 153

10.6 enable_dr_nsd.sh .................................................................................................... 156

10.7 disable_dr_nsd.sh .................................................................................................... 157

10.8 disable_dr_nsd_and_enable_fc_nsd.sh .................................................................... 160

10.9 suspend_or_resume_gpfs_fs.sh ............................................................................... 162

10.10 remote_flash_copy.sh .......................................................................................... 162

10.11 disable_repl_nsd.sh ............................................................................................. 165



DB2 PS SAP® TRBK

5/168

1 Introduction

1.1 Document purpose

This document describes IBM testing of the SAP® TRBK solution using DB2 pureScale® on

x86/Linux, based on a predefined set of tests scenario.

It was IBM managed under the leadership of the IBM Information Management Performance

and Benchmark team based in Toronto, with the support of the Information Management

Development teams based in Boëblingen, and in San Jose, and carried out in the IBM

Montpellier benchmarking centre (PSSC).

The project has been performed between July and early December 2010 for the performance

and HA tests, then between February and early May 2011 for the DR tests and finally from June

to early August 2011 for the certification test campaign.

The project is referred to as the SAP® TRBK Benchmark using DB2 pureScale® on x86/Linux.

1.2 Document content

The content of this document corresponds to the results achieved during the benchmark

activities. It covers both the SAP® results as well as the system behaviour for the most

representative runs achieved.

This paper is split in various chapters.

Chapter 2 is dedicated to the Executive Summary.

Chapter 3 is related to the overall project description and the various test scenario description

with the associated targets.

Chapter 4 covers the architecture used for this benchmark. It includes details about the level

and the configuration of the software components used (such as DB2 pureScale® and SAP® ), as

well as a description of the various hardware components configuration (servers, storage and

network).

Chapter 5 focuses on the performance results achieved, for the various batch processes

exercised as well as the results of the availability tests executed.

Chapter 6 is a summary of the outcome achieved with the DR test cases

Chapter 7 covers the certification test campaign.

Chapter 8 provides some further remarks and best practices about the DB2 pureScale®

implementation



DB2 PS SAP® TRBK

6/168

Chapter 9 draws our overall conclusion based on the test cases executed.

The Appendix chapter contains the various profiles and key parameters used to achieved the

previous described results.

1.3 Document authors

During almost 12 months of intense activity, many different organizations and people have been

involved across IBM to perform this project.

Here are listed the authors & contributors of this document:

Berni Schiefer DE, Information Management Performance and Benchmarks

Russ Stocker Information Management Performance Benchmarks and Solutions Dev. Kostas Rakopoulos Information Management DB2 Performance Quality Assurance

Wallace Chen Information Management Software Developer

Pandu Mutyala Information Management DB2 System Verification

Aruna De Silva IM Solutions - QA Specialist Software Test Specialist

Karl Fleckenstein STSM, Integration IBM Data Server solutions into SAP®

Thomas Matthae IM Senior Development Engineer, SAP® Certified Consultant

Serge Bourbonnais IM STSM, Database Replication, Software Architect

Ping Li IM, DB2 Replication Software Performance Analyst

Anand Krishniyer IM Replication Center of Competency, Software Developer: Java

Aslam Nomani DB2 LUW Quality Assurance Test and Tooling

Brent Beardsley DE, Enterprise Storage Subsystems, Master Inventor, HW Designer

Christian Hugues Performance Benchmark System x

Gael Marlin Benchmark System x

Alain Fournil Performance Benchmark DB2 UDB

Philippe Jackimczyk Certified Senior IT specialist - Storage Networking

Alexis Gausach Storage specialist

Hervé Guerin SAP® benchmark specialist – SAP® Certified consultant

Francoise Alabiso Certified Executive IT Architect – PSSC Technical Director

Hervé Sabrié Certified Senior Project Manager – PMP

Beside the people listed here, many other specialists from different teams have been involved at

some point in time to support this project. Our thanks go to all of them also.

We would like also to thank our sponsors for this project as well as the various managers who

helped to make this happen.

Vijay Lund SWG, Information management – Vice President Cross IBM Offerings

Ian Hurst S&D, General Manager, Global Financial Services Sector

Erich Baier STG, HW Development, Vice President, System x and RSS Development



DB2 PS SAP® TRBK

7/168

David Gelardi STG, Vice President, STG Competitive Labs, STG Technical Sales Centers

1.4 Reminders

Neither this document nor any part of it may be copied or reproduced in any form or by any

means or translated into another language without the prior consent of the IBM Corporation.

IBM makes no warranties or a representation with respect to the content hereof and specifically

disclaims any implied warranties of merchantability or fitness of any particular purpose.

IBM assumes no responsibility for any errors that may appear in this document. The information

contained in this document is subject to change without any notice. IBM reserves the right to

make any such changes without obligation to notify any person of such revision or changes. IBM

makes no commitment to keep the information contained herein up to date.

The results shown are based on specific workloads run in a controlled environment. The actual

throughput that any user will experience will vary considerably from these results. Therefore, no

assurance can be given that an individual user will achieve throughput equivalent to the

performance stated here.

Actual environmental costs and performance characteristics will vary depending on individual

customer configurations and conditions.

Consult your local IBM business contact for information on the product or services available in

your area.

Information about non-IBM products is obtained from the manufacturers of those products or

their published announcements. IBM has not tested those products and cannot confirm the

performance, compatibility, or any other claims related to non-IBM products. Questions on the

capabilities of non-IBM products should be addressed to the suppliers of those products.

1.5 Trademarks

IBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International

Business Machines Corporation in the United States, other countries, or both. If these and other

IBM trademarked terms are marked on their first occurrence in this information with a

trademark symbol (® or ™), these symbols indicate US registered or common law trademarks

owned by IBM at the time this information was published. Such trademarks may also be

registered or common law trademarks in other countries. A current list of IBM trademarks is

available on the web at “Copyright and trademark information” at

www.ibm.com/legal/us/en/copytrade.shtml.

SAP® , SAP® R/3, SAP® ERP, mySAP® , mySAP® .com, xApps, xApp, ABAP, BAPI, SAP®

NetWeaver and all

SAP® product and service names mentioned herein are trademarks or registered trademarks of

SAP® AG in Germany and in several other countries all over the world.



DB2 PS SAP® TRBK

8/168

Adobe is a registered trademark of Adobe Systems Incorporated in the United States, and/or

other countries.

InfiniBand is a registered trademark of the InfiniBand Trade Association.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Windows and Microsoft are registered trademarks of Microsoft Corporation in the United

States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle

and/or its affiliates.

Unicode is a trademark of Unicode, Inc.

Other product and service names might be trademarks of IBM or other companies.



DB2 PS SAP® TRBK

9/168

2 Executive Summary

The objective of this proof-of-concept was to demonstrate to a large customer the value of

IBM’s proposed architecture based on DB2 pureScale® running SAP® TRBK on Intel servers and

IBM DS8K storage systems.

Designed for organizations that run online transaction processing (OLTP) applications on

distributed systems, IBM® DB2® pureScale™ offers clustering technology that helps deliver high

availability and exceptional scalability transparent to applications. DB2 pureScale® is

conceptually based on clustering technology from IBM DB2 for z/OS® and is available as an

option on IBM DB2 9.7 Enterprise Server Edition.

The Transaction Banking (TRBK) Benchmark used in this PoC, is divided into two business

scenarios. The first scenario, day processing, reflects usage by users and integrated systems that

generally occurs during the daytime. The second scenario, night processing, reflects account

balancing that generally occurs overnight in batch mode.

The main goals of this project were to achieve some specific performance figures that were

viewed by the customer as representative of their future production workload, while

demonstrating the high-availability of the solution. In a second stage, we covered some Disaster

Recovery scenarios to further demonstrate the strength and value of the solution for a

production system.

During several months of proof-of-concept, we went through the following phases:

- Environment setup and database growth up to 90 Mio customer accounts

- Performance tests based on on-line transaction processing (‘Day runs’) and batch

processing (‘Night runs’).

- Tests to ensure the scalability of the solution

- High Availability tests

- Reconfiguration of the environment to get ready for DR test

- Disaster Recovery Test campaign.

- Setup and configuration of new platform

- Certification Test campaign.

The challenge was to demonstrate that beside the HA and DR conditions, we could also achieve

very high throughput in terms of OLTP or batch process, up to 40Mio of postings processed per

hour for the “Day run” and 22.5Mio of accounts processed per hour for the “Night run’.

In terms of day processing, we achieved 56.9Mio postings/hour, 42% higher than the initial

target. Using the same workload, we also demonstrated a linear scalability from 1 to 4

members.

For the night processing, we managed to balance 90Mio accounts within 158 minutes which

correspond to a throughput of 38.4 Mio accounts/hour. For that scenario, we also clearly over-

achieve the initial target by 65%.



DB2 PS SAP® TRBK

10/168

In terms of high availability of the solution, whatever the type of failure encountered (member

failure, coupling facility failure, malloc bomb failure, Infiniband adapter loss, Infiniband switch

failure, … ), we achieved an outage time less than the 60 second target time, and usually much

lower.

Finally, through the various test cases executed during the DR Test campaign, we stressed the

DB2 pureScale® solution, and demonstrated that in case of Site Failure or Storage interconnect

failure, we were able to recover without much impact on the production activity.

We also evaluated successfully the impact of a Backup during online activity. And we tested the

Q-Replication as an extra option to ensure double security at the table level.

During this project, we overachieved our initial targets, and demonstrated that DB2 pureScale®

is capable of delivering excellent performance when running a very challenging workload in an

environment that effectively guarantees 100% availability, with the further guarantee of

supporting the required production recovery mechanism in the event of a complete site failure.

We ended this effort with the official Certification by SAP® of our metrics. We achieved 56.5M

postings to bank accounts processed per hour and demonstrated the ability to balance 90Mio

accounts during 4 hours as a proof of stability and requirement for the certification.



DB2 PS SAP® TRBK

11/168

3 Project Overview & Requirements

3.1 Objectives

4 main objectives have been assigned to this project from a technical stand point.

- Performance

- Scalability

- High Availability

- Disaster Recovery

Based on SAP® TRBK benchmark suite, the workload was based on generic data with 90 Mio

accounts loaded in the database as a starting point.

3.2 Tests scenario description

3.2.1 Day processing scenario

During the day processing scenario, the benchmark program simulates typical daily volumes of a

retail bank, including the quantity of payment transaction operations received from external

payment transaction systems, single real-time payment transactions received from ATM

machines, real-time account statements, as well as the quantity of account information read

from the system. The scenario was constructed to mirror a distribution of typical banking tasks.

Since the usual integration into the bank IT infrastructure takes place via real-time interfaces

(BAPIs) rather than SAP® GUI, the simulation consists of a sequence of BAPI calls.

The goal of that test was to achieve a throughput of 40Mio postings per hour.

3.2.2 Night processing scenario

In the night processing scenario, account balancing is performed at regular intervals to settle an

account. The benchmark simulates account balancing by calculating interest and charges for

each account across 20 value date days.

The goal of that test was to balance the 90 Mio accounts within 4 hours, which corresponds to a

throughput of 22.5Mio accounts per hour.

3.2.3 Scalability of the architecture

The goal of that test is to ensure that we can scale from 1 member to 4 members

3.2.4 High Availability

Many tests have been identified to ensure the High Availability of the global solution. They are

listed in the table below :



DB2 PS SAP® TRBK

12/168

Test # Scenarios Customer requirement (outage time)

1 Member shutdown failure (dual member failure part 1) <= 60 seconds

2 Malloc bomb member (dual member failure part 2) <= 60 seconds

3 Primary CF shutdown failure <= 60 seconds

4 Secondary CF shutdown failure <= 60 seconds

5 Malloc bomb primary CF <= 60 seconds

6 Member IB adapter failure <= 60 seconds

7 Soft kill member <= 30 seconds

8 IB switch failure <= 60 seconds

3.2.5 Disaster Recovery

The Disaster recovery test campaign covered the following test cases:

Test# DR test case Description

1 Site Failure – storage replication TRBK Night workload can be completed on site2 in case of a failure on site1

(DWDM switched off) Based on Night Run.

2 Storage interconnect – catchup test Once the link for the storage interconnect is lost (10min delay)

monitor the performance and the time to resync. Based on Day run.

3 Database Q-Replication with 15 min replication delay

Maintain a shadow db with Qrep:

• Measure qrep impact on primary

• Demonstrate ability to keep up with primary

• Maintain shadow db 15 mins behind primary

Based on Night Run 4 Qrep + Storage replication combined Site failure test with impact of QRep

Based on Night Run

5 Backup duing on-line operations Evaluate impact on the throughput of a flashcopy on primary site or secondary

site. Based on Day Run



DB2 PS SAP® TRBK

13/168

4 Architecture – system configuration

4.1 Overall layout

This is the target Architecture used for the Performance tests, the scalability tests and the high

availability tests. Please refer to the Disaster Recovery chapter for details about the architecture

used in the DR tests.

Hardware Involved

Machine type Quantity Role

x3850 x5 (Type 7145) 6 Database servers (DB2PS)

HS22 (Type 7870)

BladeCenter H

24

2

Application servers (SAP® )

DS8700 2 Database storage

DS5300 1 Benchmark results storage

Mellanox IB switches IS5030 2 Interconnect Infiniband network (provide HA)



DB2 PS SAP® TRBK

14/168

4.2 Servers overview

4.2.1 Application servers - HS22

4.2.1.1 Technical characteristics

Element Quantity Description

Processor 2 Intel Xeon 4 Cores X5570 95W 2.93GHz/1333MHz/8MB L2

Memory 12

Express 4GB Dual Rank PC3-10600 CL9 ECC DDR3 VLP RDIMM

1333MHz

Internal disks 2 Express IBM 146 GB 2.5in SFF Slim-HS 15K 6Gbps SAS HDD

Additional cards 1 Qlogic 1 Gb Ethernet dual port CIO-v

Note: the internal disks have been configured as a RAID1 array.

Remark: Among all the HS22, one has been chosen to host the TRBK driver. This blade is also

responsible of collecting all the monitoring data (SAP® & system related) from all the machines.

Thus, this blade is attached to a DS5300 via a daughter card CFF-h Combo LAN +SAN Qlogic 8Gb.

4.2.1.2 Firmware levels

Module Version Release date

FW/BIOS P9E146A 03/01/2010 (1.07)

Diagnostics DSYT60K 02/25/2010 (3.01)

Blade Sys Mgmt Processor YUOO57H (1.10)

4.2.1.3 CPU information

Architecture: x86_64 Model: 26

Logical CPU(s): 16 Stepping: 5

Thread(s) per core: 2 CPU MHz: 2926.000

Core(s) per socket: 4 Virtualization: VT-x

CPU socket(s): 2 L1d cache: 32K

NUMA node(s): 2 L1i cache: 32K

Vendor ID: GenuineIntel L2 cache: 256K

CPU family: 6 L3 cache: 8192K

4.2.1.4 RAM configuration

HS22 are equipped with 12 DIMMs of 4GB DDR-3, resulting 48 GB total. All the slots are

populated, and the 2 nodes have an equivalent amount of memory. This is part of the Best

Practices recommended by IBM. Here is a dump of the Linux numactl command, which shows

the NUMA topology of the server.

available: 2 nodes (0-1)

node 0 size: 24196 MB


node distances:

node 0 1

0: 10 21

1: 21 10



DB2 PS SAP® TRBK

15/168

4.2.2 Database servers - x3850 x5



Processor 4 Intel Xeon 8 cores X7560 130W 2.26 GHz/1066MHz/24MB L3

Memory 64

4GB (1x4GB, Quad Rankx8) PC3-8500 CL7 ECC DDR3 1066MHz

LP RDIMM

Internal disks 2 IBM 146 GB 10K SAS 2.5" SFF Slim-HS HDD

Additional cards

1 Emulex 10Gbps Ethernet Dual ports PCI-express

2 (1) Qlogic 8 Gb FC Dual ports PCI-express

1 (2) Mellanox 40 Gbps QDR Infiniband Dual ports PCI-Express




IMM YUOO73K 06/16/2010

FPGA G0UD29C 06/16/2010

UEFI G0E122DUS 06/29/2010

DSA DSYT72C 06/16/2010











All x3850 x5 are equipped with 8 Memory cards, each containing 8 DIMMs of 4GB DDR-3.

Overall, the amount of memory is 256 GB of DDR3 RAM. As all the slots are used, the memory is

equally split between the 4 NUMA nodes. The use of 64 DIMM (8 DIMMS per memory cards)

ensures that the memory throughput is optimal, as recommended in IBM Best practices.

Here is a dump of the Linux numactl command, which shows the NUMA topology of the server.






node distances:

node 0 1 2 3

0: 10 11 11 11

1: 11 10 11 11

2: 11 11 10 11

3: 11 11 11 10



DB2 PS SAP® TRBK

16/168

4.2.2.5 uEFI Configuration

Here is an uEFI dump extract using the ASU (Advanced Setup Utility – tool provided by IBM).

The configuration has been done using the recommendations of Toronto & Intel Lab.

uEFI.OperatingMode=Performance Mode

uEFI.QuietBoot=Enable

uEFI.TurboModeEnable=Enable

uEFI.TurboBoost=Traditional

uEFI.ProcessorEistEnable=Disable

uEFI.ProcessorCcxEnable=Disable

uEFI.ProcessorC1eEnable=Disable

uEFI.HyperThreading=Enable

uEFI.EnableCoresInSbsp=All

uEFI.ExecuteDisableBit=Enable

uEFI.ProcessorVmxEnable=Enable

uEFI.ProcessorDataPrefetch=Enable

uEFI.DDRspeed=Max Performance

uEFI.CkeLowPolicy=Disable

uEFI.MirrorMigration=NonMirrored

uEFI.MapperPolicy=Closed

uEFI.SchedulerPolicy=Adaptive

uEFI.PatrolScrub=Disable

uEFI.PagePolicy=Closed

uEFI.MemoryScrambling=Disable

uEFI.Socket0Branch0SpareEn=Disable

uEFI.IdeMode=Compatibility mode

uEFI.ThunkSupport=Enable

uEFI.TpmEnable=Disable

uEFI.TpmStateUserSelection=Deactivate

uEFI.QPISpeed=Max Performance

4.2.2.6 I/O card layout for x3850 x5

Database servers require a suitable connection to the Storage Area Network (SAN) to access the

disks where the tables are stored. Moreover, DB2 pureScale® allows the different database

servers to communicate directly using RDMA, thanks to the use of Infiniband stack and uDAPL

upper protocol.

However, all the servers are not equivalent: among 6 servers, 4 are “members” (i.e. regular

database servers) and 2 are “coupling facilities” (used to manage shared data structures

between members – there are two CF’s for redundancy). Thus, these 2 kinds of servers have

different needs in terms of communication: that’s why 2 layouts have been designed for the PCI-

express cards.

Note: In both cases, the card used for the 10 Gb Ethernet is the Emulex card shipped with the

server, which is located in the slot 7.

Naming convention:

This outline illustrates the internal organization of the x3850 x5 components. The PCI slots are

represented on the right. The speed of these buses is also specified (x4 to x16).



DB2 PS SAP® TRBK

17/168

For regular members:

Card type Quantity Slot number

Infiniband (HCA) 1 6

Fiber Channel (HBA) 2 3 – 4

10 Gb Ethernet 1 7 (default)

Explanation:

- Objective: priority is given to the IB bandwidth

- Assumption: 10 Gb Eth is not resource consuming

- Load balancing on the 2 I/O Hubs:

I/O Hub 1: IB and 10 Gb Eth

I/O Hub 2: 2x FC

- Advantage : IB can benefit from « full » bandwidth of an I/O Hub

- Drawback : 2 sockets must do a additional « hop » to access FC cards

For coupling facilities:


Infiniband (HCA) 4 3 – 4 – 5 - 6

Fiber Channel (HBA) 1 2

10 Gb Ethernet 1 7 (default)



DB2 PS SAP® TRBK

18/168

Explanation:

- Hypothesis 1 : CF might be able to use 4 active IB cards

- Hypothesis 2 : CF will not have intensive SAN I/O

- All IB cards have homogeneous configuration (PCI-e 8x slots)

- Two IB on each I/O Hub with another type of interface

- Load balancing on the 2 I/O Hubs:

I/O Hub 1: IB0, IB2 and FC

I/O Hub 2: IB1, IB3 and 10 Gb Eth

- Advantage : maximize bandwidth for both IB cards

Note: the slot 1 (16x) has not been used because the Mellanox card is not currently supporting

16x speed. Thus, to have a homogeneous configuration between all the cards, all the x8 slots

have been used for the IB cards. The FC card (Qlogic) has been placed in slot 2, as this slot is

compatible with any 8x card (IBM documentation) and that the I/O on the CF is very low.

4.3 Storage overview

The storage subsystem is made of 2 DS8700 with for each DS8700 the following characteristics:

• 2 frames

• 256 GB Processor Memory (4-Way)

• 384 disks of 450 GB 15K FC

• 16 4Gb SW FCP ports connections to the SAN.

Each server (member or coupling facility) have access to the two DS8700 through a SAN (Storage

Area Network)

Some description of the two DS8700 is detailed below:

dscli showsi IBM.2107-75TX861 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_Pure_1.profile

Date/Time: 08 December 2010 16:25:50 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-

75TX861

ID IBM.2107-75TX861

Storage Unit IBM.2107-75TX860

Model 941

WWNN 500507630AFFC2BC

Signature 4d2b-1929-cad1-dee3

State Online

ESSNet Enabled

Volume Group V0

os400Serial 2BC

NVS Memory 8.0 GB

Cache Memory 235.9 GB

Processor Memory 253.4 GB

MTS IBM.2421-75TX860

numegsupported 0

ETAutoMode on



DB2 PS SAP® TRBK

19/168

ETMonitor all

dscli showsi IBM.2107-75TX901 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_pure_2.profile


75TX901

ID IBM.2107-75TX901


Model 941

WWNN 500507630AFFC2C1

Signature 8c7d-cd9d-1e17-ae88

State Online

ESSNet Enabled

Volume Group V0

os400Serial 2C1

NVS Memory 8.0 GB




numegsupported 0

ETAutoMode on

ETMonitor all



DB2 PS SAP® TRBK

20/168

SAN

x16

x16

x2 x4 x4 x4 x4x2

X3850CF0

X3850CF1

X3850M0

X3850M1

X3850M2

X3850M3

2 X 384 FC

PRA110.15.8.5375TX860WWN=2bc

PRA210.15.8.5475TX900WWN=2c1

db2PS-member010.3.67. 500x2100001b3283b04b0x2101001b32a3b04b0x21000024ff28538e0x21000024ff28538f

db2PS-member110.3.67. 510x2100001b3283d54c0x2101001b32a3d54c0x21000024ff2087100x21000024ff208711

db2PS-member210.3.67. 520x21000024ff2087180x21000024ff2087190x21000024ff2087ae0x21000024ff2087af

db2PS-member310.3.67. 530x21000024ff2087480x21000024ff2087490x21000024ff20872a0x21000024ff20872b

db2PS-cf010.3.67. 540x21000024ff2086f80x21000024ff2086f9

db2PS-cf110.3.67. 550x21000024ff2087400x21000024ff208741

DS8700 DS8700

Overview of the storage configuration

DA 2

DA 0

DA 6

DA 4

DA 7

DA 5

6PS A0

6PS A2

6PS A1

6PS A3

6PS A8

6PS A10

6PS A9

6PS A11

7P A4 7P A5

7P A77P A6

7P A12

EXP 1

7P A13

7P A14 7P A15

6PS A16 6PS A17

6PS A18 6PS A19

7P A20 7P A21

7P A22 7P A23

6PS A24 6PS A25

6PS A26 6PS A27

7P A28 7P A29

7P A30 7P A31

6PS A32 6PS A33

6PS A34 6PS A35

7P A36 7P A37

7P A38 7P A39

6PS A40 6PS A41

6PS A42 6PS A43

7P A44 7P A45

7P A46 7P A47

BASE

DS8000 arrays with 384 disks FC 450 GB

48 FC arrays

2374 GB each

24 arrays for pool1

24 arrays for pool2

All arrays for data and log



DB2 PS SAP® TRBK

21/168

The configuration of the RAID5 arrays of one DS8700 with 384 FC disks. There are 16 RAID5

arrays in the first frame and 32 in the second frame.

4.3.1 Assumptions for DB2 / GPFS / Storage design

� Assumptions:

– Shared pool of FC disks for Data/Index/Temp and logs

– Storage Pool Striping (1 GB stripe size) for Data/Index/Temp and logs

– No GPFS striping (each file system is defined / mapped on only one LUN).

– Easy configuration with only one level of stripping at DS8700 level.

� Design:

– Each file system (SAP® data1 to SAP® adata 16) is mapped on one LUN of 388 GB

size

– Each file system (log0 to log3) is mapped on one LUN of 2 TB size.

– Each LUN is stripped on 24 RAID5 arrays (192 FC disks) with a block size of 1 GB

– LUN is created with the option Rotate Extent in an Extent Pool

– 2 Extents Pool per DS8700 which consist in 24 FC RAID5 arrays each

– Each RAID5 array has 6 datas, 1 Parity and 1 spare or 7 datas and 1 parity disks.

– A total of 768 disks for the two DS8700.

DA 2

DA 0

DA 6

DA 4

DA 7

DA 5

6PS A0

6PS A2

6PS A1

6PS A3

6PS A8

6PS A10

6PS A9

6PS A11

7P A4 7P A5

7P A77P A6

7P A12

EXP 1

7P A13

7P A14 7P A15

6PS A16 6PS A17

6PS A18 6PS A19

7P A20 7P A21

7P A22 7P A23

6PS A24 6PS A25

6PS A26 6PS A27

7P A28 7P A29

7P A30 7P A31

6PS A32 6PS A33

6PS A34 6PS A35

7P A36 7P A37

7P A38 7P A39

6PS A40 6PS A41

6PS A42 6PS A43

7P A44 7P A45

7P A46 7P A47

BASE

Extent pool mapping with DS8000 RAID5 arrays

48 FC arrays

2374 GB each

24 arrays for extent pool1

24 arrays for extent pool2

All for data and log

Pool 2Pool 1

Pool 1 Pool 2



DB2 PS SAP® TRBK

22/168

An extent pool is made of 24 RAID5 arrays.

DA 2

DA 0

DA 6

DA 4

DA 7

DA 5

6PS A0

6PS A2

6PS A1

6PS A3

6PS A8

6PS A10

6PS A9

6PS A11

7P A4 7P A5

7P A77P A6

7P A12

EXP 1

7P A13

7P A14 7P A15

6PS A16 6PS A17

6PS A18 6PS A19

7P A20 7P A21

7P A22 7P A23

6PS A24 6PS A25

6PS A26 6PS A27

7P A28 7P A29

7P A30 7P A31

6PS A32 6PS A33

6PS A34 6PS A35

7P A36 7P A37

7P A38 7P A39

6PS A40 6PS A41

6PS A42 6PS A43

7P A44 7P A45

7P A46 7P A47

BASE

LUN definition in the extent pools

Pool 2Pool 1

Pool 1 Pool 2

D1

D2

D3

D4

L1

D5

D6

D7

D8

L2

Per Extent Pool:

4 LUNs of 388 GB for Data/index/Temp

1 LUN of 2 TB for logs

LUN is created with the option Rotate Extent in an Extent Pool. For each extent pool:

• 4 LUNs of 388 GB for Data/index/Temp

• 1 LUN of 2 TB for logs

For example the LUN 2000 is distributed on 24 different RAID5 arrays (R0 to R46 evenly).

This LUN of 388 GB size consists in 16 or 17 extents of 1 GB for each array.

dscli showfbvol -rank 2000 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_pure_2.profile

Date/Time: 08 December 2010 19:52:46 CET IBM DSCLI Version: 6.5.1.203 DS: IBM.2107-75TX901

Name PROD2_XXXX

ID 2000

accstate Online

datastate Normal

configstate Normal

deviceMTM 2107-900

datatype FB 512

addrgrp 2

extpool P0

exts 388

captype DS

cap (2^30B) 388.0

cap (10^9B) -

cap (blocks) 813694976

volgrp V0,V1



DB2 PS SAP® TRBK

23/168

ranks 24

dbexts 0

sam Standard

repcapalloc -

eam rotateexts

reqcap (blocks) 813694976

realextents 388

virtualextents 0

migrating 0

perfgrp PG0

migratingfrom -

==============Rank extents==============rank extents============

R0 16

R2 16

R4 16

R6 16

R8 16

R10 16

R12 17

R14 17

R16 17

R18 17

R20 16

R22 16

R24 16

R26 16

R28 16

R30 16

R32 16

R34 16

R36 16

R38 16

R40 16

R42 16

R44 16

R46 16

4.3.2 LUN and file system for DB2 with 768 disks FC

File system capacity techno NumDisks Num

LUNs

LUN size Storage Pool

Striping

/db2/D3D/ 2000 GB FC 192 1 2000 GB Yes

/db2/D3D_log0

/db2/D3D_log1

/db2/D3D_log2

/db2/D3D_log3

2000 GB

2000 GB

2000 GB

2000 GB

FC

FC

FC

FC

192 1

1

1

1

2000 GB Yes

/db2/D3D_SAP®

data1

…………………..

388 GB

388 GB

FC

FC

192

192

1

1

388 GB

388 GB

Yes

Yes



DB2 PS SAP® TRBK

24/168

/db2/D3D_SAP®

data16

4.3.3 Detailed of the LUNs configuration

SAP® fs DS8700 ID + LUN ID EP DS8700 cap

SAP® data1 2bc0000000000002100) P1 PRA1 388 GB

SAP® data2 2c10000000000002100) P1 PRA2 388 GB

SAP® data3 2c10000000000002003) P0 PRA2 388 GB

SAP® data4 2c10000000000002002) P0 PRA2 388 GB

SAP® data5 2c10000000000002001) P0 PRA2 388 GB


SAP® data7 2c10000000000002000) P0 PRA2 388 GB






SAP® data13 2c10000000000002103) P1 PRA2 388 GB


SAP® data15 2c10000000000002102) P1 PRA2 388 GB

SAP® data16 2c10000000000002101) P1 PRA2 388 GB

SAP® fs DS8700 ID + LUN ID EP DS8700 cap

log0 2c10000000000002004 P0 PRA2 2000 GB

log1 2c10000000000002104 P1 PRA2 2000 GB

log2 2bc0000000000002004 P0 PRA1 2000 GB

log3 2bc0000000000002104 P1 PRA1 2000 GB

tiebreaker 2bc0000000000002005 P0 PRA1 2000 GB

db2fs1 2c10000000000002105 P1 PRA2 2000 GB

notused 2c10000000000001005



DB2 PS SAP® TRBK

25/168

db2d3d 2c10000000000002005 P0 PRA2 2000 GB

4.4 Network fabric overview

4.4.1 Infiniband network

4.4.1.1 Topology

For HA test purposes, 2 Mellanox switches (IS5030) have been set up for providing switch failure

tolerance. Here is the topology:

In case of a switch failure, half on the cluster is brought offline.

On the outline above, 2 types of cables have been used:

- From member/switch and coupling/switch : 5m Copper cables QSFP

- Between switches : 10m Optic cables QSFP

Having four links between the 2 switches ensure an optimal throughput, as each coupling facility

has 4 HCAs.

4.4.1.2 Subnet Manager configuration

Among the 2 switches, one has been chosen to act as the Subnet Manager. It is keeping the

identifier of each HCAs and is aware of the global Infiniband topology. The switch running as a

Subnet Manager is a licensed Mellanox feature, and has been activated in mlx-switch1. The

other switch (mlx-switch2) is considering as passive, as it is only routing the packets without

having the responsibility of maintaining the IB network.

4.4.1.3 Physical connections

Here is the list of the ports that are used on the 2 switches:



DB2 PS SAP® TRBK

26/168

Port number mlx-switch-1 mlx-switch-2

1 DB2PS-member0 HCA-1 DB2PS-member1 HCA-1

3 DB2PS-member2 HCA-1 DB2PS-member3 HCA-1

7 DB2PS-cf0 HCA-2 DB2PS-cf1 HCA-2




19 mlx-switch-2:IS5030 mlx-switch-1:IS5030




4.4.2 Fiber Channel network

Each database server is connected to the storage systems (DS8700) via a storage area network.

This network is created using dedicated SAN switches in the infrastructure. As described in the

server part, each member has 4 connections as they perform intensive I/O while coupling

facilities have only 2 links on the SAN.

Another smaller SAN connection is used between the SAP® injector and a DS5000 system (peer-

to-peer connection), to store all the data coming from the run.

4.4.3 Ethernet network

4 networks are used to ensure the communication through the DB2 pureScale® architecture

components:

Network Type Machines

connected

Subnet MTU

Administration 1 Gb Ethernet All 10.3.67.0/24 1500

Application 1 Gb Ethernet SAP® servers 10.1.1.0/24 1500

Database 10 Gb Ethernet SAP® + DB2

servers

10.1.2.0/24 9000

Interconnect 10 Gb QDR

Infiniband

DB2 servers 10.9.9.0/24

(IPoIB)

65520

The administration network is reachable from the Internet thanks to a Virtual Private Network.

All other networks are isolated and dedicated to this architecture.

Remark: the gateway of the 10.3.67.0/24 network is also used for NTP (Network Time Protocol)

synchronization.



DB2 PS SAP® TRBK

27/168

For more information concerning the IP addressing map, please refer to the file pureScale – IP

Addressing.xls



DB2 PS SAP® TRBK

28/168

4.5 Operating Systems

4.5.1 Suse Enterprise Linux 10 SP3

All servers are using Suse Enterprise Linux (SLES) 10 SP3. Kernel has been upgraded on the DB2

servers from the original version to allow the newest OFED driver (for Infiniband) to be installed.

Machine Type Kernel version

Database Servers (x3850x5) 2.6.16.60-0.66.1-smp (x86_64)

Application Servers (HS22) 2.6.16.60-0.54.5-smp (x86_64)

4.5.2 I/O device drivers

4.5.2.1 Multipathing driver

The multipathing driver used for this benchmark is DM-MPIO that comes with the SLES 10 SP3

distribution. The configuration file (multipath.conf) follows IBM support recommendations. The

policy is round-robin based (rr_min_io = 100), and paths are grouped by priority. Here is an

extract of the multipath.conf file:

defaults {

polling_interval 30

failback immediate

no_path_retry 5

rr_min_io 100

path_checker tur

# features "1 queue_if_no_path"

user_friendly_names yes

# These are the default settings for 2107 (IBM DS8000)

# Uncomment them if needed on this system

device {

vendor "IBM"

product "2107900"

path_grouping_policy group_by_serial

}

4.5.2.2 Linux IO scheduler parameter

Here are Linux parameters concerning the IO scheduling, for each accessible disk:

- IO Queue size : 128

- Scheduler used : Completely Fair Queuing

At a lower level, the use of NCQ (Native Command Queuing) also comes with sets of parameters

- NCQ queue size : 32

For deeper details about the LUN mapping please refer to multipath command display or

storage configuration documentation.



DB2 PS SAP® TRBK

29/168

4.5.2.3 Infiniband driver

For the database servers, the OFED stack is used to provide the driver for Infiniband HCA.

On one node, here is the result of the infiniband driver self test command:

---- Performing Infiniband HCA Self Test ----

Number of HCAs Detected ................ 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... OFED-1.5.2-20100824-0600: 1.5.2-2.6.16.60_0.66.1_smp

Host Driver RPM Check .................. PASS

HCA Firmware on HCA #0 ................. v2.7.0

Host Driver Initialization ............. PASS

Number of HCA Ports Active ............. 1

Port State of Port #1 on HCA #0 ........ UP 4X QDR

Port State of Port #2 on HCA #0 ........ DOWN

Error Counter Check on HCA #0 .......... PASS

Kernel Syslog Check .................... PASS

------------------ DONE ---------------------

4.5.2.4 Major on-board drivers (including Emulex and firmware)

Here are some of the major drivers used in the database servers, including the 10 Gb Ethernet

Emulex NIC. The updates have been done and checked using the UpdateXpress pack provided

on the IBM support site.

BC-Emulex Device Driver 2.101.411.0 for 10Gb CNA for SLES10

Installed Version : 2.101.411.0

Broadcom bnx2 sles10 driver

Installed Version : 2.0.1-suse

IBM and LSI Basic or Integrated RAID SAS Controller Driver

Installed Version : 3.04.07_suse




IBM uEFI Flash Update

Installed Version : 1.23 (G0E122D)

IBM Dynamic System Analysis

Installed Version : 3.13 (DSYT72C)

IBM HBA/LSI Onboard 3Gb SAS/SATA BIOS/FW/UEFI Update


IBM Online SAS/SATA Hard Disk Drive Update Program

Installed Version : B536

Installed Version : B536

Online Broadcom NetXtreme and NetXtreme II Firmware Utility



DB2 PS SAP® TRBK

30/168

Installed Version : 5.2.2

Installed Version : 5.2.2

Integrated Management Module Update

Installed Version : 1.14 (YUOO73K)

IBM x3850 / x3950 X5 FPGA Flash Update


4.5.3 Settings tuned

4.5.3.1 Kernel parameters

For performance improvement, the use of several HCA (Host Channel Adapter – Infiniband

Adapter) has been recommended on the coupling facilities. As the uDAPL needs IPoIB

configuration for each HCA, several IP addresses from the same subnet are utilized on the same

physical machine.

The Linux kernel doesn’t handle this configuration by default. Thus, the kernel parameter

net.ipv4.conf.all.arp_ignore has been modified to the value “2”, as recommended in

the OFED stack documentation.

4.6 DB2 pureScale®

4.6.1 DB2 concepts DB2 pureScale® is a DB2 feature that allows the ability to scale out a database over a

number of servers in an “active-active” configuration and delivers a high level of availability and

scalability. DB2 pureScale® offers virtually unlimited capacity for growth as servers can be

dynamically added to a DB2 pureScale® instance with no disruption to cluster operation. Upon

adding additional servers, DB2 pureScale® ’s load balancing logic will automatically begin to

transparently reroute transactions to make use of the additional resources. Applications require

no knowledge of the underlying database system topology and do not require any modifications

to run on DB2 pureScale® . Thus DB2 pureScale® offers true application transparency.

Applications running on DB2 pureScale® need not be “cluster aware” in order to scale, as can be

the case with other scale-out architectures that rely on a heavy network-based messaging

infrastructure for sharing data in a cluster. As we describe below, by exploiting the latest

network and hardware architectures as well as a centralized server for locking and caching, DB2

pureScale® is able to transparently scale applications. Lastly, DB2 pureScale® features

continuous availability by being able to tolerate multiple component failures while continuing to

provide full access to data that does not need recovery. The following sections describe the

main components that make up a DB2 pureScale® instance.

A DB2 pureScale® instance consists of one or more DB2 servers (referred to as “members”)

where each member runs a full DB2 engine with its own buffer pools, agent threads, log files

and so on. DB2 pureScale® is a shared-data distributed database architecture meaning each

member has access to the same data for read and write operations. Synchronizing access to this

data is required for database consistency and being able to do so efficiently is essential for good

database performance. The solution for ensuring database consistency while maintaining high



DB2 PS SAP® TRBK

31/168

levels of performance in DB2 pureScale® is derived from DB2 for z/OS Parallel Sysplex, a shared-

data DBMS known for its exceptional availability and scalability.

DB2 for z/OS Parallel Sysplex achieves the recognition it does in large part due to its use of the

“Coupling Facility”. The Coupling Facility provides a centralized location for managing locks as

well as a global shared cache for data. Part of every DB2 pureScale® instance is the pureScale

Cluster Caching Facility (or CF) server which provides the same centralized locking and global

shared cache architecture. Whenever SQL processing on a member requires a lock such as for

updating a row on a page, it will request the lock from the CF. The CF will determine if the lock is

available and if so, the lock will be granted and execution can continue on the member.

Requests to the CF are also made when a member wants to read a page from the CF into its

local buffer pool (LBP) or commit changes to a page. During a read request, if the CF has the

page cached, it will send the page to the member, thus saving the member from having to read

the page from disk. When the CF receives a page write request, the modified page is sent to the

CF as part of the request and is cached in the Group Buffer Pool (GBP). At this point the CF has

the most recent version of the page and proceeds to “invalidate” (mark as stale) any other

copies of the page that exist in the LBPs of other members. To eliminate the CF as a single point

of failure, the DB2 pureScale® instance is generally configured to use two CFs. In a two CF

configuration, all requests are sent to the CF designated as being the “primary CF” and are also

synchronously duplexed to the “secondary CF”. In doing so, the secondary CF can quickly

takeover as the primary in case of a primary CF failure. Optimum performance requires low CF

response times, a large part of which is the cost of communicating with the CF. This

communication overhead is minimized through the use of a high performance network

architecture.

4.6.2 Integrated Technologies

The CF is connected to each member in the DB2 pureScale® instance via a low latency, high

speed Remote Direct Memory Access (RDMA) network. InfiniBand and 10 Gigabit Ethernet

support interrupt free RDMA, which plays a critical role in the scalability of DB2 pureScale® .

With RDMA, each member is able to directly access memory on the CF and the CF is able to

directly access memory on each member, without consuming any CPU at the supplying side. For

example, during commit processing all the pages that were modified during the transaction will

be sent to the CF. Upon receiving these pages, the CF will cache them and determine which (if

any) of the other members have (now stale) copies of these pages in their local buffer pools.

Once this information is obtained, the CF will directly access the pages on the remote member

machines via RDMA and mark them as “invalid”. The ability to mark these pages as stale with no

interruption to the remote member is critical for the scalability of write intensive applications.

All servers in a DB2 pureScale® instance are connected to the same disk storage via a Storage

Area Network (SAN). The IBM General Parallel File System (GPFS) is used to provide access to

the data with high scalability, high availability and seamless capacity expansion. GPFS is shipped

as part of DB2 pureScale® and is installed and configured seamlessly during DB2 pureScale®

installation. The db2cluster command is used to create and manage your GPFS file systems



DB2 PS SAP® TRBK

32/168

and will automatically configure GPFS with known best practices for settings such as file system

block size and file system cache size.

Ensuring high availability in a clustered environment like DB2 pureScale® where there are

dependencies on various resources is non-trivial. DB2 pureScale® uses IBM Tivoli Systems

Automation (TSA) and IBM Reliable Scalable Cluster Technology (RSCT) to handle the

automation of failure detection and recovery. With the ability to provide continuous high

availability and reduced duration of service disruptions, TSA and RSCT offer a way to monitor the

state of each of the components in a DB2 pureScale® environment. Upon failure detection, TSA

and RSCT will take the appropriate action to bring the failed component back to an operational

state. Examples of actions that could be taken include remounting file systems, restarting a DB2

member and rebooting a server. TSA and RSCT are shipped as part of DB2 pureScale® and are

installed and configured during DB2 pureScale® installation. Easy post installation management

of TSA and RSCT can be done through the db2cluster command.

The following diagram illustrates the topology of a DB2 pureScale® instance with two DB2

members and two DB2 pureScale® CF servers. One CF server is designated as primary (P) and the

other as secondary (S).

DB2Member 0

GPFS

TSARSCT

LBP

DB2Engine

DB2Member 1

GPFS

TSARSCT

pureScaleCF (S)

GPFS

TSARSCT

SwitchSwitch

Storage Area Network

RDMA Network

Disk Storage System

pureScaleCF (P)

GPFS

TSARSCT

Local AreaNetwork

EthernetNetwork

LBP

DB2Engine

CF Server

GBP

Locks

CF Server

GBP

Locks



DB2 PS SAP® TRBK

33/168

4.7 SAP

4.7.1 TRBK overview

The Transaction Banking Benchmark (TRBK) reflects standard business scenarios within the SAP®

Banking 7.0 application and is used to measure

all performance relevant parameters.

It is divided into two different types of workload. The first scenario, day processing,

reflects usage by users and integrated systems that generally occurs during the

daytime. The second scenario, night processing, reflects account balancing that

generally occurs overnight in batch mode.

Day Processing:

The first scenario, day processing, reflects usage by users and integrated

systems that generally occurs during the daytime.

In this scenario multiple user logins are simulated and in the “High Load Phase” a

defined number of users are working concurrently on the system.

The figure below shows the workload in day processing over time:



DB2 PS SAP® TRBK

34/168

Night Processing

In the night processing scenario, account balancing is performed at in order to settle an account.

The benchmark simulates account balancing by calculating interest and charges for each account

across 20 value date days.

In the night processing scenario, a defined number of batch processes are running in parallel

and the total amount of work (accounts to be balanced) is divided into packages that are

processed separately by the several work processes of the SAP® application servers.

4.7.2 Benchmark Configuration

Installed Software Components

Here is a list of prerequisite SAP® software components and their support package levels that

have been installed to build the SAP® Banking 7.0 System that has been used for this

benchmark:

SAP® Kernel Release: SAP® Kernel 711 Patch Level 80

Additionally the following SAP® notes have to be applied (SNOTE)

1441931 BW extraction from Bank analyzer taking long time for custom

1449022 Memory leak in contract

1449296 Performance: BAdI for BW extraction

1452459 Reordering the number assignment in benchmark tool

1452640 Performance improvement in mass generation/account migration

1461898 BCST: Synchronization of global buffer missing

1478386 Error during totals update

and manually 494171 Section to adapt DB Indexes on DFKKCOH and DFKKCOHI

3 tier SAP® Layout



DB2 PS SAP® TRBK

35/168

The figure below shows the over all topology and architecture of the

SAP® TRBK benchmark environment:

4.7.3 System Installation

The following chapter reflects the procedure that was performed in this POC due to special

situations/restrictions (e.g. hardware availability). Usually the GPFS file systems are created first

and then the system installation is performed directly on the target hardware.

For a description of the SAP® recommended general procedure to migrate an existing DB2 9.7

system to DB2 pureScale®, see also the following SDN article :

http://www.sdn.SAP® .com/irj/scn/go/portal/prtroot/docs/library/uuid/10517fd9-e2bc-2d10-

cfa1-b2c182149798?QuickLink=index&overridelayout=true

The steps that were performed to build the TRBK System on DB2 pureScale® for the POC project

are enumerated below



DB2 PS SAP® TRBK

36/168

4.7.3.1 Build SAP® TRBK system (in Walldorf)

The SAP® system with all required components was installed and prepared

in Walldorf according to the listed requirements in 4.2.7.

4.7.3.2 Build DB2 9.7 System on preliminary hardware (SAP® system copy)

The benchmark system was created by performing a SAP® system copy

with the backup/restore method on a preliminary hardware so be able to

start the data generation process as soon as possible. The final hardware was

not yet available at that point in time.

4.7.3.3 Data generation (90 Million accounts)

The TRBK Master data was generated in a staged approach. The customers

request was to generate 90 Million accounts overall. Here are the steps that

describe the data generation process:

a. generate 10 Million accounts

b. perform a “runstats with detailed indexes all” on tables BCA_CONTRACT,

BCA_CNSP_ACCT, BCA_PAYMITEM, BCA_COUNTER, , BCA_TRANSFIG

c. backup the database

d. generate another 10 Million accounts

e. backup again

f. generate 50 Million accounts (over weekend)

g. backup again

h. generate remaining 20 Million accounts

i. perform full runstats on all tables

j. final backup

k. check number of rows for the following tables (system should have 90 Million

accounts generated)

90 Million records in BCA_CONTRACT - 1 per account

90 Million records in BCA_CNSP_ACCT - 1 per account

1800 Million records in BCA_PAYMITEM - 20 per account

1800 Million records in BCA_COUNTER - 20 per account

1800 Million records in BCA_TRANSFIG - 20 per account



DB2 PS SAP® TRBK

37/168



DB2 PS SAP® TRBK

38/168

4.7.3.4 Create system on final hardware with DB2 9.7

After the target hardware was available, the database server of the SAP®

system was installed on the target hardware by a SAP® system copy with

backup/restore. To be able to migrate that system later to DB2 pureScale®,

the container paths have to exist on GPFS file systems. Therefore, on the

respective disks on the DS8700 storage system 21 GPFS file systems were

created:

/db2/D3D database directory

/db2/D3D_log_0 logfiles member 0




/db2/D3D_SAP® data1 |







/db2/D3D_SAP® data8 |----- 16 SAP® data paths









Two additional disks of the shared DS8700 storage were used for the

following purposes:

1 device as instance shared device “INSTANCE_SHARED_DEV”

1 device as tiebreaker “TIEBREAKER_DEV”



DB2 PS SAP® TRBK

39/168

4.7.3.5 Prepare GPFS file systems

First DB2 pureScale® software was installed on the member 0 of target hardware:

db2_install –p ese_dsf –b /opt/sd

The instance shared device INSTANCE_SHARED_DEV was prepared by the following

command:

/opt/sd/instance/db2cluster_prepare –instance_shared_dev

/dev/<devicefilename of instance_shared_dev>

All GPFS filesystems (see 4.7.3.4) were created – here is an example

db2cluster –cfs –create –filesystem db2d3d

–disk /dev/<devicefilename> – mount /db2/D3D



DB2 PS SAP® TRBK

40/168

4.7.3.6 SAP® system copy with backup/restore

The database server of the SAP® system was then installed on the host which

became member 0 later by performing a SAP® system copy using the backup/

restore method. The DB2 9.7 database was created with automatic storage

using the storage paths listed in 4.7.3.4.

4.7.3.7 Migrate to DB2 pureScale® with 1 member and 1 CF

Here is a description of what has to be done to migrate the existing DB2 9.7

database to DB2 pureScale® . This step consists of 2 parts, first migrating to

a DB2 9.8 ESE instance and finally change the instance type to SD (Shared

Data).

a. Migrating to a DB2 9.8 ESE instance

as db2d3d

db2 update dbm cfg using numdb 1

/opt/sd/bin/db2checkSD d3d => no problems reported

db2stop

as root

/opt/sd/instance/db2iupgrade –d –k db2d3d

as db2d3d

db2start

db2 upgrade db d3d

db2 connect to d3d

db2 terminate

db2stop

b. Change instance type to sd (data sharing instance) by performing

an instance update:

as root

/opt/sd/instance/db2iupdt –d –tbdev <device file for

tiebreaker> -cf DB2PS-cf0:DB2PS-cf0-ib0 –m DB2PS-

member0:DB2PS-member0-ib0 db2d3d

4.7.3.8 Add 1 additional CF and 3 additional members



DB2 PS SAP® TRBK

41/168

as root

# add second CF

/opt/sd/instance/db2iupdt -d -add -cf DB2PS-cf1:DB2PS-

cf1-ib0 db2d3d

# add member1

/opt/sd/instance/db2iupdt -d -add -m DB2PS-


# add member2



# add member3



4.7.3.9 Backup final configuration

After some backup and restore tuning, the time for backup and restore could

be reduced significantly (the restore time for example was reduced from

~5 hours to ~2 ¼ hours).

Here is an example of the tuned restore command:

db2 RESTORE DATABASE D3D FROM /db_backup_10, /DB_BACKUP_11,

/DB_BACKUP12, /DB_BACKUP13 TAKEN AT 20101108143425 ON

/db2/D3D_SAP® data1, /db2/D3D_SAP® data2, /db2/D3D_SAP® data3,





/db2/D3D_SAP® data16 DBPATH ON /db2/D3D WITH 128 BUFFERS

BUFFER 16384 PARALLELISM 32



DB2 PS SAP® TRBK

42/168

Here is an example for the backup command:

db2 BACKUP DATABASE D3D to /db_backup_10, /DB_BACKUP_11,

/DB_BACKUP12, /DB_BACKUP13 WITH 128 BUFFERS

BUFFER 16384 PARALLELISM 32COMPRESS



DB2 PS SAP® TRBK

43/168

5 Test Results : Performance, Scalability & Availability

The TRBK benchmark used for the performance tests during the PoC consists of 2 distinct

phases.

The Day phase is a simulation of the creation of payment items against accounts such as might

occur from, for example, ATM’s during the normal day time operation of a transactional banking

system. This workload is simulated by a configurable number of users running a 15 step script of

transactions a configurable number of times. The accounts against which each payment items

are processed are generated randomly. This means that the database activity generated by the

Day run hits values across the entire range of accounts in the database randomly.

The Night phase is a simulation of the balancing of all accounts in the system. As initially loaded

the database contains 20 payment items per account, i.e. for this PoC, the total number of

payment items reconciled was 1.8 billion. The workload consists of a configurable number of

SAP® batch processes that work on sequential ranges of accounts until the entire range is

balanced. The database activity generated by the workload is sequential in nature.

The follow-up tests (scalability, availability and Disaster recovery) were either based on the Day

or the Night workload.

5.1 Overall test configuration

5.2 Performance test results were performed using 4 DB2 pureScale® members, 2 CF’s

(primary and secondary), and 1 enqueue server/CI and 20 application servers.

5.3 TRBK Day Run

The high water mark Day run was performed using 11000 benchmark users running on 20

application server instances, each with 120 dialog work processes. TRBK allows the configuration

of benchmark think time, i.e. a time delay between transaction steps of the benchmark

workload that emulates the delay that might be seen in an interaction from an actual user. A

short time delay will cause users to do more work in any time interval. A think time of 7 seconds

gave an optimal balance between number of users and workload.

5.3.1 Day run settings + results achieved The measurement metric for the Day run is payment items per hour. Each cycle of 15 dialog

steps run by each user generates 175 payment items.

56916893 Postings per hour

Duration of high load phase in seconds: 981

Average response time per dialog step (ms): 1119.8



DB2 PS SAP® TRBK

44/168

5.3.2 Scalability Tests were done to see how well the system scaled from 1 to 4 members.

As shown in the graph below, the results indicated effectively linear scaling.

Scalability

0

10000000

20000000

30000000

40000000

50000000

60000000

1 2 3 4

Members

Pi/hr

5.3.3 Specific Day Run Tuning

5.3.3.1 BCA_RCN_SUMS_IN/OUT

These tables contain accumulation values and are updated very heavily during both the Day and

Night runs. Because these are very small tables, they are potentially the source of contention

between the many work processes doing simultaneous updates. To eliminate this contention,

we reorganized the table with PCTFREE 99 in order to place one row on each page, and used an

SAP® report to assign each work process to a specific row.



DB2 PS SAP® TRBK

45/168

Database Servers

System Summary DB2PS-member 12/4/2010

0

10

20

30

40

50

60

70

80

90

100

22:0

3

22:0

4

22:0

5

22:0

6

22:0

7

22:0

8

22:0

9

22:1

0

22:1

1

22:1

2

22:1

3

22:1

4

22:1

5

22:1

6

22:1

7

22:1

8

22:1

9

22:2

0

22:2

1

22:2

2

22:2

3

22:2

4

22:2

5

22:2

6

22:2

7

22:2

8

22:2

9

22:3

0

22:3

1

22:3

2

22:3

3

22:3

4

22:3

5

22:3

6

22:3

7

22:3

8

22:3

9

usr%

+sy

s%

CPU%

Each of the 4 DB2 database members consumed 62% user and 3% system CPU during the high

load phase.

Disk total KB/s DB2PS-member - 12/4/2010

0

100

200

300

400

500

600

700

800

900

1000

22:0

3

22:0

4

22:0

5

22:0

6

22:0

7

22:0

8

22:0

9

22:1

0

22:1

1

22:1

2

22:1

3

22:1

4

22:1

5

22:1

6

22:1

7

22:1

8

22:1

9

22:2

0

22:2

1

22:2

2

22:2

3

22:2

4

22:2

5

22:2

6

22:2

7

22:2

8

22:2

9

22:3

0

22:3

1

22:3

2

22:3

3

22:3

4

22:3

5

22:3

6

22:3

7

22:3

8

22:3

9

Thou

sand

s

KB

/sec

0

5000

10000

15000

20000

25000

30000

35000

IO/s

ec

Disk Read KB/s Disk Write KB/s IO/sec

Sustained IOPS rate during high load was 26000 per server.

Application Servers

System Summary sapservn 12/4/2010

0

10

20

30

40

50

60

70

80

90

100

22:0

3

22:0

4

22:0

5

22:0

6

22:0

7

22:0

8

22:0

9

22:1

0

22:1

1

22:1

2

22:1

3

22:1

4

22:1

5

22:1

6

22:1

7

22:1

8

22:1

9

22:2

0

22:2

1

22:2

2

22:2

3

22:2

4

22:2

5

22:2

6

22:2

7

22:2

8

22:2

9

22:3

0

22:3

1

22:3

2

22:3

3

22:3

4

22:3

5

22:3

6

22:3

7

22:3

8

22:3

9

usr%

+sy

s%

CPU%

Each of the 20 application servers consumed 65% CPU (62% user, 3% system).



DB2 PS SAP® TRBK

46/168

Disk total KB/s sapservn - 12/4/2010

0

50

100

150

200

250

22:0

3

22:0

4

22:0

5

22:0

6

22:0

7

22:0

8

22:0

9

22:1

0

22:1

1

22:1

2

22:1

3

22:1

4

22:1

5

22:1

6

22:1

7

22:1

8

22:1

9

22:2

0

22:2

1

22:2

2

22:2

3

22:2

4

22:2

5

22:2

6

22:2

7

22:2

8

22:2

9

22:3

0

22:3

1

22:3

2

22:3

3

22:3

4

22:3

5

22:3

6

22:3

7

22:3

8

22:3

9

KB

/sec

0

5

10

15

20

25

30

IO/s

ec


5.3.4 Conclusion The final Day processing number was constrained more by the amount of ime the hardware was

available than by any machine resource constraint. The final result was achieved with a well

balanced system, with approximately equal and optimal CPU usage on the DB server and the

application servers. The IO rate of 104000 IOPS gave average read times of 12ms and minimal

write times.



DB2 PS SAP® TRBK

47/168

5.4 TRBK Night Run

The high water mark Night run was performed using 720 batch work processes on 20 application

servers each with one SAP® instance. The number of accounts processed in a single transaction

by a work process (a package) is configurable. We found that the default of 50 accounts per

package worked well.

5.4.1 Night run settings + results achieved The measurement metric for the Night run is accounts per hour. The entire loaded range of 90

million accounts is processed during the Night run.

Night run duration: 2 hours, 38 minutes

34300233 accounts per hour

5.4.1.1 Night Run processing in detail

During the night run, the total number of packages (in our case1.8 million packages with

packagesize 50) is distributed over the available application servers. So each of the 20

application servers gets a range of package numbers to process.

The following figure shows the distribution of packages over application servers

1-

90000

90001-

180000

180001-

270000

270001-

360000

…… 1620001-

1710000

1710001-

1800000 app server 1 app server 2 app server 3 app server 4 …… app server 19 app server 20

All the work processes of a respective application server then pick a free package out the range,

process it and write back status information into the table BANK_WORKL_PACK.

Package is not processed ‘ ‘

Package has been picked – processing started ‘X’

Package is completed ‘R’

The distribution of package number ranges over the involved application servers can be

checked in table BANK_WL_SRVREL.

Generally speaking, that account master data that is used to balance a package is aligned

with the package numbers. So neighboring packages refer to neighboring data.



DB2 PS SAP® TRBK

48/168

5.4.1.2 Problem in night run end processing

A default behavior of the SAP® benchmarking application was that work processes on

application servers that have completed their amount of work (number range of packages

assigned to that application server) take over unprocessed packages of other ranges.

That caused serious contention problems in the DB2 Pure Scale architecture, especially because

after some time, a massive number of work processes connected to all 4 members are working

on packages of the same range (and therefore need to access the same – or at least – colocated

data).

The throughput went down to nearly 0 within the last 0.5 % of the packages to process.

So solve this problem, SAP® provided a fix at the on ABAP level, to avoid having application

servers that have finished their work taking over packages of other application servers. The

ABAP modification is officially available and is described in SAP® note 1929954.

5.4.1.3 UUID Generation

Within the project we observed that neither the application servers nor the database servers

were heavily utilized (app server ~ 60%, database server ~25%). This picture also did not change

when trying to increase the number of work processes on the application servers. But during

this test the problem was showing up more clearly. Almost all work processes were stuck in a

ABAP class called CL_SYSTEM_UUID. The SAP® kernel uses an OS functionality to determine

unique IDs. It turned out that the SLES 10 implementation of this feature is the bottleneck. In

SAP® note 1391070 there is a description how to solve this problem (use the UUID deamon).

The result of the new implementation was that the UUID generation speeded up dramatically,

our CPU utilization on the application servers jumped to 99% and the throughput went up to the

final result values.



DB2 PS SAP® TRBK

49/168

5.5 Experiments with packagesize and number of work processes

As described above, during the project there was a problem with the night run end processing.

Increasing the package size (reducing the over all number of packages) first seemed to be a valid

method to reduce the end processing problem.

But it turned out, that the actual problem could not be removed, even though the point in time

when the problem occured, could be moved to 0.05% of left packages. Database parameters like

LOCKLIST had to be adapted when using higher package sizes because the processing of one

package is done within one database transaction. Over all, we experimented with the following

package sizes:

Packagesize 50 � default at the beginning, also used for the golden run

Packagesize 100

Packagesize 300 � LOCKLIST had to be increased

Packagesize 600

In the night run scenario we also experimented with several settings for the number of used

work processes. But all these tests were bottlenecked by the described problem with UUID

generation, so the changes in number of work processes did not really have big effects. Here is

an overview of which work process configurations were used:

640 batch work processes � initial setting

720 batch work processes � setting for the golden run

800 batch work processes � setting used to try to increase workload

1280 batch work processes � roll in / roll out problems

5.5.1 Resource usage

Database Servers

Each DB member consumed 33% CPU and drove ~8000 IOPS

System Summary DB2PS-member 12/5/2010

0

10

20

30

40

50

60

70

80

90

100

22:4

6

22:5

0

22:5

4

22:5

8

23:0

2

23:0

6

23:1

0

23:1

4

23:1

8

23:2

2

23:2

6

23:3

0

23:3

4

23:3

8

23:4

2

23:4

6

23:5

0

23:5

4

23:5

8

00:0

2

00:0

6

00:1

0

00:1

4

00:1

8

00:2

2

00:2

6

00:3

0

00:3

4

00:3

8

00:4

2

00:4

6

00:5

0

00:5

4

00:5

8

01:0

2

01:0

6

01:1

0

01:1

4

01:1

8

01:2

2

01:2

6

01:3

0

01:3

4

usr%

+sy

s%

0

2000

4000

6000

8000

10000

12000D

isk

xfer

s

CPU% IO/sec



DB2 PS SAP® TRBK

50/168

Disk total KB/s DB2PS-member - 12/5/2010

0

50

100

150

200

250

300

350

400

450

500

22:4

6

22:5

0

22:5

4

22:5

8

23:0

2

23:0

6

23:1

0

23:1

4

23:1

8

23:2

2

23:2

6

23:3

0

23:3

4

23:3

8

23:4

2

23:4

6

23:5

0

23:5

4

23:5

8

00:0

2

00:0

6

00:1

0

00:1

4

00:1

8

00:2

2

00:2

6

00:3

0

00:3

4

00:3

8

00:4

2

00:4

6

00:5

0

00:5

4

00:5

8

01:0

2

01:0

6

01:1

0

01:1

4

01:1

8

01:2

2

01:2

6

01:3

0

01:3

4

Tho

usan

ds

KB

/sec

0

2000

4000

6000

8000

10000

12000

IO/s

ec


Application Servers

System Summary sapservn 12/5/2010

0

10

20

30

40

50

60

70

80

90

100

22:4

6

22:5

0

22:5

4

22:5

8

23:0

2

23:0

6

23:1

0

23:1

4

23:1

8

23:2

2

23:2

6

23:3

0

23:3

4

23:3

8

23:4

2

23:4

6

23:5

0

23:5

4

23:5

8

00:0

2

00:0

6

00:1

0

00:1

4

00:1

8

00:2

2

00:2

6

00:3

0

00:3

4

00:3

8

00:4

2

00:4

6

00:5

0

00:5

4

00:5

8

01:0

2

01:0

6

01:1

0

01:1

4

01:1

8

01:2

2

01:2

6

01:3

0

01:3

4

usr%

+sy

s%

0

2

4

6

8

10

12

14

16

18

20

Dis

k xf

ers

CPU% IO/sec

5.5.2 Conclusion

At the end that night run CPU usage on the database servers was ~33 % while all 20 application

servers were at 99%. So the conclusion is that we were application server bound in this

situation, and higher throughput might be reached by adding additional application servers.

DB2 Pure Scale performs very well in the TRBK night run scenario especially because of the

nature of the workload. The packages that are processed separately on the several servers allow

nearly full parallelization.



DB2 PS SAP® TRBK

51/168

5.6 High Availability Test

5.6.1 Results

The table below contains the results of the HA scenarios performed:

Test

# Scenarios Customer requirement

(outage time) DB2 pureScale® result

(outage time) 1 Member shutdown failure (dual

member failure part 1) <= 60 seconds 30 seconds

2 Malloc bomb member (dual member failure part 2)

<= 60 seconds 40 seconds

3 Primary CF shutdown failure <= 60 seconds 20 seconds

4 Secondary CF shutdown failure <= 60 seconds 10 seconds

5 Malloc bomb primary CF <= 60 seconds 20 seconds

6 Member IB adapter failure <= 60 seconds 30 seconds

7 Soft kill member <= 30 seconds 30 seconds

8 IB switch failure <= 60 seconds 30 seconds

The high availability (HA) tests were done using the same settings as the day/night run except

the following:

- Single HCA was used (HA support with multiple HCA will be in db2_v98fp4)

- The (primary/secondary) CFs’ memory were set to 70% of the amount of physical

memory available on the CF machines to match the HA test minimum requirement

of running the TRBK workload on half the throughput level of the day run

- 13 app servers with 6000 users were used to match the HA test minimum

requirement of running the TRBK workload on half the throughput level of the day

run

5.6.2 GPFS IO fencing

All the day/night/HA runs were run with the following TSA/GPFS settings which reduced the IO

fencing time when a hard failure occur. The settings below reduced the IO fencing time from 50

seconds to about 10 seconds.

hostfailuredetectiontime = 2

gpfs failureDectionTime = 10

leaseRecoveryWait =10

SCSI 3 PR support for IBM storage DS8700 series is now available in db2_v98fp4 and it will not

be necessary to alter the TSA/GPFS settings above anymore. With this change a much improved



DB2 PS SAP® TRBK

52/168

IO fencing time approximately in a range of 2 to 5 seconds will be seen, with a corresponding

improvement in outage times.

5.6.3 Measuring the outage time

The outage time is defined by measuring the zero throughput period from the SAP®

simstat data.

5.6.4 Methods to generate the failure

Shutdown failure: echo o > /proc/sysrq-trigger Software failure: kill -9 on an actively running db2sysc, or ca-server process Malloc bomb: malloc.pl 999999 (see appendix for the malloc.pl code)

IB adapter failure: ifdown <ib-interface> IB switch failure: Manually unplug the power cable of the IB switch

5.6.5 Behaviors in different failure scenarios

Hard member failure: applications connected to the failed member will be rerouted to

one of the active members as defined in the db2dsdriver.cfg file (see the appendix for the

db2dsdriver.cfg content used in all the HA scenarios). SQL30108 (i.e. “A connection failed but

has been re-established”) will be returned to the applications which have in-flight transactions

running against the failing member, and the applications should handle this error gracefully,

resubmit the work which have not yet been committed and continue to proceed. Notice that

the TRBK benchmark program would actually treat the SQL30108 as errors, so SQL30108 will

show up as errors in the simstat data. The behavior of the benchmark program is to continue

the remaining of the 15 steps loop -- the first remaining step will get SQL30108, saying the

connection has been rerouted. And the remaining of the 15 steps will get SAP® logical errors.

Once the 15-step loop ended and a new loop started, the applications will stop generating

errors.

On the DB2 pureScale® server side, all the in-flight transactions running against the

failing member will be rolled back and resources (e.g. locks) held by the failing member will be

released by member crash recovery which will be done on one of the live members. Once the

machine that failed comes back, it will rejoin the cluster. Connections which got rerouted to

other live members due to the hard member failure will get rerouted back to the newly joined

member.

Soft member failure: The behavior of soft member failure will be the same as hard

member failure except DB2 will be able to restart the failing member locally (instead of one of

the live members) and perform member crash recovery. Once member crash recovery is

completed, the member will be able to start serving connections immediately. Connections

which previously got rerouted to other live members due to the soft kill failure will get rerouted

back.



DB2 PS SAP® TRBK

53/168

Primary/Secondary CF failure: CF failure will be transparent to the applications, i.e.

simstat data will show no errors returned. In the case of primary CF failure, the secondary CF

will take over as the primary CF in a short period of time. In the case of secondary CF failure, the

primary CF will stay as the primary. Once the failed CF comes back online, reintegration and

catchup will occur and the CF will be marked as peer and will act as the secondary CF.

Malloc bomb: This is to simulate a machine to become unresponsive due to all the

physical memory and paging space being used up. A program (malloc.pl) listed in the appendix

was used to simulate this situation. In this PoC a TSA policy was set on the DB2 pureScale®

servers such that when 90% of the physical memory is used, a reboot will be triggered by TSA.

This is to avoid the situation when the machine becomes unresponsive and processes starting to

get killed randomly by the OS due to running out of memory. Once the machine is rebooted by

TSA, the behavior of the clients will be the same as a member failure scenario with clients

getting rerouted to the active members.

IB Switch failure: Powering off one of the IB switches will result in (soft) killing of half

the members and a CF (in our case, they are member 1, member 3, and the secondary CF)

simultaneously – essentially half of the cluster will be lost. Member failovers will occur (i.e.

member 1 failover to db2ps-member2, member 3 failover to db2ps-member0). Applications

connected to members 1 and 3 will get rerouted accordingly.

5.6.6 Reduce the client reroute time

The following parameters were adjusted on the client side to reduce the client reroute

time:

db2dsdriver.cfg:

<parameter name="keepAliveTimeout" value="5"/>

/etc/sysctl.conf:

net.ipv4.tcp_retries2 = 3

net.ipv4.tcp_syn_retries=2

5.6.7 Reduce the number of SAP® errors for read only transactions

A new DBSL driver was provided by SAP® to handle the SQL30108 gracefully on the

DBSL level without propagating the errors back to the user level for read only

transactions.



DB2 PS SAP® TRBK

54/168

6 Disaster Recovery tests

6.1 Layout change

In order to execute the various DR test cases identified, the existing equipment was divided into

2 sites, each with 2 DB2 pureScale® members and 1 CF, as described the following architecture

diagram.

6.2 Results

The DR tests performed were defined from customer requirements for the Proof of Concept. For

the main disaster tests, the workload was specified to be 50% of the performance target for the

Night run (approximately 12m accounts per hour. For the Storage Interconnect interruption, the

workload was defined to be the Day run running at 50% of the performance target

(approximately 20m payitems/hour). For the QReplication test (maintenance of a logical copy 15



DB2 PS SAP® TRBK

55/168

minutes “behind” the primary system), there was no requirement to use the logical copy to

restart the primary, and so this test was not done. However, the logical copy was consistent, and

could have been used for restart.

All the tests were completed successfully. The following table summarizes the tests and results

with more detailed descriptions in the later sections of this chapter.

6.3 Site Failure – Storage replication

The objective of this test was to show that the TRBK Night run could be restarted and completed

on the secondary site in the event of a complete failure of the primary site.

Methodology

• DS8800 Storage replication is in operation between the DS8800’s on the primary and

secondary sites

• TRBK Night executing on the primary site, with workload of approximately 17.5m

accounts/hour

• Connection to the ADVA device physically severed

• Consistency copy taken, primary workload continues until manually terminated

• Database started on secondary site

• Group crash recovery processed – 10 minutes

• SAP® started on secondary application servers

• Previous in-progress run terminated using RBANK_DELETE_RUNS



DB2 PS SAP® TRBK

56/168

• New run started, and run to completion

• Rate 18m accounts/hour, slightly faster due to quicker completion of already

processed accounts

6.3.1 Site Failure Implementation

Primary site : Primary SAP® transaction banking running normally

Disaster recovery site: Devices are down except storage devices receiving data replication

(DS8800 Metro Mirror)

6.3.2 Disaster recovery mechanism

DB2 storage devices are replicated to the Recovery site using DS8000 Metro Mirror.

(Tivoli Storage productivity Center for replication is used to manage the advanced copy

services).

Configuration for DR

l ll l

Storage replication

l

DB : D3D DB : D3D

Member0 Member1 cf0 Member2 Member3 cf1

QcQ

l

m

m

m

m

SAP®

50 51 54 52 53 55

SAP®

DS8800

Storage

10.7.21.0

10.1.2.0 10.1.21.0

10.1.3.0 10.1.3.0

Latency

Latency

10.9.9.0 10.9.9.0

DS8800

Admin network attached to all pieces

DB : D3D

fenced



DB2 PS SAP® TRBK

57/168

6.3.3 High level process of recovery.

� DR Simulation

• SAP® TRBK night run workload running on primary site

• To simulate loss of primary site, ADVA (DWDM) will be switched off.

� RPO measurement

Number of processed accounts versus targeted accounts (measurement of accounts that were

not processed). The test specification required RPO=0. Running the restarted Night run to

completion demonstrated that RPO=0 was achieved.

� Process to restart on DR site

• Recover metro Mirror session to re-enable access to target devices ( Using TPCR Web

GUI)

• Replicated devices are re-mapped to gpfs devices (see procedure in appendix )

• gpfs started

• DB2 database recataloged on dbpath

• DB2 instance started

• DB2 crash recovery started

• SAP® restarted

• Cleanup of SAP® tables

• Restart on TRBK night run

� Prerequisites on DR site

Disaster recovery site must have DB2 pureScale® installed at same level as on primary site,

DS8000 Metro Mirror

• DB2 instance created ( see procedure in appendix )

• DB2 path created ( see procedure in appendix )

• DS8000 metro mirror activated and source-target volume pairs are fully synchronized

(i.e. in “Prepared” state)

• GPFS meta-data definitions must be imported from primary cluster (see procedure in

appendix).

o Note: This is a one-time process unless NDS definitions got changed in the

primary site ( more details in section 2.2.2 Exporting the filesystem definitions )

• Imported GPFS meta-data is masked to avoid GPFS daemons from accessing those

metro-mirror replicated devices until we fail-over to the DR site ( see procedure in

appendix )



DB2 PS SAP® TRBK

58/168

� List of storage devices replicated

Running df command should return the following information

The replicated are the ones which are grey shaded.

Filesystem 1K-blocks Used Available Use% Mounted

/dev/sda3 138195432 29875488 101299944 23% / Internal disk

devtmpfs 132266056 736 132265320 1% /dev Internal disk

tmpfs 132266056 0 132266056 0% /dev/shm Internal disk

/dev/sda1 72102 28760 43342 40% /boot/efi Internal disk

/dev/d3d 2147483648 1777664 2145705984 1% /db2/D3D DB2 home

/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413

DB2 shared instance

/dev/log0 2147483648 1930240 2145553408 1% /db2/D3D_log0 DB2 logs member0


/dev/SAP® data1 1073741824 346840064 726901760 33% /db2/D3D_SAP® data1

DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container



DB2 PS SAP® TRBK

59/168


DB2 container

These devices are mapped to GPFS nsd ( 1 to 1 )

List of gpfs nsd

DB2PS-member0:~ # mmlsnsd

File system Disk name NSD servers

--------------------------------------------------------------------------

d3d gpfs2nsd (directly attached) DB2 base directory Replicated

db2fs1 gpfs1nsd (directly attached) DB2 shared instance Not replicated

log0 gpfs22nsd (directly attached) Logs for member0 Replicated


SAP®

data1 gpfs16nsd (directly attached) DB2 container

Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP® gpfs7nsd (directly attached) DB2 container Replicated



DB2 PS SAP® TRBK

60/168

data8

SAP®


Replicated

6.3.4 GPFS configuration

GPFS configuration must be propagated to the remote site.

� Prepare remote nodes file

This file will be used as input for the mmfsctl command .

add the remote nodes into this file .

vi remotenodefile

DB2PS-member2

DB2PS-member3

DB2PS-cf1

� Propagating GPFS filesystem definitions to the DR cluster

Prerequisites:

• Configure password-less rsh for root on machines between the primary site and the

target site.

• All source-target volume pairs in the Metro-Mirror session have reached “Prepared”

State (fully synchronized).

For each filesystem

Run the following command

/usr/lpp/mmfs/bin/mmfsctl gpfs_filesystem_name syncFSconfig -n remotenodefile

Example:

/usr/lpp/mmfs/bin/mmfsctl SAP® data9 syncFSconfig - n remotenodefile

Note : whenever a GPFS NSDfilesystem definition changes in the primary site, this command

must be run again .

All commands are prepared in the /root/DR/fs_sync.sh on DB2PS-member0.

This script must be run as root.



DB2 PS SAP® TRBK

61/168

o fs_sync.sh DB2PS-member0:~/DR # cat fs_sync.sh mmfsctl d3d syncFSconfig -n remotenodefile mmfsctl log0 syncFSconfig -n remotenodefile mmfsctl log1 syncFSconfig -n remotenodefile mmfsctl log2 syncFSconfig -n remotenodefile mmfsctl log3 syncFSconfig -n remotenodefile mmfsctl SAP® data1 syncFSconfig -n remotenodefile mmfsctl SAP® data10 syncFSconfig -n remotenodefile mmfsctl SAP® data11 syncFSconfig -n remotenodefile mmfsctl SAP® data12 syncFSconfig -n remotenodefile mmfsctl SAP® data13 syncFSconfig -n remotenodefile mmfsctl SAP® data14 syncFSconfig -n remotenodefile mmfsctl SAP® data15 syncFSconfig -n remotenodefile mmfsctl SAP® data16 syncFSconfig -n remotenodefile mmfsctl SAP® data2 syncFSconfig -n remotenodefile mmfsctl SAP® data3 syncFSconfig -n remotenodefile mmfsctl SAP® data4 syncFSconfig -n remotenodefile mmfsctl SAP® data5 syncFSconfig -n remotenodefile mmfsctl SAP® data6 syncFSconfig -n remotenodefile mmfsctl SAP® data7 syncFSconfig -n remotenodefile mmfsctl SAP® data8 syncFSconfig -n remotenodefile mmfsctl SAP® data9 syncFSconfig -n remotenodefile

o Execution of fs_sync.sh

DB2PS-member0:~/DR # ./fs_sync.sh > fs_sync.out 2>&1

Survey of output in a different window

DB2PS-member0:~/DR # tail -f fs_sync.out mmexportfs: Processing file system d3d ... mmfsctl: Importing file system information into th e target cluster on node DB2PS-member2 . . . mmimportfs: Processing file system d3d ... mmimportfs: Processing disk gpfs2nsd mmimportfs: Committing the changes ... mmimportfs: The following file systems were success fully imported: …/…

Note:

• We propagated the GPFS meta-data into the Secondary Clusters as part of the initial

configuration.

• Since NSDs on the primary site remain unchanged from the time we did the initial GPFS

DR configuration on the standby side, there was no need to re-sync the NSD meta-data

again. Hence we skipped this step during the DR fail-over testing.



DB2 PS SAP® TRBK

62/168

• When NSD meta-data is propagated to the DR cluster via mmfsctl .. sysFSconfig , all the block devices that map to the meta-data definitions (which will

be imported from the primary site via mmfsctl ) must be “visible” to the GPFS

daemons . Hence, in-order to do the meta-data definition we need to ensure that step

2.2.7.2 Remap GPFS nsd is executed before hand.

• At a later time, if there is ever a need to propagate any NSD changes to the DR site, for

example due to a new NSD/GPFS filesystem being added as a storage path to D3D db,

you need to ensure all NSDs are mapped (unmasked) – i.e. execute step 2.2.7.2 Remap

GPFS nsd , before executing mmfsctl .. sysFSconfig ..

• Note that this type of resynchronization of GPFS meta-data between primary and

secondary clusters are required only for those actions that modify the file system

information maintained in the local GPFS configuration data, which includes:

o Additions, removals, and replacements of disks (mmadddisk, mmdeldisk, mmrpldisk )

o Modifications to disk attributes (mmchdisk )

o Changes to the file system's mount point (mmchfs -T )

o Changing the file system device name (mmchfs -W )



DB2 PS SAP® TRBK

63/168

6.3.5 Recovery Procedure on DR Site

Initial status

• Verify GPFS daemons are up

DB2PS-member2:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member2 1* 3 3 active quorum node 2 DB2PS-cf1 1* 3 3 active quorum node 3 DB2PS-member3 1* 3 3 active quorum node

• Uncatalog DB D3D used for Qrep

db2d3d@DB2PS-Member2:~> db2 uncatalog db d3d DB20000I The UNCATALOG DATABASE command completed successfully.

• Filesystems already mounted Filesystem 1K-blocks Used Available Use% Mo unted on /dev/sda3 138195432 21604884 109570548 17% / devtmpfs 132266056 1056 132265000 1% /de v tmpfs 132266056 0 132266056 0% /dev/sh m /dev/sda1 72102 28762 43340 40% /boot/efi /dev/db2fs1 524288000 1528832 522759168 1% /db2sd_20110401115043

Qrep FS still mounted.

db2d3d@DB2PS-cf1:~> df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 138195432 19466016 111709416 15% / devtmpfs 132266056 740 132265316 1% /dev tmpfs 132266056 0 132266056 0% /dev/shm /dev/sda1 72102 28768 43334 40% /boot/efi /dev/db2fs1 524288000 1623040 522664960 1% /db2sd_20110401115043 /dev/qrlog0 524288000 14642176 509645824 3% /QR_log0 /dev/qrlog1 524288000 585728 523702272 1% /QR_log1 /dev/qrdbpath 52428800 1273856 51154944 3% /db2/QR_D3D /dev/qrSAP® data3 524288000 346584064 17770393 6 67% /QR_SAP® data3 /dev/qrSAP® data7 524288000 346584064 17770393 6 67% /QR_SAP® data7 /dev/qrSAP® data2 524288000 345648640 17863936 0 66% /QR_SAP® data2 /dev/qrSAP® data6 524288000 346584064 17770393 6 67% /QR_SAP® data6 /dev/qrSAP® data5 524288000 346584064 17770393 6 67% /QR_SAP® data5 /dev/qrSAP® data8 524288000 346584064 17770393 6 67% /QR_SAP® data8 /dev/qrSAP® data4 524288000 346584064 17770393 6 67% /QR_SAP® data4 /dev/qrSAP® data1 524288000 345648640 17863936 0 66% /QR_SAP® data1 /dev/qrSAP® data14 524288000 346584064 17770393 6 67% /QR_SAP® data14 /dev/qrSAP® data15 524288000 346584064 17770393 6 67% /QR_SAP® data15 /dev/qrSAP® data13 524288000 346584064 17770393 6 67% /QR_SAP® data13 /dev/qrSAP® data12 524288000 345648640 17863936 0 66% /QR_SAP® data12



DB2 PS SAP® TRBK

64/168

/dev/qrSAP® data11 524288000 346584064 17770393 6 67% /QR_SAP® data11 /dev/qrSAP® data10 524288000 346584064 17770393 6 67% /QR_SAP® data10 /dev/qrSAP® data16 524288000 346584064 17770393 6 67% /QR_SAP® data16 /dev/qrSAP® data9 524288000 346584064 17770393 6 67% /QR_SAP® data9

• DB2 installed and configured As user db2d3d check existence of instance

db2d3d@DB2PS-member2:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NET NAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ --- ---- 0 MEMBER STOPPED DB2 PS-member2 DB2PS-member2 NO 0 0 DB2PS- member2-ib0 1 MEMBER STOPPED DB2 PS-member3 DB2PS-member3 NO 0 0 DB2PS- member3-ib0 128 CF STOPPED DB2 PS-cf1 DB2PS-cf1 NO - 0 DB2PS- cf1-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf1 ACTIVE NO NO DB2PS-member3 ACTIVE NO NO DB2PS-member2 ACTIVE NO NO

Check that node state is active.

• Replication

Replication is suspended (loss of PPRC paths)

Recover Metro-Mirror session, prior configuring storage devices.

o start a Remote Desktop Connection to the machine on which TPCR (Tivoli Productivity

Centre for Replication) is installed. Note that TPCR should be not be installed on either

the primary or secondary systems, for obvious reasons.

o On the Desktop there is a shortcut to "Tivoli Storage Productivity Center for

Replication"



DB2 PS SAP® TRBK

65/168

o go to Sessions - there is only 1 session ( DB2PS_PPRC)



DB2 PS SAP® TRBK

66/168

o Select DP2PS_PPRC

o Check that storage replication is in status Normal

Primary Site Failure

� Start TRBK night run and leave it running 1 hour

� Disconnect interconnect fibre between the 2 ADVA

o Check that storage replication is in status Suspended



DB2 PS SAP® TRBK

67/168

� Recover session

Recover H2 (DR site) to make the MM replicated LUNs host-addressible from DR cluster



DB2 PS SAP® TRBK

68/168



DB2 PS SAP® TRBK

69/168

� TPCR Console log

Now devices are ready to be re-mapped to gpfs

� Prepare restart of recovery site

� Check accessibility of storage devices

Multipath –l should return information similar to the following.

Check presence of all replicated devices.

DB2PS-member2:~ # multipath –l SAP® data14 (3600507630bffc09c0000000000001113) dm- 41 IBM,2107900 size=1.0T features='1 queue_if_no_path' hwhandler=' 0' wp=rw `-+- policy='round-robin 0' prio=-1 status=active |- 4:0:0:39 sdao 66:128 active undef running |- 5:0:0:39 sdcl 69:144 active undef running `- 6:0:0:39 sdei 128:160 active undef running

• Remap (Unmask) storage replicated GPFS nsds

Make Metro-Mirror replicated GPFS NSDs visible to the DR cluster and mount those replicated

file systems

- On DB2PS-member2 as root, issue /root/scripts/enable_dr_nsd.sh

See contents of enable_dr_nsd.sh in appendix.

DB2PS-member2:~ # /root/scripts/enable_dr_nsd.sh sda generic sda1 generic sda2 generic …/…



DB2 PS SAP® TRBK

70/168

GPFS NSDs visible to the cluster ... Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 /dev/dm-37 DB2 PS-member2.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-26 DB2 PS-member2.localdomain …/… gpfs8nsd 0A0102324D80C475 /dev/dm-40 DB2 PS-member2.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-27 DB2 PS-member2.localdomain Tue Apr 26 19:43:35 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:37 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:38 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:44 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:47 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:53 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:43:55 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:00 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:02 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:04 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:11 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:12 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:14 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:21 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:23 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:25 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:32 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:34 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:36 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:43 CEST 2011: mmmount: Mounting fi le systems ... Tue Apr 26 19:44:45 CEST 2011: mmmount: Mounting fi le systems ... Mount status of all GPFS filesystems defined in the cluster ... File system d3d is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system db2fs1 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system log0 is mounted on 3 nodes: 10.1.2.55 DB2PS-cf1 10.1.2.52 DB2PS-member2 10.1.2.53 DB2PS-member3



DB2 PS SAP® TRBK

71/168

…/… File system SAP® data9 is mounted on 3 nodes: 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 10.1.2.52 DB2PS-member2

Issuing mmlsnsd –m should now return all the MM replicated GPFS NSDs on secondary/DR site

DB2PS-member2:~/scripts # mmlsnsd -m Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 /dev/dm-37 DB2 PS-member2.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-26 DB2 PS-member2.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-36 DB2 PS-member2.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-25 DB2 PS-member2.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-35 DB2 PS-member2.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-42 DB2 PS-member2.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-24 DB2 PS-member2.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-31 DB2 PS-member2.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-41 DB2 PS-member2.localdomain gpfs19nsd 0A0102324D80C61E /dev/dm-44 DB2 PS-member2.localdomain gpfs1nsd 0A0102344D959FDD /dev/dm-8 DB2 PS-member2.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-33 DB2 PS-member2.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-43 DB2 PS-member2.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-32 DB2 PS-member2.localdomain gpfs23nsd 0A0102344D95E0ED /dev/dm-1 DB2 PS-member2.localdomain gpfs24nsd 0A0102344D95E0FF /dev/dm-12 DB2 PS-member2.localdomain gpfs25nsd 0A0102344D95E112 /dev/dm-2 DB2 PS-member2.localdomain gpfs26nsd 0A0102344D95E126 /dev/dm-13 DB2 PS-member2.localdomain gpfs27nsd 0A0102344D95E139 /dev/dm-3 DB2 PS-member2.localdomain gpfs28nsd 0A0102344D95E150 /dev/dm-14 DB2 PS-member2.localdomain gpfs29nsd 0A0102344D95E166 /dev/dm-4 DB2 PS-member2.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-34 DB2 PS-member2.localdomain gpfs30nsd 0A0102344D95E180 /dev/dm-15 DB2 PS-member2.localdomain gpfs31nsd 0A0102344D95E19A /dev/dm-10 DB2 PS-member2.localdomain gpfs33nsd 0A0102344D95E1D3 /dev/dm-5 DB2 PS-member2.localdomain gpfs34nsd 0A0102344D95E1F1 /dev/dm-17 DB2 PS-member2.localdomain gpfs35nsd 0A0102344D95E210 /dev/dm-6 DB2 PS-member2.localdomain gpfs36nsd 0A0102344D95E230 /dev/dm-18 DB2 PS-member2.localdomain gpfs37nsd 0A0102344D95E253 /dev/dm-7 DB2 PS-member2.localdomain gpfs39nsd 0A0102344D95E29C /dev/dm-9 DB2 PS-member2.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-30 DB2 PS-member2.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-39 DB2 PS-member2.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-29 DB2 PS-member2.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-28 DB2 PS-member2.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-38 DB2 PS-member2.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-40 DB2 PS-member2.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-27 DB2 PS-member2.localdomain



DB2 PS SAP® TRBK

72/168

Issuing mmlsmount all_local will display all the mounted gfs filesystems, which should

now show MM replicated mounts as well.

DB2PS-member2:~/scripts # mmlsmount all_local File system d3d is mounted on 3 nodes. File system db2fs1 is mounted on 3 nodes. File system log0 is mounted on 3 nodes. File system log1 is mounted on 3 nodes. File system log2 is mounted on 3 nodes. File system log3 is mounted on 3 nodes. File system qrdbpath is mounted on 3 nodes. File system qrlog0 is mounted on 3 nodes. File system qrlog1 is mounted on 3 nodes. File system qrSAP® data1 is mounted on 3 nodes. File system qrSAP® data10 is mounted on 3 nodes. File system qrSAP® data11 is mounted on 3 nodes. File system qrSAP® data12 is mounted on 3 nodes. File system qrSAP® data13 is mounted on 3 nodes. File system qrSAP® data14 is mounted on 3 nodes. File system qrSAP® data15 is mounted on 3 nodes. File system qrSAP® data16 is mounted on 3 nodes. File system qrSAP® data2 is mounted on 3 nodes. File system qrSAP® data3 is mounted on 3 nodes. File system qrSAP® data4 is mounted on 3 nodes. File system qrSAP® data5 is mounted on 3 nodes. File system qrSAP® data6 is mounted on 3 nodes. File system qrSAP® data7 is mounted on 3 nodes. File system qrSAP® data8 is mounted on 3 nodes. File system qrSAP® data9 is mounted on 3 nodes. File system SAP® data1 is mounted on 3 nodes. File system SAP® data10 is mounted on 3 nodes. File system SAP® data11 is mounted on 3 nodes. File system SAP® data12 is mounted on 3 nodes. File system SAP® data13 is mounted on 3 nodes. File system SAP® data14 is mounted on 3 nodes. File system SAP® data15 is mounted on 3 nodes. File system SAP® data16 is mounted on 3 nodes. File system SAP® data2 is mounted on 3 nodes. File system SAP® data3 is mounted on 3 nodes. File system SAP® data4 is mounted on 3 nodes. File system SAP® data5 is mounted on 3 nodes. File system SAP® data6 is mounted on 3 nodes. File system SAP® data7 is mounted on 3 nodes. File system SAP® data8 is mounted on 3 nodes. File system SAP® data9 is mounted on 3 nodes.

• Restart database

• Catalog MM replicated copy of D3D database

o On DB2PS-member2 as DB2 instance ID (db2d3d), catalog D3D db

db2d3d@DB2PS-Member2:~> db2 catalog db D3D on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.

• Start instance db2d3d

On DB2PS-member2 as DB2 instance ID (db2d3d), start DB2

db2d3d@DB2PS-member2:~> db2start 04/26/2011 19:51:56 0 0 SQL1063N DB2START processing was successful.



DB2 PS SAP® TRBK

73/168

04/26/2011 19:51:56 1 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful.

� Verify status of instance db2d3d

db2d3d@DB2PS-member2:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED DB2 PS-member2 DB2PS-member2 NO 0 0 DB2PS-member2-ib0 1 MEMBER STARTED DB2 PS-member3 DB2PS-member3 NO 0 0 DB2PS-member3-ib0 128 CF PRIMARY DB2 PS-cf1 DB2PS-cf1 NO - 0 DB2PS-cf1-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf1 ACTIVE NO NO DB2PS-member3 ACTIVE NO NO DB2PS-member2 ACTIVE NO NO

• Restart database

On DB2PS-member2 as DB2 instance ID (db2d3d),

o db2 restart db D3D Check the status of the Group Crash Recovery (GCR) by

db2d3d@DB2PS-member2:~/sqllib/db2dump> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:04:40 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9DC81A8CC Current LRI 00000000000000010000000003324C0 4000003E9DC81A8CC Current LSO 4432765186522 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1303840522) Tue Apr 26 19:55:2 2 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000205E4F8A8 1 Forward Tue Apr 26 19:55:22 182194440 bytes 1735886880 bytes 0x0000000205E4FA30 2 Backward NotStarted 0 bytes 1735886880 bytes Check DB2 diag log to verify GCR completed successfully

2011-04-26-20.06.04.718193+120 E215808E450 LEVEL: Info PID : 15930 TID : 4800789452569 6 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11042 6175157



DB2 PS SAP® TRBK

74/168

EDUID : 32 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.

2011-04-26-20.06.04.718801+120 I216259E462 LEVEL: Info PID : 15930 TID : 4800789452569 6 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11042 6175157 EDUID : 32 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3170 DATA #1 : <preformatted> Crash recovery completed. Next LSN is 000003E9DCB12 427.

Re-Start the SAP® run on the DR site

• Restart the SAP® application servers

• Use transaction SM37 to clean up the status of the batch jobs that were running on the

primary

• Run report RBANK_DELETE_RUN to reset the status of the interrupted run

• Start a new TRBK Reconciliation (Night) run in the normal way

• Elapsed time from the time the “disaster” (severing of the link between the primary and

secondary systems) until the start of the new Night run was approximately 40 minutes

• The new Night run reprocesses all accounts. For those accounts already processed by

the original run, the workload is minimal, leading to a quicker completion of the new

Night run.

6.3.6 Going back to the production configuration

Initial status All the GPFS file systems are mounted, i.e. both Q-replicated filesystems and Metro-Mirror replicated filesystems are mounted on the DR GPFS cluster.

Restore the replication link Power on the ADVA ( DWDM ) equipment and validate the FC links used for the Metro-Mirror copy session is functional. You can use DSCLI to query the Metro-Mirror copy-session path status using:

lspprcpath -dev storage_image_ID -l Source_LSS_ID Failed reason should indicate System Reserved Path. See following IBM System Storage Information Centre link for more details on the command and the path status codes:

http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/topic/com.ibm.storage.ssic.help.doc/f2c_clilspprcpath_1kz8kv.html

For, example in our test configuration, running DSCLI from TPCR desktop:



DB2 PS SAP® TRBK

75/168

Note : If the link is not online, when you try to re-start the MM session it will fail.

Reset DB2 and GPFS back to production configuration � Update the DB2 Configuration on DR site

Before we switch over to the production configuration: o Update the GPFS Configuration on DR site Stop SAP® run o Deactivate D3D database:

o db2 terminate; force applications all; db2 deactiva te db D3D o Uncatalog D3D database:

o db2 uncatalog db D3D

� Before we switch over to the production configuration, unmount the Metro-Mirror

replicated filesystems and unmap (mask) those replicated NSDs on DR cluster.

- On DB2PS-member2 as root, issue /root/scripts/disable_dr_nsd.sh DB2PS-member2:~/scripts # ./disable_dr_nsd.sh Wed Apr 27 16:01:31 CEST 2011: mmumount: Unmounting file systems ... Wed Apr 27 16:01:32 CEST 2011: mmumount: Unmounting file systems ... Wed Apr 27 16:01:33 CEST 2011: mmumount: Unmounting file systems ... …/… Wed Apr 27 16:01:41 CEST 2011: mmumount: Unmounting file systems ... Mount status of all GPFS filesystems defined in the cluster ... File system d3d is not mounted. File system db2fs1 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system log0 is not mounted. File system log1 is not mounted. File system log2 is not mounted. File system log3 is not mounted.



DB2 PS SAP® TRBK

76/168

File system qrdbpath is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 …/… File system qrSAP® data9 is mounted on 3 nodes: 10.1.2.52 DB2PS-member2 10.1.2.55 DB2PS-cf1 10.1.2.53 DB2PS-member3 File system SAP® data1 is not mounted. File system SAP® data10 is not mounted. …/… dm-0 dmm dm-1 dmm dm-10 dmm dm-11 dmm ../… GPFS NSDs visible to the cluster ... Disk name NSD volume ID Device Nod e name Remarks --------------------------------------------------- ------------------------------------ gpfs1001nsd 0A0102344D963C72 /dev/dm-0 DB2 PS-member2.localdomain gpfs1002nsd 0A0102344D963C73 /dev/dm-11 DB2 PS-member2.localdomain gpfs1006nsd 0A0102344D9656A6 /dev/dm-16 DB2 PS-member2.localdomain gpfs1007nsd 0A0102344D9656A7 /dev/dm-19 DB2 PS-member2.localdomain gpfs10nsd 0A0102324D80C4A2 - DB2 PS-member2.localdomain (not found) directly attached gpfs11nsd 0A0102324D80C4BC - DB2 PS-member2.localdomain (not found) directly attached gpfs12nsd 0A0102324D80C4D9 - DB2 PS-member2.localdomain (not found) directly attached gpfs13nsd 0A0102324D80C4F5 - DB2 PS-member2.localdomain (not found) directly attached gpfs14nsd 0A0102324D80C512 - DB2 PS-member2.localdomain (not found) directly attached gpfs15nsd 0A0102324D80C530 - DB2 PS-member2.localdomain (not found) directly attached gpfs16nsd 0A0102324D80C54F - DB2 PS-member2.localdomain (not found) directly attached gpfs17nsd 0A0102324D80C570 - DB2 PS-member2.localdomain (not found) directly attached gpfs18nsd 0A0102324D80C594 - DB2 PS-member2.localdomain (not found) directly attached gpfs19nsd 0A0102324D80C61E - DB2 PS-member2.localdomain (not found) directly attached gpfs1nsd 0A0102344D959FDD /dev/dm-8 DB2 PS-member2.localdomain gpfs20nsd 0A0102324D80C641 - DB2 PS-member2.localdomain (not found) directly attached gpfs21nsd 0A0102324D80C665 - DB2 PS-member2.localdomain (not found) directly attached gpfs22nsd 0A0102324D80C68B - DB2 PS-member2.localdomain (not found) directly attached gpfs23nsd 0A0102344D95E0ED /dev/dm-1 DB2 PS-member2.localdomain gpfs24nsd 0A0102344D95E0FF /dev/dm-12 DB2 PS-member2.localdomain gpfs25nsd 0A0102344D95E112 /dev/dm-2 DB2 PS-member2.localdomain



DB2 PS SAP® TRBK

77/168

gpfs26nsd 0A0102344D95E126 /dev/dm-13 DB2 PS-member2.localdomain gpfs27nsd 0A0102344D95E139 /dev/dm-3 DB2 PS-member2.localdomain gpfs28nsd 0A0102344D95E150 /dev/dm-14 DB2 PS-member2.localdomain gpfs29nsd 0A0102344D95E166 /dev/dm-4 DB2 PS-member2.localdomain gpfs2nsd 0A0102324D80C2BA - DB2 PS-member2.localdomain (not found) directly attached gpfs30nsd 0A0102344D95E180 /dev/dm-15 DB2 PS-member2.localdomain gpfs31nsd 0A0102344D95E19A /dev/dm-10 DB2 PS-member2.localdomain gpfs33nsd 0A0102344D95E1D3 /dev/dm-5 DB2 PS-member2.localdomain gpfs34nsd 0A0102344D95E1F1 /dev/dm-17 DB2 PS-member2.localdomain gpfs35nsd 0A0102344D95E210 /dev/dm-6 DB2 PS-member2.localdomain gpfs36nsd 0A0102344D95E230 /dev/dm-18 DB2 PS-member2.localdomain gpfs37nsd 0A0102344D95E253 /dev/dm-7 DB2 PS-member2.localdomain gpfs39nsd 0A0102344D95E29C /dev/dm-9 DB2 PS-member2.localdomain gpfs3nsd 0A0102324D80C412 - DB2 PS-member2.localdomain (not found) directly attached gpfs4nsd 0A0102324D80C424 - DB2 PS-member2.localdomain (not found) directly attached gpfs5nsd 0A0102324D80C438 - DB2 PS-member2.localdomain (not found) directly attached gpfs6nsd 0A0102324D80C449 - DB2 PS-member2.localdomain (not found) directly attached gpfs7nsd 0A0102324D80C45F - DB2 PS-member2.localdomain (not found) directly attached gpfs8nsd 0A0102324D80C475 - DB2 PS-member2.localdomain (not found) directly attached gpfs9nsd 0A0102324D80C489 - DB2 PS-member2.localdomain (not found) directly attached

Re-start the Metro-Mirror replication session (DP2_PS) from Primary (H1) to DR site (H2) Login to the TPCR Web GUI and Start the replication session in the original direction:

o select session DP2PS_PPRC -> Start H1 ---> H2



DB2 PS SAP® TRBK

78/168



DB2 PS SAP® TRBK

79/168



DB2 PS SAP® TRBK

80/168



DB2 PS SAP® TRBK

81/168

o Once the MM session complete synchronizing all the copy-pairs defined in the session, the replication Session state will change from “Preparing” to “Prepared ” and Copy progress will be 100%.



DB2 PS SAP® TRBK

82/168

For details about the the script used to re-map the replicated devices back to the NSD meta-

data that was imported as part of DR site setup process, please refer to the appendix section of

this document.

Same for the script used to mask those Metro-Mirror replicated NSD definitions from the DR site

GPFS cluster by creating a nsd devices user-exit and initiating mmdevdiscover to rescan the

devices from GPFS.



DB2 PS SAP® TRBK

83/168

6.4 Storage interconnect failure and catchup

The objective of this test was to measure the effect on the TRBK Day run of a storage

interconnect failure and the further impact of the storage replication resynchronization

Methodology

• DS8800 Storage replication is in operation between the DS8800’s on the primary and

secondary sites

• TRBK Day run executing on the primary site, with workload of approximately 20m

payitems/hour

• Connection to the ADVA device physically severed

• Consistency copy taken, primary workload continues

• After 10 minutes, the connection is manually reestablished

• Time measured until TPCR shows storage replication shows catchup

6.4.1 Implementation

Primary site: Primary SAP® transaction banking. The TRBK day run is running.

Disaster recovery site: Storage mirroring in Synchronous mode (DS8800 Metro Mirror )



DB2 PS SAP® TRBK

84/168

Configuration for Storage Interconnect

l ll l

Storage replication

l

DB : D3D DB : D3D


QcQ

l

m

m

m

m

SAP®

50 51 54 52 53 55

SAP®

DS8800

Storage

10.7.21.0

10.1.2.0 10.1.21.0

10.1.3.0 10.1.3.0

Latency

Latency

10.9.9.0 10.9.9.0

DS8800


DB : D3D

fenced



DB2 PS SAP® TRBK

85/168

Configuration on Production site

ADVA FSP 3000

ADVA Optical Networking’s FSP 3000 is a scalable, modular optical transport solution based on

Dense Wavelength Division Multiplexing (DWDM) technology, providing both Optical and

Ethernet end-to-end connectivity (from the access to the metro and on to long haul).

The DWDM technology offers almost unlimited bandwidth (up to 120 wavelengths, each

carrying 10G) and it also enables native transport of all relevant protocols in the data center

space: Ethernet, ESCON, Fibre Channel, InfiniBand.

The systems were configured so that all communication between the 2 sites passed through the

ADVA switches.

ADVA FSP 3000 product webpage: http://www.advaoptical.com/en/products/scalable-optical-

transport/fsp-3000.aspx



DB2 PS SAP® TRBK

86/168

6.4.2 Storage Interconnect test mechanism

DB2 storage devices are replicated toward Recovery site using DS8000 metro mirror.

(Tivoli Storage Productivity Center for Replication is used to manage the advanced copy

services).

Once the link for the storage interconnect is lost, monitor the performance and the time to re-

sync.

6.4.3 High level process

� Procedure

• SAP® TRBK benchmark runs with the day load at 50% of performation target (20m

payitems/hour)

• Once the load is in the high load phase, remove the link between the two ADVA

equipment to stop the mirror for 10 minutes

• After the 10 minutes reconnect the link between the two ADVA equipment and

restart the metro-mirror

• For completeness, we also did a Storage Replication Disconnect/Reconnect test with

the Night run running at 12m accounts/hour

• Success Criteria

• Time to re-sync, (no more than 30 minutes, check this with business and experts

confirming the feasibility to re-sync in this time window given the amount of postings)

• No downtime, no aborts, no coredumps, no timeouts from the application side

• After re-sync, check that the secondary site is in sync without any data loss

Prerequisites

Storage mirror in Synchronous mode – all copy sets in the Metro Mirror consistency group

are in state “Prepared”

List of storage devices replicated

Running df command should return the following information



DB2 PS SAP® TRBK

87/168

The replicated are highlighted in grey.







/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413

DB2 shared instance



/dev/SAP® data1 1073741824 346840064 726901760 33% /db2/D3D_SAP® data1 DB2 container


















DB2 PS SAP® TRBK

88/168



DB2 PS SAP® TRBK

89/168

6.4.4 Storage Interconnect test results

Step 1

The TRBK day run is running and the Metro Mirror session is synchronized (prepared)

Step 2

We remove the link between the two ADVA equipments by physically disconnecting the cable

between the two switches.



DB2 PS SAP® TRBK

90/168

The Metro Mirror session is now in Suspended state.



DB2 PS SAP® TRBK

91/168

Step 3

The TRBK day run continues to run and after 10 minutes 524GB of data has been modified



DB2 PS SAP® TRBK

92/168

Step 4

The fiber is reconnected and the Metro Mirror session re-started (start H1-H2).

Step 5

The session if full synchronized 10 minutes after the fiber reconnection (and re-start of the

Metro Mirror session).



DB2 PS SAP® TRBK

93/168

TPCR Console Error Apr 14, 2011 5:12:19 PM : Server : IWNR2052E : The pair in session DP2PS _PPRC for copy set

DS8000:2107.WR541:VOL:1107 with source DS8000:2107.WR541:VOL:1107(SAP® data8) and target DS8000:2107.WR821:VOL:1107(SAP® data8_dr) in role pair H1-H2 was suspended with a reason code of 8 and is consistent.

normal Apr 14, 2011 5:12:30 PM : Server : IWNR6002I : Suspending all pairs in role pair H1-H2 ... normal Apr 14, 2011 5:12:30 PM : Server : IWNR1041I : The command freeze was successfully issued to all pairs

under role pair H1-H2 for session DP2PS _PPRC. normal Apr 14, 2011 5:12:30 PM : Server : IWNR6020I : Releasing I/O for all pairs in role pair H1-H2... normal Apr 14, 2011 5:12:30 PM : Server : IWNR1041I : The command thaw was successfully issued to all pairs

under role pair H1-H2 for session DP2PS _PPRC. normal Apr 14, 2011 5:12:30 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Prepared state to

the Suspended state. normal Apr 14, 2011 5:12:30 PM : Server : IWNR6019I : Verifying the consistency of role pair H1-H2... normal Apr 14, 2011 5:26:24 PM : administrator : IWNR1028I : The command Start H1->H2 in session DP2PS

_PPRC has been run. normal Apr 14, 2011 5:26:24 PM : administrator : IWNR6000I : Starting all pairs in role pair H1-H2 ... normal Apr 14, 2011 5:26:24 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Suspended state

to the Preparing state. normal Apr 14, 2011 5:26:25 PM : administrator : IWNR6006I : Waiting for all pairs in role pair H1-H2 to reach a

state of Prepared ... normal Apr 14, 2011 5:26:25 PM : administrator : IWNR1026I : Command Start H1->H2 in session DP2PS _PPRC

has completed successfully.



DB2 PS SAP® TRBK

94/168

warning Apr 14, 2011 5:26:25 PM : Server : IWNR1959W : Session DP2PS _PPRC has changed from Severe status to Warning status.

normal Apr 14, 2011 5:36:28 PM : Server : IWNR1950I : Session DP2PS _PPRC changed from the Preparing state to the Prepared state.

Test Results

• Time to resynchronize: 10 minutes

• No downtime, no aborts, no coredumps, no timeouts from the application side

• After re-sync no data loss in secondary site

• Day run disconnect time – 10 minutes

• Catchup time – 9 minutes

• Night run disconnect time – 30 minutes

• Catchup time – 10 minutes

6.5 Queue Replication to logical copy

The requirement for this test was to use Queue Replication software to maintain a logical copy

of the database that represented the state 15 minutes before the current primary state. Storage

replication propagates all changes to the primary database. The existence of a delayed logical

copy provides a mechanism to cope with and recovery from mass changes to the primary

system introduced in error.

Software levels (MQ): Websphere MQ 7.0.1-3

Queue Replication software works by monitoring the DB2 transaction log with a Capture

process, and inserting the harvested information about committed transactions to an MQ

queue. MQ causes this data to be moved to the secondary system, where it is read from the MQ



DB2 PS SAP® TRBK

95/168

queue by the Apply process, and inserted/updated to the logical copy database using regular

SQL statements.

DB2 pureScale® implements a new Log Read API interface that provides a merged view of the

transaction logs of all members. This allows the Capture process to read transaction log records

in commit order (across all members), and maintains the logical consistency of the updates to

the target database.

The tables that should be maintained by Queue Replication in the logical copy are registered.

For our tests, we registered all tables referenced during our target TRBK workloads – about 2000

in total.

6.5.1 Methodology

To perform the Queue Replication test, we:

• created a third database (QREP) on the secondary system (the other 2 being the primary

database, and the storage replicated secondary database).

• The QREP database was synchronized with the primary database using a common

restore image

• Queue Replication was initialized so that all changes to the primary database would be

replicated to the logical copy

• The Night run was executed at a rate of 12m accounts per hour

• The Night run completes

Log

Member0

Member1

Log0

Log1

QRep Capture

MQ Queues

Site 1

Site 2

Membe

Membe DB

MQ Queues

QRep Apply

DB



DB2 PS SAP® TRBK

96/168

6.5.2 Results

To achieve the necessary throughput we defined 2 queues (ASN1 and ASN2). Having 2 queues

implies also having 2 MQ queues, 2 Capture processes on the source system and 2 Apply

processes on the target system. As the workload starts, the Capture processes read transactions

from the DB2 transaction log via the Read Log API, and inserting these to the MQ queues. MQ

moves the messages to the target system. Due to the requirement to have a delay on the logical

copy, the Apply processes on the target do not start to remove messages from the MQ queues

for 15 minutes. This means that the MQ queues build to a considerable size (approximately

90GB). Once the Apply processes start their work, the MQ queue size stabilized for the

remainder of the run.

Qdepth (MQ messages)

020000400006000080000

100000120000140000160000

23.3

6.56

23.5

8.31

00.2

0.07

00.4

1.42

01.0

3.17

01.2

4.52

01.4

6.27

02.0

8.02

02.2

9.37

02.5

1.13

03.1

2.48

03.3

4.23

03.5

5.58

04.1

7.33

04.3

9.08

05.0

0.43

05.2

2.19

05.4

3.54

06.0

5.29

06.2

7.04

06.4

8.39

asn1

asn2

Qlatency (ms)

0

200000

400000

600000

800000

1000000

23.3

6.56

23.5

8.46

00.2

0.37

00.4

2.27

01.0

4.17

01.2

6.07

01.4

7.57

02.0

9.47

02.3

1.37

02.5

3.28

03.1

5.18

03.3

7.08

03.5

8.58

04.2

0.48

04.4

2.38

05.0

4.29

05.2

6.19

05.4

8.09

06.0

9.59

06.3

1.49

06.5

3.39

asn1

asn2



DB2 PS SAP® TRBK

97/168

Rows Applied

0

200000400000

600000

800000

10000001200000

1400000

23.3

6.56

23.5

8.41

00.2

0.27

00.4

2.12

01.0

3.57

01.2

5.42

01.4

7.27

02.0

9.12

02.3

0.57

02.5

2.43

03.1

4.28

03.3

6.13

03.5

7.58

04.1

9.43

04.4

1.28

05.0

3.14

05.2

4.59

05.4

6.44

06.0

8.29

06.3

0.14

06.5

1.59

asn2

asn1

The disparity between the work processed between queues ASN1 and ASN2 is due to an

imbalance in the work done on the tables registered in each queue.

Primary Site:

• Member 0 CPU: 40%, Member 1 CPU: 28% (difference is Capture overhead)

• 1.8 billion rows inserted/member

• 115 million rows updated/member

• 1.8 million rows deleted/member

• 75k statements/member/sec

• 33.9 mb/sec/member logged

On the logical copy site, all work was applied on one member, using 100 Apply threads.

Logical copy site:

• Member 2 CPU: 55%

• 3.6 billion rows inserted

• 406 million rows updated

• 5.3 million rows deleted

• 134k statements/sec

• 66.7mb logged/sec



DB2 PS SAP® TRBK

98/168

6.6 Site failure with Storage Replication and QRep

6.6.1 QRep & Site Failure Implementation

For this test, a Site Failure test was done when both Storage Replication and Queue Replication

was running. As the details of both these tests have been described individually earlier in this

document, they will not be described here, other than where this test differed

• Due to resource limitations, the QREP database was on the secondary system. In a real

world configuration, the QREP and secondary would be different systems

• The devices used by the QREP database were not Storage Replicated

• After the Site Failure was triggered by disconnecting the link between the ADVA devices,

the Apply processes were stopped and the QREP database was deactivated and

uncataloged

6.6.2 Results

The Night run was successfully restarted and ran to completion. The presence of the Queue

Replication workload and database did have any perceivable impact on the recovery process.

6.7 Backup during on-line operations

6.7.1 On line backup Implementation

Primary site : Production SAP® transaction banking

Secondary / Disaster recovery site : All Devices are up, i.e. storage devices receiving data

replication ( DS8800 Metro Mirror ) and devices used as remote FlashCopy targets.



DB2 PS SAP® TRBK

99/168

This document applies only to storage replication

Note:

The presence of the QREP database volumes on the Site 2 complex left us with insufficient space

on that system to accommodate the Flash Copy targets. Due to this storage space issue, we had

to reverse the roles of the sites from the original DR configuration we used for previous DR

tests., i.e. the workload was run on Site2, and the FlashCopy backup was taken on Site 1.

Configuration for Metro -Mirror replication and Remote FlashCopy

l l

Storage replication DB : D3D


m

m

SAP®

52 53 55 50 51 54

SAP®

Storage

10.7.21.0

10.1.2.0 10.1.2.0

10.1.3.0 10.1.3.0

L a t e n c y

10.9.9.0 10.9.9.0

DS8800


l lm

m

DB : D3D

l l

DB : D3D

fenced

F l a s h C o p y

DS8800

Primary / Production Site

Secondary / DR Site

FlashCopy Targets



DB2 PS SAP® TRBK

100/168

6.7.2 On-line Backup mechanism

DB2 storage devices are replicated toward Recovery site using DS8000 metro mirror.

(Tivoli Storage Productivity Center for Replication is used to manage the advanced copy

services).

From the primary site, we take an online backup of the D3D database at the remote DR site

using Remote FlashCopy via the existing metro mirror session (DP2PS_PPRC). That is, the Metro

Mirror target volumes on the recovery site’s DS8K frame are used as the source volumes for the

FlashCopy targets on the same frame.

High level process of taking an On-line backup

The following diagram describes the process of taking an online backup via PPRC link directly on

the remote frame.



DB2 PS SAP® TRBK

101/168

� On-line Backup Methodology

• SAP® TRBK day run workload running on primary site

• All the changes to primary volumes are replicated over to recovery site via long-

distance Fibre link through ADVA (DWDM).

• Issue DB2 Suspend I/O and freeze GPFS filesystems on Primary site and issue Remote

FlashCopy create operation (mkremoteflash) to create a persistent FlashCopy

relationship with the PPRC target volumes (which is also the source for FC targets) on

the recovery site’s DS8K frame.

• Issue DB2 Resume I/O and thaw GPFS filesystems on Primary site and access the

consistent Point-In-Time (PIT) FlashCopy backup clone on the recovery site.

• This Remote FlashCopy session can be re-synched (resyncremoteflash) to refresh the

Clone volumes to create any number of PIT backups as required.

� Process to access the Clone copy on DR site

• Stop DB2, uncatalog DR copy/Q-rep copy of D3D database and shutdown GPFS daemons

• Unmap the Metro-Mirror replicated copy of the gpfs devices and map the FlashCopy

target devices (with the same NSD meta-data) on the new block devices assigned to FC

targets (see procedure in appendix )

• DB2 database recataloged on dbpath

• DB2 instance started

• DB2 crash recovery started (via activate db)

• Connect to the database and do a row count on key tables (Clone read test)

� Prerequisites on DR site

Disaster recovery site must have DB2 pureScale® installed at same level as on primary site and

DS8000 Metro Mirror

• DB2 instance created

• DB2 path created

• DS8000 metro mirror activated and all source-target copy sets are fully synchronized

(i.e. in “Prepared” state)

• Gpfs meta-data definitions are already propagated from the primary site (See Disaster

Recovery procedure doc)

• FlashCopy Target LUNs are mapped to hosts in DR site

� List of storage devices replicated (via Metro-Mirror)



DB2 PS SAP® TRBK

102/168

Running df command should return the following information -- the replicated are the ones

highlighted in grey.







/dev/db2fs1 2147483648 117846016 2029637632 6% /db2sd_20110311231413

DB2 shared instance




DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container


DB2 container



DB2 PS SAP® TRBK

103/168

These devices are mapped to GPFS nsd ( 1 to 1 )

List of gpfs nsd

DB2PS-member0:~ # mmlsnsd

File system Disk name NSD servers

--------------------------------------------------------------------------

d3d gpfs2nsd (directly attached) DB2 base directory Replicated

db2fs1 gpfs1nsd (directly attached) DB2 shared instance Not replicated



SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP®


Replicated

SAP® gpfs5nsd (directly attached) DB2 container Replicated



DB2 PS SAP® TRBK

104/168

data9

GPFS configuration

GPFS configuration must be updated to map FlashCopy target volumes when accessing the clone

copy and un-map when not using them on the remote site.

Note:

• GPFS configuration is already propagated from the Primary site to the Secondary / DR

site before online backup process is started ( see Disaster Recovery procedure

document for how to propagate GPFS meta-data using mmfsctl ... syncFSconfig )

6.7.3 Online Backup Procedure

Initial status

� Check accessibility of FlashCopy target storage devices DB2PS-member0:~/alain # multipath -l | grep 0000004 fSAP® data11 (3600507630bffc06c0000000000004010) dm -40 IBM,2107900 fSAP® data10 (3600507630bffc06c0000000000004109) dm -57 IBM,2107900 fSAP® data9 (3600507630bffc06c0000000000004008) dm- 39 IBM,2107900 fdb2d3d (3600507630bffc06c0000000000004022) dm-56 I BM,2107900 fSAP® data8 (3600507630bffc06c0000000000004107) dm- 55 IBM,2107900 fSAP® data7 (3600507630bffc06c0000000000004006) dm- 45 IBM,2107900 fSAP® data6 (3600507630bffc06c0000000000004105) dm- 49 IBM,2107900 fSAP® data5 (3600507630bffc06c0000000000004004) dm- 43 IBM,2107900 fSAP® data4 (3600507630bffc06c0000000000004103) dm- 46 IBM,2107900 fSAP® data16 (3600507630bffc06c0000000000004115) dm -59 IBM,2107900 fSAP® data3 (3600507630bffc06c0000000000004002) dm- 41 IBM,2107900 flog3 (3600507630bffc06c0000000000004119) dm-60 IBM ,2107900 fSAP® data15 (3600507630bffc06c0000000000004014) dm -52 IBM,2107900 fSAP® data2 (3600507630bffc06c0000000000004101) dm- 47 IBM,2107900 flog2 (3600507630bffc06c0000000000004018) dm-50 IBM ,2107900 fSAP® data14 (3600507630bffc06c0000000000004113) dm -58 IBM,2107900 flog1 (3600507630bffc06c0000000000004117) dm-54 IBM ,2107900 fSAP® data1 (3600507630bffc06c0000000000004000) dm- 42 IBM,2107900 fSAP® data13 (3600507630bffc06c0000000000004012) dm -44 IBM,2107900 flog0 (3600507630bffc06c0000000000004016) dm-48 IBM ,2107900 fSAP® data12 (3600507630bffc06c0000000000004111) dm -51 IBM,2107900



DB2 PS SAP® TRBK

105/168

� Verify GPFS daemons are up (on DR site) DB2PS-member2:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node 3 DB2PS-member1 1* 3 3 active quorum node

� Filesystems already mounted (on DR site) Filesystem 1K-blocks Used Available Use% Mo unted on /dev/sda3 138195432 21604884 109570548 17% / devtmpfs 132266056 1056 132265000 1% /de v tmpfs 132266056 0 132266056 0% /dev/sh m /dev/sda1 72102 28762 43340 40% /boot/efi /dev/db2fs1 524288000 1528832 522759168 1% /db2sd_20110401115043

� DB2 installed and configured As user db2d3d check existence of instance

db2d3d@DB2PS-member0:~> db2instance -list ID TYPE STATE HOM E_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --- ------ ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED DB2 PS-member0 DB2PS-member0 NO 0 0 DB2PS-member0-ib0 1 MEMBER STARTED DB2 PS-member1 DB2PS-member1 NO 0 0 DB2PS-member1-ib0 128 CF PRIMARY DB2 PS-cf0 DB2PS-cf0 NO - 0 DB2PS-cf0-ib0 HOSTNAME STATE INS TANCE_STOPPED ALERT -------- ----- --- ------------- ----- DB2PS-cf0 ACTIVE NO NO DB2PS-member0 ACTIVE NO NO DB2PS-member1 ACTIVE NO NO

Check that node state is active.

� Replication

Replication is in full-duplex state (i.e. source and target copy sets are in full sync)



DB2 PS SAP® TRBK

106/168

o Check that storage replication is in status Normal

o Query the pairing status of all the copy sets in the PPRC Consistency Group /opt/ibm/dscli/dscli [-cfg Primary_Storage_profile] lspprc -l -dev Primary_Storage_Image_ID -remotedev Standby_Storage _Image_ID Range_of_Source_volume_IDs

o Login to DB2PS-member2 as root and run the following DSCLI command:

DB2PS-member2:~/scripts # ./PR_run_dscli.sh lspprc -l -dev IBM.2107-75WR821 -remotedev IBM.2107-75WR541 0000-FFFF Date/Time: May 6, 2011 9:36:26 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR821 ID State Reason Type Out Of Sync Tracks Tgt R ead Src Cascade Tgt Cascade Date Suspended SourceLSS Timeou t (secs) Critical Mode First Pass Status Incremental Resync Tgt Write GMIR CG PPRC CG isTgtSE DisableAutoResync =================================================== ================================================================================ ================================================================================ ========================= 1000:1000 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1002:1002 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1004:1004 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1006:1006 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1008:1008 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1010:1010 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1012:1012 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1014:1014 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1016:1016 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1018:1018 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1022:1022 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 10 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1101:1101 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1103:1103 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown -



DB2 PS SAP® TRBK

107/168

1105:1105 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1107:1107 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1109:1109 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1111:1111 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1113:1113 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1115:1115 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1117:1117 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown - 1119:1119 Full Duplex - Metro Mirror 0 Enabled En abled Invalid - 11 60 Disabled Invalid Disabled Disabled N/A Enabled Unknown –

o PPRC CG column in the above table for all the SOURCE:TARGET volume pairs (copy sets)

should be set to ‘Enable’.

6.7.4 Create Online backup using Remote FlashCopy

Do the following steps on the primary production site.

� Start SAP® TRBK day run Start SAP® TRBK day run performance benchmark workload and leave it running till SAP®

throughput enters the high load phase

� Create Remote FlashCopy relationship On the primary production site

� Issue DB2 suspend I/O On DB2PS-member2 as db2 instance owner (db2d3d), issue DB2 suspend I/O on the D3D

database :

db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v conne ct to d3d db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite suspend for database



DB2 PS SAP® TRBK

108/168

� Issue GPFS I/O suspend Issue GPFS I/O suspend on the file systems used as dbpath and database storage paths

mmfsctl <gpfs filesystem name> suspend

- Issue the above command on all GPFS file systems used by D3D db

- The following script with ‘suspend’ option was used to execute GPFS filesystem freeze ( see

appendix ):

/root/scripts/suspend_or_resume_gpfs_fs.sh suspend

� Create Remote FlashCopy Create a permanently attached (persistent) Remote FlashCopy session

dscli mkremoteflash -dev storage_image_ID -conduit source_storage_image_ID/LSS_ID -freeze

-record -persist source_volume_ID:target_volume_ID

- The following script with ‘mkremoteflash’ option was used to create the Remote FlashCopy

relationship ( see appendix ):

/root/scripts/remote_flash_copy.sh mkremoteflash

Note: To prevent TPC-R from suspending the PPRC session, immediately issue ‘unfreezeflash’

command on the remote frame. Refer to the notes under appendix for the above script for more

details.

� Issue GPFS I/O resume Resume I/O to suspended GPFS file systems

mmfsctl <gpfs filesystem name> resume

- Issue the above command on all GPFS file systems previously frozen

- The following script with ‘resume’ option was used to execute GPFS filesystem thaw (see

appendix):

/root/scripts/suspend_or_resume_gpfs_fs.sh resume

Issue DB2 resume I/O Resume I/O to D3D database

db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite resume for database



DB2 PS SAP® TRBK

109/168

In order to measure the performance impact of this process on the primary site, steps 3.2.2.1-

3.2.2.5 were grouped into a single script and issued as db2 instance owner “db2d3d” from

DB2PS-member2: db2d3d@DB2PS-member2:~/scripts> ./get_snapshot.sh Contents of the above script:

# We start the snapshot copy here outfile="./get_snapshot.out.`date +%F-%H:%M:%S`" echo "Starting snapshotcopy process at `date` ..." > $outfile db2 -v connect to d3d >> $outfile date >> $outfile db2 -v set write suspend for db >> $outfile date >> $outfile ssh root@DB2PS-member2 ". /root/.profile; /root/scripts/suspend_or_resume_gpfs_fs.sh suspend; date; /root/scripts/remote_flash_copy.sh mkremoteflash; d ate; /root/scripts/suspend_or_resume_gpfs_fs.sh resume" >> $outfile date >> $outfile db2 -v set write resume for db >> $outfile date >> $outfile db2 -v connect reset >> $outfile echo "Ending snapshotcopy process at `date` ..." >> $outfile

Note:

• Password-less SSH into root from db2d3d ID is required to run the above script.

From another window we examined the output of the above script

db2d3d@DB2PS-member2:~/scripts> tail -f get_snapshot.out.2011-05-07-00:19:35 Starting snapshotcopy process at Sat May 7 00:19:3 5 CEST 2011 ... connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D Sat May 7 00:19:36 CEST 2011 set write suspend for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 00:20:39 CEST 2011 suspend Issuing GPFS I/O suspend to filesystems used by dat abase D3D ... Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again …/…



DB2 PS SAP® TRBK

110/168

Quiescing all file system operations Writing dirty data to disk again Sat May 7 00:20:42 CEST 2011 mkremoteflash Executing the following DSCLI command for user opti on mkremoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile mkremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -free ze -record -persist 1000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1 010:4010 1012:4012 1014:4014 1016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 12:20:43 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1000:4000 successfully created. Use the lsremoteflash command to determine copy completion. …/… CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1022:4022 successfully created. Use the lsremoteflash command to determine copy completion. /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile mkremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -free ze -record -persist 1101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1 111:4111 1113:4113 1115:4115 1117:4117 1119:4119 Date/Time: May 7, 2011 12:20:46 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1101:4101 successfully created. Use the lsremoteflash command to determine copy completion. …/… CMUC00173I mkremoteflash: Remote FlashCopy volume p air 1119:4119 successfully created. Use the lsremoteflash command to determine copy completion. Date/Time: May 7, 2011 12:20:48 AM CEST IBM DSCLI V ersion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 10 successfully reset. CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 11 successfully reset. Sat May 7 00:20:50 CEST 2011 resume Issuing GPFS I/O resume to filesystems used by data base D3D ... Resuming operations. …/… Resuming operations. Sat May 7 00:20:53 CEST 2011 set write resume for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 00:20:53 CEST 2011 connect reset DB20000I The SQL command completed successfully. Ending snapshotcopy process at Sat May 7 00:20:53 CEST 2011 ...



DB2 PS SAP® TRBK

111/168

Accessing FlashCopy database Clone (from DR site) After successfully creating the Remote FlashCopy relationship, do the following steps on the

Secondary / DR site to access the FlashCopy database clone.

� Shutdown DB2 On DB2PS-member0 as DB2 instance ID (db2d3d), stop DB2:

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2stop force

� Shutdown GPFS Unmount all the GPFS file systems seen to the cluster (including the filesystem used by

db2instance)

DB2PS-member0:~/scripts # mmumount -a Note: Make sure that above unmount command is successful. Any open file handles can be

cleaned by using fuser –kmu /dev/<fs name>

Shutdown GPFS daemons

DB2PS-member0:~/scripts # mmshutdown -a

Check all GPFS daemons are down in the cluster by issuing:

DB2PS-member0:~/scripts # mmgetstate -a –L

Note:

• Since, the FlashCopy Targets are identical (including GPFS meta-data content) to the

source, it is recommended to shutdown GPFS before unmapping and remapping block

devices ( /dev/dm-# ) backed by GPFS nsds to avoid hosts using a cached copy or any

stale data – i.e. flush hosts file system cache before accessing FC target volumes.

� Unmap PPRC targets and map FlashCopy Targets Mask (unmap) the PPRC’d NSDs (FlashCopy source) and map FlashCopy target volumes as the

NSDs.

FC target volumes will have all the tacks copied from the source including NSD header (first 10

sectors). As long as the underlying block device contains all the NSD meta-data known to the

cluster, GPFS does not have an ability to detect that the block device paths were remapped to a

different set of DM MP devices. Key thing to remember is, we do this masking using an

nsddevices user-exit which is file local to each node. Hence this remap process needs to be done

on all nodes in the cluster.

DB2PS-member0:~/scripts # /scripts/disable_dr_nsd_a nd_enable_fc_nsd.sh



DB2 PS SAP® TRBK

112/168

DB2PS-member0:~/scripts # ssh root@DB2PS-member1 ". /root/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh" DB2PS-member0:~/scripts # ssh root@DB2PS-cf0 ". /ro ot/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh"

See Appendix for the contents of this script.

� Start GPFS Start GPFS daemons

DB2PS-member0:~/scripts # mmstartup -a Sat May 7 00:41:28 CEST 2011: mmstartup: Starting GPFS ... Check the state of the GPFS daemons

DB2PS-member0:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node 3 DB2PS-member1 1* 3 3 active quorum node

List the NSD-to-block device (DM MP device) mappings to verify that GPFS has now picked up the FlashCopy target

LUNs

DB2PS-member0:~/scripts # mmlsnsd -X Disk name NSD volume ID Device Dev type Node name Remarks --------------------------------------------------- ------------------------------------------------ gpfs10nsd 0A0102324D80C4A2 /dev/dm-52 dmm DB2PS-member0.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-41 dmm DB2PS-member0.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-51 dmm DB2PS-member0.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-40 dmm DB2PS-member0.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-50 dmm DB2PS-member0.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-57 dmm DB2PS-member0.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-39 dmm DB2PS-member0.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-46 dmm DB2PS-member0.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-56 dmm DB2PS-member0.localdomain



DB2 PS SAP® TRBK

113/168

gpfs19nsd 0A0102324D80C61E /dev/dm-59 dmm DB2PS-member0.localdomain gpfs1nsd 0A0102324D7A9E9F /dev/dm-33 dmm DB2PS-member0.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-48 dmm DB2PS-member0.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-58 dmm DB2PS-member0.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-47 dmm DB2PS-member0.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-49 dmm DB2PS-member0.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-45 dmm DB2PS-member0.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-55 dmm DB2PS-member0.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-43 dmm DB2PS-member0.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-44 dmm DB2PS-member0.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-53 dmm DB2PS-member0.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-54 dmm DB2PS-member0.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-42 dmm DB2PS-member0.localdomain

Notes:

• NSDs highlighted in gray are not part of the FlashCopy relationship and hence the block

device mappings remain unchanged.

• Since GPFS file systems are configured to Auto-Mount (during GPFS startup) on the

primary site, the replicated NSD meta-data in the FC target volumes are also set with

Auto-Mount. Therefore there is no need to manually mount the GPFS file systems on FC

target NSDs.

� Bring FlashCopy clone of D3D database online

o Catalog FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), catalog D3D db

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 catalog db d3d on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.

db2d3d@DB2PS-member0:~> db2 list db directory System Database Directory Number of entries in the directory = 1 Database 1 entry: Database alias = D3D Database name = D3D



DB2 PS SAP® TRBK

114/168

Local database directory = /db2/D3D Database release level = e.00 Comment = Directory entry type = Indirect Catalog database partition number = 0 Alternate server hostname = Alternate server port number =

o Start instance db2d3d On DB2PS-member0 as DB2 instance ID (db2d3d), start DB2 db2d3d@DB2PS-member0:~> db2start ADM12026W The DB2 server has detected that a valid license for the product "DB2 Enterprise Server Edition" has not been registered. 05/07/2011 00:47:26 0 0 SQL1063N DB2START processing was successful. 05/07/2011 00:47:27 1 0 SQL1063N DB2START processing was successful.

o Initialize the FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), initialize the FC copy of D3D database as a

snapshot

db2d3d@DB2PS-member0:~> db2inidb d3d as snapshot

The above ‘db2inidb’ command would trigger an implicit GCR operation, status of which can be monitored

via,

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:04:35 -- Date 05/07/2011 00:52:02 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9CF6608EA Current LRI 000000000000000100000000041F648 A000003E9CF6608EA Current LSO 4401095066827 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1304722286) Sat May 7 00:51:2 6 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000204A6FBC8 1 Forward Sat May 7 00:51:26 132901367 bytes 5474720160 bytes 0x0000000204A6FD50 2 Backward NotStarted 0 bytes 5474720160 bytes



DB2 PS SAP® TRBK

115/168

The DB2 diaglog should indicate when the above GCR completes

2011-05-07-01.02.25.640751+120 E213622E450 LEVEL: Info PID : 34391 TID : 4699686174080 0 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11050 6224727 EDUID : 35 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.

� Clone read test After successful ‘db2inidb’ operation, activate and connect to the FlashCopy clone of D3D

database and perform a row-count of few key tables.

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 activate db d3d DB20000I The ACTIVATE DATABASE command completed s uccessfully.

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 "select count(*) from SAP® d3d.BCA_PAYMITEM with ur" 1 ----------- 1807164585 1 record(s) selected.

6.7.5 Refresh Online backup clone using Remote FlashCopy

� Replication

Verify Metro-Mirror (PPRC) replication session is in sync via TPC-R GUI or by issuing DSCLI

command ‘lspprcpath’, and do the following steps on the primary production site.



DB2 PS SAP® TRBK

116/168

� Start TRBK day run an leave it running till SAP® throughput is stable

� Refresh Remote FlashCopy targets On the primary production site

� Issue DB2 suspend I/O On DB2PS-member2 as db2 instance owner (db2d3d), issue DB2 suspend I/O on the D3D

database :

db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v connec t to d3d db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite suspend for database

� Issue GPFS I/O suspend Issue GPFS I/O suspend on the file systems used as dbpath and database storage paths

mmfsctl <gpfs filesystem name> suspend


- The following script with ‘suspend’ option was used to execute GPFS filesystem freeze (see

appendix):

/root/scripts/suspend_or_resume_gpfs_fs.sh suspend

� Re-sync Remote FlashCopy Create a permanently attached (persistent) Remote FlashCopy session

dscli resyncremoteflash -dev storage_image_ID -cond uit source_storage_image_ID/LSS_ID -freeze

-record -persist source_volume_ID:target_volume_ID

- The following script with ‘remoteflash’ option was used to re-sync the Remote FlashCopy

relationship (see appendix):

/root/scripts/remote_flash_copy.sh remoteflash

Note: To prevent TPC-R from suspending the PPRC session, immediately issue ‘unfreezeflash’

command on the remote frame. Refer to the notes under appendix for the above script for more

details.



DB2 PS SAP® TRBK

117/168

� Issue GPFS I/O resume Resume I/O to suspended GPFS file systems

mmfsctl <gpfs filesystem name> resume


- The following script with ‘resume’ option was used to execute GPFS filesystem thaw (see

appendix):

/root/scripts/suspend_or_resume_gpfs_fs.sh resume

� Issue DB2 resume I/O Resume I/O to D3D database

db2d3d@DB2PS-member2:~/sqllib/db2dump> db2 -v set w rite resume for database In order to measure the performance impact of this process on primary site, steps 3.4.3.1-

3.4.3.5 were grouped into a single script and issued as db2 instance owner “db2d3d” from

DB2PS-member2 : db2d3d@DB2PS-member2:~/scripts> ./refresh_snapshot.sh Contents of the above script:

# We start the snapshot copy here outfile="./refresh_snapshot.out.`date +%F-%H:%M:%S` " echo "Starting snapshotcopy process at `date` ..." > $outfile db2 -v connect to d3d >> $outfile date >> $outfile db2 -v set write suspend for db >> $outfile date >> $outfile ssh root@DB2PS-member2 ". /root/.profile; /root/scripts/suspend_or_resume_gpfs_fs.sh suspend; date; /root/scripts/remote_flash_copy.sh remoteflash; dat e; /root/scripts/suspend_or_resume_gpfs_fs.sh resume" >> $outfile date >> $outfile db2 -v set write resume for db >> $outfile date >> $outfile db2 -v connect reset >> $outfile echo "Ending snapshotcopy process at `date` ..." >> $outfile Note:

• Password-less SSH into root from db2d3d ID is required to run the above script.



DB2 PS SAP® TRBK

118/168

From another window we examined the output of the above script

db2d3d@DB2PS-member2:~/scripts> tail -f refresh_snapshot.out.2011-05-07-16:31:27 Starting snapshotcopy process at Sat May 7 16:31:2 7 CEST 2011 ... connect to d3d Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D Sat May 7 16:31:28 CEST 2011 set write suspend for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 16:42:25 CEST 2011 suspend Issuing GPFS I/O suspend to filesystems used by dat abase D3D ... Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again …/… Writing dirty data to disk Quiescing all file system operations Writing dirty data to disk again Sat May 7 16:42:28 CEST 2011 remoteflash Executing the following DSCLI command for user opti on remoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile resyncremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -free ze -record -persist 1000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1 010:4010 1012:4012 1014:4014 1016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 4:42:29 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1000:4000 successfully resynchronized. Use the lsremoteflash command to determine copy completion. …/… CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1022:4022 successfully resynchronized. Use the lsremoteflash command to determine copy completion. /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile resyncremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -free ze -record -persist 1101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1 111:4111 1113:4113 1115:4115 1117:4117 1119:4119 Date/Time: May 7, 2011 4:42:32 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1101:4101 successfully resynchronized. Use the lsremoteflash command to determine copy completion. …/…



DB2 PS SAP® TRBK

119/168

CMUC00175I resyncremoteflash: Remote FlashCopy volu me pair 1119:4119 successfully resynchronized. Use the lsremoteflash command to determine copy completion. Date/Time: May 7, 2011 4:42:34 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 10 successfully reset. CMUC00172I unfreezeflash: FlashCopy consistency gro up for logical subsystem 11 successfully reset. Sat May 7 16:42:36 CEST 2011 resume Issuing GPFS I/O resume to filesystems used by data base D3D ... Resuming operations. …/… Resuming operations. Sat May 7 16:42:39 CEST 2011 set write resume for db DB20000I The SET WRITE command completed successfu lly. Sat May 7 16:42:39 CEST 2011 connect reset DB20000I The SQL command completed successfully. Ending snapshotcopy process at Sat May 7 16:42:39 CEST 2011 ...

� Query status of Clone copy refresh You can query the status of the FlashCopy progress to figure out when the targets achieve the

full copied state. Note that, since FC will copy the tracks in background, and will use Copy-On-

Access (COA), you do not need to wait till targets are fully synced to access the data on them.

However, there may be slight performance degradation whenever a track that is not yet copied

is accessed.

DB2PS-member2:~/scripts # ./remote_flash_copy.sh ls remoteflash lsremoteflash Executing the following DSCLI command for user opti on lsremoteflash: /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile lsremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 -l 1 000:4000 1002:4002 1004:4004 1006:4006 1008:4008 1010:4010 1012:4012 1014:4014 1 016:4016 1018:4018 1022:4022 Date/Time: May 7, 2011 5:01:52 PM CEST IBM DSCLI Version: 6.6.0.305 DS: IBM.2107-75WR541 ID SrcLSS SequenceNum ActiveCopy Recording P ersistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCop y OutOfSyncTracks DateCreated DateSynced State isTgtSE Pmir =================================================== ================================================================================ ================================================================================ ======== 1000:4000 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1002:4002 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1004:4004 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No



DB2 PS SAP® TRBK

120/168

1006:4006 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1008:4008 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1010:4010 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1012:4012 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1014:4014 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1016:4016 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1018:4018 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No 1022:4022 10 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:06 CEST 2011 Sat May 07 16:43:54 CEST 2011 Valid No No /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile lsremoteflash -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 -l 1 101:4101 1103:4103 1105:4105 1107:4107 1109:4109 1111:4111 1113:4113 1115:4115 1 117:4117 1119:4119 Date/Time: May 7, 2011 5:01:55 PM CEST IBM DSCLI Ve rsion: 6.6.0.305 DS: IBM.2107-75WR541 ID SrcLSS SequenceNum ActiveCopy Recording P ersistent Revertible SourceWriteEnabled TargetWriteEnabled BackgroundCop y OutOfSyncTracks DateCreate d DateSynced State isTgtSE Pmir =================================================== ================================================================================ ================================================================================ ======== 1101:4101 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1103:4103 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1105:4105 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1107:4107 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1109:4109 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1111:4111 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1113:4113 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1115:4115 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No



DB2 PS SAP® TRBK

121/168

1117:4117 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No 1119:4119 11 0 Disabled Enabled E nabled Disabled Enabled Enabled Enabled 0 S at May 07 00:22:09 CEST 2011 Sat May 07 16:43:57 CEST 2011 Valid No No

Notes:

• From output above:

o DateSynced : Indicated the timestamp of either a create FC command or a

resync FC command was issued ( May 07 16:43:54 CEST 2011 )

o OutOfSyncTracks : Indicate how many tracks are copy pending. This

column value needs to reach ‘0’ to achieve fully sync status. We reached this

fully sync state at May 7, 2011 5:01:52 PM CEST.

• Therefore time for FC target to get 100% copied is ~ 17min.

6.7.6 Accessing FlashCopy database Clone (from DR site)

After successfully re-syncing the Remote FlashCopy relationship, do the following steps on the

Secondary / DR site to access the clone copy.

� Shutdown DB2 On DB2PS-member0 as DB2 instance ID (db2d3d), stop DB2:

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2stop force

� Shutdown GPFS Unmount all the GPFS file systems seen to the cluster (including the filesystem used by

db2instance)

DB2PS-member0:~/scripts # mmumount -a Note: Make sure that above unmount command is successful. Any open file handles can be

cleaned by using fuser –kmu /dev/<fs name>

Shutdown GPFS daemons

DB2PS-member0:~/scripts # mmshutdown -a

Check all GPFS daemons are down in the cluster by issuing:

DB2PS-member0:~/scripts # mmgetstate -a -L



DB2 PS SAP® TRBK

122/168

Note:

• It is recommended to flush hosts file system cache before accessing refreshed FC target

volumes to avoid accessing any stale data – i.e. stop/start GPFS.

� Unmap PPRC targets and map FlashCopy Targets Mask (unmap) the PPRC’d NSDs (FlashCopy source) and map FlashCopy taget volumes as the

NSDs.

FC target volumes will have all the tacks copied from the source including NSD header (first 10

sectors). As long as the underlying block device contains all the NSD meta-data known to the

cluster, GPFS does not have an ability to detect that the block device paths were remapped to a

different set of DM MP devices. Key thing to remember is, we do this masking using an

nsddevices user-exit which is file local to each node. Hence this remap process needs to be done

on all nodes in the cluster.

DB2PS-member0:~/scripts # disable_dr_nsd_and_enable _fc_nsd.sh DB2PS-member0:~/scripts # ssh root@DB2PS-member1 ". /root/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh" DB2PS-member0:~/scripts # ssh root@DB2PS-cf0 ". /ro ot/.profile; /root/scripts/disable_dr_nsd_and_enable_fc_nsd.sh"

See Appendix for the contents of this script.

Note:

o This step should already have been done when the remote FC was created. Unless there

was a change that required reverting back to the source volumes (for example like a

disaster recovery operation), there is no need to execute this step again.

� Start GPFS Start GPFS daemons

DB2PS-member0:~/scripts # mmstartup -a

Check the state of the GPFS daemons

DB2PS-member0:~/scripts # mmgetstate -a -L Node number Node name Quorum Nodes up Tot al nodes GPFS state Remarks --------------------------------------------------- --------------------------------- 1 DB2PS-member0 1* 3 3 active quorum node 2 DB2PS-cf0 1* 3 3 active quorum node



DB2 PS SAP® TRBK

123/168

3 DB2PS-member1 1* 3 3 active quorum node

List the NSD-to-block device (DM MP device) mappings to verify that GPFS has now picked up

the FlashCopy target LUNs

DB2PS-member0:~/scripts # mmlsnsd -X Disk name NSD volume ID Device Dev type Node name Remarks --------------------------------------------------- ------------------------------------------------ gpfs10nsd 0A0102324D80C4A2 /dev/dm-52 dmm DB2PS-member0.localdomain gpfs11nsd 0A0102324D80C4BC /dev/dm-41 dmm DB2PS-member0.localdomain gpfs12nsd 0A0102324D80C4D9 /dev/dm-51 dmm DB2PS-member0.localdomain gpfs13nsd 0A0102324D80C4F5 /dev/dm-40 dmm DB2PS-member0.localdomain gpfs14nsd 0A0102324D80C512 /dev/dm-50 dmm DB2PS-member0.localdomain gpfs15nsd 0A0102324D80C530 /dev/dm-57 dmm DB2PS-member0.localdomain gpfs16nsd 0A0102324D80C54F /dev/dm-39 dmm DB2PS-member0.localdomain gpfs17nsd 0A0102324D80C570 /dev/dm-46 dmm DB2PS-member0.localdomain gpfs18nsd 0A0102324D80C594 /dev/dm-56 dmm DB2PS-member0.localdomain gpfs19nsd 0A0102324D80C61E /dev/dm-59 dmm DB2PS-member0.localdomain gpfs1nsd 0A0102324D7A9E9F /dev/dm-33 dmm DB2PS-member0.localdomain gpfs20nsd 0A0102324D80C641 /dev/dm-48 dmm DB2PS-member0.localdomain gpfs21nsd 0A0102324D80C665 /dev/dm-58 dmm DB2PS-member0.localdomain gpfs22nsd 0A0102324D80C68B /dev/dm-47 dmm DB2PS-member0.localdomain gpfs2nsd 0A0102324D80C2BA /dev/dm-49 dmm DB2PS-member0.localdomain gpfs3nsd 0A0102324D80C412 /dev/dm-45 dmm DB2PS-member0.localdomain gpfs4nsd 0A0102324D80C424 /dev/dm-55 dmm DB2PS-member0.localdomain gpfs5nsd 0A0102324D80C438 /dev/dm-43 dmm DB2PS-member0.localdomain gpfs6nsd 0A0102324D80C449 /dev/dm-44 dmm DB2PS-member0.localdomain gpfs7nsd 0A0102324D80C45F /dev/dm-53 dmm DB2PS-member0.localdomain gpfs8nsd 0A0102324D80C475 /dev/dm-54 dmm DB2PS-member0.localdomain gpfs9nsd 0A0102324D80C489 /dev/dm-42 dmm DB2PS-member0.localdomain

Notes:



DB2 PS SAP® TRBK

124/168

• NSDs highlighted in gray are not part of the FlashCopy relationship and hence the block

device mappings remain unchanged.

• Since GPFS file systems are configured to Auto-Mount (during GPFS startup) on the

primary site, the replicated NSD meta-data in the FC target volumes are also set with

Auto-Mount. Therefore there is no need to manually mount the GPFS file systems on FC

target NSDs.

� Bring FlashCopy clone of D3D database online

o Catalog FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), catalog D3D db

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 catalog db d3d on /db2/D3D DB20000I The CATALOG DATABASE command completed su ccessfully.

db2d3d@DB2PS-member0:~> db2 list db directory System Database Directory Number of entries in the directory = 1 Database 1 entry: Database alias = D3D Database name = D3D Local database directory = /db2/D3D Database release level = e.00 Comment = Directory entry type = Indirect Catalog database partition number = 0 Alternate server hostname = Alternate server port number =

o Start instance db2d3d On DB2PS-member0 as DB2 instance ID (db2d3d), start DB2 db2d3d@DB2PS-member0:~> db2start ADM12026W The DB2 server has detected that a valid license for the product "DB2 Enterprise Server Edition" has not bee n registered. 05/07/2011 17:05:02 0 0 SQL1063N DB2START processing was successful. 05/07/2011 17:05:03 1 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful.

o Initialize the FlashCopy clone of D3D database On DB2PS-member0 as DB2 instance ID (db2d3d), initialize the FC copy of D3D database as a

snapshot



DB2 PS SAP® TRBK

125/168

db2d3d@DB2PS-member0:~> db2inidb d3d as snapshot

The above ‘db2inidb’ command would trigger an implicit GCR operation, status of which can be

monitored via,

db2d3d@DB2PS-member0:~> db2pd -db d3d -recovery Database Member 0 -- Database D3D -- Active -- Up 0 days 00:03:54 -- Date 05/07/2011 17:08:56 Recovery: Recovery Status 0x410A8015 Current Log Current LSN 000003E9D3E2309B Current LRI 0000000000000001000000000471911 9000003E9D3E2309B Current LSO 4416510133036 Job Type GROUP CRASH RECOVERY Job ID 1 Job Start Time (1304780916) Sat May 7 17:08:3 6 2011 Job Description Group Crash Recovery Invoker Type User Total Phases 2 Current Phase 1 Progress: Address PhaseNum Description StartTime CompletedWork TotalWork 0x0000000204A6FBC8 1 Forward Sat May 7 17:08:36 137987677 bytes 20162993760 bytes 0x0000000204A6FD50 2 Backward NotStarted 0 bytes

The DB2 diaglog should indicate when the above GCR completes

2011-05-07-17.41.36.954133+120 E212632E450 LEVEL: Info PID : 40764 TID : 4743261637196 8 PROC : db2sysc 0 INSTANCE: db2d3d NODE : 000 DB : D3D APPHDL : 0-52 APPID: *N0.DB2.11050 7150502 EDUID : 35 EDUNAME: db2agnti (D 3D ) 0 FUNCTION: DB2 UDB, recovery manager, sqlpresr, prob e:3110 MESSAGE : ADM1528I Group crash recovery has comple ted successfully.

� Clone read test After successful execution of ‘db2inidb’ operation, activate and connect to the FlashCopy clone

of D3D database and perform a row-count of a key table to demonstrate that the flashcopy

clone of the database is usable.

db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 activate db d3d DB20000I The ACTIVATE DATABASE command completed s uccessfully. db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 connect to d3d



DB2 PS SAP® TRBK

126/168

Database Connection Information Database server = DB2/LINUXX8664 9.8.4 SQL authorization ID = DB2D3D Local database alias = D3D db2d3d@DB2PS-member0:~/sqllib/db2dump> db2 "select count(*) from SAP® d3d.BCA_PAYMITEM with ur" 1 ----------- 1805273916 1 record(s) selected.

6.7.7 Going back to the production configuration

While this section is not part of the POC’s Online backup or DR test cases, it is useful to

understand what got changed in the production configuration after adding FC targets on the DR

site – since those new NSDs have changed the GPFS configuration on the DR site. In addition, it

would be useful to briefly address how to operate with this new configuration.

Initial status Some of the GPFS file systems are mounted, i.e. both Q-replicated filesystems and FlashCopy

Target clone filesystems (created via Metro-Mirror) are mounted on the DR GPFS cluster.

Reset DB2 and GPFS back to production configuration The “new” production configuration would have only filesystem with db2 instance / shared

sqllib and Q-rep file systems mounted – i.e. both PPRC target volumes and FC target volumes

must be unmapped (i.e. made invisible) from GPFS.

� Update the DB2 Configuration on DR site Before we switch over to the production configuration:

o Stop SAP® run

o Deactivate FC clone copy of D3D database:

o db2 terminate; force applications all; db2 deactiva te db D3D o Stop Db2

o db2stop force

o Uncatalog FC clone copy of D3D database:

o db2 uncatalog db D3D



DB2 PS SAP® TRBK

127/168

� Update the GPFS Configuration on DR site Before switching to the production configuration, unmount the FC clone target filesystems and

unmap (mask) both Metro-Mirror and FC replicated NSDs on the DR cluster.

- From a node on DR site, as root issue: /root/scripts/disable_repl_nsd.sh - See Appendix for script details

6.7.8 Conclusion

No impact to the primary system was observed during the flashcopy backup.



DB2 PS SAP® TRBK

128/168

7 SAP® TRBK Certification results

7.1 Architecture – system configuration

7.1.1 Overall layout

Here is the target Architecture used for the Certification test campaign.

Hardware Involved

Machine type Quantity Role

x3690 x5 ( 7147-H1G) 5 Database servers (DB2PS)

HS22 (Type 7870)

BladeCenter H

22

2

Application servers (SAP® )

DS8800 (2421-951 ) 1 Database storage

DS5300 1 Benchmark results storage

Mellanox IB switches IS5030 2 Interconnect Infiniband network (provide HA)

10 Gb Eth

Uplink 10G

Uplink 10G

HS22 x 10

HS22 x 10

X3690 eX5

1Infiniban

SA

1 Gb Eth

SAP® TRBK Certification

8Gb

CF 0

Member 0

Member 1

Member 2

Member 3

SA

DS8800 4 way 384 GB cache

90 Ranks Drives 146GB

15k RPM 48 ports FC

SLES 11 SP 1

SLES 11 SP 1 DB2PS FP4

Micro-code 6.1

s2&3 s5

s1

s3 s1

s2

s2&3 s5

s1

s2&3 s

5

s1

s2&3

s5

s1



DB2 PS SAP® TRBK

129/168

7.1.2 Application servers - HS22



Processor 2 Intel Xeon 4 Cores X5570 95W 2.93GHz/1333MHz/8MB L2

Memory 12

Express 4GB Dual Rank PC3-10600 CL9 ECC DDR3 VLP RDIMM

1333MHz

Internal disks 2 Express IBM 146 GB 2.5in SFF Slim-HS 15K 6Gbps SAS HDD

Additional cards 1 Qlogic 1 Gb Ethernet dual port CIO-v


Remark: Among all the HS22, one has been chosen to host the TRBK driver. This blade is also

responsible of collecting all the monitoring data (SAP® & system related) from all the machines.

Thus, this blade is attached to a DS5300 via a daughter card CFF-h Combo LAN +SAN Qlogic 8Gb.



FW/BIOS P9E146A 03/01/2010 (1.07)

Diagnostics DSYT60K 02/25/2010 (3.01)

Blade Sys Mgmt Processor YUOO57H (1.10)











HS22 are equipped with 12 DIMMs of 4GB DDR-3, resulting 48 GB total. All the slots are

populated, and the 2 nodes have an equivalent amount of memory. This is part of the Best

Practices recommended by IBM. Here is a dump of the Linux numactl command, which shows

the NUMA topology of the server.




node distances:

node 0 1

0: 10 21

1: 21 10

7.1.3 Database servers - x3690 x5





DB2 PS SAP® TRBK

130/168

Processor 2 Intel Xeon Processor E7-2870 10C (2.40GHz 30MB L3 130w 2S)

Memory 32 8 GB DDR3 technology and operates at memory speeds of 1066 MHz.

Internal disks 8 IBM 200 GB SDD

Additional cards

1 SAS Raid

1 Emulex 10Gbps Ethernet Dual ports PCI-express

1 Qlogic 8 Gb FC Dual ports PCI-express

1 (2) Mellanox 40 Gbps QDR Infiniband Dual ports PCI-Express



All x3690 x5 are equipped with 2 Memory cards, each containing 16 DIMMs of 8GB DDR-3.

7.1.3.3 Overall, the amount of memory is 256 GB of DDR3 RAM. uEFI Configuration

Here is an uEFI dump extract using the ASU (Advanced Setup Utility – tool provided by IBM).

The configuration has been done using the recommendations of Toronto & Intel Lab.

uEFI.OperatingMode=Performance Mode

uEFI.QuietBoot=Enable

uEFI.TurboModeEnable=Enable

uEFI.TurboBoost=Traditional

uEFI.ProcessorEistEnable=Disable

uEFI.ProcessorCcxEnable=Disable

uEFI.ProcessorC1eEnable=Disable

uEFI.HyperThreading=Enable

uEFI.EnableCoresInSbsp=All

uEFI.ExecuteDisableBit=Enable

uEFI.ProcessorVmxEnable=Enable

uEFI.ProcessorDataPrefetch=Enable

uEFI.DDRspeed=Max Performance

uEFI.CkeLowPolicy=Disable

uEFI.MirrorMigration=NonMirrored

uEFI.MapperPolicy=Closed

uEFI.SchedulerPolicy=Adaptive

uEFI.PatrolScrub=Disable

uEFI.PagePolicy=Closed uEFI.MemoryScrambling=Disable

uEFI.Socket0Branch0SpareEn=Disable

uEFI.IdeMode=Compatibility mode

uEFI.ThunkSupport=Enable

uEFI.TpmEnable=Disable

uEFI.TpmStateUserSelection=Deactivate

uEFI.QPISpeed=Max Performance

7.1.3.4 I/O card layout for x3690 x5

Database servers require a suitable connection to the Storage Area Network (SAN) to access the

disks where the tables are stored. Moreover, DB2 pureScale® allows the different database

servers to communicate directly using RDMA, thanks to the use of Infiniband stack and uDAPL

upper protocol.



DB2 PS SAP® TRBK

131/168

However, all the servers are not equivalent: among 5 servers, 4 are “members” (i.e. regular

database servers) and 1 is “coupling facility” (used to manage shared data structures between

members). Thus, these 2 kinds of servers have different needs in terms of communication: that’s

why 2 layouts have been designed for the PCI-express cards.

Naming convention:

This outline illustrates the internal organization of the x3690 x5 components. The PCI slots are

represented on the right. The speed of these buses is also specified (x4 to x16).

For regular members:



Fiber Channel (HBA) 2 2 - 3

Server RAID 1 4

10 Gb Ethernet 1 5

Explanation:

- Objective: priority is given to the IB bandwidth

- Assumption: 10 Gb Eth is not resource consuming

- Advantage : IB can benefit from « full » bandwidth of an PCI bus



DB2 PS SAP® TRBK

132/168

For coupling facilities:



Fiber Channel (HBA) 2 2


Server RAID 1 4

10 Gb Ethernet 1 5

Explanation:

- Hypothesis 1 : CF might be able to use 2 active IB cards

- Hypothesis 2 : CF will not have intensive SAN I/O

- All IB cards have homogeneous configuration (PCI-e 8x slots)

- Advantage : maximize bandwidth for both IB cards

-

Note: the slot 1 (16x) has not been used because the Mellanox card is not currently supporting

16x speed. Thus, to have a homogeneous configuration between all the cards, all the x8 slots

have been used for the IB cards. The FC card (Qlogic) has been placed in slot 2, as this slot is

compatible with any 8x card (IBM documentation) and that the I/O on the CF is very low.

7.1.4 Storage overview

The storage subsystem is made of 1 DS8800 following characteristics:

• 3 frames

• 384 GB Processor Memory (4-Way)

• 768 disks of 450 GB 15K FC

• 16 x 8Gb SW FCP ports connections to the SAN.

Each server (member or coupling facility) have access to the DS8800 through a SAN (Storage

Area Network)

Some description of the two DS8800 is detailed below:

dscli showsi IBM.2107-75TX861 -cfg C:\Progra~1\IBM\dscli\profile\DB2_PRA_Pure_1.profile


75TX861

ID IBM.2107-75TX861


Model 941

WWNN 500507630AFFC2BC

Signature 4d2b-1929-cad1-dee3

State Online



DB2 PS SAP® TRBK

133/168

ESSNet Enabled

Volume Group V0

os400Serial 2BC

NVS Memory 8.0 GB




numegsupported 0

ETAutoMode on

ETMonitor all

Overview of the storage configuration

DS8800 Layout

Ran

k RAID Extentpo

ol Arra

y Device Adapter

ExtentPool ID

ExtentPool Name

Lun 1000

Lun1101

Lun1002

Lun1103

R0 RAID5 6+P+S 0 A0 0 P0 pure-pool1 x





DB2 PS SAP® TRBK

134/168


R4 RAID5 7+P 0 A4 0 P0 pure-pool1 x










































DB2 PS SAP® TRBK

135/168











































DB2 PS SAP® TRBK

136/168





R88 RAID5 7+P - A88 7 - Not used

R89 RAID5 7+P - A89 7 - Not used

The configuration of the RAID5 arrays of one DS8800 with 720 FC disks.

• 90 RAID5 arrays.

• 4 extent pools

• 22 ranks / extent pool

• Luns created with rotateexts option

7.1.4.1 Assumptions for DB2 / GPFS / Storage design

� Assumptions:

– Shared pool of FC disks for Data/Index/Temp and logs

– Storage Pool Striping (1 GB stripe size) for Data/Index/Temp and logs

– No GPFS striping (each file system is defined / mapped on only one LUN).

– Easy configuration with only one level of stripping at DS8800 level.

� Design:

– Each file system (SAP® data1 to SAP® adata 16) is mapped on one LUN of 388 GB

size

– Each file system (log0 to log3) is mapped on one LUN of 2 TB size.

– Each LUN is stripped on 22 RAID5 arrays (176 FC disks) with a block size of 1 GB

– LUN is created with the option Rotate Extent in an Extent Pool

– 4 Extents Pools on DS8800 which consist in 22 FC RAID5 arrays each

– Each RAID5 array has 6 data, 1 parity and 1 spare or 7 datas and 1 parity disks.

– A total of 720 disks.

LUN is created with the option Rotate Extent in an Extent Pool. For each extent pool:

• 4 LUNs of 388 GB for Data/index/Temp

• 1 LUN of 2 TB for logs

For example the LUN 2000 is distributed on 22 different RAID5 arrays

This LUN of 388 GB size consists in 16 or 17 extents of 1 GB for each array.

7.1.4.2 LUN and file system for DB2 with 720 disks FC



DB2 PS SAP® TRBK

137/168

File system capacity Techno NumDisks Num

LUNs

LUN size Storage Pool

Striping

/db2/D3D/ 2000 GB FC 176 1 2000 GB Yes

/db2/D3D_log0

/db2/D3D_log1

/db2/D3D_log2

/db2/D3D_log3

2000 GB

2000 GB

2000 GB

2000 GB

FC

FC

FC

FC

176 1

1

1

1

2000 GB Yes

/db2/D3D_SAP® data1

…………………..

/db2/D3D_SAP® data16

388 GB

388 GB

FC

FC

176

176

1

1

388 GB

388 GB

Yes

Yes

7.1.4.3 Detailed of the LUNs configuration

SAP® fs DS8800 ID + LUN ID EP cap

SAP® data1 2bc0000000000002100) P0 388 GB

SAP® data2 2c10000000000002100) P1 388 GB

SAP® data3 2c10000000000002003) P2 388 GB

SAP® data4 2c10000000000002002) P3 388 GB

SAP® data5 2c10000000000002001) P0 388 GB

SAP® data6 2bc0000000000002103) P1 388 GB

SAP® data7 2c10000000000002000) P0 388 GB

SAP® data8 2bc0000000000002102) P3 388 GB

SAP® data9 2bc0000000000002101) P0 388 GB

SAP® data10 2bc0000000000002003) P1 388 GB

SAP® data11 2bc0000000000002000) P2 388 GB

SAP® data12 2bc0000000000002002) P3 388 GB

SAP® data13 2c10000000000002103) P0 388 GB

SAP® data14 2bc0000000000002001) P1 388 GB

SAP® data15 2c10000000000002102) P2 388 GB

SAP® data16 2c10000000000002101) P3 388 GB

SAP® fs DS8800 ID + LUN ID EP cap

log0 2c10000000000002004 P0 2000 GB

log1 2c10000000000002104 P1 2000 GB

log2 2bc0000000000002004 P2 2000 GB



DB2 PS SAP® TRBK

138/168

log3 2bc0000000000002104 P3 2000 GB

tiebreaker 2bc0000000000002005 P0 2000 GB

db2fs1 2c10000000000002105 P1 2000 GB

notused 2c10000000000001005 P2

db2d3d 2c10000000000002005 P3 2000 GB

7.1.5 Network fabric overview

7.1.5.1 Infiniband network Topology

For HA test purposes, 2 Mellanox switches (IS5030) have been set up for providing switch failure

tolerance. Here is the topology:

In case of a switch failure, half on the cluster is brought offline.

On the outline above, 2 types of cables have been used:

- From member/switch and coupling/switch : 5m Copper cables QSFP

- Between switches : 10m Optic cables QSFP

Having four links between the 2 switches ensure an optimal throughput, as each coupling facility

has 4 HCAs.



DB2 PS SAP® TRBK

139/168

7.1.5.2 Infiniband Subnet Manager configuration

Among the 2 switches, one has been chosen to act as the Subnet Manager. It is keeping the

identifier of each HCAs and is aware of the global Infiniband topology. The switch running as a

Subnet Manager is a licensed Mellanox feature, and has been activated in mlx-switch1. The

other switch (mlx-switch2) is considering as passive, as it is only routing the packets without

having the responsibility of maintaining the IB network.

7.1.5.3 Fiber Channel network

Each database server is connected to the storage system (DS8800) via a storage area network.

This network is created using dedicated SAN switches in the infrastructure. As described in the

server part, each member has 4 connections as they perform intensive I/O while coupling

facilities have only 2 links on the SAN.

Another smaller SAN connection is used between the SAP® injector and a DS5000 system (peer-

to-peer connection), to store all the data coming from the run.

7.1.5.4 Ethernet network

4 networks are used to ensure the communication through the DB2 pureScale® architecture

components:

Network Type Machines

connected

Subnet MTU

Administration 1 Gb Ethernet All 10.7.21/24 1500

Application 1 Gb Ethernet SAP® servers 10.1.1.0/24 1500

Database 10 Gb Ethernet SAP® + DB2

servers

10.1.2.0/24 1500

Interconnect 10 Gb QDR

Infiniband

DB2 servers 10.9.9.0/24

(IPoIB)

65520

The administration network is reachable from the Internet thanks to a Virtual Private Network.

All other networks are isolated and dedicated to this architecture.

Remark: the gateway of the 10.7.21.0/24 network is also used for NTP (Network Time Protocol)

synchronization.

For more information concerning the IP addressing map, please refer to the file pureScale – IP

Addressing.xls



DB2 PS SAP® TRBK

140/168

7.1.6 Operating Systems

7.1.6.1 Suse Entreprise Linux 11 SP1

All servers are using Suse Enterprise Linux (SLES) 11 SP1. Kernel has been upgraded on the DB2

servers from the original version to allow the newest OFED driver (for Infiniband) to be installed.

Machine Type Kernel version

Database Servers (x3690 x5) 2.6.32.24-0.2-default

Application Servers (HS22) 2.6.32.16-0.54-default

7.1.6.2 I/O device drivers

• Multipathing driver

The multipathing driver used for this benchmark is DM-MPIO that comes with the SLES 11 SP1

distribution. The configuration file (multipath.conf) follows IBM support recommendations. The

policy is round-robin based (rr_min_io = 100), and paths are grouped by priority. Here is an

extract of the multipath.conf file:

defaults {

polling_interval 30

failback immediate

no_path_retry 5

rr_min_io 100

path_checker tur

# features "1 queue_if_no_path"

user_friendly_names yes

# These are the default settings for 2107 (IBM DS8000)

# Uncomment them if needed on this system

device {

vendor "IBM"

product "2107900"

path_grouping_policy group_by_serial

}

• Linux IO scheduler parameter

Here are Linux parameters concerning the IO scheduling, for each accessible disk:

- IO Queue size : 128

- Scheduler used : Completely Fair Queuing

At a lower level, the use of NCQ (Native Command Queuing) also comes with sets of parameters

- NCQ queue size : 32

For deeper details about the LUN mapping please refer to multipath command display or

storage configuration documentation.

• Infiniband driver



DB2 PS SAP® TRBK

141/168

For the database servers, the OFED stack is used to provide the driver for Infiniband HCA.

On one node, here is the result of the infiniband driver self test command:

---- Performing Infiniband HCA Self Test ----

Number of HCAs Detected ................ 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... OFED-1.5.2-20100824-0600: 1.5.2-2.6.16.60_0.66.1_smp

Host Driver RPM Check .................. PASS

HCA Firmware on HCA #0 ................. v2.7.0

Host Driver Initialization ............. PASS

Number of HCA Ports Active ............. 1

Port State of Port #1 on HCA #0 ........ UP 4X QDR

Port State of Port #2 on HCA #0 ........ DOWN

Error Counter Check on HCA #0 .......... PASS

Kernel Syslog Check .................... PASS

------------------ DONE ---------------------

7.1.6.3 Settings tuned

• Kernel parameters

For performance improvement, the use of several HCA (Host Channel Adapter – Infiniband

Adapter) has been recommended on the coupling facilities. As the uDAPL needs IPoIB

configuration for each HCA, several IP addresses from the same subnet are utilized on the same

physical machine.

The Linux kernel doesn’t handle this configuration by default. Thus, the kernel parameter

net.ipv4.conf.all.arp_ignore has been modified to the value “2”, as recommended in

the OFED stack documentation.

7.2 Certification results

On September 12, 2011, SAP® published the following press release which summarize the key

results achieved on the previously described configuration.

The SAP® TRBK (Transaction Banking) Standard Application Benchmark performed on August 09,

2011 by IBM in Montpellier, France, has been certified with the following data:

The scenario contained 90,000,000 accounts and 1,800,000,000 postings.

Part I: Day Processing

No. of postings to bank accounts/hour: 56,518,000

CPU utilization database servers: 89% (CF: 59%, Member 0: 97%,

Member 1: 97%, Member 2: 97%,



DB2 PS SAP® TRBK

142/168

Member 3: 96%)

CPU utilization application servers: 62%

Part II: Night Processing

No. of balanced accounts/hour: 22,382,000

CPU utilization database servers: 36% (CF: 75%, Member 0: 26%,

Member 1: 26%, Member 2: 26%,

Member 3: 25%)

CPU utilization application servers: 47%

The software configuration for all steps of the SAP® TRBK benchmark:

Operating system, all servers: SUSE Linux Enterprise Server 11 SP1

RDBMS database servers: IBM DB2 9.7 ESE with pureScale

feature

SAP® Business Suite software: Banking Services from SAP® 7.0

Configuration:

5 Database Servers:

(1 CF, 4 Members)

IBM x3690 X5, 2 processors / 20 cores /

40 threads, Intel Xeon Processor E7-

2870, 2.40 GHz, 64 KB L1 cache and 256

KB L2 cache per core, 30 MB L3 cache

per processor, 256 GB main memory

21 Application Servers:

1 Dialog/Batch/Message/Enq. server:

IBM BladeCenter HS22 7870, 2

processors / 8 cores /

16 threads, Intel Xeon Processor X5570,

2.93 GHz,

64 KB L1 cache and 256 KB L2 cache per

core, 8 MB L3 cache per processor, 48

GB main memory

20 Dialog/Batch server: IBM BladeCenter HS22 7870, 2

processors / 8 cores /

16 threads, Intel Xeon Processor X5570,

2.93 GHz,

64 KB L1 cache and 256 KB L2 cache per

core, 8 MB L3 cache per processor, 48

GB main memory

Certification number: 2011035

7.3 Tips & recommendations

7.3.1 Database Configuration

Single CF



DB2 PS SAP® TRBK

143/168

Only a single CF was used for the certification runs. We were unable to measure the impact of

running with a secondary CF in the benchmark configuration. However, single versus duplexed

CF tests on the PoC configuration showed only about a 1% overhead.

BCA_BCAS_EVBST Table

Space Map Page (SMP) contention on the BCA_BCAS_EVBST table caused INSERT and UPDATE

performance on this table to suffer due to reclaims. After moving this table from a tablespace

with extent size 2 to a tablespace with an extent size 256, SMP contention was eliminated since

there was no need to frequently extend the table. As an additional tuning step we enabled

APPEND mode for this table which provided an 8.5% improvement to Day run throughput.

DFKKCOHARC Table

Contention on DFKKCOHARC data pages was resolved by partitioning this table into a 16-arm

union-all-view. INSERT times improved by about 10x.

7.3.2 SAP® Application Server

SAP® Enqueue Server Bottleneck

The number of SAP® Enqueue work processes on the SAP® Central Instance needed to be

increased from 1 to 4 in order to address poor Enqueue Server performance. This bottleneck

initially limited Day run throughput to 36 Million Postings per Hour.

7.3.3 Operating System

SLES11 SP1 CPU Scheduler

We made use of the sched_compat_yield kernel tunable to revert to the SLES10 version of the

CPU scheduler. As a result of this we observed a significant reduction in system CPU time from

about 40% to 15-20% and Day run throughput increased by about 10%.

SLES11 SP1 IO Wait Accounting

This is more of an observation. During this certification effort we observed 10-15% IO wait time

during the Day run. This was a change from the earlier performance tests we did on SLES10,

where reported IO wait time was minimal. We observed that it is only the castout engines that

are in iowait and looking at their kernel stack, they are collecting IOs in the "get_events" kernel

function. This function was changed in the SLES11 kernel such that it now accounts for iowait,

thus we now see threads in this function in an iowait state. In the SLES10 kernel, this function

did not account for iowait.



DB2 PS SAP® TRBK

144/168

7.3.4 Network Topology

Three InfiniBand Cards in CF

Three IB cards were used in the CF. Moving from 2 IB cards to 3 IB cards reduced CF response

times by about 50%. Because of the limited number of PCI-X slots in the x3690s, the 10 Gigabit

Ethernet card was removed and replaced by the third IB card.



DB2 PS SAP® TRBK

145/168

8 Best Practices for DB2 pureScale® and SAP® TRBK

DB2 pureScale® considerations

DB2 pureScale® facilitates the linear growth of workloads across multiple systems in a fashion

not previously achievable with DB2 for Linux, Unix, Windows. The existence of an extremely

fast-access shared memory system like the Coupling Facility enables one data image to be

effectively shared across many systems. The more update intensive a workload is, however, the

more likelihood there is of contention for data pages between members. This contention can be

controlled and tuned, and this tuning is the key to achieving excellent performance with TRBK.

There is a distinct difference in this respect between the Day and Night processing of TRBK.

Night processing can be segmented to allow difference application servers to process different

ranges of accounts in parallel. If the work coming from an application server aligns with the

partitioning of certain key tables in TRBK, there need be no sharing of table data between

database members. This facilitates isolated parallel processing of accounts and near-perfect

workload scaling with the number of members.

The Day workload is somewhat different. The payitem processing being performed by an

arbitrary benchmark user on an arbitrary application server can reference any account number.

With a busy system processing millions of payitems per hour, there is inevitably going to be

some degree of contention on account data. This is particularly true for index pages, due to their

tree structure. This contention can be controlled with the use of table partitioning and table and

index freespace values.

8.1 Tablespace Definitions

8.1.1 Extent Size

Members maintain an extent sized space map buffer for local space allocation. For tables where

there is a high volume of inserts or page allocation changing updates, increasing extent size of

the tablespace containing the table to the maximum value (255) is strongly recommended.

Examples of TRBK tables that fall into this category and benefit from a large extent size are

BCA_PAYMITEM, BCA_GL_PAYMITEM, and BCA96.

8.1.2 Increase Size

Extending tablespaces is an expensive operation with DB2 pureScale®, and has negative impacts

on performance. For this reason, a reasonable INITIALSIZE should be set, and INCREASESIZE

should not be left at the default value. By default, we found INCREASESIZE to be a relatively

small value of around 1GB. With heavy insert activity such as is seen during the TRBK Night



DB2 PS SAP® TRBK

146/168

processing, INCREASESIZE should be set to at least 50GB in order to keep the number of increase

actions to a minimum.

8.2 Table Compression

Traditionally, from the performance perspective, tables benefit from the use of DB2’s row

compression feature when the cpu cost of decompressing a row for use is outweighed by the

overall savings flowing from a reduced memory footprint and from reduced transfer times from

disk. Tables that benefited from table compression in the PoC fell into 3 categories:

� Tables with high ongoing IO requirements, such as PAYMITEM

� Tables whose data are not frequently re-referenced after insert, such as DFKKCOH

� Tables that are effectively read-only, like BUT000

8.3 Data and Index Percent Free

Allocating tables with a portion of data and index pages unused (achieved by use of the PCTFREE

table and index attribute) is beneficial in allowing subsequent row growth (update) and insert

activity to proceed without reference to space map pages.

Tables benefiting from use of PCTFREE fell into different categories

� For tables inserted randomly, like PAYMITEM, PCTFREE changes on both indexes and

data were found to be effective

� Tables that are heavily inserted and rarely re-referenced, like DFKKCOH benefit from

index (leaf and level2) compression

� Tables that are effectively read-only, i.e. heavily referenced but rarely updated, such as

BUT000 are candidates for minimal PCTFREE (i.e, zero).

8.4 Data Partitioning

A key technique that can be used to minimize contention between members is table

partitioning. For tables that have column data that is uniform, and where the activity against

that column data is evenly spread by member, the table can be divided into segments and

logically joined with a view that unions all the pieces. An example of such a column in TRBK is

CONTRACT_INT. This column contains an effectively random GUID value. It exists (with some

variation in the name) in most of the large TRBK tables.

SAP® code contains an optimization for shared data systems that allows the allocation of ranges

of data to the batch processes running on specific application servers. As SAP® application

server work processes connect to specific members, this means that, with the right partitioning

schema, Night processing activity is predominantly local, with very little or no contention for

table data or index pages between members.

Day processing cannot be set up in the same way, as each benchmark user on any appserver can

reference or insert to part of the major tables. However, having partitioned tables reduces the

incidence of data contention between members.



DB2 PS SAP® TRBK

147/168

9 Conclusions The combined set of results (Day, Night, scalability and HA) demonstrate that DB2 pureScale® is

capable of delivering excellent performance when running a very challenging workload in an

environment that effectively guarantees 100% availability.

The very large scale of database (90 million accounts) coupled with the realistic nature of the

SAP® TRBK benchmark makes this exercise a very concrete proof point of the exceptional

robustness and capability of the DB2 pureScale® solution.

Some of the metrics achieved give an indication of the scale of the workload:

Statements per second (Day run): 169000

Statements per second (Night run): 444500

Transactions per second (Day run): 4889

Log writes (Day run): 53 megabytes per second

Log writes (Night run): 200 megabytes per second

The high availability testing done demonstrates that the DB2 pureScale® solution can suffer the

complete failure of major components such as a database member of a CF, and recover to the

same transaction rate with minimal transaction loss and with complete data integrity.



DB2 PS SAP® TRBK

148/168

10 Appendix

10.1 db2dsdriver.cfg

<configuration>

<DSN_Collection>

<dsn alias="D3D" name="D3D" host="DB2PS-member0" port="5912" />

</DSN_Collection>

<databases>

<database name="D3D" host="DB2PS-member0" port="5912">

<parameter name="keepAliveTimeout" value="5"/>

<parameter name="TcpipConnectTimeout" value="2"/>

<acr>

<parameter name="enableAcr" value="true"/>

<parameter name="enableSeamlessAcr" value="true"/>

<parameter name="affinityFailbackInterval" value="180"/>

<parameter name="maxAcrRetries" value="1"/>

<alternate_server_list>

<server name="server1" hostname="DB2PS-member0" port="5912" />




</alternate_server_list>

<affinity_list>

<list name="list1" serverorder="server1,server2,server3,server4" />
























</affinity_list>

<client_affinity_defined>

<client name="client1" hostname="SAP® serv1" listname="list1" />






DB2 PS SAP® TRBK

149/168










</client_affinity_defined>

<client_affinity_roundrobin>

</client_affinity_roundrobin>

</acr>

</database>

</databases>

<parameters>

<parameter name="CommProtocol" value="TCPIP"/>

<parameter name="enableAcr" value="true"/>

<parameter name="enableSeamlessAcr" value="true"/>

</parameters>

</configuration>



DB2 PS SAP® TRBK

150/168

10.2 Malloc Bomb

malloc.pl:

#!/usr/bin/perl -w # # Very simple memory filling mechanism # # Usage: muse <size_to_fill> (in mb) # # The script will fork processes if the amount of m emory is # too large (to cope with the 2GB process limits) # my $MAX_SINGLE_PROC = 100; my $LONG_TIME = 300; if( $#ARGV != 0 ) { print STDERR "Usage: $0 <size_to_al loc_in_mb>\n"; exit( 1 ); } my $size = shift @ARGV; my $procs = int( $size / $MAX_SINGLE_PROC ); my $remain = $size - ($procs * $MAX_SINGLE_PROC ); if( $remain == 0 ) { $procs--; $remain = $MAX_SINGLE_PROC; } print "Will allocate approximately $size mb.\n"; print "Forking $procs children at $MAX_SINGLE_PROC mb each.\n"; print "And allocating $remain mb in parent.\n"; for( my $i=0; $i < $procs; $i++ ) { my $p = fork(); if( $p == 0 ) { # Child my $a = malloc( $M AX_SINGLE_PROC ); sleep $LONG_TIME; exit( 0 ); } } my $a = malloc( $remain ); sleep $LONG_TIME; exit 0; sub malloc { my $mb = shift @_; $mb--; return( "." x (1024*512*$mb)); }



DB2 PS SAP® TRBK

151/168

10.3 DB2 Registry Settings [e] DB2DBDFT=D3D

[i] DB2_CA_INITIAL_CONNECTION_POOL_SIZE=2

[i] DB2_INDEX_BRANCH_LOCKING=ON

[i] DB2_CA_PORTS=56001,56002,56003,56004

[i] DB2_BLOCKING_WITHHOLD_LOBLOCATOR=NO [DB2_WORKLOAD]

[i] DB2_AGENT_CACHING_FMP=OFF [DB2_WORKLOAD]

[i] DB2_SD_PAGE_REUSE_THRESHOLD=100000

[i] DB2_CA_CONNECTION_POOL_SIZE=120

[i] DB2_NUM_NOTIFY_BUFFERS=256

[i] DB2_TRUST_MDC_BLOCK_FULL_HINT=YES [DB2_WORKLOAD]

[i] DB2_CREATE_INDEX_COLLECT_STATS=YES [DB2_WORKLOAD]

[i] DB2_ATS_ENABLE=YES [DB2_WORKLOAD]

[i] DB2_RESTRICT_DDF=YES [DB2_WORKLOAD]

[i] DB2_DUMP_SECTION_ENV=YES [DB2_WORKLOAD]

[i] DB2_OPT_MAX_TEMP_SIZE=10240 [DB2_WORKLOAD]

[i] DB2_WORKLOAD=SAP®

[i] DB2_TRUNCATE_REUSESTORAGE=IMPORT [DB2_WORKLOAD]

[i] DB2_MDC_ROLLOUT=DEFER [DB2_WORKLOAD]

[i] DB2RSHCMD=/usr/bin/ssh

[i] DB2_ATM_CMD_LINE_ARGS=-include-manual-tables [DB2_WORKLOAD]

[i] DB2_SKIPINSERTED=YES [DB2_WORKLOAD]

[i] DB2_VIEW_REOPT_VALUES=YES [DB2_WORKLOAD]

[i] DB2_OBJECT_TABLE_ENTRIES=65532 [DB2_WORKLOAD]

[i] DB2_OPTPROFILE=YES [DB2_WORKLOAD]

[i] DB2_IMPLICIT_UNICODE=YES [DB2_WORKLOAD]

[i] DB2STMM=APPLY_HEURISTICS:YES [DB2_WORKLOAD]

[i] DB2_INLIST_TO_NLJN=YES [DB2_WORKLOAD]

[i] DB2_MINIMIZE_LISTPREFETCH=YES [DB2_WORKLOAD]

[i]

DB2_REDUCED_OPTIMIZATION=4,INDEX,JOIN,NO_TQ_FACT,NO_HSJN_BUILD_FACT,STARJN_CARD_SKEW,NO_SORT_

MGJOIN,CART OFF,CAP OFF [DB2_WORKLOAD]

[i] DB2NOTIFYVERBOSE=YES [DB2_WORKLOAD]

[i] DB2_INTERESTING_KEYS=YES [DB2_WORKLOAD]

[i] DB2_EVALUNCOMMITTED=YES [DB2_WORKLOAD]

[i] DB2_EXTENDED_OPTIMIZATION=NLJOIN_KEYCARD [DB2_WORKLOAD]

[i] DB2_ANTIJOIN=EXTEND [DB2_WORKLOAD]

[i] DB2COMPOPT=327685,704 [DB2_WORKLOAD]

[i] DB2ATLD_PORTS=60000:65000

[i] DB2ENVLIST=INSTHOME SAP® SYSTEMNAME dbs_db6_schema DIR_LIBRARY LD_LIBRARY_PATH

[i] DB2COUNTRY=1

[i] DB2COMM=TCPIP [DB2_WORKLOAD]

[i] DB2AUTOSTART=NO

[g] DB2SYSTEM=DB2PS-member0

[g] DB2INSTDEF=db2d3d

10.4 DB Manager Configuration Settings

Database Manager Configuration

Node type = Enterprise Server Edition with local and remote clients



DB2 PS SAP® TRBK

152/168

Description Parameter Current Value

------------------------------------------------------------------------------------ Database manager configuration release level = 0x0e00

CPU speed (millisec/instruction) (CPUSPEED) = 4.133012e-07

Communications bandwidth (MB/sec) (COMM_BANDWIDTH) = 1.000000e+02

Max number of concurrently active databases (NUMDB) = 1

Federated Database System Support (FEDERATED) = NO

Transaction processor monitor name (TP_MON_NAME) =

Default charge-back account (DFT_ACCOUNT_STR) =

Java Development Kit installation path (JDK_PATH) = /db2/db2d3d/sqllib/java/jd

Diagnostic error capture level (DIAGLEVEL) = 3

Notify Level (NOTIFYLEVEL) = 3

Diagnostic data directory path (DIAGPATH) = /db2/db2d3d/sqllib/db2dump

Size of rotating db2diag & notify logs (MB) (DIAGSIZE) = 0

Default database monitor switches

Buffer pool (DFT_MON_BUFPOOL) = ON

Lock (DFT_MON_LOCK) = ON

Sort (DFT_MON_SORT) = ON

Statement (DFT_MON_STMT) = ON

Table (DFT_MON_TABLE) = ON

Timestamp (DFT_MON_TIMESTAMP) = ON

Unit of work (DFT_MON_UOW) = ON

Monitor health of instance and databases (HEALTH_MON) = OFF

SYSADM group name (SYSADM_GROUP) = DBD3DADM

SYSCTRL group name (SYSCTRL_GROUP) = DBD3DCTL

SYSMAINT group name (SYSMAINT_GROUP) = DBD3DMNT

SYSMON group name (SYSMON_GROUP) =

Client Userid-Password Plugin (CLNT_PW_PLUGIN) =

Client Kerberos Plugin (CLNT_KRB_PLUGIN) =

Group Plugin (GROUP_PLUGIN) =

GSS Plugin for Local Authorization (LOCAL_GSSPLUGIN) =

Server Plugin Mode (SRV_PLUGIN_MODE) = UNFENCED

Server List of GSS Plugins (SRVCON_GSSPLUGIN_LIST) =

Server Userid-Password Plugin (SRVCON_PW_PLUGIN) =

Server Connection Authentication (SRVCON_AUTH) = NOT_SPECIFIED

Cluster manager = TSA

Database manager authentication (AUTHENTICATION) = SERVER_ENCRYPT

Alternate authentication (ALTERNATE_AUTH_ENC) = NOT_SPECIFIED

Cataloging allowed without authority (CATALOG_NOAUTH) = NO

Trust all clients (TRUST_ALLCLNTS) = YES

Trusted client authentication (TRUST_CLNTAUTH) = CLIENT

Bypass federated authentication (FED_NOAUTH) = NO

Default database path (DFTDBPATH) = /db2/D3D

Database monitor heap size (4KB) (MON_HEAP_SZ) = 32768

Java Virtual Machine heap size (4KB) (JAVA_HEAP_SZ) = 2048

Audit buffer size (4KB) (AUDIT_BUF_SZ) = 0

Size of instance shared memory (4KB) (INSTANCE_MEMORY) = 62914560

Instance memory for restart light (%) (RSTRT_LIGHT_MEM) = 3

Backup buffer default size (4KB) (BACKBUFSZ) = 1024

Restore buffer default size (4KB) (RESTBUFSZ) = 1024

Agent stack size (AGENT_STACK_SZ) = 1024

Sort heap threshold (4KB) (SHEAPTHRES) = 0

Directory cache support (DIR_CACHE) = NO

Application support layer heap size (4KB) (ASLHEAPSZ) = 16

Max requester I/O block size (bytes) (RQRIOBLK) = 65000

Query heap size (4KB) (QUERY_HEAP_SZ) = 1000



DB2 PS SAP® TRBK

153/168

Workload impact by throttled utilities(UTIL_IMPACT_LIM) = 10

Priority of agents (AGENTPRI) = SYSTEM

Agent pool size (NUM_POOLAGENTS) = 100

Initial number of agents in pool (NUM_INITAGENTS) = 5

Max number of coordinating agents (MAX_COORDAGENTS) = AUTOMATIC(200)

Max number of client connections (MAX_CONNECTIONS) = AUTOMATIC(MAX_COORDAGENTS)

Keep fenced process (KEEPFENCED) = NO

Number of pooled fenced processes (FENCED_POOL) = MAX_COORDAGENTS

Initial number of fenced processes (NUM_INITFENCED) = 0

Index re-creation time and redo index build (INDEXREC) = RESTART

Transaction manager database name (TM_DATABASE) = 1ST_CONN

Transaction resync interval (sec) (RESYNC_INTERVAL) = 180

SPM name (SPM_NAME) =

SPM log size (SPM_LOG_FILE_SZ) = 256

SPM resync agent limit (SPM_MAX_RESYNC) = 20

SPM log path (SPM_LOG_PATH) =

TCP/IP Service name (SVCENAME) = SAP® db2D3D

Discovery mode (DISCOVER) = SEARCH

Discover server instance (DISCOVER_INST) = ENABLE

SSL server keydb file (SSL_SVR_KEYDB) =

SSL server stash file (SSL_SVR_STASH) =

SSL server certificate label (SSL_SVR_LABEL) =

SSL service name (SSL_SVCENAME) =

SSL cipher specs (SSL_CIPHERSPECS) =

SSL versions (SSL_VERSIONS) =

SSL client keydb file (SSL_CLNT_KEYDB) =

SSL client stash file (SSL_CLNT_STASH) =

Maximum query degree of parallelism (MAX_QUERYDEGREE) = ANY

Enable intra-partition parallelism (INTRA_PARALLEL) = NO

Maximum Asynchronous TQs per query (FEDERATED_ASYNC) = 0

No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) = AUTOMATIC(9182)

No. of int. communication channels (FCM_NUM_CHANNELS) = AUTOMATIC(2048)

Node connection elapse time (sec) (CONN_ELAPSE) = 10

Max number of node connection retries (MAX_CONNRETRIES) = 5

Max time difference between nodes (min) (MAX_TIME_DIFF) = 1

db2start/db2stop timeout (min) (START_STOP_TIME) = 10

CF Server Configuration:

Memory size (4KB) (CF_MEM_SZ) = 62000128

Number of worker threads (CF_NUM_WORKERS) = 48

Number of connections (CF_NUM_CONNS) = AUTOMATIC(14)

Diagnostic error capture level (CF_DIAGLEVEL) = 2

Diagnostic data directory path (CF_DIAGPATH) =

10.5 DB2 Configuration Settings

Database Configuration for Database d3d

Description Parameter Current Value

------------------------------------------------------------------------------

Database configuration release level = 0x0e00

Database release level = 0x0e00



DB2 PS SAP® TRBK

154/168

Database territory = en_US

Database code page = 1208

Database code set = UTF-8

Database country/region code = 1

Database collating sequence = IDENTITY_16BIT

Alternate collating sequence (ALT_COLLATE) =

Number compatibility = OFF

Varchar2 compatibility = OFF

Date compatibility = OFF

Database page size = 16384

Dynamic SQL Query management (DYN_QUERY_MGMT) = DISABLE

Statement concentrator (STMT_CONC) = OFF

Discovery support for this database (DISCOVER_DB) = ENABLE

Restrict access = NO

Default query optimization class (DFT_QUERYOPT) = 5

Degree of parallelism (DFT_DEGREE) = 1

Continue upon arithmetic exceptions (DFT_SQLMATHWARN) = NO

Default refresh age (DFT_REFRESH_AGE) = 0

Default maintained table types for opt (DFT_MTTB_TYPES) = SYSTEM

Number of frequent values retained (NUM_FREQVALUES) = 10

Number of quantiles retained (NUM_QUANTILES) = 20

Decimal floating point rounding mode (DECFLT_ROUNDING) = ROUND_HALF_EVEN

Backup pending = NO

All committed transactions have been written to disk = NO

Rollforward pending = NO

Restore pending = NO

Multi-page file allocation enabled = YES

Log retain for recovery status = NO

User exit for logging status = NO

Self tuning memory (SELF_TUNING_MEM) = OFF

Size of database shared memory (4KB) (DATABASE_MEMORY) = 60293120

Database memory threshold (DB_MEM_THRESH) = 10

Max storage for lock list (4KB) (LOCKLIST) = 200000

Percent. of lock lists per application (MAXLOCKS) = 97

Package cache size (4KB) (PCKCACHESZ) = 120000

Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) = 2400000

Sort list heap (4KB) (SORTHEAP) = 2048

Database heap (4KB) (DBHEAP) = AUTOMATIC(10000)

Catalog cache size (4KB) (CATALOGCACHE_SZ) = 30000

Log buffer size (4KB) (LOGBUFSZ) = 4096

Utilities heap size (4KB) (UTIL_HEAP_SZ) = 10000

Buffer pool size (pages) (BUFFPAGE) = 10000

SQL statement heap (4KB) (STMTHEAP) = AUTOMATIC(38563)

Default application heap (4KB) (APPLHEAPSZ) = AUTOMATIC(256)

Application Memory Size (4KB) (APPL_MEMORY) = AUTOMATIC(241024)

Statistics heap size (4KB) (STAT_HEAP_SZ) = AUTOMATIC(4384)

Interval for checking deadlock (ms) (DLCHKTIME) = 10000

Lock timeout (sec) (LOCKTIMEOUT) = 3600

Changed pages threshold (CHNGPGS_THRESH) = 40

Number of asynchronous page cleaners (NUM_IOCLEANERS) = 12

Number of I/O servers (NUM_IOSERVERS) = 20

Index sort flag (INDEXSORT) = YES

Sequential detect flag (SEQDETECT) = NO

Default prefetch size (pages) (DFT_PREFETCH_SZ) = AUTOMATIC

Track modified pages (TRACKMOD) = NO

Default number of containers = 1

Default tablespace extentsize (pages) (DFT_EXTENT_SZ) = 2



DB2 PS SAP® TRBK

155/168

Max number of active applications (MAXAPPLS) = AUTOMATIC(4747)

Average number of active applications (AVG_APPLS) = AUTOMATIC(3)

Max DB files open per application (MAXFILOP) = 61440

Log file size (4KB) (LOGFILSIZ) = 32760

Number of primary log files (LOGPRIMARY) = 100

Number of secondary log files (LOGSECOND) = 154

Changed path to log files (NEWLOGPATH) =

Path to log files = /db2/D3D/logs/DBPART

Overflow log path (OVERFLOWLOGPATH) =

Mirror log path (MIRRORLOGPATH) =

First active log file =

Block log on disk full (BLK_LOG_DSK_FUL) = YES

Block non logged operations (BLOCKNONLOGGED) = NO

Percent max primary log space by transaction (MAX_LOG) = 0

Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0

Group commit count (MINCOMMIT) = 1

Percent log file reclaimed before soft chckpt (SOFTMAX) = 1000

Log retain for recovery enabled (LOGRETAIN) = OFF

User exit for logging enabled (USEREXIT) = OFF

HADR database role = STANDARD

HADR local host name (HADR_LOCAL_HOST) =

HADR local service name (HADR_LOCAL_SVC) =

HADR remote host name (HADR_REMOTE_HOST) =

HADR remote service name (HADR_REMOTE_SVC) =

HADR instance name of remote server (HADR_REMOTE_INST) =

HADR timeout value (HADR_TIMEOUT) = 120

HADR log write synchronization mode (HADR_SYNCMODE) = NEARSYNC

HADR peer window duration (seconds) (HADR_PEER_WINDOW) = 0

First log archive method (LOGARCHMETH1) = OFF

Options for logarchmeth1 (LOGARCHOPT1) =

Second log archive method (LOGARCHMETH2) = OFF

Options for logarchmeth2 (LOGARCHOPT2) =

Failover log archive path (FAILARCHPATH) =

Number of log archive retries on error (NUMARCHRETRY) = 5

Log archive retry Delay (secs) (ARCHRETRYDELAY) = 20

Vendor options (VENDOROPT) =

Auto restart enabled (AUTORESTART) = ON

Index re-creation time and redo index build (INDEXREC) = SYSTEM (RESTART)

Log pages during index build (LOGINDEXBUILD) = OFF

Default number of loadrec sessions (DFT_LOADREC_SES) = 1

Number of database backups to retain (NUM_DB_BACKUPS) = 12

Recovery history retention (days) (REC_HIS_RETENTN) = 60

Auto deletion of recovery objects (AUTO_DEL_REC_OBJ) = OFF

TSM management class (TSM_MGMTCLASS) =

TSM node name (TSM_NODENAME) =

TSM owner (TSM_OWNER) =

TSM password (TSM_PASSWORD) =

Automatic maintenance (AUTO_MAINT) = OFF

Automatic database backup (AUTO_DB_BACKUP) = OFF

Automatic table maintenance (AUTO_TBL_MAINT) = ON

Automatic runstats (AUTO_RUNSTATS) = ON

Automatic statement statistics (AUTO_STMT_STATS) = ON

Automatic statistics profiling (AUTO_STATS_PROF) = OFF

Automatic profile updates (AUTO_PROF_UPD) = OFF

Automatic reorganization (AUTO_REORG) = OFF

Auto-Revalidation (AUTO_REVAL) = DEFERRED

CF Resource Configuration:



DB2 PS SAP® TRBK

156/168

CF database memory size (4KB) (CF_DB_MEM_SZ) = 61380000

Group buffer pool size (4KB) (CF_GBP_SZ) = 55994112

Global lock memory size (4KB) (CF_LOCK_SZ) = 2000128

Shared communication area size (4KB) (CF_SCA_SZ) = 2337792

Catchup target for secondary CF (mins)(CF_CATCHUP_TRGT) = AUTOMATIC(15)

Currently Committed (CUR_COMMIT) = ON

CHAR output with DECIMAL input (DEC_TO_CHAR_FMT) = NEW

Enable XML Character operations (ENABLE_XMLCHAR) = YES

WLM Collection Interval (minutes) (WLM_COLLECT_INT) = 0

Monitor Collect Settings

Request metrics (MON_REQ_METRICS) = BASE

Activity metrics (MON_ACT_METRICS) = BASE

Object metrics (MON_OBJ_METRICS) = BASE

Unit of work events (MON_UOW_DATA) = NONE

Lock timeout events (MON_LOCKTIMEOUT) = NONE

Deadlock events (MON_DEADLOCK) = WITHOUT_HIST

Lock wait events (MON_LOCKWAIT) = NONE

Lock wait event threshold (MON_LW_THRESH) = 5000000

Number of package list entries (MON_PKGLIST_SZ) = 32

Lock event notification level (MON_LCK_MSG_LVL) = 1

SMTP Server (SMTP_SERVER) =

SQL conditional compilation flags (SQL_CCFLAGS) =

Section actuals setting (SECTION_ACTUALS) = NONE

10.6 enable_dr_nsd.sh

The purpose of this script is to re-map the replicated devices back to the NSD meta-data that

was imported as part of DR site setup process.

DB2PS-member2:~/scripts # vi enable_dr_nsd.sh

#!/bin/ksh # Remove nsddevices userexit rm -f /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover



DB2 PS SAP® TRBK

157/168

# List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m # Mount the GPFS filesystems cat > /tmp/drfs.out <<EOF d3d log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/drfs.out` do mmmount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all –L

10.7 disable_dr_nsd.sh

GPFS daemons during startup find NSDs by reading an NSD ID (in first two sectors) from each

block device visible to it. However, unless a site-failure is initiated LUNs replicated via Metro-

Mirror is not host-addressible on the DR site. Therefore we need to ensure that GPFS is not able

to see those Metro-Mirror replicated NSD definitions (we imported during DR site setup), during

the non-site failure times.

The following script masks those Metro-Mirror replicated NSD definitions from the DR site GPFS

cluster by creating a nsddevices user-exit and initiating mmdevdiscover to rescan the devices

from GPFS.

DB2PS-member2:~/scripts # vi disable_dr_nsd.sh

#!/bin/ksh # Unmount the PPRC'd GPFS filesystems cat > /tmp/drfs.out <<EOF d3d



DB2 PS SAP® TRBK

158/168

log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/drfs.out` do mmumount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all -L # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: # # a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name.



DB2 PS SAP® TRBK

159/168

# # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES: # # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]] then # Add function to discover disks in the Lin ux environment. multipath -l | awk '$3 ~/dm-/ && $1~/[_qr]$ / || $1~/^[mq_ backup]/ {print $3}' | sort | xargs -I {} echo "{} dmm" fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit



DB2 PS SAP® TRBK

160/168

# Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m

10.8 disable_dr_nsd_and_enable_fc_nsd.sh

The purpose of this script is to unmap block devices backed by PPRC replicated NSDs and map

block devices backed by FC target devices.

DB2PS-member0:~/scripts # cat disable_dr_nsd_and_enable_fc_nsd.sh #!/bin/ksh # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: # # a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name. # # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES:



DB2 PS SAP® TRBK

161/168

# # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]] then # Add function to discover disks in the Lin ux environment. multipath -l | awk '($1 ~/^f/ || $1 ~/^db2f s1/) && $3 ~/dm-/ {print $3}' | sort | xargs -I {} echo "{} dmm" fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit # Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m



DB2 PS SAP® TRBK

162/168

exit 0

10.9 suspend_or_resume_gpfs_fs.sh

This script allows, to:

a) suspend/freeze

b) resume/thaw

GPFS file systems replicated via PPRC link from Primary site to the DR site. The script must be

run from one of the cluster nodes on the primary site.

DB2PS-member2:~/scripts # cat suspend_or_resume_gpfs_fs.sh #! /bin/ksh if [ $# -ne 1 ] then echo "Usage $0 <suspend|resume>" exit fi option=$1 case $1 in suspend|resume ) echo $option ;; *) echo "Wrong option : $option"; exit ;; esac echo "Issuing GPFS I/O $option to filesystems used by database D3D ..."; mmlsnsd | awk '$1 ~/^[^qr File -]/ {print $1}' | se d '/db2fs1/'d | xargs -I {} mmfsctl {} $option

10.10 remote_flash_copy.sh

This script allows:

a) Creating a Remote FlashCopy relationship

b) Refreshing / re-syncing an already established Remote FC relationship

c) Listing the status of a Remote FC relationship / copy-session

d) Deleting an already established Remote FC relationship

The script should be run from a node on primary site with DSCLI installed and has a network

connection to the remote DS8K frame as well.

DB2PS-member2:~/scripts # cat remote_flash_copy.sh #target disk = source disk + 3000 if [ $# -ne 1 ] then echo "Usage $0 <mkremoteflash|rmremoteflash|remotef lash|lsremoteflash>" exit fi option=$1 case $1 in



DB2 PS SAP® TRBK

163/168

lsremoteflash|remoteflash|rmremoteflash|mkremotefla sh ) echo $option ;; *) echo "Wrong option : $option" exit ;; esac list="1000 1002 1004 1006 1008 1010 1012 1014 1016 1018 1022" fc_list1="" for source in $list do (( target = $source + 3000 )) fc_list1="$fc_list1 $source:$target" done list="1101 1103 1105 1107 1109 1111 1113 1115 1117 1119" fc_list2="" for source in $list do (( target = $source + 3000 )) fc_list2="$fc_list2 $source:$target" done case $option in mkremoteflash) cmd=mkremoteflash; opt="-freeze -record -persist" ;; remoteflash) cmd=resyncremoteflash; opt="-freeze -record -persist" ;; rmremoteflash) cmd=rmremoteflash; opt="-quiet" ;; lsremoteflash) cmd=lsremoteflash; opt="-l";; esac echo "Executing the following DSCLI command for use r option $option:" # Execute DSCLI command echo "/opt/ibm/dscli/dscli -cfg /root/scripts/PR_ds cli.profile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 $opt $fc_list1" /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/10 $opt $fc_list1 echo "/opt/ibm/dscli/dscli -cfg /root/scripts/PR_ds cli.profile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 $opt $fc_list2" /opt/ibm/dscli/dscli -cfg /root/scripts/PR_dscli.pr ofile $cmd -dev IBM.2107-75WR541 -conduit IBM.2107-75WR821/11 $opt $fc_list2 # Unfreeze the FlashCopy session to prevent TPCR ti mingout and suspending copy-pairs in the PPRC consistency group case $option in mkremoteflash|remoteflash) ssh root@DB2PS-member0 ". /root/.profile; /opt/ibm/dscli/dscli -cfg /root/scripts/DR_dscli.pr ofile unfreezeflash -dev IBM.2107-75WR541 10 11";; esac

Notes:

Work-around to avoid PPRC session being suspended when TPC-R is managing the copy

session



DB2 PS SAP® TRBK

164/168

• Both mkremoteflash and resyncremoteflash are issued using '-freeze ' option in

our script to ensures that volumes on the target LSSs are consistent with the source LSSs

volumes.

• The -freeze option creates a Freeze Consistency Group condition -- i.e. it'll trigger a queue

full condition for the source volumes,

o All writes to the source volumes are queued by the host and are written after the queue

full condition is reset.

o This queue full condition is reset by an extended long busy timeout condition -- default

for the timeout is 2 min.

o The timeout condition affects all FlashCopy source volumes that are contained within a

respective logical subsystem (LSS) and that are established or modified with the -freeze parameter.

• When -freeze option is issued to a PPRC target which is the case when using Remote FC,

the following is the behaviour:

o The Freeze Consistency Group condition stops further I/O to PPRC target volumes (which

in turn act as the source for FC).

o Until extended long busy timeout condition is exceeded, those volumes will be frozen.

o Note that TPC-R monitoring timeout (default is 30 sec) is less than long busy timeout

(default is 2min).

o We issued Remote FC commands (mkremoteflash / resyncremoteflash )

directly on the remote from using DSCI, which is outside of TPC-R – i.e. TPC-R did not

issue the I/O freeze on the PPRC pipe.

o Hence, that will cause TPC-R to detect this I/O freeze on the remote frame as an error

condition, which will in turn result in TPC-R cutting the PPRC link and suspending the

replication copy sets.

o The I/O freeze will be reset after extended long busy timeout condition is exceeded.

However, that will not automatically re-start/re-sync PPRC session managed by TPC-R.

• To avoid having to manually restart the PPRC session every time a Remote FC resync is done,

the following work-around is done:

o Issue the FC command with consistency group to all of the FC pairs

o Issue a resume to each of those LSSes once all of the FC commands finished -- a FC

'unfreezeflash' command

o Assuming this was done quickly enough (i.e. before TPC-R detects), then non of the

PPRC copy sets would be suspended

• This is the reason for the unfreezeflash command issued after, either a mkremoteflash or a resyncremoteflash command in our script.

Work-around to use Remote FlashCopy when the PPRC copy-set spans over multiple LSSes

• Remote FC command –conduit option limits source:target pairs to one LSS.

• Hence if your PPRC copy-set spans over multiple LSSes, then you need to issue Remote FC

commands grouped by copy-sets within a single LSS. For example, since our test

configuration had copy sets in two LSSes, (10 and 11) we had to issue two Remote FC

commands in our script, one for each LSS.



DB2 PS SAP® TRBK

165/168

• Note that the FC Freeze Consistency is only applicable for copy-sets within that LSS. Hence,

this would mean we are staggering our consistency over a period of time when more than

one LSS is involved.

• Hence to guarantee the integrity of the FC target clone, we suspend both DB2 and GPFS I/O

before issuing either a mkremoteflash or a resyncremoteflash command.

The two profile scripts that describe the configuration used to connect to each Storage frame, is

as follows: DB2PS-member2:~/scripts # cat DR_dscli.profile # # DS CLI Profile for ATS_MOP # # # created and modified by easy_dscli hmc1: 10.15.8.70 hmc2: username: db2 password: xxxxxxxx devid: IBM.2107-75WR541 remotedevid: IBM.2107-WR541 fullid: off banner: on verbose: off paging: off olc:off # End of Profile DB2PS-member2:~/scripts # cat PR_dscli.profile # # DS CLI Profile for ATS_MOP # # # created and modified by easy_dscli hmc1: 10.15.8.71 hmc2: username: db2 password: xxxxxxxx devid: IBM.2107-75WR821 remotedevid: IBM.2107-75WR821 fullid: off banner: on verbose: off paging: off olc:off # End of Profile

10.11 disable_repl_nsd.sh

An nsddevices user-exit to unmap (mask) both Metro-Mirror and FC replicated NSDs on DR

cluster. This script must be run on all DR cluster nodes.



DB2 PS SAP® TRBK

166/168

GPFS daemons during startup find NSDs by reading an NSD ID (in first two sectors) from each

block device visible to it. However, unless a site-failure is initiated LUNs replicated via Metro-

Mirror is not host-addressable on the DR site. Similarly, unless there is a need to access the

clone copy, the FC targets should be accessed as well. Therefore we need to ensure that GPFS is

not able to see both Metro-Mirror replicated NSD definitions (we imported during DR site

setup), and FC target NSD definitions, during the normal operational state.

/root/scripts/ disable_repl_nsd.sh #!/bin/ksh # Unmount the PPRC'd/FC GPFS filesystems (note both PPRC and FC targets have same NSD meta) cat > /tmp/dr_or_fc_fs.out <<EOF d3d log0 log1 log2 log3 SAP® data1 SAP® data10 SAP® data11 SAP® data12 SAP® data13 SAP® data14 SAP® data15 SAP® data16 SAP® data2 SAP® data3 SAP® data4 SAP® data5 SAP® data6 SAP® data7 SAP® data8 SAP® data9 EOF for f in `cat /tmp/dr_or_fc_fs.out` do mmumount $f -a done # List all mounted GPFS file systems echo "\nMount status of all GPFS filesystems define d in the cluster ...\n" mmlsmount all -L # Create nsddevices script cat > /var/mmfs/etc/nsddevices <<"EndofUserExit" #!/bin/ksh # @(#)53 1.5 src/avs/fs/mmfs/ts/config/nsdd evices.sample, mmfs, avs_rche, rche0933a 8/9/04 16:51:23 ################################################### ########################### # # When properly installed, this script is invoked b y the # /usr/lpp/mmfs/bin/mmdevdiscover script. # # INSTALLATION GUIDELINES FOR THIS SCRIPT: #



DB2 PS SAP® TRBK

167/168

# a) edit this script using the configuration gui delines below # b) copy this script to /var/mmfs/etc/nsddevices # c) ensure this script is executable (chmod +x / var/mmfs/etc/nsddevices) # # DESCRIPTION OF NSD DEVICE DISCOVERY: # # The mmdevdiscover script and conversely this sc ript are invoked by # the mmfsd daemon when it tries to discover or v erify physical # devices previously defined to GPFS with the mmc rnsd command. These # scripts identify devices found in the /dev file system on the local # machine that may correlate to NSDs defined to G PFS. # # GPFS uses the list of devices output by these s cripts in mapping # the NSD name listed in the configuration databa se to a local device # in the /dev file system. When an NSD is create d via the mmcrnsd # command it is marked with a unique identifier w ritten to sector # two of the device. This unique identifier is r ecorded in the # configuration database along with the user reco gnizable NSD name. # # During GPFS disk discovery each device name out put by mmdevdiscover # and nsddevices is opened in turn and sector two of each device is # read. If a match between an NSD identifier on the device and an # identifier recorded in the configuration databa se is found, then # this machine has local access to the NSD device . I/O is thus # subsequently performed via this local /dev inte rface. # # CONFIGURATION AND EDITING GUIDELINES: # # If this script is not installed then disk disco very is done # only via the commands listed in mmdevdiscover. # # If this script is installed and returns a NON Z ERO return code # then the disk discovery commands listed in mmde vdiscover will ALSO # be run. # # If this script is installed and returns a ZERO return code # then the disk discovery commands listed in mmde vdiscover will NOT # be run. # # The output from both this script and nsddevices is a number # of lines in the following format: # # deviceName deviceType # # where (deviceName) is a device name such as (hd isk1) # and (deviceType) is a set of known disk types. Consult # # /usr/lpp/mmfs/bin/mmdevdiscover # # for a list of currently known deviceTypes # # Example output: # # hdisk1 hdisk # hdisk2 hdisk # ################################################### ########################### osName=$(/bin/uname -s) if [[ $osName = Linux ]]



DB2 PS SAP® TRBK

168/168

then # Add function to discover disks in the Lin ux environment.

multipath -l | awk '($1 ~/^db2fs1/ || $1~/[_qr]$/ | | $1~/^[mq_ backup]/ && $3 ~/^dm-/ {print $3})' | sort | xargs -I {} ech o "{} dmm"

fi if [[ $osName = AIX ]] then :# Add function to discover disks in the AI X environment. fi # To bypass the GPFS disk discovery (/usr/lpp/mmfs/ bin/mmdevdiscover), return 0 # To continue with the GPFS disk discovery steps, # return 1 EndofUserExit # Set executable permissions chmod +x /var/mmfs/etc/nsddevices # Run GPFS device discovery /usr/lpp/mmfs/bin/mmdevdiscover # List the direct-attached nsds echo "\nGPFS NSDs visible to the cluster ...\n" mmlsnsd -m exit 0

Documents

SAP® TRBK Banking Benchmark using IBM DB2 pureScale® for ... · The project is referred to as the SAP® TRBK Benchmark using DB2 pureScale® on x86/Linux. 1.2 Document content The