© Copyright IBM Corporation 2019.

Copy Services Manager Update and

Bidirectional PPRC links Best Practices

Randy Blea, Copy Services Software Architect

Craig Gordon, WSC DS8000 Subject Matter Expert

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Accelerate with IBM Storage Webinars

The Free IBM Storage Technical Webinar Series Continues in 2019...

Washington Systems Center – Storage experts cover a variety of technical topics.

Audience: Clients who have or are considering acquiring IBM Storage solutions. Business Partners and IBMers are also welcome.

To automatically receive announcements of upcoming Accelerate with IBM Storage webinars, Clients, Business Partners and IBMers are welcome to send an email request to accelerate-join@hursley.ibm.com.

Located in the Accelerate with IBM Storage Blog: https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en

Also, check out the WSC YouTube Channel here: https://www.youtube.com/channel/UCNuks0go01_ZrVVF1jgOD6Q

2019 Upcoming Webinars:

December 10 – Copy Services Manager Update and Bidirectional PPRC links Best PracticesRegister Here: https://ibm.webex.com/ibm/onstage/g.php?MTID=e6037edc31ba8c6d8e6e6670c39134d5d

January 7 – What's New in IBM Spectrum Discover v2.0.2Register Here: https://ibm.webex.com/ibm/onstage/g.php?MTID=ef6d6e0a10c7d839c67765f00825c03ce

IBM Copy Services Manager 6.2.7 New FeatureseGA Dec 13, 2019

IBM Systems

Agenda

Quick Product overview

What’s New in 6.2.7

IBM Copy Services Manager

Installable across all major platforms:

Windows, Linux, AIX, z/OS, Linux for z Systems

And now pre-installed on the DS8000 HMC

Manages z Systems HyperSwap and

supports Power HA HyperSwap

to create highly available

solutions in Metro Mirror and

Metro Global Mirror

Environments

Practice As You Would Recover with

CSM Practice Volumes

Issue the ‘Flash’ command to

back up your remote site and

automatically restart replication

so that you can practice while

maintaining DR capabilities

Protect Your Data

Warning Prompts

User Roles

Site Awareness

Dynamic Images

Event and Threshold Alerts

* See your IBM Representative for further details.

Disaster Recovery and High Availability

One Step RecoveryProvides Automation for Advanced Copy Services

Single Point of Control Across Environment

Designed to Scale For Thousands of Relationships

RPO Monitoring and Historical Reporting

Simplifies the recovery process in multi-site replication,

protecting customer data while keeping costs down

Provides support for both distributed and mainframe data

Supports DS8880 and Spectrum storage systems

Easy to install. Easy to Setup. Easy to Manage.Simplify Your Replication

IBM SystemsIBM Systems | 6

IBM Systems | 6

What’s New in IBM Copy Services Manager V6.2.7

Support for new MT MM-MM 4 Site session type

Scheduled Task Improvements

Support for Email Alert Categories

Ability to “Lock” Practice Copy

Remove Copy sets with CSV from CSMCLI

Launch CSMCLI with -Xquickstart

Support for Linux PPC-LE

GUI Enhancements

Logging improvements for performance

IBM Systems

Scheduled Task Improvements

| 7

• In 6.2.7 CSM plans to support two new types of actions that can

be used when creating a Scheduled Task

• Wait for Percent Complete - when used this action will wait for

the given role pair in a given session to reach the provided

percent complete.

• Validate Role Pair Consistency – when used this action will

check whether the given role pair in the given session is

considered consistent before moving to the next action. If the

role pair is not consistent then the Scheduled Task will quit.

IBM Systems

Support for Email Categories

| 8

• In 6.2.7 CSM has added Alert Categories for different types of

notifications

• Different email addresses can be assigned to different Alert

Categories thus preventing a user from having to sift through all

notifications from the CSM server

• Types of categories are

• Session state change events

• Configuration change events

• Unexpected Severe status change

• Communication failures

• Active/Standby server connected

• Active/Standby server disconnected

• Scheduled Task completed

• Scheduled task failed

• Session RPO Trigger

• Logical Path change event

IBM Systems

Ability to ”Lock” Practice Copy

| 9

• In 6.2.7 CSM has the ability to lock the Practice copy of a

session.

• When the option “Prevent Reflash of Practice volume

after Flash or Recover” is set on a session, the ”Flash”

command will no longer be available until the customer

manually specifies they are ready by issuing the “Allow

Practice Flash” command.

• This feature helps ensure that if the Flash is taken

through scheduled batch processing, that a new Flash

does not overwrite a Flash that is currently being used for

test purposes.

IBM Systems

Launch CSMCLI with –Xquickstart option

| 10

• The CSMCLI runs in its own Java runtime which may be

on a different server than the CSM server.

• When running automated batch jobs against the CSMCLI,

there is a performance hit today as every command has

to startup Java.

• In CSM 6.2.7, the CSMCLI now supports usage of the -

Xquickstart option which enables Java to start up faster

• This option and other Java properties can be specified by

using a javaoptions.properties file

IBM Systems

Remove copy sets with CSV from CSMCLI

| 11

• In CSM 6.2.7 customers will now be able to use the

csmcli importcsv command specifying a csv file, in order

to remove copy sets

• The importcsv file supports adding copy sets via CSV by

default today

• Now an option –action tag can be specified for either

“add” or “remove” to indicate what the csv file should be

used for

IBM Systems

GUI Enhancements

| 12

• In CSM 6.2.7 improvements have been made to CSM

GUI

• New sorting and filtering capabilities on the View Copy

Set panel allow customers to see full name and nick

names for volumes as well as quickly find certain volumes

in a session or determine if there are volumes missing

• User defined Consistency Group names have been added

to the Session Overview panel to allow customers to see

which individual sessions are tied together by the

consistency group name for GM or HyperSwap

environments

IBM Systems

Support for Linux PPC Little Endian

| 13

• In CSM 6.2.7 CSM has added a new install option for

Linux PPC Little Endian servers

• In addition, customers can also choose to load the CSM

CLI on a Linux PPC Little Endian server for remote

access to a CSM server

IBM Systems

Logging improvements for performance

| 14

• In CSM 6.2.7 improvements have been made to CSM

logging

• On some platforms in previous releases CSM logging

could lead to a lot of system resource being used as CSM

writes to the file system.

• This release should limit the logging on those platforms to

help minimize the system resources and improve overall

performance of the CSM server

IBM Systems

New MT MM-MM 4 Site session type

| 15

• New session type Metro Mirror – Metro Mirror 4

site session for DS8000 storage systems

• Provides Metro Mirror replication or Global Copy

replication to two different targets from the same

source, and a cascaded Global Copy replication to

a fourth site

• Symmetrical configuration allows failover to

cascaded fourth site replication such that a MT

MM-MM setup can still be maintained

• HyperSwap capabilities are allowed for both Metro

Mirror legs allowing high level of high availability

IBM SystemsIBM Systems | 16

IBM Systems | 16

Reference Material

•CSM Log gathering DCF doc (coming soon) – Useful for tailoring log collection for what support needs to see

•CSM Support Portal - http://www.ibm.com/support/knowledgecenter/SSESK4/csm_kcwelcome.html

•CSM online Support Matrix - http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005402

•CSM Latest Downloads - http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005482

•CSM Flashes Alerts and Bullentins - https://www-947.ibm.com/support/entry/myportal/alerts/system_storage/storage_software/storage_infrastructure_management/ibm_copy_services_manager?productContext=332956944

•Knowledge Center – http://www.ibm.com/support/knowledgecenter/SSESK4/csm_kcwelcome.html

•RFE Community -https://www.ibm.com/developerworks/rfe/execute?use_case=changeRequestLanding&BRAND_ID=352&PROD_ID=1467&x=19&y=22

•Youtube channel - https://www.youtube.com/channel/UC2KaP7qpv4Z9yJ75YioI-pA

•Storageneers Blog - https://www.ibm.com/developerworks/community/blogs/storageneers?lang=en_us

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Host Adapters – Best Practices

17

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

16Gb Host Adapter – FCP and FICON

18

• 16Gb connectivity reduces latency and provides faster single stream and per port throughput

• 8GFC, 4GFC compatibility (no FC-AL Connections)

• Quad core Power PC processor upgrade

• Dramatic (2-3x) full adapter IOPS improvements compared to existing 8Gb adapters (for both

CKD and distributed FCP)

• Enables FC Endpoint Security

• Authentication

• Lights on Fastload avoids path disturbance during code loads

• Forward Error Correction (FEC) for the utmost reliability

• Additional functional improvements for IBM Z environments combined with z13/z14 host channels

• zHPF extended distance performance feature

• (zHPF Extended Distance II)

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

DS8900F 32Gb Host Adapter – FCP / FICON

• 32Gb FC / FICON connectivity

• 4 port adapter

• Up to 32 HA, 128 ports

• Auto-negotiate to 16Gb FC, 8Gb FC

• No 2/4Gb or FC-AL connections

• Enables FC Endpoint Security

• Authentication

• Encryption of Data in Flight (EDiF)

• Preservation of prior adapter capabilities:

• Forward Error Correction (FEC) for the utmost reliability at 16Gbps and 32Gbps

speeds

• Lights on Fastload avoids path disturbance during code loads

• Read Diagnostic Parameters (RDP) support

• zHPF extended distance performance feature

• (zHPF Extended Distance II) using FC-SB-6 FTBC to prevent SAN congestion and

eliminate long distance handshake latency

19

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

DS8900F 32Gb vs 16Gb FC HA – MB/s - CKD

0

1000

2000

3000

4000

5000

6000

7000

Single HA Read Single HA Write

MB

ps

CKD/zHPF – Sequential Read/Write MB/s

32 GFC 16 GFC2x increase in both read/write bandwidth

20

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

DS8900F 32Gb vs 16Gb FC HA – IOPs - CKD

0

50

100

150

200

250

Single HA Read Single HA Write

KIO

ps

CKD/zHPF - 4 KB Read/Write IOPs

32 GFC 16 GFC

21

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

DS8900F 32Gb vs 16Gb FC HA – Port Latency - CKD

0

0.02

0.04

0.06

0.08

0.1

0.12

4KB Single Port Read 4KB Single Port Write

Respo

nse T

ime (

msec)

CKD/zHPF - 4 KB Read/Write Base Latency (Server to DS8900F)

32 GFC

16 GFC

22

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Port / Host Adapter Utilization Guidelines

Metric Green Amber Red

FICON Host Adapter Utilization Percent < 50% 50% 70%

FICON Port Utilization Percent < 50% 50% 70%

Fibre Host Adapter Utilization Percent < 60% 60% 80%

Fibre Port Utilization Percent < 60% 60% 80%

Metro and Global Mirror Link Utilization Percent < 60% 60% 80%

zGM (XRC) Link Utilization Percent < 60% 60% 80%

• Leverage RMF or Spectrum Control performance data and use Disk Magic or StorM to model required resources

• Size the number of host adapters needed based on expected aggregate maximum bandwidth and maximum IOPS (use Disk Magic or other common sizing methods based on actual or expected workload)

• Best practice is to keep utilization not to exceed Amber levels

• Start planning for additional resources when you reach approximately 50% of the maximum values during some intervals (and not a single number for one day)

23

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

General Host Adapter Performance Planning Considerations

• A single 16/32Gb host adapter will not provide full line rate bandwidth with all ports active

• 16Gb HA - 3300MB/s Read / 1730MB/s Write

• 32Gb HA - 6500MB/s Read / 3500MB/s Write

• Spread the paths from all host systems across the available I/O ports, HA cards, and I/O Enclosure

• Spread the 32Gb HA’s across as many IO Enclosures as possible

• Maximum of 4 HA’s per I/O Enclosure but for optimal performance, recommend max of 2x32Gb HA’s per IO Enclosure

• Always have symmetric pathing by connection type (i.e. use the same number of paths on all host adapters used by each

connection type). For z/OS, all path groups should be symmetric (i.e. uniform number of ports per HA) and spread path

groups as widely as possible across all CKD host adapters

• Dynamic Path Selection (DPS) allows the channel subsystem to select any available (non-busy) path to initiate an operation to the

storage system

• Dynamic Path Reconnect (DPR) allows the DS8000 storage system to select any available path to a host to reconnect and resume a

disconnected operation

• When possible, use the same number of host adapter connections (especially for IBM z) as the number of connections

coming from the hosts

24

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Host Adapter Performance Planning Considerations - continued

• Don’t share a mainframe CHPID with tape and Flash devices

• Keep large block and small block transfers isolated to different host adapter cards

• For Distributed systems, each host system must have a minimum of two host connections to HA cards in

different I/O Enclosures on the DS8900F storage system and must use a multipathing device driver

• Share or isolate

• Share example

• Four host connections from each of four servers, all running a similar type workload, and four HAs. Spread the host connections with

each host attached to one port on each of four adapters

• Isolate example

• If one of the four servers is running an I/O-intensive workload, acquire two additional HAs and attach the four connections of the I/O-

intensive server to these adapters. Connect two host connections on each adapter. The other three servers remain attached to the

original four adapters

• Even though I/O ports on the same HA can be configured independently, recommend using separate HA

cards for the FICON protocol and FCP for host connectivity

25

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Replication Port Best Practice (1 of 2)

Replication – for best performance:

• Replication port pairs (primary to secondary) should be zoned 1:1

• With 16/32Gb HA’s, OK to use 1 port for replication and 1 / 2 ports for Host access (assuming network can exploit full replication port speed)

• For Global Copy or Global Mirror, assign even and odd LSS/LCU to different ports

• No benefit for MM

Odd LSSs

Ho

st Ad

ap

ter

0

1

2

3

Ho

st Ad

ap

ter

0

1

2

3

Even LSSs

Odd LSSs

Ho

st Ad

ap

ter

0

1

2

3

Ho

st Ad

ap

ter

0

1

2

3

SAN A

SAN B

Even LSSs

26

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Replication Port Best Practice (2 of 2)

Replication – for best performance:

• Avoid using an adapter port for both replication send and receive

27

I0000

I0130

Site 1 DS8000 Site 2 DS8000 Site 3 DS8000

I0000

I0130

I0000

I0130

I0000

I0130

I0000

I0130

Site 1 DS8000 Site 2 DS8000 Site 3 DS8000

I0000

I0130

I0001

I0131

I0000

I0130

Metro Global Mirror example

I0000

I0130

Site 1 DS8000 Site 2 DS8000

I0000

I0130

I0000

I0130

Site 1 DS8000 Site 2 DS8000

I0000

I0130

Bi-directional Global Mirror example

I0001

I0131

I0001

I0131

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

Metro Mirror Performance

0

2

4

6

8

10

12

1 HA 1 HA 2 HA 4 HA 8 HA

1 path 2 paths 2 paths 4 paths 8 paths

Seq

ue

nti

al W

rite

GB

/s

• The performance of a Metro Mirror path at

16Gb reflects the write performance of the

Host Adapter on the Metro Mirror secondary.

Using 2 ports on a Host Adapter does not

provide significant throughput increase

• Metro Mirror sequential performance scales

with the number of Host Adapters and for a

DS8888 can reach more than 10GB/s with 8

Host Adapters used for Metro Mirror

• With the 16Gb adapters sharing Host

Adapters (but not ports) between host and

replication traffic can provide more balanced

throughput given a particular number of host

adapters

28

Washington Systems Center - Storage

© Copyright IBM Corporation 2019.

0

2

4

6

8

10

12

14

16

18

20

1 Path 2 Paths 2 HA 4 Paths 4 HA 8 Paths 8 HA

GB

/se

c

Sequential Write / zHPF – MMDS8950F 40 Core – 8 HPFE Gen2

DS8950F (32GFC)

DS8950F (16GFC)

DS8900F Metro Mirror Bandwidth

29

Washington Systems Center - Storage

© Copyright IBM Corporation 2019. 30