So What is FlashCopy Doing? - GSE Homepageconferences.gse.org.uk/attachments/presentations/I0o5aw_1477897514… · So What is FlashCopy Doing? John Ticic IntelliMagic Date of presentation

So What is FlashCopy Doing?

John Ticic

IntelliMagic

Date of presentation (1/11/2016)

Session DB

http://tyc.gse.org.uk/

https://instagram.com/gse_uk/

https://www.linkedin.com/grp/home?gid=1741877

https://www.facebook.com/pages/Guide-Share-Europe-UK-Region/174907039241415

https://twitter.com/gseukc

Agenda

• Observations

– Volume FlashCopy

– Data Set FlashCopy

• Summary

• Appendix

• What is FlashCopy?

– Features

– Differences

• FlashCopy Interfaces

– Commands

• Disk Back-end


John Ticic – Senior Technical Consultant

Started in Systems Programming in 1984.

Joined IntelliMagic in 2008 as a Senior Consultant

Specialties include:

Disk/Tape performance

z/OS Performance

z/OS, zSeries implementation

Presenting (I/O classes, SHARE, GSE,..)


What is FlashCopy?


FlashCopy®

• Creates image of volume or data set – With or without full physical copy

– Source & target are immediately available for use

– Does not consume resources outside DSS

Source

Target


FlashCopy – Many Features

• Copy / NoCopy

• Persistent or Temporary

Relationship

• Data set copy in addition to

volume copy

– DFSMSdss allocates and

catalogs the copy

• Multiple relationships:

– Associate single source with

more than one target

• Incremental FlashCopy:

– FC command triggers

transmission of changes since

previous FC command

• FlashCopy Consistency Group

– Create group of target

volumes that is consistent

• Reverse Restore – The relationship can be

reversed by copying modified tracks from the target to the source volume


FlashCopy – Even More Features

• Fast Reverse Restore – Used by Global Mirror

• Virtual Concurrent Copy – Concurrent Copy (CC) first

issues FlashCopy and then copies data

• Space Efficient FlashCopy – Space Efficient means target

disk space is only required when write data needs to be hardened

• Other vendors support

FlashCopy, but also have their

own point-in-time technology

• EMC

– Timefinder Mirror/Clone/Snap

• HDS/HP

– ShadowImage

Track SE not available on DS8880 family


FlashCopy - Copy

COPY option initiates full source to target copy

• Sequential process, no RAID-5 write penalty

On-demand copy when updating tracks not yet copied

• ODC rate probably decreases over time

FlashCopy relationship automatically ended when full copy is complete

Full copy & ODC may impact application I/O performance

• But, when it’s over, it’s over


FlashCopy - Copy

Process target

Process source

FlashCopy / Copy

initiated

copy

FC relationship ended (automatically)

On-demand Copy copy Copy tracks


FlashCopy - Copy

Copy activity is background, low priority.

Internal copy tasks exist per disk adapter and per disk rank.

The maximum number of tasks per adapter/rank are limited.

I/O is optimized (sequential).


FlashCopy - Nocopy

On-demand copy for tracks updated on source or target

• ODC rate fully depends on source / target write activity

• Worst case: multiple sequential write processes in parallel to volumes in same RAID array

FC relationship should be ended ‘manually’ when finished processing target

• Avoid unnecessary ODC activity

• Logical status of target is unpredictable


FlashCopy - Nocopy

On-demand Copy

Process target

Process source

FlashCopy / Nocopy

Initiated

FC relationship ended manually


FlashCopy - Nocopy

All I/O is based on destaging modified data.

Before the destage, data needs to be copied from source to target disk.

Can affect random I/O.

I/O is background and limited in concurrency.


Incremental FlashCopy

Requires persistent relationship & change recording must be on

Sets up 2 track bitmaps on source

• (1) preserves ‘t0’ status of target

• (2) keeps a record of updates after ‘t0’

Initial FC invocation results in full copy

• ODC may occur during full / delta copy; driven by (1)

Incremental FC results in ‘delta’ copy based on (2)


Incremental FlashCopy

Process target

Process source

Initial FC Incremental FC

copy

On-demand Copy copy Full Copy of tracks

Delta Copy of tracks


FlashCopy Interfaces


Interaction

• TSO/E Command – Note: Starting z/OS 2.3, only QUERY

commands remain supported.

• ANTRQST API

• REXX ANTTREXX

• DFSMSdss

• ICKDSF

• Web Service – DS8K interface

Note: RACF and/or IKJTSOxx changes may

be needed.


Sample Commands

FCESTABL Establish a FlashCopy relationship

FCESTABL SDEVN(2001) TDEVN(2804) MODE(COPY)

ANTF0001I FCESTABL COMMAND COMPLETED FOR DEVICE 2001, COMPLETION CODE: 00

Instantaneous reply. Command accepted and executed.

Copy initiated, but not finished.

When copy is complete, the relationship will be terminated. No

message will be generated.

Caution: target device will have source volume label. Possible IPL

considerations.


Sample Commands

FCWITHDR Withdraw a FlashCopy relationship

FCWITHDR TDEVN(2A06)

ANTF0001I FCWITHDR COMMAND COMPLETED FOR DEVICE 2A06,

COMPLETION CODE: 00

Terminates the FlashCopy relationship, regardless of the current status. i.e. does not wait for complete volume to be copied when COPY is specified.

Status of target volume is unknown.

But, it may have the source volume label! Possible IPL considerations.


Sample Commands

FCQUERY Query a device

FCQUERY DEVN(2001) SHOWRELS(ALL)

ANTF0421I FCQUERY Relationship 1

DEVN SSID LSS CCA CU SERIAL ACT MAX XC PC CC RV SE SEQNUM

2001 1000 00 01 2107 0000000DDK61 0 65534 N N N N NN 00000000

RELATIONSHIP DETAIL STARTING TRACK: 00000000

DEVICE LONG BUSY FOR CG: NO WRITE INHIBITED: NO

---------------------------------------------------

NO RELATIONSHIPS FOUND FOR SPECIFIED DEVICE OR STARTING ADDRESS

ANTF0001I FCQUERY COMMAND COMPLETED FOR DEVICE 2001, COMPLETION CODE: 00


Sample Commands

SSID + LSS + CCA + Serial Number uniquely identify a device.

SSID Disk Subsystem Identifier. This must be unique within an LPAR.

LSS Logical Subsystem. Unique within a disk system.

CCA Channel Connection Address. Device number within an LSS

Some customers have LSS + CCA = z/OS Device address. Nice, but not required.


Sample Commands




2006 1000 00 06 2107 0000000DDK61 1 65534 N P N N NN 00000000



---------------------------------------------------

PARTNER SOURCE TARGET S F C C P C T S F P

LSS CCA SSID START START O V O A R R W E S M

--- --- ---- -------- -------- - - - - - - - - - -

0A 06 100A 00000000 00000000 Y Y N N N N Y N N N

NO. OF TRACKS: 0006E0CD TRACKS TO COPY: 0002E16C

ESTABL: 2015/07/27 13:43:19 LAST INCR: 2015/07/27 13:43:19

ANTF0001I FCQUERY COMMAND COMPLETED FOR DEVICE 2006, COMPLETION CODE: 00


Sample Commands




2006 1000 00 06 2107 0000000DDK61 1 65534 N P N N NN 00000000

z/OS

Device

Disk

Device

(Source)


Sample Commands


---------------------------------------------------



--- --- ---- -------- -------- - - - - - - - - - -

0A 06 100A 00000000 00000000 Y Y N N N N Y N N N

NO. OF TRACKS: 0006E0CD TRACKS TO COPY: 0002E16C

Disk

Device

(Target)

Full

Volume See Appendix for full

description

30051 Cyl 12585 Cyl


Disk Back-end


What is Available?

• RMF (CMF) provide performance data –

– From the z/OS LPAR

• Based on Start Subchannels (SSCH)

• Collected per logical volume

• FICON channel data available

– From the Disk Cache statistics

• Not SSCH but Locate Records (LR)

– From the Disk internal Ranks/Arrays

SMF 74.1

SMF 73

SMF 74.5

SMF 74.8


Front-end activity Front-end activity is

the count of SSCH

issued by z/OS

(SMF74SSC.)

We need to

consolidate data from

all LPARs and isolate

single disk systems.

Will all of these I/O’s

create disk internal

I/O?


I/O Types

HDD (arrays)

Read hit Read miss

Random Write

Cache

Seq. Write

Front-end I/Os

Reads/Writes

Sequential/Random

Cache hits/misses

Back-end I/Os

Reads/Writes

Cache staging

Additional I/O for Parity

Optimized for sequential

writes

How

many?


Front-end vs Back-end activity Very application and

I/O dependent.

Sometimes the back-

end I/O rate (blue) is

higher than the front-

end (red).

Being able to see the

back-end activity

allows us to observe

FlashCopy activity.


Sample FlashCopy

Let’s look at some disk performance data that contains FlashCopy

activity.

Using RMF/CMF data, we can observe the z/OS initiated I/Os, and look

at the back-end rank data.

The back-end rank data will show us the results of z/OS initiated I/Os

(those that made it to the back-end) and any other I/O activity (e.g.

FlashCopy.)


Front-end FlashCopy is a disk

internal I/O

technology.

We will not see it in

the front-end

performance data.

But – we may suffer

performance due to

FlashCopy back-end

activity.

~14000 I/Os


Front-end and Back-end We can see significant

back-end activity

(blue).

Note. Both of these

are to the same scale.

What about read/write

MB/s? ~14,000 I/Os

~130,000 I/Os


Back-end MB/s We can also see the

significant throughput,

by drive tier (IBM

DS8870).

1200 GB 10K

RAID6

300 GB 15K

RAID6

400 GB SSD

RAID5

Let’s break it down

(Read/Write).


Back-end MB/s We see a very close

correlation between

read (red) and write

(blue) MB/s (we

expected it for

FlashCopy activity.)

Let’s break it down

further into the 3 Tiers.


Back-end Read MB/s Most of the MBs read

activity is from the

SSD Tier.

What about writes?


Back-end Write MB/s Very few MBs are

being written to SSD!

This is to be expected

but has implications.

What do you want to

use this data for?

A large amount is on

1200GB 10K HDD!

What about the I/O

breakdown?


Back-end Read I/Os This is similar to the

Read throughput

distribution (by Tier.)

What about write

I/Os?


Back-end Write I/Os This is similar to the

Write throughput

distribution (by Tier.)

Note: The scale is

different.

How do they

compare?

Let’s put them onto

the same scale.


Back-end I/O Comparison Many fewer write I/Os

(Red). But, the same

amount of data.

FlashCopy is not

simple I/O!

What does the back-

end balance look like?


Back-end Storage Pool Balance All 4 Extent Pools are

reasonable well

balanced.

It is very important to

know where the

secondary FlashCopy

devices will be.

Spread evenly to

minimize disruption.

What about z/OS

Response time?


z/OS Response Time We clearly see an

increase in z/OS

response time (e.g.

between 7:00 and

9:00 PM)

The breakdown shows

us that it is Disconnect

time.

Lets look a little

deeper.


Disconnect Time Breakdown Looking closer at

Disconnect time

shows us that our

peak is caused by

HDD read response

time.

But – is it

FlashCopy?


Conclusive Evidence?

• We’ve seen what we think is FlashCopy activity affecting our Disk

Back-end response times, and causing z/OS response time

increases.

• But, how can we be sure?

• The previously shown commands can show FlashCopy activity.


FlashCopy Sessions I’m using data

gathered by our z/OS

Collector to show

FlashCopy activity.

We clearly see that we

have FlashCopy

relationships.

And, they don’t end!


FlashCopy Relationships

• Each FlashCopy source to target activity is assigned to a

relationship (see FCQUERY output.)

• Multiple relationships per volume are possible

– COPY relationships dissolve automatically after data has been copied.

– Other relationships remain indefinitely.


FlashCopy Relationships

• Active relationships does not mean that data is always being copied.

• But, if relationships exists, data may be copied (on demand) when

not expected.

• How much data is being copied?


Sample FlashCopy - Cylinders to Copy The total number of

cylinders isn’t really

too significant, but

how long it takes until

all activity ceases can

be critical.

Note: Data is interval

based so peaks can

(will) be much higher.

What have we

observed?


FlashCopy Observations

• “Kicking off” a FlashCopy process can certainly generate a large

back-end load.

• This load is not visible from z/OS, but can be observed with the

RMF/CMF 74.8 records (ESS – rank statistics.)

• But, you can’t observe and control FlashCopy if you can’t tell what is

generating the I/O load.

• The Relationships (in this case) remain even after the data has been

copied.


Observations


Observations

• In this section, I’d like to look at some of the different FlashCopy

executions and what the results are.

• FlashCopy.

– Full Volume COPY.

– Full Volume NOCOPY.

– Incremental + Data set.


Full Volume COPY

• FlashCopy COPY

– We expect a lot of Back-end activity, but it will cease when the copies

are complete and the relationships will be dissolved.

– Let’s look at an example:

• 42 x 3390-27. 83% full

• Approximately 1200 Thousand Cylinders. (~850 GB)

• IBM DS8870. 5 minute intervals.


Full Volume COPY There are some

existing sessions, but

the 42 Full Volume

COPY sessions start

at 12:15 PM.

We are done by 4:15

PM.

How fast are the

cylinders copied?


Full Volume COPY A quick start and then

very linear until all

sessions are

completed.

Let’s look at the disk

performance data.


Full Volume COPY This is the I/O rate

generated from z/OS

to this disk system.

A spike around 3:00

PM, but very little I/O.

What does the back-

end look like?


Full Volume COPY Only 2 drive tiers.

Read MB/S.

Initial spike, but it

settles down quickly.

Initially, data is read

from the SSD, but

older data is read from

the 7.2K drives.

Note: 3:00 PM z/OS

spike not even visible!


Full Volume COPY Write MB/s.

Initially, SSDs are

used, but most of the

write I/O is to the 7.2K

drives.

This is where most of

the new data will

reside until EasyTier

moves it around!

You need to be aware

of this!


Full Volume COPY The z/OS response

time (red) isn’t

affected in this

sample.

This disk system is

not heavily loaded, so

it can cope well with

the FlashCopy load.

But it still took 4 hours

to complete (42 x

3390-27)


Full Volume NOCOPY

• FlashCopy NOCOPY

– We actually expect little activity. Only on-demand-copy.

– Note: The relationships will need to be manually terminated.

– Same example:

• 42 x 3390-27. 83% full

• Approximately 1200 Thousand Cylinders. (~850 GB)


Full Volume NOCOPY We see the additional

42 sessions starting

at 7:50.

At 10:25, the sessions

are manually

terminated.

So, how many

cylinders need to be

copied?


Full Volume NOCOPY Almost no data is

being copied.

How busy is the disk

system?


Full Volume NOCOPY Very little z/OS

initiated I/O.

How much work do

the internal disk drives

need to do?


Full Volume NOCOPY Almost no back-end

I/O!

FlashCopy NOCOPY

only generates I/O

when demand-on-

copy is required.

Combine with

INCREMENTAL(YES)

to only copy the tracks

that have changed

since the last

FlashCopy command.


Customer FlashCopy

COPY INCREMENTAL(YES)

• What does a real customer environment look like?

• IBM DS8870 hardware

• FlashCopy COPY INCREMENTAL(YES) and Data Set FlashCopy.


Data Set Copy We see very many

relationships.

It looks like it takes 1

½ hours until all

extents are copied.

Each data set extent

is defined by a

relationship.

How many cylinders

need to be copied?

Existing

COPY +

Incremental

Data Set FC


Data Set Copy Many Cylinders!

Around 14,500

Thousand Cylinders

(~ 60 x 3390-A)

Start time is around

9:10

End time is around

10:45

What does the back-

end look like?


Data Set Copy Back-end activity has

a clear starting time.

Start time is around

9:10

At 10:45, we still have

significant back-end

activity.

End time is around

11:50

Compare to Response

Time.

10:45


Data Set Copy The z/OS I/O

response time (red)

clearly spikes when

the back-end (blue)

activity increases

significantly.

It does recover, but it

will take time.

The main component

was Disconnect time

(back-end HDD

access) – not shown.

3 ms


Data Set Copy Note:

This disk systems has

a relatively low read

cache hit rate (%).

This means it will be

more influenced by

the back-end read

response times.

(i.e. more read I/Os

requiring HDD

access)


FlashCopy

Overhead

• How much overhead is there in obtaining FlashCopy information?

• The IBM DS8K ensures that host I/O activity has a higher priority

than the FlashCopy workload.

• The FCQUERY requests are also prioritized to not impact disk

response time and disk activity.


FCQUERY Time (Sec.) Be aware that the

FCQUERY command

may sometimes take a

long time to complete.

The disk system

needs to prioritize the

internal activity and

favor response time.


Other Observations

• Each data set extent to be copied is represented by a unique

relationship.

• If PPRC (Metro Mirror) replication is being used, you can query the

remote device.

– Referred to as being an “Inband” command.

– E.g.

FCQUERY DEVN(2603) +

REMOTE(YES) QRYDVC(DDK61 X'0A' X'03') +

QRYSSID(100A)


Summary


Summary

• Be aware of location of FlashCopy target volumes.

– It may be on a slower tier.

– This may be relevant when using these volumes.

• If you want to dump (e.g. to tape) the target devices, wait until a

Copy is completely finished. This will stress the back-end disk

system much less.

• Results may be different for other vendors.


Summary

• FlashCopy may impact your z/OS response time.

– Ensure that any “Pre-online” FlashCopy activities are finished well

before online start.

• Define FlashCopy source and target devices in different ranks.

– Especially important for COPY activity.


Documentation IBM DFSMS Advanced Copy Services SC23-6847

IBM IBM DS8870 Copy Services for IBM z Systems SG24-6787

IBM IBM System Storage DS8870 Performance

Whitepaper

WP102183

IBM FlashCopy in a zSeries Environment Charlie Burger

ATS

HDS ShadowImage User Guide MK-92RD8021

EMC Implementing Local Replication using EMC

Timefinder and Recoverpoint and coexistance of

RP/CDP with SRDF on Symmetrix VMAX

H8981.2

IntelliMagic Monitoring FlashCopy


Thank you

www.intellimagic.com

Note: See appendix for code

samples and additional information


Session feedback

• Please submit your feedback at

http://conferences.gse.org.uk/2016/feedback/DB

• Session is DB


Appendix


Sample TSO Command //*

//* ISSUE FC COMMANDS FOR DATA AND FTP VOLUMES

//*

//DATA EXEC PGM=IKJEFT1A

//SYSTSPRT DD SYSOUT=*

//SYSPRINT DD SYSOUT=*

//SYSTSIN DD *

FCESTABL SDEVN(2005) TDEVN(2039) MODE(COPY)

/*


Sample REXX /* REXX */

command = 'FCQUERY'

args.0 = 3

args.1 = 'DEVN(2000)'

args.2 = 'QRYINFO()'

args.3 = 'QRYSIZE(200)'

call anttrexx command,'args'

say result

say args.2


Sample Assembler ANTRQST ILK=ESSRVCS,REQUEST=FCQUERY,FORMAT=FFQMAP, X

DEVN=Dev_No,SUBCHSET=Subchan, X

QRYSIZE=FQRY_Size,QRYINFO=(4), X

RETCODE=Pret,RSNCODE=Prsn,RETINFO=Pretinfo, X

MF=(E,PLIST)


Sample DFDSS //DSCOPY EXEC PGM=ADRDSSU


//SYSIN DD *

COPY DATASET(INCLUDE(IMUSJT1.**) -

EXCL(IMUSJT1.BRODCAST, -

IMUSJT1.ZF*.**) ) -

RENAMEU(IMUSJT2 ) -

FR(REQ) FCNOCOPY -

NOPACKING(**) -

ADMIN -

CATALOG -

ALLDATA(*) ALLEXCP TOL(ENQF)

/*


Sample ICKDSF //*

//* FCQUERY

//*

//QUERY EXEC PGM=ICKDSF


//VOL1 DD DISP=SHR,VOL=SER=JOHN01,UNIT=3390

//SYSIN DD *

FLASHCPY QUERY -

DDNAME(VOL1)

/*


Detailed Output - Data Set Copy Device 6020 SS=0 ABCDEF Source/Target SSID=6000/0000 CCA=20/00 LSS=60/00 Rels = 235

Rel 1 LSS=62 CCA=22 SSID=6200/6200

FC cap. PPRC Sess. FCPY Sess. SRC. Full Vol. Persist. INCR(YES).

SEQ= 0 TRKS= 3606120 LSIG= 0 MSIG 0

Established = 2016/06/13 17:59:18 Recent Increment = 2016/06/13 17:59:18

Source CCHH = 00000000 Target CCHH = 00000000

Source device 6020 SSID 6000 LSS 60 + CCA 20

Target device SSID 6200 LSS 62 + CCA 22

Full Volume. Incremental = yes.

Source device = 3,606,120 tracks (3390-A 229,763 Cylinders)


Data Set Copy Device 6020 SS=0 ABCDEF Source/Target SSID=6000/0000 CCA=20/00 LSS=60/00 Rels = 235

Rel 2 LSS=62 CCA=21 SSID=6200/6200

FC cap. PPRC Sess. FCPY Sess. SRC.



Source CCHH = 00780009 Target CCHH = 00780009

Disk

Timestamp

Tracks = 30 30 Tracks will be copied


Data Set Copy Device 6020 SS=0 ABCDEF Source/Target SSID=6000/0000 CCA=20/00 LSS=60/00 Rels = 235

Rel 235 LSS=62 CCA=21 SSID=6200/6200

FC cap. PPRC Sess. FCPY Sess. SRC.



Source CCHH = 2859000C Target CCHH = 2859000C

Disk

Timestamp

Tracks= 3,446,448 3,446,448 Tracks will be copied


ANTF042I



2006 1000 00 06 2107 0000000DDK61 1 65534 N P N N NN 00000000



---------------------------------------------------



--- --- ---- -------- -------- - - - - - - - - - -

0A 06 100A 00000000 00000000 Y Y N N N N Y N N N

SO is one of the following value

Y Indicates that this is a source relationship.

N Indicates that this is a target relationship.


ANTF042I

FV is one of the following values:

Y Indicates that this is a full volume relationship.

N Indicates that this is not a full volume relationship.

CO is one of the following values:

Y Indicates that this is a COPY relationship.

N Indicates that this is a NOCOPY relationship.

CA is one of the following values:

Y Indicates that the background copy is currently active for this relationship.

N Indicates that the background copy is not currently active for this relationship.

PR is one of the following values:

Y Indicates that this is a persistent relationship.

N Indicates this is not a persistent relationship.


ANTF042I

CR is one of the following values

Y Indicates that change recording is active for this relationship (incremental).

N Indicates that change recording is not active for this relationship.

TW is one of the following values:

Y Indicates that the target of the relationship is writable.

N Indicates that the target is write-inhibited.

SE is one of the following values:

N Indicates that the target of this relationship is not a space efficient volume.

F Indicates that the target of this relationship is space efficient and the relationship will be

failed if the repository becomes full.

I Indicates that the target of this relationship is space efficient and the source will become

write-inhibited if the repository becomes full.


ANTF042I

FS is one of the following values:

Y Indicates that the relationship is currently in a failed state.

N Indicates that the relationship is not currently in a failed state.

PM represents the Preserve Mirror state of the relationship. r one of the following

values:

N Indicates that the relationship was not established as a Preserve Mirror operation because

Preserve Mirror was not requested or Preserve Mirror Preferred was specified but could not be

accomplished.

P Indicates that the relationship was established as a Preserve Mirror operation because of a

Preserve Mirror Preferred request.

R Indicates that the relationship was established as a Preserve Mirror operation because of a

Preserve Mirror Required request.

S Indicates that this is a remote relationship established between two PPRC secondary devices as

part of a Preserve Mirror operation or the PM relationship type cannot be determined because of a

conflict in the source and/or target PPRC copy status.


Documents

So What is FlashCopy Doing? - GSE Homepageconferences.gse.org.uk/attachments/presentations/I0o5aw_1477897514… · So What is FlashCopy Doing? John Ticic IntelliMagic Date of presentation