View
220
Download
0
Category
Preview:
Citation preview
Host Based Volume Shadowing Update
Jeff FriedrichsVolume Shadowing Project Leader
Jeff.Friedrichs@hp.com
May 2004
Agenda
• Review of Recent Features• New Features to look forward to
May 2004
HBMM
Host Based Mini Merge
Available on V7.3-2 (with TIMA kits)
May 2004
What is Mini Merge?
• Full merge requires comparing entire shadow set
• But only blocks with I/O in progress need to be merged
• MSCP Based mini merge – Supported on HSCxx / HSDxx / HSJxx controllers– Controller tracks in-progress writes– Host can get list of writes from the controller
• Host Based Mini Merge (HBMM)– Selected cluster hosts track recent writes using write
bitmap– Bitmap is periodically reset to flush out old writes– Contents of bitmap drive mini merge operation
May 2004
Write Bitmap for Mini Merge
• Write bitmap originally released on V7.2-2 for mini copy– Each system has an in-memory bitmap to track write I/O– Each bitmap has one system that is the master– 2KB memory per Gbyte of storage per bitmap per system
• Originally 6 bitmaps, HBMM extended to 12– There are 6 bitmaps are available (per VU) for mini merge use– There are also 6 bitmap slots reserved for mini copy use
• HBMM recovery must be by a bitmap master system
• Use multiple bitmap masters for availability of the bitmap after a system crash
May 2004
MINI COPY Bitmaps
• Created with $DISMOUNT/POLICY=MINICOPY
• NO redundancy
• Uses bitmap first 6 bitmap slots
May 2004
HBMM Bitmap Master Policy
• The policy defines– number of bitmap masters for a VU in the cluster– the location of masters (in counted groups)– the bitmap reset threshold
• Named policies– Are known cluster wide– Any named policy can be assigned to an individual
VU or to multiple VUs– A named policy can be deleted at any time
• That does not affect VUs that had it applied
May 2004
Bitmap Master Policy
• A policy must be directly assigned to VU for HBMM to be enabled on that VU
• With the following exception– If a DEFAULT policy has been defined, then every VU in
the cluster, that does not have a named policy, will automatically “pick up” the DEFAULT policy
– To prevent a VU from “picking up” the DEFAULT policy use • SET SHADOW DSA1: / POLICY = HBMM = NODEFAULT
• In summary– the DEFAULT policy and a mix of VU specific policies
can be used in the cluster at the same time
May 2004
Policy Definition
SET SHADOW /POLICY = HBMM = (
(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2),
(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2),
RESET_THRESHOLD=50000 ) / NAME = POLICY_1
SET SHADOW DSA1: / POLICY = HBMM = POLICY_1
May 2004
Policy Definition continued
SET SHADOW DSA1: /POLICY = HBMM = (
(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2),
(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2),
RESET_THRESHOLD=50000 )
May 2004
Other HBMM Controls
SET SHADOW DSA1: / DISABLE = HBMM– Disables HBMM on DSA1
SET SHADOW DSA1: / ENABLE = HBMM– Enables HBMM on DSA1– If there is a policy for the VU to enable
SET SHADOW DSA1: / POLICY=HBMM=(MASTER=*)– Allows all systems in the cluster to be bitmap masters – First six to mount DSA1: will become masters
– Others eligible when a master dismounts VU or crashes
May 2004
Other HBMM Controls
SET SHADOW / POLICY = HBMM = (MASTER_LIST = …. )/ NAME = DEFAULT
– Creates default policy for all shadow sets that are mounted in the cluster that do not have a VU specific policy in place
SET SHADOW DSA1: / POLICY = HBMM = NODEFAULT– Disables HBMM for DSA1– Can be used to override a DEFAULT policy on a specific VU– Can be viewed as a policy governor
SET SHADOW / POLICY = HBMM / NAME=POLICY_1 / DELETE– Will delete the policy named “POLICY_1”
May 2004
HBMM rules
– If a policy is associated with a VU, HBMM is automatically enabled upon first mount on a system that has been named as bitmap master in that policy
– If a master system ceases to be a master (due to dismount or crash), a new master bitmap will be automatically started on another master system, subject to the policy in force on the VU
– Devices capable of MSCP Based (HSC/HSJ/HSD) mini merge are not eligible for HBMM
– To enable host based mini merge on a VU, all systems that mount the VU must be HBMM capable
May 2004
AMCVP
• Automatic Mini Copy on Volume Processing• aka Multiuse Bitmaps• Available in V8.3• Extension of HBMM
May 2004
AMCVP Considerations
• Multiuse bitmaps are valid for both mini MERGE and mini COPY operations
• Multiuse bitmaps are NOT zeroed, therefore mini MERGEs will take longer and longer
• Multiuse bitmaps must be deleted manually if the removed member is never coming back to the set
May 2004
Policy Definition
SET SHADOW /POLICY = HBMM = (
(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2, MULTIUSE=2),
(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=2, MULTIUSE=1),
RESET_THRESHOLD=50000 ) / NAME = POLICY_1
SET SHADOW DSA1: / POLICY = HBMM = POLICY_1
May 2004
Write Bitmap Message Modes
• Single Message Mode– Only one logical WBM write request per SCS
message
• Buffered Message Mode– Supports up to nine logical WBM write requests per
SCS message– Message dispatching occurs
• when the message is full
• when an internal WBM timer expires
May 2004
Write Bitmap Message Mode Control
SYSGEN Parameters for WBM
• WBM_MSG_UPPER– The upper threshold for the number of messages
sent during the calibration interval that will initiate buffered message mode
• WBM_MSG_LOWER– The lower threshold for the number of messages
sent during the calibration interval that will initiate single message mode
V8.3 and V8.3-1H1 – CHECK YOUR PARAMETERS!
May 2004
Merge and Copy Prioritization
May 2004
Current Merge and Copy Controls
• Management controls to determine order and choice of system for copy and merge operations are incomplete.
• Important volumes may be merged after less important volumes
• Systems better suited to perform merge or copy operations on some volumes are not always selected
May 2004
Merge and Copy Control
• Allow user to assign a priority to every VU
• Better predict which system will perform any transient state operations (merge or copy operations) – requires SYSGEN settings
• Utilize SHADOW_MAX_COPY dynamic characteristic
May 2004
Shadow Priority
• New command qualifier– $ SET SHADOW /PRIORITY = n DSAnnnn:
• A range of 0 through 10,000
• Default is 5000
• 1 is the lowest priority
• Zero has special meaning
• At MOUNT time each VU will be placed in system wide priority linked list by this value– VUs at the same priority have an undefined ordering
• Governs merge and copy priority for VUs on this system
May 2004
Show Shadow Priority
New command $ SHOW SHADOW /BY_PRIORITY
• Lists the DSA devices on this system using the priority assigned to each, highest to lowest
• Shows transient state % and system performing operation
Device Priority Virtual Unit State % Completed on Node
DSA3233: 3233 Steady State DSA2325: 2325 Not Mounted DSA42: 42 Full Merge Active 14% on CSGF1
May 2004
SYSGEN parameter
SHADOW_REC_DLY (Shadow Recovery Delay)
• This parameter governs how many seconds, after the VU enters a merge transient state, that this system will wait before it attempts to manage that state
• Default of 20 seconds
• Making this value different across the cluster will guide which system will manage transient state operations
May 2004
Using Priority List
Until all SHADOW_MAX_COPY threads are used on a system, the priority list is processed in two phases:
– Host Based Mini Merge or MSCP Based Mini MergeThen
• Full Copy state
or • Full Merge state
This implies that all HBMM VUs are processed before any full copy or full merge VUs … regardless of the priority value for any VU
May 2004
Per SSM Served Delay
Each SSM on an MSCP served path has an additional delay added before attempting to obtain the _COPIER lock to manage a merge operation.
If the default value (30 seconds) is not suitable it can be changed
• SET SHADOW/<tbs>=n DSAn:– Will use this value for all SSMs on an MSCP served
path• SET SHADOW/<tbs>=n /ALL
– Will change the default value of 30 that HBVS will use for all VUs that are mounted on this system
– Will change the value on all VUs currently mounted on this system
May 2004
Managing Transient State Operations
New SET SHADOW command qualifiers:/EVALUATE=RESOURCES
– Is a system specific command … i.e. not cluster specific • Thus it only affects the VUs that are being merged or
copied by this system
– Useful if the command is issued before• SHADOW_MAX_COPY is increased or is decreased
– To start or stop a merge or a copy operation
• The priority of a VU is changed– Priority of 0 now non-zero or the inverse– Priority value has been raised or lowered
May 2004
Managing Transient State Operations
New SET SHADOW command qualifier:/DEMAND_MERGE DSAn:Changes the state of the VU to a merge required state
• The type of merge initiated will depend on merge recovery characteristics that are enabled currently on the VU
• To insure that a full merge is initiated on a VU• Disable host based mini merge that it is currently enabled
• This is especially useful if the shadow set had been created with INIT/ SHADOW without / ERASE
• Can be used if differences are found on the members with ANALYZE /DISK /SHADOW
May 2004
The Future of Shadowing
V8.4 Enhancements
May 2004
HBMM Dismount
• Further extension of HBMM and AMCVP
• Allows all 12 bitmap slots to be used by HBMM or MiniCopy
• Provides bitmap redundancy for removed members
May 2004
Policy Definition
SET SHADOW /POLICY = HBMM = (
(MASTER_LIST = (NODE1,NODE2,NODE3), COUNT=2, MULTIUSE=2, DISMOUNT=2),
(MASTER_LIST = (NODE4,NODE5,NODE6), COUNT=3, MULTIUSE=2, DISMOUNT=2),
RESET_THRESHOLD=50000 ) / NAME = POLICY_1
May 2004
Fast MINI operations
Performance enhancement for Mini Copy and Mini Merge
Look ahead in the bitmap
No user intervention required
Significantly reduces I/O from SHADOW_SERVER
May 2004
Write Bitmap Enhancements
• Fix buffered mode transitions
• Multicast “set bit” messages
• Optimize sequential writes
May 2004
SPLIT_READ_LBNS
• Attempt to utilize read ahead controller cache
• Currently reads are “round robin” among members
• Simply divide the virtual unit into smaller chunks based on LBN and send all reads for the same area to the same member
• Only members with the lowest and same READ_COST are used
May 2004
SPLIT_READ_LBNS
$ show shad dsa33/full
Device $1$DGA33 Index 0 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DCA540 Error Count 0 LBN_LIMIT: 00000000
Device $1$DGA41 Master Member Index 1 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DB8B80 Error Count 0 LBN_LIMIT: 00000000
Device $1$DGA34 Index 2 Status 200000A0 mbr_src,mbr_valid,user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81DD5E40 Error Count 0 LBN_LIMIT: 00000000
May 2004
SPLIT_READ_LBNS
$ SET SHADOW/ENABLE=SPLIT_READ_LBNS$ SHOW SHADOW/FULL DSA33
Device $1$DGA33 Index 0 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DCA540 Error Count 0 LBN_LIMIT: 000F4240
Device $1$DGA41 Master Member Index 1 Status 000000A0 mbr_src,mbr_valid Ext. Member Status 00 Read Cost 2 Site 0 Member Timeout 120 UCB 81DB8B80 Error Count 0 LBN_LIMIT: 001E8480
Device $1$DGA34 Index 2 Status 200000A0 mbr_src,mbr_valid,user_supplied_read_bia Ext. Member Status 00 Read Cost 3 Site 0 Member Timeout 120 UCB 81DD5E40 Error Count 0 LBN_LIMIT: 00000000
May 2004
SPLIT_READ_LBNS
$ SET SHADOW –_$ /DISABLE=SPLIT_READ_LBN DSA1:
May 2004
One More!
May 2004
XMBRS
$ show dev DSA6
Device Device Error Volume Free Trans Mnt Name Status Count Label Blocks Count CntDSA16: Mounted 0 XMBRS 49954 1 1$1$DGA1: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA2: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA3: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA4: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA5: (CSGF2) ShadowSetMember 0 (member of DSA6:)$1$DGA6: (CSGF2) ShadowSetMember 0 (member of DSA6:)
May 2004
XMBRS
• To turn it on, just specify more than 3 members for the shadow set
$ MOUNT/SHADOW DSA16 –_$ /SHADOW=($1$dga1, $1$dga2, $1$dga3,
$1$dga4) –_$ DSA6
• No new qualifiers• No changes to DISMOUNT
May 2004
XMBRS Compatibility
• Mixed version, 3 member shadow sets will continue to work
• To use 4 or more members, all systems that have the VU MOUNTed, must have the new software
• MOUNT/INCLUDE of a once XMBR virtual unit on an older version may not find all of the members
May 2004
XMBRS Performance
• Performance testing in progress• Reads could be faster• Writes *WILL* be slower
• Important to use SET SHADOW commands– Noticeable difference with /COPY_SOURCE
May 2004
Questions??
Recommended