Advanced Technical Skills (ATS) North America€¦ · SAN1 SAN2 SVC node1 SVC node2 Power Server server port 1 server port 4 SVC IO Group N1P1 Node 1 Port 1 Assumes 2 host ports per

Advanced Technical Skills (ATS) North America

© 2012 IBM Corporation

Overview

Dan Braden and Bill WiegandIBM Advanced Technical SkillsMay 29, 2012

Dienstag, 29. Mai 12


© 2012 IBM Corporation2




Power VIO and SVC Disk Path Design

Dan Braden – IBM Power ATSBill Wiegand – IBM Storage ATS

December 14, 2011




Agenda

Determine and examine all possible 4 path designs Dual VIOS, dual SAN fabric and SVC Examine each for availability characteristics

Handling multiple VIOCsExtend to more SVC IO groupsIncreasing bandwidth




Objective

The objective of this presentation is to show alternative disk path designs to setup a dual VIOS environment with SVC storage, and their availability characteristics.

This will examine zoning with 4 vFCs, each connected to a single SVC port to generate 4 paths/LUN.




Definitions

SAN link – a cable connecting a storage port to the SAN or a server port to the SAN Disk path – a logical path from a specific host port to a specific storage port WWPN – World Wide Port Name – unique 16 digit ID for a host or storage port Active/passive disk subsystem – a disk subsystem with dual controller/processors

where a controller handles all IOs for a LUN except under failure scenarios Half the LUNs are typically owned by each controller to use all resources

VIOS – Virtual IO Server VIOC – Virtual IO Client vFC – Virtual Fibre Channel adapter created with VIO for a VIOC

Tied to a specific real FC adapter in a VIOS Has an online and offline WWPN to facilitate LPM The active WWPN changes with each LPM

LPM – Live Partition Mobility – the ability to move a running VIOC LPAR from one Power system to another Requires all IO for the VIOC to be virtualized thru VIOSs

WWPN zone: Typically a single initiator (host port) zone indicating which storage ports with which the host port may communicate: host port WWPN, storage port WWPN, storage port WWPN, ….




Zoning nomenclature

Single initiator zone – a zone with one host port, and one or more storage ports A best practice

Alias – Used to give a descriptive name to WWPNs Samples:

vFC0_myhost – the first virtual adapter port on the myhost LPARLPARa_fcs0 - fcs0 on a LPAR representing a host portN1_P3 – SVC Node 1 Port 3

This presentation assumes WWPN zoning




Availability Requirements

We’d like to setup the zoning such that we can survive a double failure Failure of two of : VIOS, SAN fabric or SVC node

But not failure of both VIOSs, both SAN fabrics or both SVC nodes which will always cause an outage

With 2 ports per VIOS or SVC node, we can ignore failure of individual ports Two ports failing in a VIOS or SVC is equivalent to the VIOS/SVC failing




Why 4 paths?

Minimizing the number of paths minimizes the time needed by AIX to determine paths have failed and to recover AIX must distinguish between a slow IO and a path failure

We don’t want to mark paths as failed if the IO is just slow




VIOC

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

Power Server

server port 1

server port 4

SVC IO Group

N1P1Node 1 Port 1

Assumes 2 host ports per VIOSAdditional ports can be used for additional bandwidth

One may create up to 4 vFCs here per VIOC

Availble host HBAs have 1, 2 or 4 ports

Each SVC node has 4 ports

SAN cabling

N2P3Node 2 Port 3




Given 4 host ports and 4 storage ports we have 16 potential paths Paths reduced by half due to dual SAN fabric Can be further reduced by SAN zoning at the switch Can be further reduced by LUN masking at the SVC

A LUN served from the SVC is served from one IO group only Connections to other IO groups do not increase the number of paths

Options for 4 paths: 4 vFCs each connected to a different storage port 2 vFCs each connected to two storage ports

Doesn’t meet availability requirements as each vFC only uses one SAN fabric, so failure of a SAN fabric and the real adapter the other vFC uses causes an outage

1 vFC connected to 4 storage portsDoesn’t meet availability requirements as this can only use one SAN fabric

Thus we’ll use sets of ports that include 4 unique host ports and 4 unique storage ports

We’ll examine all the possible ways to use 4 vFCs connected to 4 storage ports

Skip to slide 28 to see designs meeting the requirement Or go thru them to understand what designs to avoid and why

How do we get 4 paths?




Given 4 host ports and 4 storage ports, with one host port connected to one storge port, there are 24 ways to pair up the ports 4!=24 Not all can use all 4 storage ports Many are equivalent

How many ways are there to set this up?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 11 11 11 11 11 11 11 11 11 11 11 11 12 12 12 12 12 12 12 12 12 12 12 1211 11 12 12 12 12 11 11 12 12 12 12 11 11 11 11 12 12 11 11 11 11 12 1222 22 21 22 21 22 22 22 21 22 21 22 21 22 21 22 21 21 21 22 21 22 21 2122 22 22 21 22 21 22 22 22 21 22 21 22 21 22 21 21 21 22 21 22 21 21 21

Table entries are VIOS#, SVC node#

Design#

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 11 11 11 11 11 11 12 12 12 12 12 12 13 13 13 13 13 13 14 14 14 14 14 1422 22 23 23 24 24 21 21 23 23 24 24 21 21 22 22 24 24 21 21 22 22 23 2333 34 32 34 32 33 33 34 31 34 31 33 32 34 31 34 31 32 32 33 31 33 31 3244 43 44 42 43 32 44 43 44 41 43 41 44 42 44 41 42 41 43 42 43 41 42 41

Design#

Table entries are host port#, storage port#

Since there’s no difference between ports on a VIOS, or ports on a SVC node, we can redo the table showing VIOS# and SVC node# for the ports




How many ways are there to set this up?

Examining each design for unique ones, and reducing the design name to (SVCN#, SVCN#, SVCN#, SVCN#) for connection to VIOS1, VIOS1, VIOS2, VIOS2 respectively yields these 6 designs:

1. 1122 2. 12123. 12214. 21125. 21216. 2211

VIOS1 port 1Connected to


VIOS2 port1Connected to


Design SVCN# SVCN# SVCN# SVCN#

1 1 1 2 2

2 1 2 1 2

3 1 2 2 1

4 2 1 1 2

5 2 1 2 1

6 2 2 1 1

Or simply:




LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

SAN zoningZone1: vFC1 WWPN1, N1P1Zone2: vFC1 WWPN2, N1P1Zone3: vFC2 WWPN1, N1P3 Zone4: vFC2 WWPN2, N1P3Zone5: vFC3 WWPN1, N2P1Zone6: vFC3 WWPN2, N2P1Zone7: vFC4 WWPN1, N2P3Zone8: vFC4 WWPN2, N2P3 Italic zones are for inactive WWPNs

Design 1 with 4 paths/LUN - 1122

vFC2

vFC3

vFC4

Disk PathsvFC1 - N1P1vFC2 – N1P3vFC3 – N2P1vFC4 – N2P3




LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 1 with 4 paths/LUN VIOS failure

vFC2

vFC3

vFC4


Lost zones and paths shown in red

Failure of SVC node2 results in an outage




LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

SAN zoningZone1: vFC1 WWPN1, N1PxZone2: vFC1 WWPN2, N1PxZone3: vFC2 WWPN1, N2Px Zone4: vFC2 WWPN2, N2PxZone5: vFC3 WWPN1, N1PxZone6: vFC3 WWPN2, N1PxZone7: vFC4 WWPN1, N2PxZone8: vFC4 WWPN2, N2Px Italic zones are for inactive WWPNs


vFC2

vFC3

vFC4

Disk PathsvFC1 - N1PxvFC2 – N2PxvFC3 – N1PxvFC4 – N2Px

Not possible to zone this to use all storage ports






LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO GroupvFC

2vFC

3vFC

4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 3 with 4 paths/LUN – VIOS failure

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 3 with 4 paths/LUN – SAN fabric failure

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 3 with 4 paths/LUN – SVC node failure

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

SAN zoningZone1: vFC1 WWPN1, N2P1Zone2: vFC1 WWPN2, N2P1Zone3: vFC2 WWPN1, N1P3Zone4: vFC2 WWPN2, N1P3Zone5: vFC3 WWPN1, N1P1Zone6: vFC3 WWPN2, N1P1Zone7: vFC4 WWPN1, N2P3Zone8: vFC4 WWPN2, N2P3 Italic zones are for inactive WWPNs


vFC2

vFC3

vFC4

Disk PathsvFC1 – N2P1vFC2 – N1P3vFC3 – N1P1vFC4 – N2P3




LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group



vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 4 with 4 paths/LUN – SAN fabric failure

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 4 with 4 paths/LUN – SVC node failure

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

SAN zoningZone1: vFC1 WWPN1, N2PxZone2: vFC1 WWPN2, N2PxZone3: vFC2 WWPN1, N1PxZone4: vFC2 WWPN2, N1PxZone5: vFC3 WWPN1, N2PxZone6: vFC3 WWPN2, N2PxZone7: vFC4 WWPN1, N1PxZone8: vFC4 WWPN2, N1Px Italic zones are for inactive WWPNs


vFC2

vFC3

vFC4

Disk PathsvFC1 – N2PxvFC2 – N1PxvFC3 – N2PxvFC4 – N1Px

Not possible to zone this to use all storage ports




LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group


Design 6 with 4 paths/LUN

vFC2

vFC3

vFC4





LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group



vFC2

vFC3

vFC4





Only designs 3 and 4 provide availability in case of a double failure There are inferior ways to zone

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

Design 3 Design 4

Designs meeting availability requirements




Designs meeting availability requirements





Design SVCN# SVCN# SVCN# SVCN#

1 1 1 2 2

2 1 2 1 2

3 1 2 2 1

4 2 1 1 2

5 2 1 2 1

6 2 2 1 1

Only designs 3 and 4 meet availability requirements – avoid the others




Adding more VIOC LPARs

Using the same FCs on the VIOSRotate LPARs across SVC ports

Using other FCs on the VIOSRotate LPARs across port sets

Using a balanced resource approach is cost effective Where bandwidth of host ports = bandwidth of storage ports Bandwidth of ports varies depending on the model Latest HW: SVC port bandwidth of about 40,000 IOPs, Host port bandwidth around 50,000 IOPS

Option on right is more balanced

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1 SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

LPAR2

vFC1

vFC2

vFC3

vFC4

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1 SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

LPAR2

vFC1

vFC2

vFC3

vFC4

4 host ports 8 host ports




Adding SVC IO Groups

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

SVC node3

SVC node4

SVC IO Group

Allows additional SVC resources to be usedNode processors and cache

LUNs are served up from one IO group onlyStill 4 paths/LUN

The zoning changes:

Zone1: vFC1 WWPN1, N1P1, N3P1Zone2: vFC1 WWPN2, N1P1, N3P1Zone3: vFC2 WWPN1, N2P3, N4P3Zone4: vFC2 WWPN2, N2P3, N4P3Zone5: vFC3 WWPN1, N2P1, N4P1Zone6: vFC3 WWPN2, N2P1, N4P1Zone7: vFC4 WWPN1, N1P3, N3P3Zone8: vFC4 WWPN2, N1P3, N3P3 Italic zones are for inactive WWPNs




To add bandwdith to a VIOC, one must add more physical adapter ports Add one port with 1 additional path or Add a set of ports

Adding IO bandwidth to a VIOC LPAR

LPAR1

VIOS1 VIOS2

SAN1 SAN2

SVC node1

SVC node2

vFC1

Power Server

SVC IO Group

vFC2

vFC3

vFC4

vFC5

vFC6

vFC7

vFC8

Zone the new ports like the old portsStorage administrator assigns LUNs to new host ports, orMove half or some of the LUNs to the new port set




Error message from SVC

The GUI panel shows the 4 paths to the inactive host WWPNs used by LPM as Offline This is normal and expected behavior with LPM

The GUI panel also shows the 4 active host paths used for I/O as “Degraded” This is expected behavior but development is looking at a change in the future

The “# Nodes Logged In” column shows how many nodes in cluster see the host ports

The current SVC code (as of 12/2011) displays a “Degraded” path warning message when a host port is zoned to only one node in an I/O group, even if the second host port is zoned to the other node in the same I/O group providing path redundancy to a volume:




Disk Path Design tech doc: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101914

SDDPCM manual: http://www-01.ibm.com/support/docview.wss?rs=540&context=ST52G7&uid=ssg1S7000303

Dynamically adding a FC adapter to a partition tech doc: http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105218

vFC adapter properties using NPIV tech doc: http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FQ128819

Systems Hardware information center: Virtual Fibre Channelhttp://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphat/iphblconfigvfc.htm

References


http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101914




http://www-01.ibm.com/support/docview.wss?rs=540&context=ST52G7&uid=ssg1S7000303




http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105218

http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105218

http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FQ128819

http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FQ128819

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphat/iphblconfigvfc.htm

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphat/iphblconfigvfc.htm

Documents

Advanced Technical Skills (ATS) North America€¦ · SAN1 SAN2 SVC node1 SVC node2 Power Server server port 1 server port 4 SVC IO Group N1P1 Node 1 Port 1 Assumes 2 host ports per