PAN Manager SAN Integration Guide - Fujitsumanuals.ts.fujitsu.com/file/3644/bf200-400_san-integration-en.pdf · Red Hat is a registered trademar k of Red Hat, Inc. in the Un ited

PAN Manager SAN Integration Guide

For BladeFrame® BF400 S2 and BladeFrame® BF200

Document Number 430-SB0074

August 2010

PM5.2_BF

CopyrightCopyright © 2010 Egenera, Inc. All rights reserved.

This document, and the product described in it, is furnished under license and may only be used in accordance with the terms of such license. The content of this document is furnished for information purposes only and is subject to change without notice.

Egenera, Egenera stylized logos, BladeFrame, BladeLatch, BladeMate, BladePlane, cBlade, Control Blade, PAN Manager, pBlade, Processing Blade, sBlade, and Switch Blade are either registered trademarks or trademarks of Egenera, Inc. in the United States and/or other countries.

PRIMERGY is a registered trademark of Fujitsu Technology Solutions.

AMD, AMD Opteron, and AMD Athlon are trademarks of Advanced Micro Devices, Inc..

EMC, CLARiiON, and Symmetrix are registered trademarks of EMC Corporation.

The IBM Developer Kit for Linux, Java 2 Technology Edition contains software which is copyright IBM Corporation, Sun Microsystems Inc., Hewlett-Packard Co., and X Consortium.

Intel and Itanium are registered trademarks and Xeon is a trademark of the Intel Corporation in the United States and other countries.

Sun, Sun Microsystems, the Sun Logo, Solaris, and the Java logo are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.

Linux is a registered trademark of Linus Torvalds.

Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The virtual VGA console uses Microsoft Terminal Services Advanced Client (TSAC), which is a copyright of Microsoft Corporation.

MindTerm is copyright AppGate AB.

Nero is a trademark of Nero AG.

NetApp is a registered trademark and Network Appliance is a trademark of Network Appliance, Inc.

Oracle9i is a trademark of Oracle Corporation.

Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries.

SUSE is a registered trademark of SUSE LINUX Products GmbH, a Novell business.

VMware, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc.

Xen, XenSource, XenServer, and XenEnterprise are either registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries.

All other company and product names are trademarks or registered trademarks of their respective holders.

Printed in the United States of America.[100715:112658]

Egenera, Inc., 165 Forest Street, Marlboro, Massachusetts 01752.

iii

Contents

Preface

Customer Support ...................................................................................................... ix

Document Conventions ............................................................................................. xi

Chapter 1: Understanding the PAN and SAN Environments

Designing for No Single Point of Failure ................................................................ 1-2

Supported SAN Ports (HBAs) ................................................................................. 1-3

Storage Connection Types ....................................................................................... 1-5Fibre Channel Topologies ................................................................................ 1-5

Multipath on cBlades and pServers ......................................................................... 1-9

Supported I/O Pathing Models for Disks .............................................................. 1-12

Configuring SCSI Tape Devices in the SAN ........................................................ 1-13

Chapter 2: Integrating with a SAN

Connecting a Platform to the SAN .......................................................................... 2-1

Zoning and Scalability ............................................................................................. 2-2Switched Fabric Zoning Suggestions ............................................................... 2-2Planning for Growth ......................................................................................... 2-3


iv

Managing the Generic Multipath Software ............................................................. 2-3

Managing SCSI Device Numbering ........................................................................ 2-5About Device Numbering................................................................................. 2-5About Path Device Names................................................................................ 2-7About PAN Manager Device Names ............................................................... 2-8

Controlling SAN Device Names ........................................................................... 2-10Reasons for Renaming SAN Devices............................................................. 2-11Guidelines for Renaming SAN Devices......................................................... 2-12Rename SAN Devices .................................................................................... 2-12Maintaining SAN Device Names ................................................................... 2-14

About SCSI Reservations ...................................................................................... 2-15About SCSI-3 Persistent Reservations ........................................................... 2-15About SCSI-2 Reserve/Release Reservations ................................................ 2-16

Support for SCSI Tape Resources ......................................................................... 2-16Best Practices for Configuring SCSI Tape Resources ................................... 2-16Technical Considerations When Using Tape Devices ................................... 2-17

Chapter 3: Testing your SAN Configuration

Validate the Physical Connectivity for SCSI Devices ............................................ 3-2Check Physical Connectivity to SCSI Disk Devices........................................ 3-2Check Physical Connectivity to SCSI Tape Devices ....................................... 3-3

Test I/O on SCSI Devices ....................................................................................... 3-4Test I/O on Each Disk Device .......................................................................... 3-4Test I/O on Each Tape Device.......................................................................... 3-8

Manage SAN I/O on the cBlades .......................................................................... 3-10Check the Load Balancing on the cBlades ..................................................... 3-11Check I/O Statistics on Both cBlades............................................................. 3-16

Take a Snapshot of the Finished SAN Configuration ........................................... 3-17

Detect and Correct I/O Imbalances ....................................................................... 3-18Detect an I/O Imbalance ................................................................................. 3-18

Contents

v

Correct an I/O Imbalance ............................................................................... 3-19Detecting a Possible Zoning Issue.................................................................. 3-21

Appendixes

Appendix A: Troubleshooting and Maintenance

SAN Configuration Issues ...................................................................................... A-2

Troubleshooting ...................................................................................................... A-2

Recoverability ........................................................................................................ A-3

Maintenance Mode ................................................................................................. A-4

Diagnosing and Correcting Performance Issues .................................................... A-5Diagnostic Tools.............................................................................................. A-5

SAN Failover .......................................................................................................... A-6Conditions That Disrupt All SAN Connectivity ............................................. A-7Connection Reinitialization ............................................................................. A-8Connection Failback........................................................................................ A-8

Appendix B: Using SAN Diagnostic Tools

Monitoring I/O Paths with the giscsimon Daemon ................................................ B-2About the giscsimon Daemon ......................................................................... B-2Examining Log Messages................................................................................ B-4Displaying Statistics for I/O Paths .................................................................. B-6Editing the giscsimon.conf File ....................................................................... B-9Using the Start/Stop Script ............................................................................ B-12

Enabling and Disabling HBA Ports with the pathctl Utility ................................ B-12About the pathctl Utility ................................................................................ B-12pathctl Utility Arguments .............................................................................. B-13


vi

Displaying the Current HBA Configuration.................................................. B-14Disabling an HBA Port.................................................................................. B-15Enabling an HBA Port................................................................................... B-15Trespassing a Path ......................................................................................... B-16

Displaying LUN Performance Statistics with the pathstat Utility ....................... B-18About the pathstat Utility .............................................................................. B-18pathstat Utility Options.................................................................................. B-19Wildcards in LUN Names ............................................................................. B-21Columns in pathstat Reports.......................................................................... B-22pathstat Multipath Report (Default) .............................................................. B-23pathstat Multipath Report for Specific LUNs ............................................... B-25pathstat Subpath Report................................................................................. B-25pathstat Topology Report .............................................................................. B-26

Displaying HBA Port Performance Statistics with the pathtop Utility ................ B-27About the pathtop Utility............................................................................... B-27pathtop Utility Defaults ................................................................................. B-28pathtop Statistics Format ............................................................................... B-29pathtop Utility Options .................................................................................. B-30pathtop Example............................................................................................ B-32

Appendix C: SAN Resource Guidelines and Limitations

Index

PM5.2_BF ix

Preface

Welcome to PAN Manager PM5.2_BF. PAN Manager SAN Integration Guide is part of the PAN Manager documentation set. Its purpose is to describe the best practices to employ when integrating a Processing Area Network (PAN) with a data center’s storage area network (SAN) environment.

Audience — PAN Manager SAN Integration Guide is for PAN Administrators and LPAN Administrators.

Topics — Read this book to learn about the following:

• Understanding the PAN and SAN Environments

• Integrating with a SAN

• Testing your SAN Configuration

• Troubleshooting and Maintenance

• Using SAN Diagnostic Tools

• SAN Resource Guidelines and Limitations

Customer Support

For customer support regarding this product, please use the following contact information:


x PM5.2_BF

Fujitsu Technology Solutions customer support

Important: Fujitsu Technology Solutions supports only storage devices that it has explicitly certified. Before consulting Fujitsu Technology Solutions customer support, see the Storage Interoperability Matrix at http://www.egenera.com/support-services-overview.htm for a list of supported storage devices.

Internet http://support.ts.fujitsu.com

Telephone See the Help Desk information at http://manuals.ts.fujitsu.com/primergyservers.html, under Contact.

http://www.egenera.com/support-services-overview.htm

http://www.egenera.com/support-services-overview.htm

Preface

PM5.2_BF xi

Document Conventions

Convention Description

> Directory-level delimiter used to navigate the left pane of the PAN Manager GUI.

Example: Resources > Ethernet Connections

Sans serif italics Variable text, such as a path, a filename, or an LPAN name.

Example: lpan -c lpanname

Sans serif Text that must be typed as shown.

Example: Type root at the login prompt.

Bold The name of a field or window element appearing in a GUI. It also highlights default values in PAN Manager man pages.

Example: In the Users page...

Italics Text that is emphasized.

Example: Do not connect the power.

[text] Text that is optional to a command.

{text} A set of choices, one of which is required.

| Separation of mutually exclusive choices in syntax lines.

Example: lpan [-aD | -rD]{switch | SCSI_ID} lpanname

Note Information of importance or that may not fit in main text.

Caution Failure to heed a caution could result in loss of data.

Warning — Failure to heed a warning could result in physical harm to the user or the hardware.!


xii PM5.2_BF

PM5.2_BF 1-1

BetaChapter 1

Understanding the PANand SAN Environments

This chapter describes various SAN features and how to configure them.

• Designing for No Single Point of Failure

• Supported SAN Ports (HBAs)

• Storage Connection Types

• Multipath on cBlades and pServers

• Supported I/O Pathing Models for Disks

• Configuring SCSI Tape Devices in the SAN


1-2 PM5.2_BF

BetaDesigning for No Single Point of Failure

The ultimate goal when configuring a SAN, and integrating the platform and its PAN into the SAN, is to create sufficient redundancy that the failure of a single component will not interrupt connectivity between servers and storage. The components of an integrated PAN and SAN are as follows:

• A cBlade

• A SAN switch

• A control port on a storage array

• Fibre Channel cables

This goal of designing the SAN for no single point of failure is met by:

• the inherent redundancies in the physical architecture of the PAN and its virtualization software

• the use of multipathing software for pServers when accessing storage resources

• configuring the SAN with redundancy in mind

Figure 1.1 on page 1-3 shows how the PAN and Fibre Channel SAN environments interact, and suggests a configuration in which a component failure would not interrupt connectivity between

About SAN Components of the PAN

PM5.2_BF 1-3

BetaFigure 1.1 Fibre Channel SAN: A Multiple-Switch Fabric

Supported SAN Ports (HBAs)

The following table lists the SAN ports (host bus adapters) available on the cBlades.


1-4 PM5.2_BF

BetaTable 1.1 HBA Configurations, by Platform

Figure 1.2 shows the location of the Fibre Channel ports on the front of a single cBlade EX.

Figure 1.3 shows the location of the Fibre Channel ports on the back of a BladeFrame BF200 chassis.

Figure 1.2 BladeFrame BF400 S2: HBA Locations on a cBlade

Figure 1.3 BladeFrame BF200: HBA Locations on each cBlade

Platform BladeType

Port Numbers

Total Portsper System

BladeFrame BF400 S2

cBlade EX SCSI 4, 3, 2, 1 8

BladeFrame BF200 cBlade ES SCSI 3, 4 4

SAN ports: SCSI 4SCSI 3

SCSI 2SCSI 1

SAN ports:SCSI 3 SCSI 4 SCSI 3 SCSI 4


PM5.2_BF 1-5

BetaStorage Connection Types

This section provides information about the following SAN topologies:

• Fibre Channel Topologies

Refer to the appropriate section, according to the type of SAN ports provided on the platform’s cBlades.

Fibre Channel Topologies

A platform can connect to a Fibre Channel SAN environment in one of three ways:

• Through a direct and arbitrated loop connection

• Through a single-switch fabric connection

• Through a multi-fabric connection

The following sections provide information about these connection types:

• Direct Attach and Arbitrated Loop Configurations

• Using Fabric Switches in a Fabric

• Single-Fabric Configuration

• Multi-Fabric Configuration

Direct Attach and Arbitrated Loop Configurations

The simplest connection types are the direct attach and arbitrated loop from the platform to one, or more, storage arrays (see Figure 1.4 on page 1-6). The disadvantage to the direct attach connection type is that you are limited by the number of HBA ports available on the platform. You can only connect to the same number of SAN storage arrays.


1-6 PM5.2_BF

BetaIn this connection type, each cBlade HBA connects (through physical cables) directly to the storage devices. This storage connection type allows the HBAs on each cBlade to see the same storage device.

Figure 1.4 Direct Attach Connections

Using Fabric Switches in a Fabric

Fibre Channel networks built with switches are called fabrics. Using fabric switches provides multiple ways to route traffic from an HBA port to a target array on the SAN. (See Figure 1.1 on page 1-3.)

A limitation to connecting to one fabric switch (instead of multiple fabric switches): if the switch fails, access to the data is lost. In a properly configured multi-switch fabric, there is no single point of failure: if one switch fails, access to the data is maintained through an alternate path over one of the other switches.

Single-Fabric Configuration

A platform supports direct connection through a single fabric switch (Figure 1.5).

StorageArray

cBlade 2

cBlade 1Storage

Array


PM5.2_BF 1-7

BetaFigure 1.5 A Single Fabric Switch Configuration

Each cBlade HBA connects to the same fabric switch. This storage connection type allows the adapters on each cBlade to see the same storage devices that connect to the fabric.


1-8 PM5.2_BF

BetaMulti-Fabric Configuration

A platform also supports connection through two or more fabrics (see Figure 1.6 on page 1-8).

Figure 1.6 A Multiple-Switch Fabric

In the example in this figure, the following are true:

• HBAx from cBlade A and cBlade B connects to fabric switch 1

• HBAy from cBlade A and cBlade B connects to fabric switch 2


PM5.2_BF 1-9

BetaThis storage connection type allows the two HBAx initiator ports (one on each cBlade) to see the same storage devices that connect to the fabric. The same is true for both HBAy initiator ports for the other fabric. The storage devices themselves connect to both fabrics.

Multipath on cBlades and pServers

This section describes the features available with generic multipath support:

• How multipath choices influence which connection type to use

• How a SAN disk array’s support of certain modes of operation (active/active, active/passive, or preferred path) affects which multipath option to use

Multipath support is provided for paths to SAN tape devices. Refer to “Configuring SCSI Tape Devices in the SAN” on page 1-13 for more information.


1-10 PM5.2_BF

BetaMultipath on the pServers

Multipath on the pServers is enabled by default, and provides a path for I/O requests to be sent through either cBlade to external SAN disk storage. Multipath on pServers allows pServers to continue operation even if one cBlade fails or reboots. Figure 1.7 illustrates multipath from a pServer to the cBlades.

Figure 1.7 Multipath From a pServer to the cBlades

For example, if cBlade 1 fails, multipath allows I/O requests from the pServer to be retried on cBlade 2.

pServer

cBlade 1

cBlade 2

Multipath frompServer


PM5.2_BF 1-11

BetaMultipath on the cBlades

Figure 1.8 depicts multipath from the cBlades to external storage.

Figure 1.8 Multipath From cBlades to External Storage

This figure shows one possible configuration type (a multi-switch topology) that supports multipath. The host bus adapters on each cBlade connect to the storage environment through SAN switches, so that the HBAx and HBAy adapters on each cBlade see the same disk storage.

Multipath distributes the I/O requests among the four paths from cBlade 1 and cBlade 2 to SAN storage. This increases the bandwidth of the I/O and the number of I/O operations per second. Also, if one cBlade fails or reboots, the other can fulfill any I/O requests.

PAN Manager assigns a name to each multipath device on the cBlades during the initial discovery process. This discovery takes place when the driver loads on the cBlade at boot time. Adding or removing storage arrays could change the discovery order of the target controllers. (See “Managing SCSI Device Numbering” on page 2-5 about the impact that adding or removing storage arrays can have on SCSI device numbering.)

pServer

cBlade 1

cBlade 2 SAN Switch 1

SAN Switch 2

Storage Array

HB

Ay

HB

Ay

HBAx

HBAxMultipath from

pServer

Multipath fromcBlade


1-12 PM5.2_BF

BetaSupported I/O Pathing Models for Disks

Pathing models determine how a storage array presents its disk LUNs to the network.

The PAN environment supports the following I/O pathing models for SAN disk resources:

• Active/active

• Active/passive

• Primary/secondary or “Preferred path”

In an optimal situation, I/O uses active and primary paths, instead of passive or secondary paths. Passive and secondary paths remain available to assume I/O in suboptimal situations.

• Active/active — Allows a LUN to be visible on all ports of the storage array. This is a fully symmetric pathing model in which all paths are available for I/O to the disk LUN. In this model, if an active path fails, multipath retries the I/O on the other active path.

Multipathing software can be very useful to manage the complexities that this model allows. For information about multipathing software, see PAN Manager Technical Overview.

• Active/passive — Multipath tries I/O on all active paths on both cBlades. If I/O fails on an active path, multipath activates the corresponding passive path (a process sometimes called trespassing), and then retries I/O on the newly active path. (Multipath must issue specific commands in order to change a passive path to an active one.) For information about active and passive paths, see “Detect and Correct I/O Imbalances” on page 3-18.

• Primary/secondary or “Preferred path” — Multipath tries I/O on all optimal (primary) paths on both cBlades. If I/O fails on a primary path, multipath retries the I/O on a secondary path.


PM5.2_BF 1-13

BetaA primary path offers faster access to a disk LUN. A secondary path typically offers a slower path to a disk LUN, and is used as a standby path.

Note: Pathing models do not apply to SAN tape devices because each SAN tape device operates through a single I/O path.

Configuring SCSI Tape Devices in the SAN

Supported tape devices (including tape drives and any tape libraries) can connect directly to a fabric switch, typically a core switch. Because connecting a tape drive directly to an HBA on the platform can diminish the bandwidth available to disk I/O, we do not recommend connecting a tape device directly to the platform.

Figure 1.9 shows a typical SAN configuration for tape resources.


1-14 PM5.2_BF

BetaFigure 1.9 Tape Devices in a SAN Environment

Not all pServer releases support SCSI tape devices. For more information about which pServer software supports tape devices, see PAN Manager Release Notes.

In PAN Manager, the target IDs for SCSI tape devices range from 224 to 255; the LUN number of each SCSI tape device is zero.

For SCSI tape devices, as with all fabric devices, use single-initiator zoning. Each tape device must be zoned to a single HBA on a single cBlade. For more details about zoning, see “Switched Fabric Zoning Suggestions” on page 2-2.


PM5.2_BF 1-15

BetaMultipath software is not supported for Fibre Channel tape devices. Because a pServer accesses each SCSI tape device through only one cBlade, automatic load balancing and failover for SCSI tape devices is not supported. “Testing your SAN Configuration” on page 3-1 describes how to use cBlade utilities to detect failures and to balance I/O manually across HBA ports.

For more information and guidelines about configuring SCSI tape devices for use by pServers, see “Best Practices for Configuring SCSI Tape Resources” on page 2-16.


1-16 PM5.2_BF

Beta

PM5.2_BF 2-1

BetaChapter 2

Integrating with a SAN

This chapter describes how to integrate one or more platforms into a SAN environment.

• Connecting a Platform to the SAN

• Zoning and Scalability

• Managing the Generic Multipath Software

• Managing SCSI Device Numbering

• Controlling SAN Device Names

• About SCSI Reservations

• Support for SCSI Tape Resources

Connecting a Platform to the SAN

Important: Your SAN administrator should have already attached the SAN cables to the hardware platform’s cBlades. If the SAN cables are not connected to the cBlades, see READ THIS FIRST: PAN Manager Configuration Guide for information about the cBlade SCSI ports.


2-2 PM5.2_BF

BetaZoning and Scalability

This section describes the best practices and recommended configurations to use when creating SAN zones, and provides suggestions on how to ensure that the integration supports flexible and scalable configurations.

The purpose of zoning is to isolate storage resources for use by some servers, and to exclude the use of these storage resources from use by other servers. Zoning is configured in the switch director software on the fabric switches in your SAN.

The following sections provide the background and guidelines for configuring SAN zoning:

• Switched Fabric Zoning Suggestions

• Planning for Growth

For details about troubleshooting zoning issues, see “Detecting a Possible Zoning Issue” on page 3-21.

Switched Fabric Zoning Suggestions

Storage administrators frequently segment their switched fabric environments into soft or hard zones. This limits access to certain ports, and distributes I/O traffic among various controllers in the SAN.

Single Initiator Zoning

We recommend that the SAN administrator implement single-initiator-to-single-target zoning. For example, an initiator port with a unique worldwide name (WWN) would have a single target with a different WWN. In general, single-initiator-to-single-target zoning is a good isolation mechanism and can better withstand fabric changes than other types of zoning.


PM5.2_BF 2-3

BetaHard Zoning

Hard zones provide the highest security and help prevent unauthorized entry into a zone. In this zoning model, an initiator cannot see outside the zone(s) to which it belongs.

In multiswitched environments, it is best to zone to segregate storage I/O traffic between groups of servers and their storage subsystems and devices.

Planning for Growth

The platform integrates seamlessly into SAN environments, and provides scalability through its multipath I/O driver support.

As your SAN and enterprise networks grow, the extensible PAN architecture can scale to accommodate new configurations.

Managing the Generic Multipath Software

When PAN Manager is started or does a disk scan to discover new disks or paths, the cBlade software creates the file /etc/sysconfig/ifc/multipath_devices on each cBlade. This file maps the unique SAN device IDs to their respective path devices; the kernel path driver uses the file to reference SAN path devices.

We recommend configuring SCSI disk devices symmetrically on the platform(s) in your PAN. This means that each Control Blade in a platform has access to an identical set of SCSI disk devices.

The following is an example of the contents of the multipath_devices file:

HITACHI-OPEN3-VID-HITACHIR401ED2A0090:(9,0,0,30):(3,0,0,30)HITACHI-OPEN3-VID-HITACHIR401ED2A0091:(9,0,0,31):(3,0,0,31)HITACHI-OPEN3-VID-HITACHIR401ED2A0092:(9,0,0,32):(3,0,0,32)


2-4 PM5.2_BF

BetaHITACHI-OPEN3-VID-HITACHIR401ED2A0093:(9,0,0,33):(3,0,0,33)HITACHI-OPEN3-VID-HITACHIR401ED2A0097:(9,0,0,34):(3,0,0,34)HITACHI-OPEN3-VID-HITACHIR401ED2A0098:(9,0,0,35):(3,0,0,35)

Caution: Because the PAN Manager software and the kernel can independently assign SAN device IDs, we strongly recommend that you save a copy of each cBlade multipath_devices file off the platform for safekeeping.

If, after adding or removing SAN disks, the kernel’s list of device IDs and the PAN Manager’s list of device IDs do not match, do the following:

1. Determine the cause of the differences between the device mappings. There may be valid or benign reasons for the differences.

2. If you cannot determine the reason for the difference, copy the archived file back to /etc/sysconfig/ifc, and then reboot the cBlades to restore the previous device ID mappings.

If PAN Manager determines that the contents of this file changed since the last reboot, it creates a backup (multipath_devices.pm-saved) of the current file before writing the new file and changing its contents.

To keep a copy of the last known state of the multipath device mapping, archive this backup file. You can then use the backup copy if you need to recover the information for any reason.


PM5.2_BF 2-5

BetaManaging SCSI Device Numbering

Under some circumstances, the names of some devices can have an unexpected form, or can change after a cBlade reboots. This can occur in the following instances:

• You change the SAN configuration (including adding or removing one or more disks).

• You change the kernel’s SCSI settings.

• You change path groups on existing devices, and then reboot (which may change the UIDs of existing devices).

These changes can impact device names in two different ways:

• A device’s path device name can change from 9.n.n.n to 9.n.128.n.

• A device’s PAN Manager device name can change from 9.n.n.n to 10.n.n.n.

Although this condition is harmless, it may cause some confusion. This section describes how to identify and correct this condition, as desired, and where possible, how to prevent it.

About Device Numbering

The cBlade software (kernel and PAN Manager) names SCSI devices in various ways. The following summarizes these names and how they affect each other.

• PAN Manager device name — Appears in the same format as a SCSI ID (n.n.n.n), and used in PAN Manager for display and configuration. PAN Manager assigns this name to the disk when it discovers the disk for the first time.

• Path device name — The name of the disk as recognized by the cBlade kernel and which represents the aggregate paths (subpaths) by which the device is accessible. The multipath driver creates this name each time the kernel boots.


2-6 PM5.2_BF

BetaIn most cases, a device’s PAN Manager device name and path device name are identical. However, if there are SAN configuration changes or changes in the kernel SCSI settings, a physical device’s path device name can change after a cBlade reboots. (It is also possible for one cBlade kernel to recognize a physical device by one path device name, and for the other cBlade kernel to recognize the same device by a different path device name.)

• Unique identifier (UID) — A long string used by the kernel to identify the device. This UID persists for the lifetime of the device. Because PAN Manager persists the mapping between the UID and the PAN Manager device name, PAN Manager configurations are not affected by any changes in the device’s path device name.

For example, in PAN Manager, you use the PAN Manager device name to display a device’s characteristics:

# san -l "(9.0.6.9)"

In the command output, the PAN Manager device name appears in the ID column, the Unique ID appears on its own line, and the path device names, as recognized by each cBlade kernel, appear in the Path Device column:

Description:

ID Capacity Type S/N Paths LPAN(9.0.6.9) 20.0GB 3PARdata/VV 000C022B 2 my_lpan

Unique ID: 3PARdata-VV-WWN-50002ac0000c022b0000000000000000

I/O Paths:Cblade Path Device Subpathsminigolf/c2 (9.0.6.9) (3.0.2.9),(4.0.2.9)minigolf/c1 (9.0.6.9) (3.0.3.9),(4.0.2.9)

Partitions:N/A

Assignments:my_lpan (pserver4:(56.0))


PM5.2_BF 2-7

BetaAbout Path Device Names

This section describes how path device names can change, and how to help prevent this from happening.

The Cause of Path Device Name Changes

When PAN Manager scans and detects a new device, the cBlade software establishes two subpaths for it, for example 3.0.1.0 and 4.0.1.0. The cBlade software then attempts to construct a single path device name (such as 9.0.0.1) that corresponds with each subpath.

For any of the reasons listed under “Managing SCSI Device Numbering” on page 2-5, the cBlade software may associate one subpath (3.0.1.0) with a path device name (9.0.0.1), but be unable to associate the other subpath (4.0.1.0) with this same path device name.

For example, if a device has these names:

PAN Manager device name: (9.0.0.1)Path device name: (9.0.0.1)UID: uid-1

and the SAN in which the device resides is reconfigured (a device is added), the new device might have these names:

PAN Manager device name: (?.?.?.?)Path device name: (9.0.0.1)UID: uid-2

Because the path device name of the new device (9.0.0.1) would be identical to the path device name of an existing device, this causes the path device name of the new device to become 9.0.128.n (in this case 9.0.128.0):


2-8 PM5.2_BF

BetaPAN Manager device name: (9.0.0.1)Path device name: (9.0.128.0)UID: uid-1

and

PAN Manager device name: (?.?.?.?)Path device name: (9.0.0.1)UID: uid-2

To prevent a device from being recognized by two different path device names, be sure to adhere to the guidelines described in the section “Preventing Path Device Name Changes”.

Preventing Path Device Name Changes

To help prevent the situation in which a single device is identified by multiple path device names, check to see that the LUNs you expose are visible on all paths as the same LUN number.

About PAN Manager Device Names

This section describes how PAN Manager device names can change, how to detect this condition, and if necessary, how to synchronize PAN Manager device names with their associated path device names.

The Cause of PAN Manager Device Name Changes

When PAN Manager discovers a new device, it attempts to generate a PAN Manager device name that is identical to the new device’s path device name.


PM5.2_BF 2-9

BetaIf the new device’s PAN Manager device name is already in use in PAN configurations, PAN Manager generates an easily noticeable PAN Manager device name for the new device, in the form 10.n.n.n, for example 10.0.0.0. This yields the following names for the two devices:


and


Important: Before importing SCSI devices into the PAN, be sure that each discovered device’s PAN Manager device name and path device name are identical.

To check that PAN Manager device names and path device names are identical:

1. In PAN Manager, turn the auto import feature off:# san -m

2. Display all devices (those in the PAN and those not in the PAN):# san -A

Comparing PAN Manager Device Names and Path Device Names

This section describes how to determine whether the PAN Manager device names and path device names of one or more devices are identical, and if they are not, how to make them identical.

To compare the PAN Manager device names and path device names for all SAN devices, enter:

# san -l


2-10 PM5.2_BF

BetaOutput similar to the following appears for each device:

Description:

ID Capacity Type S/N Paths LPAN(9.0.6.9) 20.0GB 3PARdata/VV 000C022B 2 my_lpan

Unique ID: 3PARdata-VV-WWN-50002ac0000c022b0000000000000000

I/O Paths:Cblade Path Device Subpathsminigolf/c2 (9.0.6.9) (3.0.2.9),(4.0.2.9)minigolf/c1 (9.0.6.9) (3.0.3.9),(4.0.2.9)

Partitions:N/A

Assignments:my_lpan (pserver4:(56.0))

In this case, the PAN Manager device name (at top left, in the ID column) is identical to each path device listed in the Path Device column.

If there are differences between the path device names and PAN Manager device names of one or more devices in your SAN, follow the procedure “Rename SAN Devices” on page 2-12.

Controlling SAN Device Names

This section describes how to use the PAN Manager option for explicitly renaming SCSI disk and tape devices.

Note: Support for Fibre Channel tape devices requires having Fibre Channel HBAs on the platform’s cBlades.

• Reasons for Renaming SAN Devices

• Guidelines for Renaming SAN Devices


PM5.2_BF 2-11

Beta• Rename SAN Devices

• Maintaining SAN Device Names

PAN Manager provides an optional way to persist a SAN device’s SCSI ID as presented by PAN Manager, despite configuration changes in the SAN.

Although there is no technical requirement for doing this, some administrators may find this feature useful for maintaining a consistent device naming policy in the PAN. This is particularly useful when managing PAN archives (disaster recovery files) across multiple hardware platforms.

Important: Rebooting the cBlades will not change any SCSI IDs unless there have been changes in the SAN’s configuration (devices added or removed).

Reasons for Renaming SAN Devices

Renaming SAN devices is useful for sites that:

• Use the disaster recovery (DR) feature to create PAN archives for the purpose of replicating PAN configurations on other hardware platforms.

For example, if a particular pServer's disk device is named 9.0.1.2 on the production hardware platform, but is named 9.0.128.0 on the DR hardware platform, you can rename one or the other device names so that they match. This can help prevent confusion during a PAN DR swap operation and simplify management of your hardware platforms. (It makes PAN archives easier to use during the "map" phase if the SCSI IDs are the same on both platforms.)

• Want the PAN Manager device names (SCSI IDs) for SAN devices to remain the same, despite changes to the PAN’s overall SAN configuration.

• Want PAN Manager SCSI IDs of devices newly added to the PAN to conform to the site’s device naming conventions.


2-12 PM5.2_BF

BetaGuidelines for Renaming SAN Devices

This section describes under what conditions you can rename SAN devices.

In order to rename a SAN device:

• The device must already have been imported to the PAN.

• The device must not be allocated either to an LPAN or configured for use in disaster recovery.

• You must use the PAN Manager command line interface (CLI). (Renaming SAN devices is not supported in either the PAN Manager GUI or the PAN API.)

• The new name you select for the device must:

• not already be used by a SAN device in the PAN

• follow the PAN Manager 4-tuple SCSI ID naming convention (w.x.y.z)

• fall in the acceptable range for SCSI IDs, where

– w is from 0 to 255

– x is from 0 to 255

– y is from 0 to 511

– z is from 0 to 4095

Rename SAN Devices

This procedure describes how to change any devices in the PAN not to have PAN Manager device names of the form 10.n.n.n.

Consider performing this procedure if one or more reasons for doing so applies to your site; see “Reasons for Renaming SAN Devices” on page 2-11.

To change one or more devices in the PAN not to have PAN Manager device names of the form 10.n.n.n:

1. If you are renaming one or more SAN devices that are already configured in an LPAN/pServer, remove each device from its LPAN/pServer or from its configuration as a DR device.


PM5.2_BF 2-13

Beta2. Capture the SCSI IDs in your current SAN configuration, (and

the kernel Unique ID (UID) to which each SAN device corresponds).# san -l | egrep -A 3 '^\ID' > anyfile

Output similar to the following appears:ID Capacity Type S/N Paths LPAN(9.0.0.0) 7MB EMC/SYMMETRIX 502917000000 2

Unique ID: EMC-SYMMETRIX-WWN-6006048000018450291753594d303230--ID Capacity Type S/N Paths LPAN(9.0.0.99) 8.4GB EMC/SYMMETRIX 502917072000 2

Unique ID: EMC-SYMMETRIX-WWN-6006048000018450291753594d304632--...

You will use this information in Step 4 when renaming your SAN devices.

3. PAN Manager does not support persistent bindings. You should have removed any persistent bindings before upgrading PAN Manager to Release 5.2.n.

If you are not sure whether you had persistent bindings configured before upgrading PAN Manager to Release 5.2.n, open the /crash_dumps/backup_data_files/etc/modules.conf file. (The PAN Manager upgrade process archived this file to this location from /etc/modules.conf.) If there are any lines in modules.conf that are similar to the following:scsi-qla0-tgt-10-di-0-port=50060e8000031560;\scsi-qla0-tgt-11-di-0-port=500601601020d12a;\

then one or more of your SAN devices were configured to have persistent bindings. Because the SCSI IDs of these devices can change if there is a change to your overall SAN configuration, you must now set a specific SCSI ID for each device. To do so, go on to Step 4.

4. For each SAN device, specify the following:# san -n “(new_SCSI_ID)” {“(SCSI_ID)” | “UID”}


2-14 PM5.2_BF

Betawhere

new_SCSI_ID is the SAN device’s new PAN Manager SCSI ID

SCSI_ID is the SAN device’s current PAN Manager SCSI ID

UID is the SAN device’s Unique Identifier: a long string used by the kernel to identify the SAN device. (This string never changes.)

When renaming the device’s SCSI ID, you can either specify its current SCSI ID, or its UID.

For example, to rename the device 9.0.1.25 to 12.3.0.58, enter:

# san -n “(12.3.0.58)” “(9.0.1.25)”

To specify a new SCSI ID for a device whose UID is EMC-SYMMETRIX-WWN-6006048000038722003753594d323741, enter:

# san -n “(9.0.3.78)” “EMC-SYMMETRIX-WWN-6006048000038722003753594d323741”

5. If you have renamed any SAN devices that were configured in an LPAN/pServer or configured as a DR device, return each device to its original configuration.

Maintaining SAN Device Names

To maintain naming control of your SAN devices, perform the procedure in “Rename SAN Devices” on page 2-12 on an ongoing basis, at the following times:

• Immediately after the first cBlade reboot following a SAN configuration change

• Immediately after performing a manual disk scan (san -s) of the SAN following a SAN configuration change.


PM5.2_BF 2-15

BetaAbout SCSI Reservations

The cBlade software provides support for two types of SCSI reservations:

• A persistent reservation — As defined by the SCSI-3 standard, is an association between a pServer running on any Processing Blade and a SAN disk, and allows more than one booted pServer to simultaneously access the same disk. For the persistent reservation feature, PAN Manager lists both the pServer registrations and reservations associated with the disk.

• A reserve/release reservation — As defined by the SCSI-2 standard, allows only a single booted pServer to access a single device. By default, PAN Manager enables all devices with reserve/release reservations.

Some applications that you install on pServers may require SCSI reservations. For instructions for enabling and disabling persistent reservations, and for clearing stale SCSI-2 or SCSI-3 reservation data from disks, see PAN Manager Administrator’s Guide.

To use the persistent reservation feature on any disk, you must use the array vendor’s command set to enable persistent reservations for that disk. For details about enabling persistent reservations, refer to your array vendor’s documentation.

About SCSI-3 Persistent Reservations

For Linux pServers, the SCSI-3 persistent reservation feature allows more than one booted pServer to share the same data disk (LUN) in a controlled fashion, primarily for clustering applications.

Typically, a persistent reservation is an association between the ports of a server and the ports of an array. In the PAN environment, a persistent reservation is actually an association between a pServer running on any pBlade and a SAN disk device. The persistent reservation is used to control access between the pServers and the disk over any of the configured paths between the cBlade HBA ports (initiator ports) and the SAN ports (target ports).


2-16 PM5.2_BF

BetaAbout SCSI-2 Reserve/Release Reservations

SCSI-2 (reserve/release) reservations prevent unwarranted access to a pServer’s SCSI devices by other pServers. SCSI-2 reservations govern how pServers can or cannot access the same disk LUN.

The cBlade enforces the reservation with respect to I/O from other pServers. The cBlade does not forward reserve (and release) commands to the device itself. This allows the cBlade to continue to loadbalance among all the paths to the device even when the device is reserved. This results in better performance from some storage arrays in the presence of a SCSI-2 reservation.

Support for SCSI Tape Resources


Best Practices for Configuring SCSI Tape Resources

This section describes various ways to optimize your use of tape backup resources in a PAN environment:

• Distribute tape drives across the cBlades. This means that if your SAN has multiple zones, try to distribute the tape devices evenly across these zones.

• For security reasons, it is desirable for each SCSI tape device in the PAN to have one hard zone path. For more information about zoning, see “Zoning and Scalability” on page 2-2.

• Configure tapes on one HBA on a single cBlade. For performance reasons, it is best to configure all SCSI tape devices to use as few cBlade HBAs as possible, without exceeding the I/O capacity of individual HBAs.

Use the pathtop and pathstat utilities to determine the I/O performance of specific SCSI devices and specific HBAs. For additional information about using these utilities, see Chapter 3, “Testing your SAN Configuration”.


PM5.2_BF 2-17

Beta• It is possible to configure a disk device and a tape device on the

same HBA, but only one of these devices can perform I/O operations at any given moment.

In general, if a disk is using an HBA, and a tape device that is configured on that same HBA starts processing I/O, all disk I/O on the disk ceases. (This behavior helps maximize the tape device’s throughput.) An exception is when the shared HBA is the disk’s only functioning device path. In that case, disk I/O and tape I/O can occur simultaneously on the same HBA.

• When perfoming tape backup operations, you can use bandwidth best by ensuring that the disk you are reading from, and the tape you are writing to, are configured on different pServers.

• If you have multiple pServers that will act as SCSI tape device media servers on a platform, you can optimize performance by ensuring that their pBlades are evenly divided between odd and even slots in the platform. For example, if you have four pServers that will be media servers, select two even numbered pBlades (such as p2 and p6), and two odd numbered pBlades (such as p3 and p5).

• You can assign a SCSI tape device to multiple pServers within an LPAN; however, be sure that the applications installed on these pServers use reserve/release (SCSI-2) reservations to prevent unwarranted access to a pServer’s SCSI tape device(s) by another pServer. For more information about reserve/release reservations for SCSI disk and tape device resources, see “About SCSI Reservations” on page 2-15.

Following these practices will help optimize the performance and reliability of your data backups onto tape.

Technical Considerations When Using Tape Devices

If you are using tape devices, keep in mind the following:

• Multipath support is not provided for paths to SAN tape devices, because a SCSI tape device uses only one path.

• The pServers of an LPAN can share a specific SCSI tape device.

• A SCSI tape device cannot be shared across LPANs.


2-18 PM5.2_BF

BetaCaution: The Linux tape class driver gets its maximum transfer size from the SCSI tape device. If the maximum transfer size for a tape device is greater than 128 KB, the I/O operation fails. To avoid this issue when running applications, use a transfer size of 128 KB or smaller.

PM5.2_BF 3-1

BetaChapter 3

Testing your SANConfiguration

This chapter describes how to test the combined PAN and SAN environments that you have configured. Although these procedures are of greatest value when you first configure your SAN, you can also use them to test your SAN connectivity at any time in an existing environment.

Various factors can affect the I/O behavior you may see when you perform the procedures in this chapter. If you are experiencing uneven results, capture any applicable utility output, and contact Fujitsu Technology Solutions customer support or your authorized support vendor for assistance.

This chapter includes the following tests:

• Validate the Physical Connectivity for SCSI Devices

• Test I/O on SCSI Devices

• Manage SAN I/O on the cBlades

• Take a Snapshot of the Finished SAN Configuration

• Detect and Correct I/O Imbalances


3-2 PM5.2_BF

BetaThe examples provided in this chapter are limited in scope by the following factors:

• The examples assume that you are configuring Linux pServers. (Your own pServer operating system may be different.)

• The examples assume that you are running the generic multipath software rather than third-party multipath software.

• The examples are valid for SCSI IDs (LUNs) that are accessed through active/active controllers. (Output results differ if your pServers access LUNs through active/passive controllers or preferred path controllers.)

• The examples assume that you are using the PAN Manager GUI to determine the SCSI IDs (and subpaths) of SCSI devices. (You can also use the PAN Manager CLI for this purpose.)

Validate the Physical Connectivity for SCSI Devices

The goal of the following two procedures is to determine whether you have physically configured the SAN arrays, fabric switches, and cables correctly.

Check Physical Connectivity to SCSI Disk Devices

Perform this procedure for each SCSI disk device in the PAN.

To check physical connectivity to SCSI disk devices:

1. Using a web browser, log on to the PAN Manager GUI.

2. In the PAN Manager GUI left pane, select Resources > Disks.

The SCSI Disks page displays all SCSI disk devices in the PAN.

3. Select a disk by clicking its SCSI ID in the ID column.

4. On the disk_name page, in the Type field, be sure that the Vendor and Model of the disk’s storage array are accurate.

Testing your SAN Configuration

PM5.2_BF 3-3

Beta5. On this same GUI page, in the IO Paths area, click the icon for

each cBlade. The information that appears includes the subpaths associated with this disk.

a. Be sure that there is a subpath for each HBA configured on your cBlades. (The first number in a subpath’s SCSI ID represents the HBA number on the cBlade. For example, the subpath 3.0.0.47 indicates that I/O passes through HBA 3.).

b. Be sure that both cBlades display the same subpaths and SCSI ID for the disk, indicating that the device is accessible to both cBlades.

If the number of subpaths you see is more than expected, the disk’s unique ID may not be unique.

If the number of subpaths you see is less than expected:

• There may be a cabling issue.

• There may be an issue with an array controller.

• Zoning may be improperly configured.

If the number of subpaths continues to be more or less than you expect, contact Fujitsu Technology Solutions customer support or your authorized support vendor for assistance.

Check Physical Connectivity to SCSI Tape Devices

Perform this procedure for each SCSI tape device in the PAN.


To check physical connectivity to SCSI tape devices:


2. In the PAN Manager GUI left pane, select Resources > Tapes.

The SCSI Tapes page displays all SCSI tape devices in the PAN.

3. Select a tape device by clicking its SCSI ID in the ID column.

4. On the SCSI Tape tape_name page, in the Type field, be sure that the Vendor and Model of the tape device are accurate.


3-4 PM5.2_BF

Beta5. On this same GUI page, in the IO Paths area, click the icon for

each cBlade.

The information that appears includes the subpath associated with this tape device. There should be only one subpath. The first number in the subpath’s SCSI ID represents the HBA number on the cBlade through which the tape device is connected.

If more than one subpath appears:

• The SCSI tape device’s unique ID may not be unique.

• Zoning may be improperly configured.

If this behavior persists, contact Fujitsu Technology Solutions customer support or your authorized support vendor for assistance.

Test I/O on SCSI Devices

The goal of this procedure is to test sample I/O traffic between pServers and the storage devices configured on them. The procedure includes creating one or more pServers for the sole purpose of testing all storage devices in the PAN. For information about creating an LPAN and pServers, see PAN Manager Administrator’s Guide.

Test I/O on Each Disk Device

To ensure the flow of I/O to and from a specific SCSI disk device:


2. Create an LPAN, and then create one or more pServers in the LPAN. (See PAN Manager Administrator’s Guide, if necessary.).

3. Add the same number of SCSI disk devices to each pServer that you created.


PM5.2_BF 3-5

Beta4. Install the same pServer software on each pServer. (This

example uses the Linux operating system. Refer to the appropriate pServer Guide for instructions for installing the pServer software.)

5. Install an application or use an operating system utility that generates I/O. (In this case, we use the Linux dd command.)

6. Use the network utility of your choice (such as ssh or PuTTY) to open a remote network console on each cBlade.

7. Display all disks assigned to a specific pServer. To do this, in the PAN Manager GUI left pane, select LPANs > lpan_name > pserver_name, where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively.

8. Open a console on the pServer, by doing one of the following:

• Using a network utility of your choice (such as ssh or puTTY), log on to one of the cBlades, and then enter console lpan_name/pserver_name where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively.

• In the PAN Manager GUI left pane, select LPANs > lpan_name > pserver_name, where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively. On the pServer pserver_name page, in the bottom border, click the console icon.

9. On the pServer pserver_name page, in the Current Blade area, determine the name of the pBlade that is configured on the pServer. The pBlade name appears as a link, in the format platform/pN, where N is the slot number of the pBlade in the platform chassis.

10. To use this pBlade’s slot number to display I/O statistics for this pServer, issue this command, as appropriate, for your hardware platform, in both cBlade consoles:# pathstat -T p3 -u -t -i 1

This example displays a topology report at one-second intervals for the pBlade in Slot 3, and shows that this pBlade is configured on the pServer mylpan/ps5.


3-6 PM5.2_BF

BetaThu Feb 2 15:09:45 2009pBlade topologypb saddr LUN subpath state req/s MB/s err--------------------------------------------------------------- 3 0.0 9.0.1.54 3.0.4.54 act 0.0 0.000 0 1.0 9.0.0.0 3.0.3.0 act 0.0 0.000 0 4.0.2.0 act 0.0 0.000 0 2.0 9.0.0.1 3.0.3.1 act 0.0 0.000 0 4.0.2.1 act 0.0 0.000 0 3.0 9.0.0.2 3.0.3.2 act 0.0 0.000 0 4.0.2.2 act 0.0 0.000 0 120.0 9.0.0.9 3.0.2.9 PAS 0.0 0.000 0 4.0.4.9 ACT 0.0 0.000 0 121.0 9.0.0.11 3.0.2.11 PAS 0.0 0.000 0 4.0.4.11 ACT 0.0 0.000 0 ifcb mylpan-ps5 9/11 0.0 0.000 0===============================================================TOTAL 9/11 0.0 0.000 0

In the SCSI address column (saddr), note the device IDs of the devices configured on the pServer. Devices appear in numeric order. The top device, 0.0, corresponds to device sda on the pServer, followed by sdb, and so on.

11. In the pServer console window, enter the following:# dd if=/dev/sdc of=/dev/null

12. Examine the output that appears in the cBlade console windows. Generating I/O for device sdc in this way appears as requests per second (req/s) and megabytes per second (MB/s) for SCSI address 2.0:

Thu Feb 2 15:10:25 2009pBlade topologypb saddr LUN subpath state req/s MB/s err----------------------------------------------------------------- 1 0.0 9.0.1.54 3.0.4.54 act 0.0 0.000 0 1.0 9.0.0.0 3.0.3.0 act 0.0 0.000 0 4.0.2.0 act 0.0 0.000 0 2.0 9.0.0.1 3.0.3.1 act 238.2 7.445 0 4.0.2.1 act 237.3 7.414 0 3.0 9.0.0.2 3.0.3.2 act 0.0 0.000 0 4.0.2.2 act 0.0 0.000 0 120.0 9.0.0.9 3.0.2.9 ACT 0.0 0.000 0 4.0.4.9 PAS 0.0 0.000 0 121.0 9.0.0.11 3.0.2.11 PAS 0.0 0.000 0


PM5.2_BF 3-7

Beta 4.0.4.11 ACT 0.0 0.000 0 ifcb mylpan-ps5 9/11 475.5 14.860 0=================================================================TOTAL 9/11 475.5 14.860 0Thu Feb 2 15:10:25 2009

Compare the numbers that appear: if the disk device on the storage array is configured on an active/active controller, within each column, the values displayed on all HBAs should be roughly equivalent, indicating that disk device’s I/O is balanced properly.

In this example, generated on a plaform whose cBlades contain Fibre Channel HBAs, the command output shows that the pServer has a LUN that has a state of active/passive (9.0.0.9). (Specifically, its active subpath is 3.0.2.9; its passive subpath is 4.0.4.9.)

To verify that this LUN is handling I/O properly, issue the following command on the pServer (using sde because it is the pServer’s fifth device) to generate I/O on that device:

# dd if=/dev/sde of=/dev/null

Output similar to the following appears:

Thu Feb 2 15:12:27 2009pBlade topologypb saddr LUN subpath state req/s MB/s err--------------------------------------------------------------- 1 0.0 9.0.1.54 3.0.4.54 act 0.0 0.000 0 1.0 9.0.0.0 3.0.3.0 act 0.0 0.000 0 4.0.2.0 act 0.0 0.000 0 2.0 9.0.0.1 3.0.3.1 act 0.0 0.000 0 4.0.2.1 act 0.0 0.000 0 3.0 9.0.0.2 3.0.3.2 act 0.0 0.000 0 4.0.2.2 act 0.0 0.000 0 120.0 9.0.0.9 3.0.2.9 PAS 0.0 0.000 0 4.0.4.9 ACT 380.9 11.903 0 121.0 9.0.0.11 3.0.2.11 PAS 0.0 0.000 0 4.0.4.11 ACT 0.0 0.000 0 ifcb mylpan-ps5 9/11 380.9 11.903 0===============================================================TOTAL 9/11 380.9 11.903 0


3-8 PM5.2_BF

BetaNotice that I/O is only occurring on HBA port 4 (the active port) of SCSI address (120.0). This means that I/O is functioning as expected.

In general:

• You should not see I/O on any secondary paths

• You should not see the active/passive states change or the error count increase

• Each active (or primary) path on which you generate I/O should display non-zero statistics. The values for MBs/s across these active subpaths should be approximately equivalent.

Certain configurations will not have the I/O load distributed equally across all active paths:

• On an active/active array, there should be equal I/O distribution across the LUNs’ subpaths

• On an active/passive array, or on a primary/secondary array, in which the number of active (or primary) paths does not equal the number of passive (or secondary) paths, the I/O may not be equally distributed. This is benign and can be ignored.

Test I/O on Each Tape Device


To ensure the flow of I/O to and from a specific SCSI tape device:

1. Insert a scratch tape in each tape device that you want to test.


3. Create an LPAN, and then create one or more pServers in the LPAN. (See PAN Manager Administrator’s Guide, if necessary.).

4. Assign all SCSI tape devices equally among the pServers that you have created.


PM5.2_BF 3-9

Beta5. Install the same pServer software on each pServer. (This

example uses the Linux operating system. Refer to the appropriate pServer Guide for instructions for installing the pServer software.)

6. Install an application or use an operating system utility that generates I/O. (In this case, we use the Linux dd command.)


8. Display all SCSI tape devices assigned to a specific pServer. To do this, in the PAN Manager GUI left pane, select LPANs > lpan_name > pserver_name, where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively.

9. Open a console on the pServer, by doing one of the following:

• Using a network utility of your choice (such as ssh or PuTTY), log on to one of the cBlades, and then enter console lpan_name/pserver_name.


10. On the pServer pserver_name page, in the SCSI Tapes area, select a tape device by clicking its SCSI ID.

11. Use this tape device’s SCSI ID to display I/O statistics. (Issue the command that is appropriate for your hardware platform.)# pathstat -S 9.0.3.0 -u -i 1 -t

This example displays I/O statistics at one-second intervals for SCSI ID 9.0.3.0, and includes the current time.

12. On the SCSI Tape tape_device_name page, in the Assignments area, note the target ID of the device on the pServer, which appears in parentheses.

13. In the pServer console window, issue the Linux dd command to generate I/O on the pServer.


3-10 PM5.2_BF

BetaCaution: The Linux tape class driver gets its maximum transfer size from the SCSI tape device. If the maximum transfer size for a tape device is greater than 128 KB, the I/O operation fails. To avoid this issue when running applications, use a transfer size of 128 KB or smaller.

To do this, enter the following:# dd if=/dev/zero of=/dev/st0 bs=128k

14. Examine the output that appears in the cBlade console windows:Thu Feb 2 15:50:35 2009multipath device w/ subpaths LUN subpath state req/s MB/s err----------------------------------------------------------9.0.3.0 3.0.1.0 act 295.1 36.885 09.0.3.0 ifcb 1/1 295.1 36.885 0------------==========================================================TOTAL 1/1 295.1 36.885 0

Each SCSI tape device uses only a single I/O path. This command output indicates that the pServer is successfully writing data to the tape device through HBA port 3.

Manage SAN I/O on the cBlades

In previous sections in this chapter, you verified the physical connectivity of storage resources to the PAN environment (see “Validate the Physical Connectivity for SCSI Devices” on page 3-2) and verified that each individual storage resource was handling I/O properly (see “Test I/O on SCSI Devices” on page 3-4).

This section describes how to examine I/O on all configured storage devices simultaneously, by displaying statistics for an entire cBlade.


PM5.2_BF 3-11

BetaCheck the Load Balancing on the cBlades

This section describes how to examine I/O statistics for the following storage environment(s):

• One that contains only SCSI disk resources

• One that contains SCSI disk and SCSI tape resources


Check Disk I/O Only

This section describes how to examine I/O statistics on cBlades that are processing I/O for all disk devices available to the PAN.

To check I/O statistics for disk devices:


2. In each cBlade console, run the pathtop utility:# pathtop

3. Open a console on each pServer on which SCSI disk devices are configured, by doing one of the following:

• Using a network utility of your choice (such as ssh or PuTTY), log on to one of the cBlades, and then enter console lpan_name/pserver_name, where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively.


4. On each pServer, install an application or use an operating system utility that generates I/O using each SCSI disk device configured on it.


3-12 PM5.2_BF

BetaYou can write and run a script on each pServer that performs a basic operation using the Linux dd command. For example, on a pServer that has eight SCSI disks configured, this script would generate read operations on each configured device:dd if=/dev/sda of=/dev/null &dd if=/dev/sdb of=/dev/null &dd if=/dev/sdc of=/dev/null &dd if=/dev/sdd of=/dev/null &dd if=/dev/sde of=/dev/null &dd if=/dev/sdf of=/dev/null &dd if=/dev/sdg of=/dev/null &dd if=/dev/sdh of=/dev/null &

Output similar to the following appears in the cBlade console windows. Each row of data pertains to a specific HBA port, numbered in the leftmost column.

multipath meter Wed Jan 11 14:22:57 2009

hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb

3 20000005e600867c use up 4 392.8 12.136 31.634 31.646 8.0004 21000005e600867c use up 6 392.6 12.120 31.609 31.623 4.000

Total all LUNs 10 785.5 24.256 31.621 31.634 6.000

5. In one of the cBlade console windows, check the following numbers:

a. For each HBA port, check the value of trq/s (total requests per second). The values that appear should be roughly equal for each HBA port on the cBlade.

b. For each HBA port, check the value of tmb/s (total megabytes per second). The values that appear should be roughly equal for each HBA port on the cBlade.

6. In the other cBlade console window, check the same values that you did on the first cBlade. The values that appear for trq/s (and also tmb/s) should be roughly equal for each HBA port on this cBlade.

7. If you plan to test cBlade I/O for SCSI disk and tape devices simultaneously (recommended if you have SAN tape devices), do the following:

a. Determine on which cBlade(s) you have configured SCSI tape devices.


PM5.2_BF 3-13

Betab. On each cBlade on which SCSI tape devices are configured,

in the row labeled Total all LUNs, note the total value that appears in the tmb/s column, in this case 24.256.

This value represents the total megabytes per second for all HBA ports on the cBlade that are processing I/O for disk devices.

You will use this value as a baseline in the next section when comparing this figure to a cBlade on which SCSI tape resources are active.

Check Disk I/O and Tape I/O Simultaneously


This section describes how to examine I/O statistics on cBlades that are processing I/O for disk and tape devices simultaneously. The purpose is to verify that you have configured your tape devices according to the best practices described in “Best Practices for Configuring SCSI Tape Resources” on page 2-16.

Caution: The Linux tape class driver gets its maximum transfer size from the SCSI tape device. If the maximum transfer size for a tape device is greater than 128 KB, the I/O operation fails. To avoid this issue when running applications, use a transfer size of 128 KB or smaller.

To check I/O statistics for disk and tape devices:

1. Use the network utility of your choice (such as ssh or puTTY) to open a remote network console on each cBlade.

2. In each cBlade console, run the pathtop utility:# pathtop

3. Open a console on each pServer on which SCSI disk and SCSI tape devices are configured, by doing one of the following:


3-14 PM5.2_BF

Beta• Using a network utility of your choice (such as ssh or

puTTY), log on to one of the cBlades, and then enter console lpan_name/pserver_name, where lpan_name and pserver_name indicate the name of your LPAN and pServer, respectively.


4. On each pServer, install an application or use an operating system utility that generates I/O using each SCSI disk and tape device configured on it.

You can write and run a script on each pServer that performs a basic operation using the Linux dd command. For example, on a pServer that has eight SCSI disks and one SCSI tape device configured, this script would generate read operations on each configured device:dd if=/dev/sda of=/dev/null &dd if=/dev/sdb of=/dev/null &dd if=/dev/sdc of=/dev/null &dd if=/dev/sdd of=/dev/null &dd if=/dev/sde of=/dev/null &dd if=/dev/sdf of=/dev/null &dd if=/dev/sdg of=/dev/null &dd if=/dev/sdh of=/dev/null &dd if=/dev/zero of=/dev/st0 bs=128k &

(This sample script is identical to the one we suggest in the previous section, except that it has an extra line, for a tape device.)

Output similar to the following appears in the cBlade console windows. Each row of data pertains to a specific HBA port, numbered in the leftmost column.

multipath meter Wed Jan 11 14:27:40 2009

hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb3 20000005e600867b use up 0 153.5 19.187 128.000 0.000 128.000

4 21000005e600867b use up 17 1619.0 48.888 30.921 30.924 16.000

Total all LUNs 17 1772.5 68.075 39.328 30.924 127.709


PM5.2_BF 3-15

Beta5. In one of the cBlade console windows, check the following

numbers:

a. For each HBA port, check the value of trq/s (total requests per second). In this example, note that the value for HBA4 is significantly higher than that for HBA3. This indicates that SCSI tape devices are attached to HBA3.

b. For each HBA port, check the value of tmb/s (total megabytes per second). Note again that the value for HBA4 is significantly higher than that for HBA3, indicating that SCSI tape devices are attached to HBA3.

6. In the other cBlade console window, check the same values that you did on the first cBlade. If you have SCSI tape devices configured on this cBlade, the values for trq/s and tmb/s will be lower for the HBA(s) that have SCSI tape devices configured on them.

7. On each cBlade on which SCSI tape devices are configured, in the row labeled Total all LUNs, note the total value for all HBA ports that appears in the tmb/s column.

8. Identify which HBA port has tape devices configured on it, note its value for tmb/s, and then subtract this value from the total tmb/s value for all HBAs. This value represents the total megabytes per second for all HBA ports on the cBlade that are processing I/O for disk devices.

9. Compare the value you derived in Step 8 to the similar value you displayed at the end of the previous section (“Check Disk I/O Only” on page 3-11).

If the two values vary significantly, do the following, in order:

a. Be sure that you have followed the guidelines for configuring SCSI tape devices in “Best Practices for Configuring SCSI Tape Resources” on page 2-16.

b. Contact Fujitsu Technology Solutions customer support or your authorized support vendor for assistance.


3-16 PM5.2_BF

BetaCheck I/O Statistics on Both cBlades

This procedure describes how to check overall I/O statistics for an entire cBlade, and to compare this value to the same value on the other cBlade. The goal is to have the I/O load distributed evenly between the two cBlades.

If you plan to configure SCSI tape devices, we recommend that you do the following:

• Perform this procedure first without any tape devices running. This allows you to determine whether disk I/O is properly balanced across the cBlades.

• Perform this procedure a second time with SCSI tape devices running. In order for this test to be as accurate as possible, configure your pServers (tape media servers) exactly as you would in a production environment. For additional guidelines for configuring tape devices, see “Best Practices for Configuring SCSI Tape Resources” on page 2-16.

To check overall I/O statistics, run the pathtop utility in each cBlade console:

# pathtop

This generates output similar to the following:

multipath meter Fri Feb 3 13:40:44 2009hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb

3 20000005e600867c use up 2 564.9 16.967 30.757 30.804 4.000

4 21000005e600867c use up 0 386.1 11.622 30.826 30.826 0.000Total all LUNs 2 951.0 28.589 30.785 30.813 4.000

hba io requests/sec <= size (bytes)scsi type any 512 1k 2k 4k 8k 16k 32k 64k 128k

3 read 563.9 . 0.4 0.4 6.0 6.0 11.6 539.6 . .

3 write 1.0 . . . 1.0 . . . . .3 total 564.9 . 0.4 0.4 7.0 6.0 11.6 539.6 . .

4 read 386.1 . 0.6 0.4 4.2 4.2 5.0 371.7 . .4 write 0.0 . . . . . . . . .

4 total 386.1 . 0.6 0.4 4.2 4.2 5.0 371.7 . .

The total number of megabytes per second (tmb/s), here 28.589, should be roughly equivalent for cBlade A and cBlade B.


PM5.2_BF 3-17

BetaTake a Snapshot of the Finished SAN Configuration

Note: You can take a snapshot of your SAN configuration in this way only if your SAN uses the active/passive pathing model.

After you are satisfied that the SAN is configured properly, and functioning properly, we recommend that you capture its finished configuration by running the pathctl utility. This can be helpful later, especially when you want to correct I/O imbalances that result from a SAN failover.

To preserve a snapshot of the pathing models that are configured on the storage array:

1. Open a console window on a cBlade.

2. Run the pathctl utility:# pathctl list | more

Output similar to the following appears:Wed Jan 11 15:03:55 2009Active/Passive luns:Path Group 0 Path 15 ( 9, 0,2,0) serial APM00024800495 wwn: 60060178900d0000ba4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,0) ACTIVE Device 1:( 4, 0,2,0) PASSIVE Path 16 ( 9, 0,2,1) serial APM00024800495 wwn: 60060178900d0000bb4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,1) PASSIVE Device 1:( 4, 0,2,1) ACTIVE Path 17 ( 9, 0,2,2) serial APM00024800495 wwn: 60060178900d0000bc4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,2) ACTIVE Device 1:( 4, 0,2,2) PASSIVE Path 18 ( 9, 0,2,3) serial APM00024800495 wwn: 60060178900d0000bd4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,3) PASSIVE Device 1:( 4, 0,2,3) ACTIVE...


3-18 PM5.2_BF

Beta(This output is typical of an active/passive pathing model.)

Keep this output to use later, when comparing the path model for an array that has been running.

Detect and Correct I/O Imbalances

Although SAN failover allows I/O to continue, a SAN configuration in which failover has occurred may contain uneven concentrations of active LUNs. (The SAN combines LUNs originally configured as active, and LUNs that were passive but are now active.)

The following procedures:

• Help detect the active/passive status of LUNs in a SAN in which a failover event has taken place

• Describe how to trespass specific LUNs (that is, change their active/passive status)

Detect an I/O Imbalance

To detect a difference in the configuration of LUNs between the SAN’s current configuration and its original configuration:


2. Run the pathctl utility:# pathctl list | more

3. Compare this output to the output you captured before the SAN failover occurred (see “Take a Snapshot of the Finished SAN Configuration” on page 3-17).

Wed Feb 18 14:13:49 2009Active/Passive luns:Path Group 0 Path 15 ( 9, 0,2,0) serial APM00024800495 wwn: 60060178900d0000ba4bdf9d331cd811 state:NORMAL Clariion


PM5.2_BF 3-19

Beta Device 0:( 3, 0,2,0) ACTIVE Device 1:( 4, 0,2,0) PASSIVE Path 16 ( 9, 0,2,1) serial APM00024800495 wwn: 60060178900d0000bb4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,1) ACTIVE Device 1:( 4, 0,2,1) PASSIVE Path 17 ( 9, 0,2,2) serial APM00024800495 wwn: 60060178900d0000bc4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,2) ACTIVE Device 1:( 4, 0,2,2) PASSIVE Path 18 ( 9, 0,2,3) serial APM00024800495 wwn: 60060178900d0000bd4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,3) ACTIVE Device 1:( 4, 0,2,3) PASSIVE...

Reading from top to bottom, each device’s four values within parentheses indicate HBA port number, path group, target, and LUN, respectively. In this output, each path shows that all I/O is passing through HBA port 3, and none through HBA port 4. (Contrast this to the alternating active and passive status for device 0 in the example in “Take a Snapshot of the Finished SAN Configuration” on page 3-17.)

Correct an I/O Imbalance

If, after examining the output from the pathctl utility, you determine that there are concentrations of active LUNs, the cause might be one of the following:

• Some paths might be broken. If the HBA port numbers that appear are different from those in your original configuration, check the HBA connections on the cBlades, and then rerun the pathctl utility.

• A SAN failover event may have caused some passive paths to become active, and/or some active paths to become passive.

• A storage processor in the storage array may have rebooted or is down, causing LUNs to be trespassed to the remaining storage processor. (Trespassing a LUN means activating a passive path when I/O has failed on the corresponding active path.)


3-20 PM5.2_BF

BetaTo restore the active/passive status of all LUNs to their original state:

1. Determine which LUNs you want to trespass.


3. Run the pathctl utility:# pathctl trespass group x path y device z

where:

x indicates the number of the path group. (If you do not have path groups configured, this number is zero.)

y indicates the path number.

z indicates a single device number within a specific path.

For example, to restore two paths to their previous status, such as Path 16 and Path 18, enter the following:# pathctl trespass group 0 path 16 device 1# pathctl trespass group 0 path 18 device 1

4. To verify that these trespass operations were successful, issue the pathctl utility again:# pathctl list | more

Note the differences compared to the output you generated in the previous section.

Wed Feb 18 14:15:20 2009Active/Passive luns:Path Group 0 Path 15 ( 9, 0,2,0) serial APM00024800495 wwn: 60060178900d0000ba4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,0) ACTIVE Device 1:( 4, 0,2,0) PASSIVE Path 16 ( 9, 0,2,1) serial APM00024800495 wwn: 60060178900d0000bb4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,1) PASSIVE Device 1:( 4, 0,2,1) ACTIVE Path 17 ( 9, 0,2,2) serial APM00024800495 wwn: 60060178900d0000bc4bdf9d331cd811 state:NORMAL Clariion Device 0:( 3, 0,2,2) ACTIVE Device 1:( 4, 0,2,2) PASSIVE Path 18 ( 9, 0,2,3) serial APM00024800495 wwn: 60060178900d0000bd4bdf9d331cd811 state:NORMAL Clariion


PM5.2_BF 3-21

Beta Device 0:( 3, 0,2,3) PASSIVE Device 1:( 4, 0,2,3) ACTIVE...

If the results of the trespass operations are not what you expected, contact Fujitsu Technology Solutions customer support or your authorized support vendor for assistance.

Detecting a Possible Zoning Issue

This procedure describes how to use the pathstat topology report (-T) to detect a load imbalance for a device on a cBlade.

To display the topology report:

1. Use the Linux dd command to generate SCSI device I/O on a pServer.

2. Run this command:# pathstat -T 1 -u -t -i 1

Compare the output on cBlade A to the output on cBlade B. Output similar to the following appears on cBlade A:

Thu Feb 2 16:13:58 2009pBlade topologypb saddr LUN subpath state req/s MB/s err--------------------------------------------------------------- 1 0.0 9.0.1.54 3.0.4.54 act 49.0 1.532 0 1.0 9.0.0.0 3.0.3.0 act 73.5 2.241 0 4.0.2.0 act 74.5 2.223 0 2.0 9.0.0.1 3.0.3.1 act 78.4 2.381 0 4.0.2.1 act 79.4 2.406 0 3.0 9.0.0.2 3.0.3.2 act 80.4 2.483 0 4.0.2.2 act 81.4 2.482 0 120.0 9.0.0.9 3.0.2.9 ACT 186.3 5.694 0 4.0.4.9 PAS 0.0 0.000 0 121.0 9.0.0.11 3.0.2.11 PAS 0.0 0.000 0 4.0.4.11 ACT 179.4 5.351 0 ifca mylpan-ps5 9/11 882.4 26.793 0===============================================================TOTAL 9/11 882.4 26.793 0

Output similar to the following appears on cBlade B:


3-22 PM5.2_BF

BetaThu Feb 2 16:14:18 2009pBlade topologypb saddr LUN subpath state req/s MB/s err--------------------------------------------------------------- 1 0.0 9.0.1.54 3.0.4.54 act 0.0 0.000 0 4.0.3.54 act 63.7 1.991 0 1.0 9.0.0.0 3.0.3.0 act 0.0 0.000 0 4.0.2.0 act 130.4 3.960 0 2.0 9.0.0.1 3.0.3.1 act 0.0 0.000 0 4.0.2.1 act 136.3 4.204 0 3.0 9.0.0.2 3.0.3.2 act 0.0 0.000 0 4.0.2.2 act 147.0 4.461 0 120.0 9.0.0.9 3.0.2.9 ACT 189.2 5.846 0 4.0.5.9 PAS 0.0 0.000 0 121.0 9.0.0.11 3.0.2.11 PAS 0.0 0.000 0 4.0.5.11 ACT 196.1 5.890 0 224.0 3.0.1.0 3.0.1.0 act 88.2 11.028 0 ifcb mylpan-ps5 11/13 950.9 37.381 0===============================================================TOTAL 11/13 950.9 37.381 0

Notice that on cBlade A, SCSI address (saddr) 0.0 has only one path; on cBlade B the same SCSI address has two paths. This could indicate a zoning issue that prevents subpath 4.0.3.54 from appearing on cBlade A.

If this occurs:

a. Check the Fibre Channel cables leading from the fibre switch (if any) to the storage array.

b. Verify that zoning is properly configured on your fabric switch.

c. Note that because there is a tape device configured on HBA port 3 of cBlade B (SCSI address 224.0), it is processing all I/O through HBA port 3, at the expense of other active/active devices whose subpaths use this port (those numbered 3.n.n.n). It is therefore acceptable for there to be zeroes in the req/s and MB/s columns for devices that use these subpaths.

d. If the problem persists, contact Fujitsu Technology Solutions customer support or your authorized support vendor.


PM5.2_BF 3-23

Beta


3-24 PM5.2_BF

Beta

Appendixes

This section of PAN Manager SAN Integration Guide contains the following appendixes:

• Appendix A, “Troubleshooting and Maintenance”

• Appendix B, “Using SAN Diagnostic Tools”

• Appendix C, “SAN Resource Guidelines and Limitations”


PM5.2_BF A-1

BetaAppendix A

Troubleshooting andMaintenance

This appendix lists error situations that might occur during the SAN integration process, and provides some troubleshooting hints and recovery and maintenance suggestions. This appendix includes the following sections:

• SAN Configuration Issues

• Troubleshooting

• Recoverability

• Maintenance Mode

• Diagnosing and Correcting Performance Issues

• SAN Failover

The BladeFrame® BF400 S2 and BladeFrame® BF200 hardware platform integrates with a wide range of SAN storage systems and have been qualified on many hardware types and configurations. Contact Fujitsu Technology Solutions customer support or your authorized support vendor to verify that your storage hardware has been tested and qualified.


A-2 PM5.2_BF

BetaSAN Configuration Issues

If you have questions about the software or firmware versions your storage array requires, contact Fujitsu Technology Solutions customer support or your authorized support vendor.

Troubleshooting

The following table provides a list of issues that you might encounter and suggested steps to troubleshoot and resolve each issue.

Table A.1 Troubleshooting SAN Issues

Issue Observed Troubleshooting Hints

Multipath drivers are disabled

The multipath drivers are enabled by default and cannot be disabled. If you have disabled the generic multipath driver, contact Fujitsu Technology Solutions customer support or your authorized support vendor for instructions specific to your SAN hardware configuration.

Troubleshooting and Maintenance

PM5.2_BF A-3

Beta

Recoverability

You must recover any storage processor failures manually.

The following failures and errors should recover automatically, except where noted:

• Link failures

• Switch failures

• Zoning and LUN masking errors should recover automatically during system operation, but you must recover them manually at startup.

• HBA failures should recover automatically, but you must recover them manually if there are LUN mismatches.

After running the pathtop utility, the utility’s output displays the value Down in the link column.

There is no link light on the HBA adapter, and could indicate one of the following:

• Faulty HBA

• Bad SAN cable

• Bad GBIC on the switch port

• Faulty switch

Run the pathtop utility. In the command output, each row consists of data for a specific HBA port. The value in the Link column indicates whether the HBA port is 'up’ or ‘DOWN’. (See “Check I/O Statistics on Both cBlades” on page 3-16 for information about using the pathtop utility.)

Issue Observed Troubleshooting Hints


A-4 PM5.2_BF

BetaMaintenance Mode

Your cBlades are equipped with a utility called CIO (Customer Information Operation). CIO will automatically alert Fujitsu Technology Solutions customer support or your authorized support vendor in the case of a severe cBlade event, such as an unresponsive cBlade. However, there may be times when such an event is expected, typically during routine maintenance. In such cases, use the maintenance mode feature to prevent an alert being sent to service personnel.

To turn on maintenance mode:

1. Log on to a cBlade.

2. Change to the /opt/panmgr/bin directory.

3. At the prompt, enter:# maintenance on

To turn off maintenance mode:



3. At the prompt, enter:# maintenance off

To check the status of maintenance mode:



3. At the prompt, enter:# maintenance status

A message appears stating whether maintenance mode is on or off.


PM5.2_BF A-5

BetaDiagnosing and Correcting Performance Issues

Tasks common to the maintenance of an integrated SAN and platform include running reports and monitoring system logs to look for anomalies and potential SAN problems, such as failed ports or I/O bottlenecks.

This section describes the cBlade software tools and utilities you can use to diagnose and resolve performance issues related to the internal network and I/O. The cBlade software provides the utilities listed in this section to help monitor and reconfigure I/O paths.

For information about using diagnostic tools to test your SAN configuration, see Chapter 3, “Testing your SAN Configuration”.

Diagnostic Tools There are a number of diagnostic utilities in the cBlade software, as well as standard Linux and Windows diagnostic tools, that you can use to monitor traffic between a platform and a SAN.

Diagnostic Utilities in the Control Blade Software

The cBlade software contains the following diagnostic tools that can help you detect and correct integration issues with your SAN subsystem:

• giscsimon — Reads the multipath statistics and debugging files, and reports information captured by the giscsimon daemon.

• pathctl — Displays the current multipath I/O configuration, and for platforms in which the cBlades contain Fibre Channel HBAs, allows you to reconfigure (or trespass) LUNs on active/passive storage arrays.

• pathstat — Provides device performance monitoring statistics for SAN I/O subpaths, SCSI disks, and SCSI tapes (depending on your storage array type and pServer type).


A-6 PM5.2_BF

Beta• pathtop — Reports performance statistics related to specific

HBAs on the cBlade.

Examples for using pathctl, pathstat, and pathtop utilities to detect and resolve SAN configuration issues appear in Chapter 3, “Testing your SAN Configuration”. For more information about these utilities, see Appendix B, “Using SAN Diagnostic Tools”.

Linux and Windows Diagnostic Tools

Linux tools can provide a window into SAN network traffic to identify and isolate I/O bottlenecks and failed ports. Use the iostat and vmstat commands to report central processing statistics. The top command also provides a snapshot of processor activity in real-time and displays a listing of the most CPU-intensive tasks on the system. Refer to the Linux man pages for additional information about these commands.

Windows Server systems also have a strong set of performance monitoring tools (system monitor, performance logs, and alerts) that you can use to diagnose systems issues specific to pServers that are running Windows Server. See the appropriate pServer Guide for Microsoft Windows Server for details.

SAN Failover

This section describes the conditions that can disrupt all SAN connectivity, and what happens during connection reinitialization and connection failback.

Specifically, the following sections provide the background and guidelines for handling SAN failover:

• Conditions That Disrupt All SAN Connectivity

• Connection Reinitialization


PM5.2_BF A-7

Beta• Connection Failback

Conditions That Disrupt All SAN Connectivity

When properly configured, the hardware platform provides redundant I/O connectivity to a SAN. However, there are specific, unique conditions that disrupt all SAN connectivity.

Figure A.1 illustrates points in the network where SAN disruptions can occur.

Figure A.1 Possible SAN Disruption Points

The following scenarios can cause SAN disruptions:

• Removing and reconnecting one Fibre Channel cable on a cBlade and then removing the second connection in the pair before the first connection reinitializes causes a disruption on all pServer I/O requests through that connection pair.

• Removing only the send or receive portion of the Fibre Channel connection causes the entire connection to fail, and disrupts I/O. If the connection is not restored, pending I/O requests fail over to an active connection.


A-8 PM5.2_BF

Beta• Removing a cBlade from the platform causes an I/O connection

failure on all SAN connections on that cBlade, and disrupts SAN connectivity. If the connection is not restored, the I/O requests fail over to the connection on the other cBlade.

Connection Reinitialization

When a platform detects a disruption on a SAN connection, it resumes the I/O requests within a second or two of the reconnection. If the connection does not come back online within 60 seconds, the I/O requests fail over to the remaining connection.

Connection Failback

If the platform fails over I/O because a connection goes down for more than 60 seconds, the platform fails back I/O when the initial connection is reestablished. The reestablished connection is ready for I/O within a second or two.

Use the pathtop and pathstat utilities to monitor I/O on the platform. For information on using these utilities, see Appendix B, “Using SAN Diagnostic Tools”.

PM5.2_BF B-1

Appendix BUsing SAN Diagnostic

Tools

This appendix describes how to use the diagnostic tools to diagnose issues with multipath I/O to the SAN. Specifically, this appendix discusses the following topics:

• Monitoring I/O Paths with the giscsimon Daemon

• Enabling and Disabling HBA Ports with the pathctl Utility

• Displaying LUN Performance Statistics with the pathstat Utility

• Displaying HBA Port Performance Statistics with the pathtop Utility


B-2 PM5.2_BF

Monitoring I/O Paths with the giscsimon Daemon

This section explains how to monitor multipath I/O with the giscsimon daemon, and consists of the following topics:

• About the giscsimon Daemon

• Examining Log Messages

• Displaying Statistics for I/O Paths

• Editing the giscsimon.conf File

• Using the Start/Stop Script

About the giscsimon Daemon

The giscsimon daemon monitors multipath I/O to facilitate the diagnosis of problem I/O paths. Problem I/O paths can reduce the performance and reliability of pServer applications. We use the term path to specify the connection from an HBA port on the cBlade through one or more fabric switches to a target storage array. A path is numbered with the same integer as the HBA port through which it connects with the external storage. For example, path 4 uses HBA port 4.

The giscsimon daemon reads statistics and debug files maintained by the multipath driver. The giscsimon daemon maintains statistics for the last 1-minute, 5-minute, and 15-minute periods, as well as for the entire time that it runs. These statistics include mean service times, minimum/maximum error rates, and minimum/maximum service times for each time period, as well as the standard deviation of service times.

The service time is the length of time it takes for the multipath driver to send an I/O request to the SAN and receive a reply back from the SAN. Anytime an I/O request doesn't complete on a path and should be retried on the next path (to the same LUN), the giscsimon daemon increases its error count by one. An error can

Using SAN Diagnostic Tools

PM5.2_BF B-3

be caused by a disconnected or broken Fibre Channel cable, data corruption on the path, a SCSI device that is not ready, or a SCSI bus that is busy.

The giscsimon daemon automatically starts running at boot time. You control the giscsimon daemon with the following files:

• /etc/rc.d/init.d/giscsimon (a start/stop script)

• /etc/giscsimon.conf (a configuration file)

The giscsimon daemon reads the multipath I/O statistics and debug files at a frequency specified in the configuration file. If the I/O path’s service time or number of errors exceeds the parameters specified, the giscsimon daemon takes the following actions:

1. Logs messages in the /var/log/messages file (if running in either monitor mode or active mode).

The giscsimon daemon runs in monitor mode unless you explicitly changed the configuration file. By default, the giscsimon daemon logs messages about problem I/O paths once every 15 minutes. For more information on the messages, see “Examining Log Messages” on page B-4.

2. Shuts down the I/O path using the pathctl utility (if running in active mode). Initially, the giscsimon daemon is configured to run in monitor mode.

To change to active mode, you must edit the configuration file and restart the daemon, as described in “Editing the giscsimon.conf File” on page B-9 and “Using the Start/Stop Script” on page B-12. You can also change the values of fields in the configuration file to change the time periods and thresholds for logging messages and shutting down paths.

Caution: Shutting down I/O paths can disrupt a production system. Therefore, you should not configure the giscsimon daemon to run in active mode unless you are instructed to do so by Fujitsu Technology Solutions customer support.


B-4 PM5.2_BF

Although the giscsimon daemon collects statistics, it does not display them by default. To display the statistics collected, you must send a SIGUSR1 signal to the giscsimon daemon. In response, the giscsimon daemon writes the current statistics for each time period to the /var/log/messages file. For more information, see “Displaying Statistics for I/O Paths” on page B-6.

Examining Log Messages

The giscsimon daemon logs messages to the /var/log/messages file. These messages vary with the mode (monitor or active) and the field values in the /etc/giscsimon.conf file.

To examine the messages that the giscsimon daemon logs to the /var/log/messages file, use the following utility:

# tail -f /var/log/messages* | grep giscsimon

This section discusses the following kinds of log messages:

• Monitor Mode Messages

• Active Mode Messages

• Start-up Messages

Monitor Mode Messages

In monitor mode, the giscsimon daemon logs warning messages once every 15 minutes when the I/O path exceeds the values for the error-rate or svc-mean fields in the configuration file. These warning messages have the following format (respectively):

month day time platform-node giscsimon[pid]: Warning: path n exceeding error rate! (expected: x, actual: y)month day time platform-node giscsimon[pid]: Warning: path n exceeding maximum service time! (expected: x, actual: y)

where:

• month — The month.

• day — The day of the month.

• time — The time in 24-hour format.


PM5.2_BF B-5

• platform — The name of the platform.

• node — The cBlade, which is c1 for cBlade 1, or c2 for cBlade 2.

• pid — The process ID of the giscsimon daemon.

• n — The number of the I/O path.

• x — The expected value of the error-rate field (a percentage, such as 1.00 for 1%) or the svc-mean field (in milliseconds, such as 20.00 for 20,000 microseconds) from the configuration file.

• y — The actual value of the error-rate or svc-mean field, which exceeded the expected value.

Note: In the configuration file, the value of the svc-mean field is expressed in microseconds, such as 20,000 microseconds. In the warning message, the values of the svc-mean field are expressed in milliseconds, such as 20.00 milliseconds.

Active Mode Messages

In active mode, the giscsimon daemon shuts down an I/O path and logs a message when the I/O path exceeds the values for the error-rate or svc-mean fields in the configuration file. These path disabled messages have the following format (respectively):

month day time platform-node giscsimon[pid]: path n disabled due to exceeding error rate!month day time platform-node giscsimon[pid]: path n disabled due to exceeding maximum service time!

where:

• month — The month.

• day — The day of the month.

• time — The time in 24-hour format.

• platform — The name of the platform.

• node — The cBlade, which is c1 for cBlade 1, or c2 for cBlade 2.


B-6 PM5.2_BF

• pid — The process ID of the giscsimon daemon.

• n — The number of the I/O path.

In addition to the giscsimon daemon message, the multipath driver logs a message to the /var/log/messages file when the giscsimon daemon disables an I/O path. The multipath driver also displays this message on the console. The message has the following syntax, where n is the number of the I/O path:

path n disabled by path driver

Start-up Messages

Whenever the giscsimon daemon starts or restarts, it logs a series of messages to /var/log/messages that indicate its mode, configuration settings, and the paths found. For example:

Starting (mode: MONITOR)cmd file: /opt/egenera/bin/pathctlgiscsimon startup succeeded

data file: /proc/egenera/multipath/statsdebug file: /proc/egenera/multipath/debugupdate: 10 secssvc-mean: 20000 usecssvc-time: 60 secserror: 0error-stime:0verbose: 0

scan_paths: multipath is enabledscan_paths: 2 paths found, 2 active

Note: Although not shown in the preceding excerpt, each message in the /var/log/messages file begins with the following:month day time platform-node giscsimon[pid]:

Displaying Statistics for I/O Paths

The giscsimon daemon automatically collects statistics for the I/O paths on each cBlade. Specifically, the giscsimon daemon collects statistics for the last 1-minute, 5-minute, and 15-minute periods, as well as for the duration of time it has been running. To display these


PM5.2_BF B-7

statistics, you send a SIGUSR1 signal to the giscsimon daemon, and the giscsimon daemon writes the statistics to the /var/log/messages file.

To send a SIGUSR1 signal to the giscsimon daemon, use the ps, grep, and kill commands, as shown in the following example. Notice the -10 option with the kill command specifies the SIGUSR1 signal.

# ps -A | grep giscsimon 6455 ? 00:00:07 giscsimon# kill -10 6455

The following excerpt from the /var/log/messages file shows sample statistics from the giscsimon daemon:

Current stats for path 4:IOs Er/st Errs mSvcTime(ms) [2**n] Lo [2**n] Hi [2**n] Most (Time)

0/s 0 0 2.71 [ 1:3] [ 9] 23 [13] 5 [10] 47 (40.05s)

1/s 0 0 4.17 [ 1:15] [ 8] 5 [16] 1 [10] 27 (4m40s)0/s 0 0 2.80 [ 1:15] [ 8] 6 [16] 1 [ 9] 28 (14m40s)

1/s 0 0 17.36 [ 0:137] [ 7] 1 [20] 1 [13] 20 (4d0h36m)

Current stats for path 3:IOs Er/st Errs mSvcTime(ms) [2**n] Lo [2**n] Hi [2**n] Most (Time)

0/s 0 0 2.44 [ 2:3] [ 9] 23 [12] 11 [11] 35 (40.05s)

1/s 0 0 4.44 [ 1:11] [ 8] 8 [16] 1 [10] 26 (4m40s)0/s 0 0 3.25 [ 1:11] [ 7] 1 [16] 1 [10] 30 (14m40s)

1/s 0 0 17.84 [ 0:3147][ 7] 1 [23] 1 [13] 21 (4d0h36m)

dump_stats: done

Note: Although not shown in the preceding excerpt, each line of the statistics output in the /var/log/messages file begins with the following:month day time platform-node giscsimon[pid]:

In the sample statistics, notice the following:

• Statistics are grouped by path number.

• The four rows of data for each path correspond to the giscsimon collection periods of 1 minute, 5 minutes, 15 minutes, and the entire time that the giscsimon daemon has been running.


B-8 PM5.2_BF

• The statistics show the results of a histogram operation on I/O requests. The histogram shows the frequency of distribution of I/O requests based on the service time in microseconds. The histogram distributes the results in “buckets,” where each bucket represents a specific service time in microseconds, and counts the number of I/O requests that completed in that service time. The histogram buckets range from less than 1 microsecond (the Lo bucket) to 231 microseconds (the Hi bucket) in frequency.

• The “dump_stats: done” message marks the end of the statistics in the /var/log/messages file.

Table B.1 describes each column in the statistics output.

Table B.1 Columns in giscsimon Statistics

Column Description

IOs The number of I/O operations per second that occurred.

Er/st The number of I/O operations that caused an error per service time interval.

Errs The total number of I/O operations that caused an error since the giscsimon daemon started.

mSvcTime(ms) The estimated mean service time, in milliseconds (ms), which is the length of time it takes for the multipath driver to send an I/O request to the SAN and receive a reply back from the SAN. The values in square brackets show the minimum service time and maximum service time, respectively.

[2**n] Lo The first number specifies the lowest histogram bucket (enclosed in square brackets). The second number specifies the percentage of I/O operations for the bucket. The number of I/O operations in the Lo bucket should be the same across the HBA ports and cBlades.

[2**n] Hi The first number specifies the highest histogram bucket (enclosed in square brackets). The second number specifies the percentage of I/O operations for the bucket. The number of I/O operations in the Hi bucket should be the same across the HBA ports and cBlades.


PM5.2_BF B-9

Editing the giscsimon.conf File

To control how the giscsimon daemon runs, edit the /etc/giscsimon.conf file and change the values of fields as necessary. The following example shows the initial contents of the /etc/giscsimon.conf file after a cBlade software installation:

## config file for giscsimon(8)## syntax is as follows:# token: value# anything after a '#' is comment, whitespace is ignored.#

data-file: /proc/egenera/multipath/statscmd-file: /opt/egenera/bin/pathctldebug-file: /proc/egenera/multipath/debug

error-rate: 0 # absolute error ratefrequency: 10 # sample every n secondssvc-mean: 20000 # svc mean (usecs)svc-time: 60 # secondsmode: monitor # monitor == warn, active == shut down pathsverbose: 0 # verbose mode?error-stime: 0 # max errors/svc time

[2**n] Most The first number specifies the histogram bucket with the most I/O requests (enclosed in square brackets). The second number specifies the percentage of I/O operations for the bucket. The number of I/O operations in the Most bucket should be the same across the HBA ports and cBlades.

(Time) The time period for which the statistics were collected, expressed in a combination of seconds (s), hours (h), minutes (m), and days (d).

Column Description


B-10 PM5.2_BF

Table B.2 describes each field in the /etc/giscsimon.conf file.

Table B.2 Fields in the giscsimon.conf File

Field Name Description and Value

data-file The pathname of the multipath statistics file that is being sampled.

Initial value: /proc/egenera/multipath/stats

cmd-file The pathname of the command that is used by the giscsimon daemon to control the I/O paths on the cBlades.

Initial value: /opt/egenera/bin/pathctl

debug-file The pathname of the multipath debug file, which specifies whether multipath is enabled, the available paths, and the disabled paths. Path groups are separated by a forward slash (/) at the end of each group.

Note: When in active mode, the pathctl utility writes to this file when it disables paths, and it does not separate groups with a slash.

Initial value: /proc/egenera/multipath/debug

error-rate The absolute number of errors that causes the giscsimon daemon to disable the multipath I/O from a cBlade to the external storage. A value of 0 prevents the giscsimon daemon from disabling multipath I/O due to error counts. If the error-rate field is disabled, the giscsimon daemon can still disable multipath I/O when the actual service time exceeds the mean service time (that is, the svc-time and svc-mean fields).

Initial value: 0

frequency How often (in seconds) the daemon should read the file specified by the data-file token. You can specify a value from 1 to 60 seconds.

Initial value: 10

svc-mean The mean service time for the path, in microseconds. You might need to adjust the value to accommodate the type of storage system. If the value is too low, the giscsimon daemon logs warning messages every 15 minutes, which are actually false alarms. If the value is too high, the giscsimon daemon might not warn you about potential issues.

Initial value: 20000 (that is, 20000 microseconds, which is 20 milliseconds)


PM5.2_BF B-11

svc-time The amount of time that a path is in service (after system boot or after it is enabled), in seconds, before the giscsimon daemon logs a warning or disables the path. The giscsimon daemon logs a warning or disables the path when it exceeds the mean service time limit specified in the svc-mean field.

Initial value: 60

mode Whether the giscsimon daemon logs warning messages about problem I/O paths or disables problem I/O paths. This field can have the following values:

• monitor — Logs warning messages when an I/O path exceeds the values specified in the error-rate or svc-mean fields.

• active — Disables an I/O path (using the pathctl utility) when it exceeds the values specified in the error-rate, svc-mean, and svc-time field. You must specify values for all fields (error-rate, svc-mean, and svc-time).

Caution: Do not specify active for this field unless instructed to do so by Fujitsu Technology Solutions customer support. If you specify active mode, the giscsimon daemon has a safeguard so that it does not shut down the last I/O path or the last I/O path in a path group on the cBlade. However, the giscsimon daemon does not protect against the situation where an administrator manually disables one path while the giscsimon daemon automatically shuts down the other path.

Initial value: monitor

verbose Whether to log verbose messages in the /var/log/messages file. Specify the value 0 for normal mode or the value 1 for verbose mode (not recommended).

Initial value: 0

error-stime The number of I/O errors that must occur in the svc-time period to cause the giscsimon daemon to disable the I/O path. A value of 0 disables this field. You use this field in active mode.

Initial value: 0

Field Name Description and Value


B-12 PM5.2_BF

Using the Start/Stop Script

The start/stop script for the giscsimon daemon has the following syntax:

giscsimon {start|stop|restart|reload|status}

Important: If you edit the /etc/giscsimon.conf file and change the values of fields, you need to restart the giscsimon daemon for the changes to take effect. To restart the giscsimon daemon, use the following command:# /etc/rc.d/init.d/giscsimon restart

Enabling and Disabling HBA Ports with the pathctl Utility

This section describes how to enable and disable I/O paths for a specific HBA port on a cBlade with the /opt/egenera/bin/pathctl utility. This section consists of the following topics:

• About the pathctl Utility

• pathctl Utility Arguments

• Displaying the Current HBA Configuration

• Disabling an HBA Port

• Enabling an HBA Port

• Trespassing a Path

About the pathctl Utility

You use the pathctl utility for the following reasons:

• To prepare a path for servicing.

If you need to perform service on a path, you can use the pathctl utility to disable the I/O paths that are using the HBA ports before servicing the path. While the path is disabled, work


PM5.2_BF B-13

can be done on the path without causing failovers and I/O delays for pServers. Then, after service on the path is complete, you can use the pathctl utility to enable the I/O paths for the HBA ports.

• To shut down problem I/O paths that are impacting the performance or reliability of pServer applications.

Typically, you monitor I/O paths with the giscsimon daemon first, as described in “Monitoring I/O Paths with the giscsimon Daemon” on page B-2. Under the guidance of Fujitsu Technology Solutions customer support, you run the giscsimon daemon in a mode in which it shuts down I/O paths that exceed specific performance thresholds. You don’t use the pathctl utility directly; the giscsimon daemon uses the pathctl utility for you.

Caution: When you use the pathctl utility to disable an HBA port, all I/O is disabled for that HBA port.

• To trespass paths from passive to active or active to passive as part of load balancing I/O or manual failover for active/passive arrays.

pathctl Utility Arguments

The pathctl utility provides a variety of arguments and has the following command-line syntax:

# /opt/egenera/bin/pathctl [help] [disable hba] [enable hba] [list [group g path p]] [refresh group g path p] [trespass group g path p device x]

where:

• help — displays the command syntax and supported arguments.

• disable — disables I/O for a specific HBA port. For more information, see “Disabling an HBA Port” on page B-15.

• enable — enables I/O for a specific HBA port. For more information, see “Enabling an HBA Port” on page B-15.


B-14 PM5.2_BF

• list — displays all active/passive LUNs in all path groups, or a single LUN in a specific path group. For more information, see “Trespassing a Path” on page B-16.

• refresh — updates the status (active or passive) for a path that was recently trespassed (with the trespass argument) on an active/passive array. For more information, see “Trespassing a Path” on page B-16.

• trespass — moves a path from active to passive or passive to active on an active/passive array. For more information, see “Trespassing a Path” on page B-16.

If you use the pathctl utility without any arguments, it displays the current multipath I/O and HBA port configuration. For more information, see “Displaying the Current HBA Configuration”.

Displaying the Current HBA Configuration

To display the current multipath I/O and HBA port configurations, use the pathctl utility with no arguments, as follows:

# /opt/egenera/bin/pathctl

Mon Jan 2 16:27:36 2009

Available HBAs: 3 4Disabled HBAs:Disabled paths:

As this example shows, the pathctl utility (with no arguments) displays the following information:

• The date and time that you executed the utility.

• The available HBA ports. The example shows that two HBA ports are available: HBA3 and HBA4.

• The disabled HBA ports. That is, the HBA ports that were disabled using the pathctl utility. The example shows that no HBA ports are disabled.


PM5.2_BF B-15

• The disabled paths. That is, the paths that were disabled by the giscsimon daemon while it was running in active mode. (See “Monitoring I/O Paths with the giscsimon Daemon” on page B-2.) The example shows that no paths are disabled.

Disabling an HBA Port

Typically, you disable an HBA port prior to performing service or maintenance operations. To disable a specific HBA port, use the pathctl utility with the disable option followed by the number of the HBA port (for example, 4). The following example disables multipath I/O on HBA4:

# /opt/egenera/bin/pathctl disable 4

Mon Jan 2 17:46:34 2009

If you display the HBA configuration, it shows that HBA4 and path 4 are disabled:


Mon Jan 2 17:46:34 2009

Available HBAs: 3Disabled HBAs: 4Disabled paths: 4

Enabling an HBA Port

Typically, you enable an HBA port after performing service or maintenance operations. To enable a previously disabled HBA port, use the pathctl utility with the enable option followed by the number of the HBA port (for example, 4). The following example enables multipath I/O on HBA4, which was previously disabled:

# /opt/egenera/bin/pathctl enable 4

Mon Jan 2 17:55:26 2009

If you display the HBA configuration, it shows that HBA4 and path 4 are no longer disabled:



B-16 PM5.2_BF

Mon Jan 2 17:55:35 2009

Available HBAs: 3 4Disabled HBAs:Disabled paths:

Trespassing a Path

SAN arrays that operate in an active/passive configuration can fail over paths in response to errors. What was originally the active path for a LUN can fail over to passive status as a result of the error. After the error resolves itself, some arrays automatically return the passive path back to active status or provide separate administrative utilities for doing so. The act of moving a path from active to passive status is called trespassing the path. Some arrays do not provide a way to trespass a path, so you can use the trespass features of the pathctl utility to do so.

To determine the current status of each path, use the list argument with the pathctl utility. The following example shows the status of paths 35, 36, and 37 in path group 0:

# /opt/egenera/bin/pathctl listWed Oct 11 10:24:40 2009

Active/Passive luns:Path Group 0 Path 35 ( 9, 0,0,1) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,1) ACTIVE Device 1:( 4, 0,0,1) PASSIVE Path 36 ( 9, 0,0,2) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,2) ACTIVE Device 1:( 4, 0,0,2) PASSIVE Path 37 ( 9, 0,0,3) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,3) ACTIVE Device 1:( 4, 0,0,3) PASSIVE


PM5.2_BF B-17

To trespass a path, use the trespass argument and specify the group number, path number, and device number. The following example shows a trespass of device 1 on path 36 from passive status to active status:

# /opt/egenera/bin/pathctl trespass group 0 path 36 device 1Wed Oct 11 11:56:01 2009# /opt/egenera/bin/pathctl listWed Oct 11 11:56:37 2009

Active/Passive luns:Path Group 0 Path 35 ( 9, 0,0,1) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,1) ACTIVE Device 1:( 4, 0,0,1) PASSIVE Path 36 ( 9, 0,0,2) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,2) PASSIVE Device 1:( 4, 0,0,2) ACTIVE Path 37 ( 9, 0,0,3) serial P5849D5AAPL031 state:NORMAL HP EVA Device 0:( 2, 0,0,3) ACTIVE Device 1:( 4, 0,0,3) PASSIVE

In the case where an active/passive array can dynamically trespass paths back to their original status after a failover, the list argument might not show the most recently changed status of the path. In this case, you need to use the refresh argument with the pathctl utility and specify the path group and path number. The following example refreshes the status of path 36:

# /opt/egenera/bin/pathctl refresh group 0 path 36Wed Oct 11 12:12:35 2009


B-18 PM5.2_BF

Displaying LUN Performance Statistics with the pathstat Utility

When diagnosing a storage problem, you can use the pathstat utility to create reports of I/O performance statistics for each LUN. This section describes the following topics:

• About the pathstat Utility

• pathstat Utility Options

• Wildcards in LUN Names

• Columns in pathstat Reports

• pathstat Multipath Report (Default)

• pathstat Multipath Report for Specific LUNs

• pathstat Subpath Report

• pathstat Topology Report

About the pathstat Utility

The pathstat utility can either aggregate the I/O of all the paths for each LUN or show the I/O for each subpath for a LUN or SCSI device. In contrast, the pathtop utility aggregates the I/O of all LUNs for each HBA port. You can create reports with pathstat for the multipath driver, which shows the I/O performance for all paths from the cBlade to the LUN.

Note: Typically, you need to log in to each cBlade and run pathstat on both cBlades to see a complete picture of I/O performance because I/O is load balanced across both cBlades.

In the pathstat statistics, the LUN column specifies the device ID for the LUN, which consists of the following four-tuple:

host.controller.target.LUN


PM5.2_BF B-19

where:

• host — the initiator ID, which is either the HBA port number (between 1 and 8), or multipath port number (9 or greater).

• controller — the path group number, which is 0 by default.

• target — the target ID.

• LUN — the LUN for the target device.

To minimize the overhead associated with statistics collection, the multipath driver does not collect performance statistics until you actually run the pathstat utility. After you run the pathstat utility, the driver continues to collect statistics until you explicitly disable it with the -D option. See “pathstat Multipath Report for Specific LUNs” on page B-25.

pathstat Utility Options

That pathstat utility provides a variety of different options, and has the following command-line syntax:

# /opt/egenera/bin/pathstat [-DESThtu] [-i sec] [-c cnt] [-g grp] [--] [device(s)] [pBlade(s)]

where:

• -D (disable statistics) — Disables collection of performance statistics in the multipath driver. By default, the multipath driver does not collect performance statistics because of the overhead in doing so. Each time you run pathstat, it automatically enables statistics collection so it can generate the reports. If you want to disable statistics collection after you are finished using pathstat, you can use the -D option to do so. This eliminates the overhead in the driver associated with collecting statistics. Keep in mind that the -D option could interrupt the statistics display for any currently running instances of the pathstat utility.

• -E (enable statistics) — Enables collection of performance statistics in the multipath driver. Each time you use pathstat, it automatically enables statistics collection, so you don’t need to explicitly specify this option.


B-20 PM5.2_BF

• -S (subpath) — Displays multipath devices on the cBlade (specified as ifca for cBlade A or ifcb for cBlade B), along with their associated subpaths and subpath states.

• -T (topology) — Displays the topology for each pBlade you specify, listing the associated cBlade (specified as ifca for cBlade A and ifcb for cBlade B), the LPAN pServer name, and the assigned device ID of each configured SCSI device, along with its associated subpaths and subpath states.

• -h (help) — Displays the utility syntax and supported options.

• -t (current time) — Displays the current time at the top of each report. The time format includes the day, date, and time, such as the following: Tue Jan 10 16:43:40 2009. This option can be useful to distinguish between specific reports in a series of reports. Default: Do not display current time.

• -u (update display) — Clears the screen between reports. Default: Do not clear display between each report.

• -i sec (interval) — Specifies the time interval between each report, in seconds. Default: 5.

• -c cnt (iteration count) — Specifies the number of reports to be generated. Default: Continuous (that is, there is no limit; you type Ctrl-C in the shell to terminate report generation).

• -g grp (path group) — Displays a report for the specified path group number. This argument has been deprecated; it remains for backward compatibility. Its value is ignored and the utility defaults to all path groups.

• -- (separator) — Separates the utility options from the LUNs when wildcards are used in LUN names.

• [device(s)] (specific LUNs) — Displays a report for the specified LUNs. For more information on using wildcards, see “Wildcards in LUN Names” on page B-21. Default: All LUNs.

• [pBlade(s)] (specific pBlades) — Displays a report for the specified pBlades. You can specify an inclusive range of pBlades, in the form pX-pY (where the valid range is p1-p24). For example, the range “p1 p4-p6” specifies p1, p4, p5, and p6.


PM5.2_BF B-21

If desired, you can use the -c option to generate a specific number of reports, and then redirect the output of pathstat to a file. Then you could display the file with the more command or import it into another application. For example:

# /opt/egenera/bin/pathstat -c 5 -t >> multipathreport

Wildcards in LUN Names

You can use the following wildcards to specify LUNs:

• * (an asterisk) Matches any number in the device ID (that is, the controller, HBA, target, or LUN). For example, you can use 9.0.2.* to specify any LUN found on target 2.

• - (a dash) Specifies an inclusive range of HBAs, controllers, targets, or LUNs. A range can be specified in one of the following ways:

• low-high where low specifies the beginning of the range and high specifies the end of the range.

• low- where low specifies the beginning of the range, and the end of the range is implied to be 255.

• -high where high specifies the end of the range, and the beginning of the range is implied to be 0.

If you use this syntax to express a range of HBA port numbers, you must use the -- separator between the options and LUNs on the command line.

For example, you can use -8.0.*.10-19 to specify HBAs 0 through 8, path group 0, all targets, and LUNs 10 through 19, as shown in the following example:

#/opt/egenera/bin/pathstat -i 1 -c 10 -t -- -8.0.*.10-19


B-22 PM5.2_BF

Columns in pathstat Reports

The pathstat utility creates two different styles of reports, depending on the options that you use:

• Multipath reports — Display I/O statistics for each LUN for all LUNs or specific LUNs. Multipath reports can help you analyze I/O performance.

• Subpath and topology reports — Display I/O statistics for each subpath of a LUN or each SCSI device associated with a pBlade. Subpath and topology reports can help you analyze I/O load balancing across subpaths.

Table B.3 describes the columns in the Multipath reports.

Table B.3 Columns in the pathstat Multipath Report

Note: Since I/O requests include reads, writes, and other requests, such as status inquiry, the trq/s (total I/O requests) column may not be the sum of the rrq/s (read requests) and wrq/s (write requests)

Column Description Time Period

LUN The device ID for the SAN device. N/A

rrq/s The number of read requests per second. Since last report

rmb/s The number of megabytes read per second. Since last report

wrq/s The number of write requests per second. Since last report

wmb/s The number of megabytes written per second. Since last report

trq/s The number of I/O requests per second. Since last report

tmb/s The volume of I/O requests per second (in megabytes). Since last report

err The number of I/O errors (read and write). Since boot time

q The current number of I/O requests in the queue. Now

%rq The ratio of the I/O requests on a specific path to the total I/O requests for all paths.

Since last report

%mb The ratio of megabytes of I/O requests on a specific path to the total megabytes of I/O requests for all paths.

Since last report


PM5.2_BF B-23

columns. Likewise, the tmb/s (total megabytes of I/O requests) column may not be the sum of the rmb/s (megabytes of read requests) and wmb/s (megabytes of write requests) columns.

Table B.4 describes the columns in the subpath/topology reports:

Table B.4 Columns in the pathstat Subpath and Topology Reports

pathstat Multipath Report (Default)

If you run the pathstat utility without any options, it displays a report every 5 seconds showing the multipath I/O performance for all LUNs in path group 0 across all HBA ports. To terminate report generation, you type Ctrl-C in the command shell.

Note: We recommend not using path groups when configuring storage resources in the PAN. Configuring path groups is unnecessary because the PAN Manager multipath driver automatically discovers the storage settings for multipathing. (This feature was implemented because path groups can be complex and are fundamentally static in nature.)


pb (Topology only) The pBlade number. N/A

saddr (Topology only) The mapping ID of the device on the pServer. N/A

LUN The device ID for the LUN. N/A

subpath The device ID for the subpath (that is, the path through a specific HBA to the LUN). Because the cBlade load balances I/O across both HBAs, the subpath represents the portion of the load to a LUN handled by a specific HBA.

N/A

state The state of the subpath: ACT (active), PAS (passive), PRI (primary), SEC (secondary), or act (active/active).

Now

req/s The number of read/write requests per second. Since last report

MB/s The number of megabytes read/written per second. Since last report

err The total number of read/write errors that occurred. Since last report


B-24 PM5.2_BF

Path group 0 is the default path group if you have not defined any path groups.

If you have defined path groups, you must use the -g option with pathstat; otherwise, you will not see any performance statistics. That is, path group 0 shows no statistics when path groups are explicitly defined. See “pathstat Multipath Report for Specific LUNs” on page B-25.

The following example shows the initial report and second report:

# /opt/egenera/bin/pathstatmultipath hba 3 hba 4 LUN rrq/s rmb/s wrq/s wmb/s trq/s tmb/s err q %rq %mb %rq %mb9.0.0.20 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.21 0.0 0.000 0.0 0.000 0.0 0.000 0 1 0 0 0 09.0.0.22 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.23 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.24 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.25 0.0 0.000 0.0 0.000 0.0 0.000 0 0 100 100 0 09.0.0.26 0.0 0.000 0.0 0.000 0.0 0.000 0 0 100 100 0 09.0.0.27 0.0 0.000 0.0 0.000 0.0 0.000 0 4 100 100 0 09.0.0.28 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.29 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 0 TOTAL 0.0 0.000 0.0 0.000 0.0 0.000 0 5multipath hba 3 hba 4 LUN rrq/s rmb/s wrq/s wmb/s trq/s tmb/s err q %rq %mb %rq %mb9.0.0.20 0.0 0.000 1.2 0.000 1.2 0.000 0 0 50 50 50 509.0.0.21 123.1 3.791 0.0 0.000 123.1 3.791 0 0 50 50 50 509.0.0.22 0.0 0.000 0.4 0.000 0.4 0.000 0 0 50 11 50 899.0.0.23 116.7 3.591 0.0 0.000 116.7 3.591 0 1 50 50 50 509.0.0.24 0.0 0.000 1.4 0.000 1.4 0.000 0 0 57 57 43 439.0.0.25 113.7 3.392 0.0 0.000 113.7 3.392 0 1 50 50 50 509.0.0.26 223.1 5.586 0.6 0.000 223.7 5.586 0 1 50 50 50 509.0.0.27 0.0 0.000 187.7 5.387 187.7 5.387 0 8 50 50 50 509.0.0.28 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.29 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 0 TOTAL 576.6 16.360 191.3 5.387 767.9 21.747 0 11

The first report always shows zero for all columns. In this example, the second report shows the first 5 seconds of I/O measurement. In the second report, notice that significant read requests occurred for LUNs 9.0.0.21, 9.0.0.23, 9.0.0.25, and 9.0.0.26, while significant


PM5.2_BF B-25

write requests occurred for LUN 9.0.0.27. Also, notice in the hba 3 and hba 4 columns, I/O is balanced evenly (50% / 50%) for LUNs with significant reads and writes.

pathstat Multipath Report for Specific LUNs

The following example shows how to use the pathstat utility to display a series of two multipath reports for a specific group of LUNs. Each report includes the current time, and has a time interval of 20 seconds.

# /opt/egenera/bin/pathstat -i 20 -c 2 -t 9.0.0.20-23 9.0.0.26-27Tue Jan 10 17:50:59 2009multipath hba 3 hba 4 LUN rrq/s rmb/s wrq/s wmb/s trq/s tmb/s err q %rq %mb %rq %mb9.0.0.20 0.0 0.000 0.0 0.000 0.0 0.000 0 1 0 0 0 09.0.0.21 0.0 0.000 0.0 0.000 0.0 0.000 0 8 0 0 0 09.0.0.22 0.0 0.000 0.0 0.000 0.0 0.000 0 0 0 0 0 09.0.0.23 0.0 0.000 0.0 0.000 0.0 0.000 0 2 0 0 0 09.0.0.26 0.0 0.000 0.0 0.000 0.0 0.000 0 1 0 0 0 09.0.0.27 0.0 0.000 0.0 0.000 0.0 0.000 0 8 0 0 0 0TOTAL 0.0 0.000 0.0 0.000 0.0 0.000 0 20Tue Jan 10 17:51:19 2009multipath hba 3 hba 4 LUN rrq/s rmb/s wrq/s wmb/s trq/s tmb/s err q %rq %mb %rq %mb9.0.0.20 142.7 4.297 0.7 0.000 143.4 4.297 0 1 50 50 50 509.0.0.21 0.0 0.000 132.0 4.097 132.0 4.097 0 8 50 50 50 509.0.0.22 147.3 4.397 0.8 0.000 148.1 4.447 0 0 50 50 50 509.0.0.23 0.0 0.000 131.3 4.097 131.3 4.097 0 8 50 50 50 509.0.0.26 137.9 4.147 0.5 0.000 138.4 4.147 0 0 50 50 50 509.0.0.27 0.0 0.000 131.8 4.097 131.8 4.097 0 8 50 50 50 50TOTAL 428.0 12.840 397.2 12.291 825.1 25.181 0 25

pathstat Subpath Report

The following example shows how to use the pathstat utility to display the subpaths, states, and I/O performance for each LUN. When running an I/O load, you can use this report to analyze the load balancing of I/O across subpaths. During normal operation, the I/O load for each LUN is split evenly across its subpaths. If a SAN cable was disconnected from one cBlade, however, the I/O for that subpath of the LUN would be zero.

# /opt/egenera/bin/pathstat -S 9.0.1.93 9.0.1.116 9.0.2.167multipath w/subpaths LUN subpath state req/s MB/s err


B-26 PM5.2_BF

--------------------------------------------------------------9.0.1.93 3.0.1.93 act 0.2 0.001 09.0.1.93 4.0.0.93 act 0.0 0.000 09.0.1.93 ifca 2/2 0.2 0.001 0------------9.0.1.116 3.0.1.116 act 0.0 0.000 09.0.1.116 4.0.0.116 act 0.0 0.000 09.0.1.116 ifca 2/2 0.0 0.000 0------------9.0.2.167 3.0.2.167 act 87.6 0.155 09.0.2.167 4.0.1.167 act 88.2 0.184 09.0.2.167 ifca 2/2 175.8 0.339 0------------==============================================================TOTAL 6/6 176.0 0.339 0

pathstat Topology Report

The following example shows how to use the pathstat utility to display a topology map of the pBlade, including its list of device IDs and mapped SAN devices. Like the subpath report, the topology report lists each SAN device and its associated subpaths and subpath states.

# /opt/egenera/bin/pathstat -T p2-p6

pBlade topologypb saddr LUN subpath state req/s MB/s err------------------------------------------------------------------ 3 0.0 9.0.1.18 3.0.1.18 act 0.6 0.012 0 ifca nettest-ps2 1/1 0.6 0.012 0------------------------------------------------------------------ 4 0.0 9.0.1.22 3.0.1.22 act 0.2 0.001 0 ifca nettest-ps4 1/1 0.2 0.001 0------------------------------------------------------------------ 5 0.0 9.0.1.23 3.0.1.23 act 0.0 0.000 0 ifca nettest-ps5 1/1 0.0 0.000 0------------------------------------------------------------------ 6 0.0 9.0.1.93 3.0.1.93 act 0.6 0.009 0 4.0.0.93 act 0.6 0.002 0 1.0 9.0.1.116 3.0.1.116 act 0.0 0.000 0 4.0.0.116 act 0.0 0.000 0 2.0 9.0.2.167 3.0.2.167 act 89.8 0.157 0 4.0.1.167 act 89.4 0.156 0 ifca nettest-ps6 6/6 180.3 0.325 0==================================================================TOTAL 9/9 181.0 0.337 0


PM5.2_BF B-27

Displaying HBA Port Performance Statistics with the pathtop Utility

When diagnosing a storage problem, you can use the pathtop utility to create reports of I/O performance statistics for each HBA port, also known as a path. This section describes the following topics:

• About the pathtop Utility

• pathtop Utility Defaults

• pathtop Statistics Format

• pathtop Utility Options

• pathtop Example

About the pathtop Utility

The pathtop utility aggregates the I/O of all LUNs for each HBA port, whereas the pathstat utility aggregates the I/O of all the paths for each LUN. You can create reports for the multipath driver, which shows the I/O performance for the path from the cBlade to the LUNs.

Note: Typically, you need to log in to each cBlade and run the pathtop utility on both cBlades to see a complete picture of I/O performance, because I/O is load balanced across both cBlades.

To minimize the overhead associated with statistics collection, the multipath driver does not collect performance statistics until you actually run the pathtop utility. After you run the pathtop utility, the driver continues to collect statistics until you explicitly disable it with the -D option. See “pathtop Utility Options” on page B-30.


B-28 PM5.2_BF

pathtop Utility Defaults

If you run the pathtop utility without any options, it displays a report every 5 seconds showing the multipath I/O performance on each HBA port for all LUNs in path group 0. To terminate report generation, you type Ctrl-C in the command shell.

Note: We recommend not using path groups when configuring storage resources in the PAN. Configuring path groups is unnecessary because the PAN Manager multipath driver automatically discovers the storage settings for multipathing. (This feature was implemented because path groups can be complex and are fundamentally static in nature.)

Path group 0 is the default path group if you have not defined any path groups. If you have defined path groups, you must use the -g option with pathtop; otherwise, you will not see any performance statistics. That is, path group 0 shows no statistics when path groups are explicitly defined. See “pathtop Utility Options” on page B-30.

The following example shows the first report and then the second report. The first report always shows zero for all columns:

# /opt/egenera/bin/pathtopmultipath meter Mon Jan 9 13:21:36 2009hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb 3 20000005e6008682 use up 11 0.0 0.000 0.000 0.000 0.000 4 21000005e6008682 use up 12 0.0 0.000 0.000 0.000 0.000 Total all LUNs 23 0.0 0.000 0.000 0.000 0.000hba io requests/sec <= size (bytes)scsi type any 512 1k 2k 4k 8k 16k 32k 64k 128k 3 read 0.0 . . . . . . . . . 3 write 0.0 . . . . . . . 0.0 . 3 total 0.0 . . . . . . . 0.0 .4 read 0.0 . . . . . . . 0.0 .

4 write 0.0 . . . . . . . . . 4 total 0.0 . . . . . . . 0.0 .

The second report shows the first 5 seconds of I/O measurement. Notice that the reads and writes are almost evenly split across HBAs 3 and 4 and that the bulk of the requests are 64K in size:

multipath meter Mon Jan 9 13:21:41 2009


PM5.2_BF B-29

hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb 3 20000005e6008682 use up 12 379.8 11.574 31.210 31.644 30.888 4 21000005e6008682 use up 13 378.6 11.574 31.308 31.527 31.145 Total all LUNs 25 758.3 23.148 31.259 31.585 31.016hba io requests/sec <= size (bytes)scsi type any 512 1k 2k 4k 8k 16k 32k 64k 128k 3 read 161.4 . . 0.2 . . 1.2 1.4 158.6 . 3 write 218.3 . . 0.8 0.6 0.8 0.4 0.4 215.3 . 3 total 379.8 . . 1.0 0.6 0.8 1.6 1.8 374.0 .4 read 162.0 . . . . 0.4 0.2 0.6 160.8 .

4 write 216.5 . . 0.2 0.4 0.8 0.2 0.6 214.3 . 4 total 378.6 . . 0.2 0.4 1.2 0.4 1.2 375.2 .

The following section outlines the meaning of each column in the reports:

pathtop Statistics Format

Table B.5 describes the columns in the pathtop report.

Table B.5 Columns in the pathtop Report


hba The number of the HBA port that is reporting the statistics.

N/A

world-wide-name

The World Wide Name (WWN) that uniquely identifies the HBA port.

N/A

enable Whether the HBA port is enabled. This column displays use if the HBA port is enabled or OFF if it is disabled.

For example, if the giscsimon daemon is in active mode, the pathctl utility can automatically disable HBA ports as a way of disabling problem I/O paths. For more information, see “Monitoring I/O Paths with the giscsimon Daemon” on page B-2.

Now


B-30 PM5.2_BF

pathtop Utility Options

The pathtop utility provides a variety of options, and has the following command-line syntax:

# /opt/egenera/bin/pathtop [-bDEh] [-i sec] [-c cnt] [--] [luns]

where:

• -b (batch mode) — Does not clear the screen between reports.Default: clear the screen between reports.

link Whether the link used by the HBA port is up. This column displays up if the link is up or DOWN if it is down. For example, if the cable connecting the HBA port to the external storage has been removed, this column displays DOWN.

Now

que The current number of I/O requests in the queue. Now

trq/s The number of I/O requests per second. Since last report

tmb/s The volume of I/O requests per second (in megabytes). Since last report

avg-tkb The average size of I/O requests (in kilobytes). Since last report

avg-rkb The average size of read requests (in kilobytes). Since last report

avg-wkb The average size of write requests (in kilobytes). Since last report

hba scsi The number of the HBA port that is reporting the statistics.

N/A

io type The type of I/O request, which can be one of the following: read (read requests), write (write requests), spcfd (all requests for specific group of LUNs), and total (all requests for all LUNs).

N/A

requests/sec <= sizes (bytes)

The number of requests of each specific size that occurred per second. The request sizes include 512 (512 bytes), 1K, 2K, 4K, 8K, 16K, 32K, 64K, and 128K.

Since last report



PM5.2_BF B-31

• -D (disable statistics) — Disables collection of performance statistics in the multipath driver. By default, the multipath driver does not collect performance statistics because of the overhead in doing so. Each time you run the pathtop utility, it automatically enables statistics collection so it can generate the reports. If you want to disable statistics collection after you are finished using the pathtop utility you can use the -D option to do so. This eliminates the overhead in the driver associated with collecting statistics. Keep in mind that the -D option could interrupt the statistics display for any currently running instances of the pathtop utility.

• -E (enable statistics) — Enables collection of performance statistics in the multipath driver. Each time you use the pathtop utility, it automatically enables statistics collection, so you don’t need to explicitly specify this option.

• -h (help) — Displays the utility syntax and supported options.

• -i sec (interval) — Specifies the time interval between each report, in seconds. Default: 5.

• -c cnt (iteration count) — Specifies the number of reports to be generated. Default: Continuous (that is, there is no limit; you type Ctrl-C in the shell to terminate report generation).

• -- (separator) — Separates the utility options from the LUNs when wildcards are used in LUN names.

• luns (specific LUNs) — Displays a report for the specified LUNs. Default: All LUNs.

The pathtop utility supports the same wildcards as the pathstat utility. For information on using wildcards, see “Wildcards in LUN Names” on page B-21.

If desired, you can use the -c option to generate a specific number of reports, and then redirect the output of pathtop to a file. Then you could display the file with the more command or import it into another application. For example:

# /opt/egenera/bin/pathtop -c 5 >> multipathreport


B-32 PM5.2_BF

pathtop Example The following example shows how to use the pathtop utility in batch mode to display a series of two multipath reports for a specific LUN (9.0.0.26). Each report has a time interval of 20 seconds.

# /opt/egenera/bin/pathtop -b -i 20 -c 2 9.0.0.26multipath meter Wed Jan 11 12:03:21 2009hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb 3 20000005e6008682 use up 0 0.0 0.000 0.000 0.000 0.000 4 21000005e6008682 use up 1 0.0 0.000 0.000 0.000 0.000 Total specified LUNs 1 0.0 0.000 0.000 0.000 0.000 Total all LUNs 26 0.0 0.000 0.000 0.000 0.000hba io requests/sec <= size (bytes)scsi type any 512 1k 2k 4k 8k 16k 32k 64k 128k 3 read 0.0 . . . . . . . . . 3 write 0.0 . . . . . . . . . 3 spcfd 0.0 . . . . . . . . . 3 total 0.0 . . . . . . . . . 4 read 0.0 . . . . . . . . . 4 write 0.0 . . . . . . . . . 4 spcfd 0.0 . . . . . . . . . 4 total 0.0 . . . . . . . 0.0 .

The second report shows the following:

• The total read and write performance for LUN 9.0.0.26 for each HBA port (spcfd row).

• The total read and write performance for all LUNs for each HBA port (total row).

• The bulk of the I/O activity is read requests as opposed to write requests.

multipath meter Wed Jan 11 12:03:41 2009hba world-wide-name enable link que trq/s tmb/s avg-tkb avg-rkb avg-wkb 3 20000005e6008682 use up 0 80.1 1.949 24.929 25.101 0.000 4 21000005e6008682 use up 1 79.9 1.949 24.976 25.070 0.000 Total specified LUNs 1 160.0 3.898 24.952 25.085 0.000 Total all LUNs 27 864.0 24.135 28.604 25.987 31.644hba io requests/sec <= size (bytes)scsi type any 512 1k 2k 4k 8k 16k 32k 64k 128k 3 read 79.5 . . 0.3 0.6 3.9 8.2 35.1 31.3 . 3 write 0.5 . . . . 0.4 0.0 0.0 0.0 . 3 spcfd 80.1 . . 0.3 0.6 4.3 8.3 35.1 31.4 . 3 total 432.1 . . 0.8 1.6 11.0 20.7 111.4 286.5 . 4 read 79.6 . . 0.2 0.5 3.3 7.8 36.3 31.4 .


PM5.2_BF B-33

4 write 0.3 . . . . 0.2 0.1 . . . 4 spcfd 79.9 . . 0.2 0.5 3.5 7.9 36.3 31.4 . 4 total 431.9 . . 0.5 1.6 12.3 18.5 111.9 287.1 .


B-34 PM5.2_BF

PM5.2_BF C-1

BetaAppendix C

SAN Resource Guidelinesand Limitations

This appendix provides guidelines for using SAN resources in the Processing Area Network (PAN). It also includes any limitations, where applicable, on the maximum number of SAN resources available for use in the PAN.

As you configure SAN resources, be aware of the following resource limits.

Table C.1 Resource Guidelines

Characteristic Value

Maximum # of SCSI devices (disks + tapes) per PAN:

• For PAN Manager Releases lower than 4.0.1.5

• For PAN Manager Releases 4.0.1.5 through 5.1.n

• For PAN Manager Releases 5.2 and higher

688

2000

2048

Maximum # of SAN targets per HBA(For systems on which the specified version of the Red Hat kernel is running on the cBlade.)

256 (on kernel version 2.4.n) or 512 (on kernel version 2.6.n)


C-2 PM5.2_BF

Beta

Maximum # of LUNs per target:

• For Fibre Channel HBAs

(This limit derives from the HBA’s QLogic driver.)

PAN Manager cannot discover any LUN that has an ID equal to or greater than 255. For example, the LUN 9.0.3.200 would be visible, but LUN 9.0.3.300 would not be visible.

256 (numbered 0 through 255)

Maximum # of device mappings (disks + tapes + media drives + virtual CDs [VCDs] per pServer)

256

Table C.1 Resource Guidelines (Continued)

Characteristic Value

Index-1

A

active mode B-3, B-5, B-11active subpaths 3-8active/active 1-9, 1-12active/passive 1-9, 1-12

restoring LUNs to 3-20alert A-6arbitrated loop configurations 1-5auto import feature 2-9avg-rkb, avg-tkb, and avg-wkb columns

B-30

B

-b optionpathtop utility B-30

batch mode, pathtop utility B-30BladeFrame ES port number 1-4BladeFrame EX port numbers 1-4breaking connectivity A-7

C

-c optionpathstat utility B-20

pathtop utility B-31cBlades

checking I/O on both 3-16check load balancing on the cBlades

3-11checking device I/O 3-13checking disk I/O 3-11clearing screen between reports B-20cmd-file field B-10commands

console 3-5, 3-14dd 3-6, 3-14san 2-6, 2-9

component failure 1-2connecting a platform to the SAN 2-1connection failback A-8connection time to reinitialize A-8connection type 1-5, 1-6connection types, SAN 1-5connectivity disruption A-7console command 3-5, 3-14console messages B-6control port on a storage array 1-2correcting I/O imbalances 3-18

Index


Index-2

D

-D optionpathstat utility B-19pathtop utility B-31

data-file field B-10dd command 3-6, 3-14debug-file field B-10detecting I/O imbalances 3-18device I/O, checking 3-13device ID B-18device names

PAN Manager device name 2-5path device name 2-5unique identifier (UID) 2-6

device numbering, about 2-5diagnostic tools A-5

Linux A-6Windows A-6

direct attach configurations 1-5illustrated 1-6

disablingHBA ports B-15pathstat statistics B-19pathtop statistics B-31

disk devicetesting I/O on a single 3-4

disk deviceschecking connectivity to 3-2

disk I/O, checking 3-11displaying

giscsimon statistics B-4, B-6disrupting connectivity A-7dropped packet A-6

E

-E option

pathstat utility B-19pathtop utility B-31

editing the giscsimon.conf file B-9enable column B-29enabling

HBA ports B-15pathstat statistics B-19pathtop statistics B-31

Er/st column B-8err column B-22error-rate field B-4, B-5, B-10errors

LUN masking A-3zoning A-3

error-stime field B-11Errs column B-8/etc/sysconfig/ifc/multipath_devices

2-3/etc/giscsimon.conf file B-9

F

fabric switches 1-6fabrics 1-6failover A-8failures

switch A-3fibre channel cable 1-2files

/etc/sysconfig/ifc/multipath_devices 2-3

frequency field B-10

G

-g optionpathstat utility B-20

Index

Index-3

generic multipath 1-9generic multipath software

managing 2-3giscsimon daemon B-2

active mode B-5configuration file B-9modes B-11monitor mode B-4startup messages B-6statistics format B-8

giscsimon utility, defined A-5giscsimon.conf file B-9guidelines

max devices per PAN C-1max devices per pServer C-2max LUNs per target C-2max targets per HBA C-1

guidelines and limitations for SAN resources C-1

H

hard zoning 2-2hba and hba scsi columns B-29HBA configurations 1-3HBA ports

enabling and disabling B-12performance statistics B-27

High column B-8histogram buckets B-8

I

-i optionpathstat utility B-20pathtop utility B-31

I/O bottleneck, isolating A-6

I/O imbalances, correcting 3-18I/O imbalances, detecting 3-18I/O, pausing during failover A-8importing SCSI devices into the PAN

2-9initial discovery process 1-11initiator 2-2integrating with a SAN 2-1io type column B-30IOs column B-8issues, troubleshooting A-2iteration count

pathstat statistics B-20pathtop statistics B-31

L

link column B-30link failure A-3Linux diagnostic tools A-6Lo column B-8log, performance A-6LUN column B-22LUN masking error A-3LUN performance statistics, displaying

B-18

M

maintenance modedefined A-4

maintenance suggestions A-5%mb column B-22messages file

giscsimon active mode B-5giscsimon monitor mode B-4giscsimon startup B-6


Index-4

mode field B-3, B-11monitor mode B-3, B-4, B-11monitoring I/O paths B-2monitoring, SAN integration A-5Most column B-9mSvcTime column B-8multipath

on cBlade 1-11multipath I/O B-2, B-10, B-18, B-27multipath on cBlades 1-11multipath on cBlades and pServers 1-9multipath on pServers 1-10

illustrated 1-10multipathing 1-2multi-switch fabric 1-6, 1-11

illustrated 1-8

N

no single point of failure 1-6designing for 1-2

O

/opt/egenera/bin/pathctl utility B-12/opt/egenera/bin/pathstat utility B-18/opt/egenera/bin/pathtop utility B-27/opt/egenera/bin/pathctl utility B-10

P

PAN Manager device names 2-8comparing to path device names 2-9of the form 10.n.n.n 2-5of the form 9.n.n.n 2-5why they can change 2-8

PAN softwarevirtualization 1-2

path device names 2-7comparing to PAN Manager device

names 2-9of the form 9.n.128.n 2-5of the form 9.n.n.n 2-5preventing changes 2-8why they can change 2-7

path disabled messages B-5path groups B-24, B-28

pathstat statistics B-20pathctl utility 3-17, 3-20, B-10, B-12

defined A-5pathing models

active/active 1-9, 1-12active/passive 1-9, 1-12preferred path 1-9, 1-12primary/secondary (see preferred

path) 1-12pathing models, preserving 3-17pathstat utility 2-16, 3-5, 3-21, B-18

command-line options B-19defaults B-23defined A-5statistics format B-22wildcards B-21

pathtop utility 2-16, 3-13, 3-16, B-27command-line options B-30defaults B-28defined A-6example report B-32statistics format B-29wildcards B-21

pausing I/O during failover A-8performance issues

diagnosing and correcting A-5performance log A-6

Index

Index-5

performance monitoring using Windows Server 2003 A-6

performance statistics B-18, B-27persistent reservations 2-15persistent reservations, about 2-15port numbers 1-4preferred path 1-9, 1-12primary/secondary (see preferred path)

1-12/proc/egenera/multipath directory B-10puTTY utility 3-5, 3-14

Q

q column B-22que column B-30

R

READ THIS FIRSTPAN Manager Configuration Guide

2-1recoverability A-3redundancy 1-2reinitializing SAN connection A-8reloading the giscsimon daemon B-12reservations

SCSI 2-15reserve/release reservations 2-15reserve/release reservations, about 2-16restarting the giscsimon daemon B-12rmb/s, rrq/s, and %rq columns B-22

S

-S option

pathstat utility B-20SAN

connection types 1-5SAN cables 2-1san command 2-6, 2-9SAN configuration

testing 3-1SAN connection, reinitializing A-8SAN disruption points, illustrated A-7SAN I/O on the cBlades

managing 3-10SAN integration monitoring A-5SAN issues

troubleshooting A-2SAN resources

guidelines and limitations C-1SAN zoning 2-2scalability 2-3script for starting giscsimon B-3, B-12SCSI device numbering

managing 2-5SCSI devices

testing I/O on 3-4validating physical connectivity 3-2

SCSI reservations 2-15persistent 2-15persistent, about 2-15reserve/release 2-15reserve/release, about 2-16

SCSI-2 standard 2-15SCSI-3 standard 2-15sending signal to giscsimon B-4, B-6servicing HBAs B-12shutting down I/O paths B-3, B-5,

B-10, B-11, B-12SIGUSR1 signal B-4, B-6single fabric attachment 1-6single initiator zoning 1-14


Index-6

single switch fabric 1-6illustrated 1-7

soft zoning 2-2ssh utility 3-5, 3-14starting the giscsimon daemon B-12startup messages, giscsimon B-6statistics

displaying B-4, B-6HBA port performance B-27LUN performance B-18maintained by giscsimon daemon

B-2stopping the giscsimon daemon B-12storage processor failure A-3subpaths

pathstat utility B-20svc-mean field B-4, B-5, B-10svc-time field B-11switch director software 2-2switch failure A-3system monitor A-6

T

-T optionpathstat utility B-20

-t option, pathstat utility B-20tape device

testing I/O on a single 3-8tape devices 1-9, 1-13

across LPANs 2-17best practices 2-16checking connectivity to 3-3configuring 1-13LUN number (zero) 1-14maximum transfer size 2-18, 3-10optimizing performance 2-17

performing backups 2-17range of target IDs 1-14SCSI reservations 2-17single-initiator zoning 1-14technical considerations 2-17zoning 2-16

tape devices in a SAN, illustrated 1-14target 2-2testing your SAN configuration 3-1Time column B-9time interval

pathstat statistics B-20pathtop statistics B-31

time to reinitialize connection A-8tmb/s and trq/s columns B-22, B-30topology

pathstat utility B-20topology report (pathstat -T)

displaying 3-21trespassing paths B-12troubleshooting A-2troubleshooting and maintenance A-1

U

-u option, pathstat utility B-20utilities

pathctl 3-17, 3-20pathstat 2-16, 3-5, 3-21pathtop 2-16, 3-13, 3-16puTTY 3-5, 3-14ssh 3-14

utilities ssh 3-5

V

verbose field B-11

Index

Index-7

W

wildcards in LUN names B-21Windows diagnostic tools A-6wmb/s and wrq/s columns B-22worldwide name (WWN) 2-2world-wide-name column B-29WWN (worldwide name) 2-2

Z

zoning 3-4detecting issues 3-21error A-3hard 2-2, 2-3SAN 2-2single initiator 1-14, 2-2soft 2-2suggestions 2-2

zoning and scalability 2-2zoning, purpose 2-2


Index-8

Documents

PAN Manager SAN Integration Guide - Fujitsumanuals.ts.fujitsu.com/file/3644/bf200-400_san-integration-en.pdf · Red Hat is a registered trademar k of Red Hat, Inc. in the Un ited