Using ibm total storage productivity center for disk to monitor the svc redp3961

Redbooks Paper

Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC

IntroductionSuccessful businesses require real-time responsiveness to change—whether it is due to new customer needs, changes in the supply chain, unexpected competitive moves, external threats, or changes in the economic climate. Rapid response to change requires an IT infrastructure that can turn information into a competitive advantage; the IT infrastructure must provide maximum benefit at an affordable cost and must have the flexibility to support changes in business processes. An On Demand Operating Environment provides a cost effective and flexible IT environment. With information at the heart of competitiveness, storage becomes an ever more critical component of an On Demand Operating Environment.

The IBM® TotalStorage® strategy addresses some of the most pressing needs currently facing Chief Information Officers (CIOs) and IT managers. As part of its strategy, IBM intends to deliver industry-leading technologies that will help dramatically reduce the total cost of ownership (TCO) for storage and help turn fixed costs into variable costs that scale with business volume.

Success in the on demand world will depend on the ability to leverage information technology. A greater dependence on information means a greater dependence on storage. What differentiates an On Demand Business is the ability to quickly sense and rapidly respond to a dynamic marketplace—to do this, there are challenges that an On Demand Business must overcome.

At the business level, customers are faced with three major storage challenges:

� Managing storage growth: Storage needs continue to grow at more than 50% per year. Managing storage is becoming more complex than ever, because we now have to deal with multiple server platforms and different operating systems, which might be connected to a storage area network (SAN) with multiple and diverse storage platforms.

Barry MellishRob Nicholson

Steve Hurley

© Copyright IBM Corp. 2005. All rights reserved. ibm.com/redbooks 1

http://www.redbooks.ibm.com/


� Increasing complexity: Although the declining cost of storage per megabyte makes it attractive to add additional disks, the increasing complexity of managing this storage results in overused staff and under-used IT resources. Combining this with the shortage of skilled storage administrators, it is possible to add significant cost and introduce risk to storage management.

� Maintaining availability: The added complexity of 24x7 environments significantly reduces, for example, the efficiency of conducting routine maintenance, scheduling backups, migrating data, and introducing new software and hardware. This problem is compounded by the fact that as availability increases, so does the cost inherent with making it so.

These challenges still exist, although large SANs do offer desirable and tangible benefits, for example, better connectivity, improved performance, distance flexibility, and scalability. Yet even these benefits might be outweighed by the added complexity that they introduce.

As an example, large enterprise SANs often contain different types of storage devices. These differences could be in the types of disk deployed, their level of performance, or the functionality provided, such as RAID or mirroring. Often customers have different vendor storage devices as the result of mergers or consolidations. The result, however, is that storage and SAN administrators need to configure storage to servers, and then keep track of which servers own or have access to that storage. The storage administrative tasks can become daunting as the SAN grows and as the storage administrators manually attempt to manage the SAN.

Furthermore, the complexity of different file systems in the same SAN requires that storage administrators know how to administer each client operating system (OS) platform. The management interfaces for each might be different, because there is no common standard to which all vendors adhere. Lastly, because the file systems are tied to each of the servers, storage management functions potentially have to be run on hundreds of servers. It is easy to see why manageability and interoperability are the top areas for concern, especially in a SAN where the number of possible storage and OS platform permutations are considerable.

These challenges are at odds with the commonly held belief that storage is decreasing in cost per megabyte. It is clear that the cost of managing storage is greater than the initial purchase price. A strategy is needed to address storage manageability, while at the same time, addressing the need for interoperability. This strategy is the IBM TotalStorage Open Software Family.

This strategy represents the next stage in the evolution of storage networking. It affords you the opportunity to fundamentally improve your company’s effectiveness and efficiency in managing its storage resources. With the IBM SAN virtualization products, you are witnessing IBM deliver on its continued promise to provide superior on demand solutions that will assist in driving down costs and reducing TCO.

In this IBM Redpaper, we cover the interaction of two products and their components.

IBM TotalStorage SAN Volume ControllerIBM TotalStorage SAN Volume Controller (SVC) is a virtualization appliance solution that maps virtualized volumes visible to hosts and applications to physical volumes on storage devices. Each server within the SAN has its own set of virtual storage addresses that are mapped to a physical address. If the physical addresses change, the server continues running using the same virtual addresses it had before. This means that volumes or storage can be added or moved while the server is still running. The IBM virtualization technology

2 Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC

improves management of information at the “block” level in a network, enabling applications and servers to share storage devices on a network.

For more information, see the IBM Redbook, IBM TotalStorage SAN Volume Controller, SG24-6423.

IBM TotalStorage Productivity CenterThe IBM TotalStorage Productivity Center is an open storage infrastructure management solution designed to help reduce the effort of managing complex storage infrastructures to help improve storage capacity utilization and to help increase administrative efficiency. It is designed to enable the storage infrastructure to have the ability to respond to “on demand” storage needs.

IBM TotalStorage Productivity Center consists of the following four products.

� IBM TotalStorage Productivity Center for Data (formerly IBM Tivoli® Storage Resource Manager) can provide more than 300 enterprise-wide reports, monitoring and alerts, policy-based action, and file system capacity automation in a heterogeneous environment.

� IBM TotalStorage Productivity Center for Fabric (formerly IBM Tivoli Storage Area Network Manager) can provide automated device discovery, topology rendering, error detection fault isolation, SAN error predictor, zone control, real-time monitoring and alerts, and event management for heterogeneous enterprise SAN environments.

� IBM TotalStorage Productivity Center for Disk (formerly IBM TotalStorage Multiple Device Manager – Performance Manager feature) can enable device configuration and management of supported SAN-attached devices from a single console. It can discover storage and provides configuration capabilities for IBM TotalStorage Enterprise Storage Server® (ESS), IBM TotalStorage SAN Volume Controller (2145), and the IBM TotalStorage DS4000 (formerly Fibre Array Storage Technology, FAStT) storage. IBM TotalStorage Productivity Center for Disk manages the performance for the ESS and SAN Volume Controller (SVC).

� IBM TotalStorage Productivity Center for Replication (formerly IBM TotalStorage Multiple Device Manager – Replication Manager feature) can provide copy services management for the ESS. IBM TotalStorage Productivity Center for Replication provides configuration and management of the point-in-time copy (FlashCopy®) and Metro Mirror (synchronous point-to-point remote copy) capabilities of the ESS in supported configurations.

In this Redpaper, we focus on IBM TotalStorage Productivity Center for Disk.

IBM TotalStorage Productivity Center for DiskIBM TotalStorage Productivity Center for Disk is designed to centralize management of networked storage devices that implement the Storage Management Initiative Specification (SMI-S) established by the Storage Networking Industry Association (SNIA), including the IBM TotalStorage Enterprise Storage Server, IBM TotalStorage DS4000 series family (formerly the FAStT family), and IBM TotalStorage SAN Volume Controller, and the devices that they manage. The IBM TotalStorage Productivity Center for Disk is designed to:

� Help reduce storage management complexity and costs while improving data availability� Centralize management of storage devices through open standards (SMI-S)� Enhance storage administrator productivity� Improve storage resource utilization� Offer proactive management of storage devices

Using IBM TotalStorage Productivity Center for Disk to Monitor the SVC 3

Discovery of IBM storage devices that are SMI-S enabledCentralized access to storage devices information, information concerning the system attributes of connected storage devices, is available from the IBM TotalStorage Productivity Center for Disk console.

Centralized management of storage devicesThe device configuration and manager console for the SMI-S-enabled IBM storage devices can be launched from the IBM TotalStorage Productivity Center for Disk console.

Device managementIBM TotalStorage Productivity Center for Disk can provide access to single-device and cross-device configuration functionality. It enables the user to view important information about the storage devices that are discovered by IBM TotalStorage Productivity Center for Disk, examine the relationships between those devices, or change their configurations. IBM TotalStorage Productivity Center for Disk supports the discovery and logical unit number (LUN) provisioning of IBM TotalStorage DS4000 series storage systems, IBM TotalStorage SAN Volume Controller, and IBM TotalStorage ESS. The user can view essential information about the storage, view the associations of the storage to other devices, and change the storage configuration. IBM TotalStorage DS4000 and ESS, attached to the SAN or attached behind the SAN Volume Controller, can be managed by IBM TotalStorage Productivity Center for Disk. IBM TotalStorage Productivity Center for Disk and IBM TotalStorage Productivity Center for Replication also include the IBM TotalStorage Productivity Center for Fabric Bonus Kit, which supports up to 64 Fibre Channel ports. With this function, the storage administrator can view Fibre Channel as well as SMI-S enabled storage devices, automate zoning of clients, LUNs, and storage devices for the TotalStorage SAN Volume Controller environment, and drill-down from storage devices to the underlying infrastructure. If more than 64 SAN ports are required in an enterprise, you can obtain additional capacity by ordering IBM TotalStorage Productivity Center for Fabric.

Performance monitoring and managementIBM TotalStorage Productivity Center for Disk can provide performance monitoring of ESS and SVC storage devices, customization of thresholds based on your storage environment, and generation of events if thresholds are exceeded. In addition, IBM TotalStorage Productivity Center for Disk is designed to help IT administrators select the LUN for better performance. IBM TotalStorage Productivity Center for Disk enables an IT administrator to specify both when and how often the data should be collected. IBM TotalStorage Productivity Center for Disk is also designed to help support high availability or critical applications by providing customization of threshold settings and generating alerts when these thresholds are exceeded. Plus, it can provide gauges to track real-time performance. IBM TotalStorage Productivity Center for Disk is designed to enable the IT administrator to:

� Monitor performance metrics across storage subsystems from a single console� Receive timely alerts to enable event action based on customer policies� Focus on storage optimization through the identification of the best LUN

For more information, see the IBM Redbook, IBM TotalStorage Productivity Center: Getting Started, SG24-6490.

In the topics that follow, we show how IBM TotalStorage Productivity Center for Disk can be used to monitor the performance of an SVC and assist in making the correct decisions if the back-end storage systems become bottlenecked.


The test environmentAs shown in Figure 1, our environment consisted of:

� A pair of SVC nodes� SVC Master Console� One Microsoft® Windows® 2000 host, x345� One Linux® host, x345� Two Brocade 16 port switches� IBM TotalStorage Productivity Center server, x345� IBM TotalStorage Productivity Center client, x345

Figure 1 Equipment schematic

The zoning was the standard SVC zoning and is shown in Figure 2.

Note: It must be noted that the environment was deliberately set-up so that the back-end storage system would become a bottleneck. In a production environment, we would not recommend the use RAID 5 arrays consisting of only three disk drives. In addition, the workloads that were generated consisted of random reads. This workload was chosen so that the SVC cache would be bypassed, making the back-end storage a bottleneck.

SAN Volume Controller - Connections

4 ports

4 ports

DS4300

Windows 2000host

Linux host


Figure 2 SVC zoning

ScenarioWe followed this outline scenario:

1. We configured multiple RAID 5 arrays on the FAStT600 (DS4300). Each array consisted of three disk drives.

2. Multiple LUNs were created on each array so that the whole of the usable space was occupied.

3. Each of the LUNs was “imported” into the SVC as a managed disk. The LUNs for each RAID array were put into their own managed disk group (MDG).

4. Virtual disks (VDisks) were created from one of the managed disk groups and presented to the two attached hosts (Windows and Linux).

5. A workload was run on the Windows host (generated by Iometer) and a steady state condition was reached. In order to stress the back-end storage system, the workload created consisted of a high proportion (70%) of random reads, thus nullifying the effects of the SVC cache.

6. Performance gauges were created on IBM TotalStorage Productivity Center for Disk and these were used to take measurements of the steady state performance.

7. A workload was generated on the Linux host. This workload was designed so that it would stress the VDisks. Because the VDisks attached to the Linux host were created from the same MDG as the VDisks attached to the Windows host, the back-end storage array from which the MDG was created was stressed.

8. We used IBM TotalStorage Productivity Center for Disk to take performance measurements. These measurements indicated that the MDG was becoming bottlenecked.

SAN Volume Controllernode pair

SAN Volume Controller - Zoning

Windows host Linux host

Storage zone

Host zones

SVCSwitches

Storage


9. The effect of this bottleneck was to cause a drop off in performance of the Iometer workload on the Windows host.

10.Further performance measurements were taken using IBM TotalStorage Productivity Center for Disk gauges.

11.The gauges indicated that the VDisk performance on the Windows host was being affected by the bottleneck at the managed disk (MDisk) level.

12.As we knew that the performance on the Windows host was being affected by poor MDisk performance, we migrated the VDisks attached to the Windows host to another MDG.

13.The Iometer performance on the Windows host returned to its previous steady state.

14.Performance measurement showed that the input/output (I/O) bottleneck had been removed and that the system was running correctly.

DetailsThe following figures showing IBM TotalStorage Productivity Center for Disk take you through the process that we followed.

The following steps describe our process:

1. Figure 3 shows the initial IBM TotalStorage productivity Center welcome window. Click Manage Disk and Performance Replication at the top of the window.

Figure 3 Initial IBM TotalStorage Productivity Center welcome window

Note: These steps assume that you have already started collecting performance statistics from the SVC. The procedure for doing this is described in the IBM Redbook IBM TotalStorage Productivity Center: Getting Started, SG24-6490.

Important: If for some reason the Multiple Device Manager (MDM) performance statistics task is stopped, you need to abort the older process from the SVC Master Console. Log on to the SVC Master Console and go to Manage Clusters. You can then choose to stop data collection. You will then be able to restart gathering statistics using IBM TotalStorage Productivity Center for Disk.


2. Enter your password in the Multiple Device Manager (MDM) password window, as shown in Figure 4.

Figure 4 MDM password window

3. Click Next in the IBM Director Event Action Plan wizard, as shown in Figure 5.

Figure 5 Event Action Plan wizard window


4. Expand Multiple Device Manager in the Tasks pane on the right, as shown in Figure 6.

Figure 6 MDM front window

Figure 7 shows the expanded MDM tasks.

Figure 7 Multiple Device Manager expanded


5. Then, click the Gauges icon in the Tasks pane, as shown in Figure 8.

Figure 8 Gauges icon highlighted

While pressing and holding the left mouse button, move the “hand” so that it is over the SVC icon in the Group Contents pane, as shown in Figure 9.

Figure 9 “Hand” over SVC icon


A Job in Progress window opens, as shown in Figure 10.

Figure 10 Job in progress

Next, the Performance gauges window opens, as shown in Figure 11. Because at this stage no gauges have been created, the only buttons available for use are the Create and Help buttons.

Figure 11 Performance gauges window

6. Click Create, and the Create performance gauge window opens, as shown in Figure 12. It is from this window that you can create the gauges that you need. The type of gauges that you can create depends on what you are monitoring, VDisks or MDisks:

– Virtual disk (VDisk): For a single Vdisk or all Vdisks combined

• Total and average number of reads, writes• Number of 512-bytes blocks read, writes

– Managed disk (MDisk): For a single MDisk or per MDisk group

• Total and average number of reads, writes• Number of 512-bytes block read, writes• Read and write transfer rates• Total, average, minimum, maximum response time

7. In Figure 12, we selected All VDisks and the Total I/O’s/sec. This will be measured on MDG1, DS4300-RAID5_1. You have some more choices to make:

– Name of the gauge.– Whether to select the number of data points to show or the date range. We selected

the last 5 data points.– Whether to display the gauge now or store it for late use. We display the gauge now.

After making your selections, click OK, and your gauge will be created.


Figure 12 Create performance gauge window

We made an error with the name of our gauge, VDisk total I/O / sec, because blanks are not allowed in a gauge name. Therefore, the error message shown in Figure 13 appears.

Figure 13 Error message when blanks are used in a gauge name


We clicked OK, and the Create performance gauge window opens again, this time with all our entries, as shown in Figure 14.

Figure 14 Create group VDisk performance gauge


We corrected the gauge name, replacing the blanks with underscores (_) so that the name was VDisk_total_I/O_/_sec. We then clicked OK. The gauge was created and displayed, as shown in Figure 15. As you can see, the gauge shows the I/O ramping up and reaching a steady state. The only I/O activity taking place was that generated by Iometer on the Windows host, and this gives us our baseline figure.

Figure 15 VDisk I/O per second gauge


8. We then created a gauge for the single VDisk that the Windows host was using. Figure 16 shows the creation process. We used similar parameters as for the gauge created in Figure 14 on page 13, the difference being that now we chose a single VDisk, barrytest1, rather than all VDisks.

Figure 16 Single VDisk gauge create window


Because the I/O was only taking place to a single VDisk in the I/O group, the results displayed in Figure 17 are the same as for the group that we saw in Figure 15 on page 14.

Figure 17 Single VDisk I/O gauge for Windows host VDisk


9. We then created a single VDisk gauge for the Linux host and started the workload on this host. Figure 18 shows how the I/O increased on the Linux host’s VDisk.

Figure 18 Linux host VDisk I/O gauge


As we describe in “Scenario” on page 6, we deliberately created the VDisks used by the Windows and Linux hosts so that contention would occur on the storage system RAID array, which would affect I/O performance. The effect of this is clearly shown in Figure 19, which is a graph of the I/O performance of the Windows host’s VDisk. As the Linux workload increases, the MDisks that support the VDisks become staturated. The result of this is that I/O performance of the Windows host dramatically decreases.

Figure 19 Windows host’s VDisk gauge


Although the I/O rate of the Windows host decreased, the overall I/O on the back-end storage system increased. Figure 20 shows this in a gauge of the MDG from which the VDisks were created.

Figure 20 MDG I/O rate


10.We then created a gauge to measure the I/O performance of the second MDG. Figure 21 shows this process.

Figure 21 MDG 2 I/O performance gauge creation


Because no I/O was taking place on this MDG, the resulting gauge, shown in Figure 22, is rather flat.

Figure 22 MDG 2 I/O performance: No VDisks allocated

At this stage, we analyzed the results of our findings so far. We came to the conclusion that the drop off in performance of our steady state Windows workload, our main production server, was caused by the additional load on the back-end storage system caused by the Linux host. In a non-virtualized world, we would be faced with the difficult and time-consuming task of stopping production, copying the data to a different set of disks, and then resuming production. Here, we have the benefit of virtualized storage. Using the SVC migrate VDisk functionality, we migrated the offending VDisk attached to the Linux host to our second, currently inactive MDG, DS-4300_RAID5_2.

As we migrated the VDisk, we monitored the performance of the VDisk attached to the Windows host. Figure 23 shows the results.


Figure 23 Windows VDisk performance during data migration

SummaryThis paper shows how IBM TotalStorage Productivity Center for Disk can be used to monitor VDisk and MDisk performance on the SVC. If VDisk performance analysis is detected, the supporting MDG can be easily analyzed to determine the root cause. After you determine the offending storage components, you can use the VDisk migration functions of the SVC to non-disruptively correct the problem.

The team that wrote this RedpaperThis Redpaper was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center.

Barry Mellish is a certified I/T Specialist and is currently working as a Storage Software and Virtualization Specialist in North Region, EMEA. Previous to this, he was a Project Leader at the International Technical Support Organization (ITSO), San Jose Center, for four years. He has coauthored 15 Redbooks™ and has taught many classes worldwide about storage subsystems. He joined IBM United Kingdom 20 years ago. Before joining the ITSO, he worked as a Senior Storage Specialist on the Disk Expert Team in EMEA.

Rob Nicholson is based in Hursley, United Kingdom, and is an IBM Senior Technical Staff Member. He currently working on the software architecture for SAN Volume Controller and has had various roles in development and testing for IBM storage products for more than 13 years.


Steve Hurley is currently Program Manager for open systems disaster recovery software working in Tucson, Arizona. Prior to this role, he worked from Hursley, England, designing storage virtualization solutions for customers throughout Europe.

Thanks to the following people for their contributions to this project:

Elizabeth BarnesEditor, International Technical Support Organization, Austin CenterEmma JacobsGraphics Designer, International Technical Support Organization, San Jose Center



Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2005. All rights reserved. 25

Send us your comments in one of the following ways:� Use the online Contact us review redbook form found at:

ibm.com/redbooks� Send your comments in an email to:

[email protected]� Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. QXXE Building 80-E2650 Harry RoadSan Jose, California 95120-6099 U.S.A.

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Enterprise Storage Server®FlashCopy®ibm.com®

IBM®Redbooks (logo) ™Redbooks™

Tivoli®TotalStorage®

The following terms are trademarks of other companies:

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

®



http://www.ibm.com/redbooks/

http://www.ibm.com/redbooks/

http://www.redbooks.ibm.com/contacts.html