CONNECTING THE SCATTERED PIECES FOR STORAGE REPORTING · reporting should be made available to show the interrelationships from server to SAN switches, ... A database connection,

CONNECTING THE SCATTERED PIECES FOR STORAGE REPORTING

Hok Pui Chan Practice Consultant Consulting, EMC Hong Kong [email protected]

mailto:[email protected]

2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Introduction ................................................................................................................................ 4

Analyzing the Components and Inter-relationships of a Storage Infrastructure........................... 6

Extracting and Transforming the Storage Configuration Data ..................................................... 8

VMAX ..................................................................................................................................... 8

EMC ControlCenter ............................................................................................................. 8

EMC Solutions Enabler CLI ................................................................................................ 9

SAN Switches/Fabrics ...........................................................................................................10

Host Information ....................................................................................................................10

Summary ...............................................................................................................................11

Building the Storage Configuration Repository ..........................................................................12

A Simple Solution ..................................................................................................................12

A Scalable Solution ...............................................................................................................13

Automation of Data Source Collections ..............................................................................14

Extract Transform & Load Process.....................................................................................14

Database ...........................................................................................................................14

OLAP Analysis ...................................................................................................................15

Web Page ..........................................................................................................................15

Static Report ......................................................................................................................16

Real Life Use Cases of the Storage Configuration Repository ..................................................17

Use Case #1 – Capacity Trending for Storage Array .............................................................17

Use Case #2 – Capacity Used by Servers .............................................................................17

Use Case #3 – Storage/Fabric Migration and Data Center Relocation...................................18

Use Case #4 – Highlighting the Exceptions ...........................................................................18

Practical Considerations ...........................................................................................................19

Skill Set .................................................................................................................................19


Difficulty Acquiring Server Resources ....................................................................................19

Difficulty Acquiring Software Licenses ...................................................................................19

On-going Maintenance ..........................................................................................................19

Setting Up Data Import for New Storage Equipment ..............................................................20

Future Extensions .....................................................................................................................20

Include Performance Data into the Repository .......................................................................20

Correlate Other Non-Storage Related Data ...........................................................................21

Conclusion ................................................................................................................................21

Disclaimer: The views, processes or methodologies published in this article are those of the

author. They do not necessarily reflect EMC Corporation’s views, processes or methodologies.


Introduction

Every organization has its own unique needs and there are a lot of different types (and even

different brands) of storage equipment to be managed. In managing a sizable storage

infrastructure, an accurate, timely, and comprehensive storage reporting mechanism is

important to measure the performance and utilization of the storage infrastructure. Storage

reporting is also crucial for measuring the business value of storage equipment investments by

associating storage costs to different application and business units.

At present, many companies rely on native tools (e.g. EMC Unisphere®, EMC ControlCenter®,

Cisco DCNM, etc.) to manage the storage infrastructure and extract information for reporting.

These tools have done a very good job of helping storage administrators manage and provide

reporting functionalities for that particular type of equipment. However, these tools do not talk to

each other – the tools for managing fabric only provide fabric/SAN switch-related information

while the tools for storage array only provide array-specific information. Sometimes, answers to

common, simple questions cannot be easily produced by these tools, e.g. “What does the end-

to-end connectivity for this server look like?”, “When this server is retired, how much storage

resources can be freed up and re-purposed?” A holistic, interconnected dimension of storage

reporting should be made available to show the interrelationships from server to SAN switches,

SAN switches to storage array, and storage array to storage volumes provisioned.

Moreover, producing a meaningful report to senior IT management requires co-relating

company-specific information into storage reports. For example, to justify storage infrastructure

investment, an IT manager might want to know how much storage resource is being used by a

particular project, application, or line of business. Since most storage reporting tools do not

provide functionality to include customized data inputs, storage administrators struggle to

produce these reports manually at regular intervals while trying to ensure the accuracy and

timeliness of such reports.

So, as the title suggests, all the essential elements for a comprehensive storage report are

available but scattered around different places or tools. What is needed is to find a way to

connect them and provide different dimensions for storage reporting, e.g. from the project/

business line perspective, from the server/application perspective and, of course, from the

traditional storage array perspective. To produce these co-related reports, a scalable and robust


Storage Configuration Repository is required as the foundation to all of the above. This article

will focus on how a Storage Configuration Repository can be built.

EMC Symmetrix® VMAX® SAN infrastructure will be used to illustrate the technical details of

extracting the source data, processing them, storing them in the Storage Configuration

Repository, and how the reports are produced. Technical details include, “What and how can

VMAX information be extracted?”, “How can this information be transformed and imported into

the configuration repository?”, etc. Other practical considerations and future extensions will also

be discussed.


Analyzing the Components and Inter-relationships of a Storage Infrastructure

Server, SAN switches, and storage arrays are the essential components of a typical storage

infrastructure. These components are physically connected together.

Taking a deeper look, we find that all the control and configurations of the storage infrastructure

is actually built around one element – the World Wide Name (WWN). In simple words, the WWN

is a global identifier of storage Fibre Channel ports (for storage array front-end ports and server

Host Bus Adapter {HBA ports}) and storage devices. WWN is used to control which ports can be

interconnected with each other and which storage devices can be accessed by these storage

ports.

If we can gather the WWN relationship in a storage infrastructure, we can co-relate and present

a logical view of the whole storage infrastructure. For example, if we can identify the WWN of a

server HBA, the zoning configuration on the SAN switches, and the LUN masking information

on the array, we can easily know how much storage a server is using as well as the

corresponding logical connection. When building the Storage Configuration Repository for

showing the end-to-end relationships, we can focus on gathering information around the WWN

and make things co-related. The following diagram illustrates how the WWN connects each

storage infrastructure component.


In the following sections, we will discuss what information we should extract from the storage

infrastructure components to build an end-to-end relationship.

HBA WWN

Storage Devices

LUNMasking

Storage Ports

LUNMasking Switch Port

FabricLogin

SAN Switch

Production Server

Remote Storage Devices

ReplicationRelationships

HBA WWN

DR Server

LUNMasking


Extracting and Transforming the Storage Configuration Data

In this section, we discuss the information required to be collected from the storage

infrastructure components to:

1. Connect the interrelationships of the components

2. Store necessary configurations for reporting

A VMAX SAN infrastructure is used for illustration.

VMAX

For VMAX, there are two commonly used data sources:

EMC ControlCenter (ECC)

EMC Solutions Enabler CLI (SE)

Information to be collected:

Masking information

Device information

SRDF® relationship

FA port information

Pool information

FAST™ VP information

EMC ControlCenter

A component in EMC ControlCenter (ECC)—StorageScope—has its own Oracle database as

the data repository. It provides database views for accessing the data. We can use the views as

the data sources for our purposes. A database connection, e.g. ODBC, JDBC, etc., is needed in

order to access the database using SQL queries to those views. After that, the data can be

stored and manipulated further. For details about accessing these database views and the data

available, please refer to StorageScope documentation.

This method became popular few years ago as most Symmetrix environments had ECC

installed and the information provided by the views is quite comprehensive. However, other

products like Unisphere for VMAX and Symmetrix Management Console (SMC) are starting to

replace ECC as the most popular management tool for Symmetrix environments. Hence, we are


not drilling into the details for getting the required information through ECC StorageScope in this

article.

EMC Solutions Enabler CLI

In almost all Symmetrix environments, Solutions Enabler CLI will be installed. It is a good tool

for managing the Symmetrix, especially when repeated commands and scripting are required.

For collecting storage configurations, we need to execute a set of different symcli commands

and parse the text output.

There is an excellent function in Solutions Enabler CLI which can specify that the command

output results be in XML format by adding the “-output xml_element” option to the symcli

command. Using XML outputs can make automated data importing much easier.

Here are some sample commands that are useful to extract VMAX configuration data in XML

format:

Device information

o symdev –sid <sid> list -v -output xml_element > symdev.xml

SRDF relationship

o symrdf -sid <sid> list -v -output xml_element > symrdf.xml

Pool information

o symcfg -sid <sid> list -pool -thin -mb -output xml_element >

pool.xml

Masking information

o symaccess -sid <sid> list view -v -detail -output xml_element >

masking.xml

Device and Pool relationships

o symcfg -sid <sid> list -tdev -detail -output xml_element >

tdev.xml

We will use this approach for extracting configuration data from a Symmetrix VMAX.


SAN Switches/Fabrics

SAN switches and fabrics are the core connectivity component within a storage infrastructure,

providing physical connectivity (through physical ports) and logical connectivity (through zone

and zoneset configurations).

Configurations to be collected to build the end-to-end relationship:

Fabric logins – the relationship between WWN and SAN Switch port

Zones and Zonesets – the relationship between HBA and storage ports

SAN switches usually provide a command line interface and command outputs can be collected

for further processing. Here are some examples:

Fabric Logins:

o show flogi database

Zone information

o show zoneset active

Some storage administrators might think that it is relatively less important to collect SAN and

Fabric information. The configuration information they usually need for reporting is just port

utilization and zone information. This information will not change frequently and can easily

tracked manually.

Host Information

Host information includes host-related attributes like OS and Host model. The most important

information in this discussion is the HBA WWN(s) of a host. We need to have “Host Name to

HBA WWN” mappings in order to link up all other storage infrastructure components.

One of the methods is to rely on tools. Most EMC storage administrators should be familiar with

the emcgrab/emcreport tools to collect host information for a configuration snapshot or

troubleshooting. EMC also provide a nice web portal called E-Lab Advisor (ELA) to accept

emcgrab/emcreport output file uploads. In this web portal, a user can see an inventory of hosts

which emcgrab/emcreport output files have uploaded. A user can then select the hosts and

generate a SAN Summary report in Excel format. The SAN Summary report presents a list of

hosts in tabular form with WWN(s) as one of the columns.


Some organizations require internal processes in order to get approval to run

emcgrab/emcreport. Consequently, storage administrators often try to avoid using

emcgrab/emcreport unless they are really necessary, e.g. for troubleshooting issues. Some

storage administrators prefer to manually keep track on this mapping; they need to collect this

information anyway for storage provisioning and zoning.

Another possibility is to use configuration management tools like MS SCCM.

Summary

Below is a summary of information to be collected and the suggested frequency of data

collection. (We will discuss “Simple Solution” and “Scalable Solution” in the next section.)

Information to

be Collected

Data Source Data Collection

Frequency

(Simple Solution)

Data Collection

Frequency

(Scalable Solution)

VMAX configs Solution Enabler CLI output in

XML format

Weekly Daily

Switch/SAN fabric CLI command outputs Weekly Daily

Host Information Manual mapping input/SAN

Summary

Ad-hoc Ad-hoc


Building the Storage Configuration Repository

After discussing some details about the data to be collected to co-relate storage infrastructure

components, this section is going to discuss about how to make use of this data.

We will discuss two alternatives. One is simpler but with less functionality while the other is

more comprehensive and can provide more opportunities for future extensions.

A Simple Solution

This solution focuses only on storage array configurations (not much correlation with other

storage infrastructure components) and is only semi-automated. It requires some manual tasks

and provides limited functionalities. However, storage administrators might find it very useful to

keep track of storage configuration changes and produce regular storage utilization reports. This

is especially true for a storage environment having less than five arrays.

In the last section, we discussed how to collect VMAX configuration data in XML files. Now, we

can use XSL transformation to extract and transform these XML files into simpler formats such

as CSV and HTML which can be imported to Excel or directly used for reporting.


The diagram below illustrates such transformation using the XML command output of:

“symtier list –v -output xml_element > symtier.xml”.

After collecting output for several weeks, a report of the tier utilization trend can be easily made.

Some open source XML editors can provide XML/XSL editing and transformation tools.

A Scalable Solution

A more scalable option is recommended for storage infrastructures having more than five arrays.

We can build a web application with a database to store historical storage configuration. When

comparing to the simple solution discussed above, this option is more robust and automated for

data collection and reporting. The diagram below illustrates the high level architecture of this

option of the Storage Configuration Repository.

Symcli XML output XSL File

CSV output

Transform


Automation of Data Source Collections

The data collection process, except the manual input of Host and WWN mapping, can be

automated using scheduled jobs to execute the commands daily. After that, the output files will

be transferred to a shared location, e.g. a share drive, for further processing.

Extract Transform & Load Process

Simple programs can be developed to parse and import the file contents into the database.

Some tools like MS SQL Server Integration Services can be used to design and implement the

Extract Transform & Load (ETL) process easily.

Database

The database is the core component of the storage configuration repository. It stores the most

recent, as well as historical data, for reporting. The database structure reflects the relationship

between storage infrastructure components and configurations. A simplified version of the

Entity-Relationship diagram of the database is shown below.

Solution Enabler

Database

XML Files

Storage Administrator

Host-WWNMapping

SAN Switch

Command Output

Web Page

Static Report

Extract, Transform &

Load (ETL)

OLAP Analysis


OLAP Analysis

Once the database is constructed, online analytical processing (OLAP) cubes can be designed

to facilitate an interactive, responsive, and user friendly access of data inside the database. For

example, MS SQL Server Analysis Service is a popular tool to build OLAP cubes and MS Excel

pivot tables can provide a familiar interface for accessing information in the OLAP cubes.

However, building OLAP cubes require special knowledge and may not be easy.

Web Page

It is very common to use a web interface to access a database. Everyone is familiar with web

browsers and web pages are not very difficult to build. With a web interface, information in the


database can be shared easily. A web page can accept a user’s input parameters, e.g.

hostname, as query or filtering criteria.

Static Report

Sometimes, a storage administrator may need regular reports on the storage infrastructure.

The formats of these reports are not expected to change frequently. Hence, report templates

can be constructed and reports can be generated regularly to a shared location or even

published to a web portal. Here are some examples of static reports:

Storage pool utilization report with history

SAN Switch port utilization report

Hosts with highest storage capacity subscriptions

Fan-out ratio of storage front-end ports


Real Life Use Cases of the Storage Configuration Repository

This section will discuss how the Storage Configuration Repository helps in real life situations.

Use Case #1 – Capacity Trending for Storage Array

Today, storage pools with thin provisioning are usually configured on newly acquired arrays. It is

important to keep track of pool utilization and initiate the disk upgrade procurement process in

time. Storage administrators are usually required to look at capacity statistics for individual array,

arrays in a site, and all the arrays in all the sites. The Storage Configuration Repository can

provide capacity usage history and a capacity report with multiple dimensions can be generated

regularly.

Use Case #2 – Capacity Used by Servers

In real life, storage administrators need to have statistics about the capacity used by individual

or a group of servers. Storage administrators not only want to know the capacity used in the

production site but also in the DR site. From the interrelationship that we built in the repository,

we can easily identify the storage devices being used by the production servers and the DR

servers through the device replication relationships.

However, there are two major considerations here. The first is to cater for cluster servers. If we

only aggregate the storage capacity used by a list of servers, the shared devices used by

clusters could be double counted. Consider the following scenario:

A 2-node cluster

Using a shared disk 50GB in size

Node#1 and Node#2 both “see” the 50GB disk

If we just aggregate the storage capacity used by Node#1 and Node#2, the total capacity will

become 100GB instead of the real usage of 50GB. Ideally, the reporting/query facility should be

able to “de-dupe” these capacities.

Another consideration is applications that do not use array-based replication for DR, e.g. some

databases that use native data replication. In such cases, we cannot get the hostname and the

corresponding capacity used in the DR site from the repository. We need to either store these


relationships in the repository or include DR hostnames in the query/reporting and make use of

the “de-dupe” mechanism.

Use Case #3 – Storage/Fabric Migration and Data Center Relocation

In a typical storage and/or fabric migration or Data Center Relocation project, the storage

infrastructure will be very dynamic. It is extremely difficult and error-prone to manually track

inventory and migration status, especially if it is a large scale migration involving multiple arrays,

fabrics, and sites. Also, the migration project team may need to make a lot of migration

priority/scheduling decisions based on some “what-if” analysis results.

During the planning phase of a migration, we can rely on the query/reporting functionalities to

provide immediate results for decisions. As data will be refreshed regularly, the repository can

provide the most updated configurations snapshot for reporting.

When executing the migration, storage administrators can rely on the repository to provide

detailed storage configurations. For example, the repository contains information that can be

used to list the devices, the local and remote replication relationships, related cluster nodes, and

DR hostnames in a migration event. It is particularly useful for some complex storage

environments where a host is using multiple arrays with some replicated and some non-

replicated storage devices.

The migration progress can also be tracked. Based on the most updated configurations,

statistics of the host inventory on both the new and old environment can be easily extracted and

presented in migration status dashboards.

Use Case #4 – Highlighting the Exceptions

Mis-configurations or non-standard configurations can be highlighted by extracting configuration

data using pre-defined criteria. For example, it can be a mis-configuration if a storage device

does not have a remote replication relationship while the other devices belonging to the same

host do. Another example is to check whether the LUN masking configurations align to the

zoning configurations – a missing zone that makes a LUN masking ineffective can be

highlighted.


Practical Considerations

If you decide to build your own Storage Configuration Repository, look at the practical

considerations discussed in this section.

Skill Set

Storage administrators are usually not experts in application development. Building the Storage

Configuration Repository requires programming, web framework, and database design skills.

Some organizations have chosen to outsource this initiative to external professional services.

Difficulty Acquiring Server Resources

Demand for a Storage Configuration Repository is usually initiated from the storage team.

However, the storage team does not have the control over server provisioning. In some

organizations, the storage team may need to provide a lot of justifications and go through a

lengthy approval process to get a server (either a physical server or VM). Some storage teams

have given up getting a server grade environment; instead, they just use some old PCs to run

their repository.

Difficulty Acquiring Software Licenses

Usually, the software used by the storage team is acquired together with the storage hardware.

To build the Storage Configuration Repository, a database license, e.g. MS SQL Server license,

may need to be acquired. However, there is usually no precedent for the storage team to

acquire such licenses. There could be no mechanism or be very difficult to chargeback the

licensing cost to the storage team.

To avoid these “troubles”, storage teams have decided to use open source software and

platforms. For example, some choose Postgres database, JAVA, and Tomcat.

On-going Maintenance

Even when construction of the Storage Configuration Repository is outsourced to professional

services, the storage team remains responsible for the on-going maintenance and

troubleshooting. Maintenance tasks could include backup, performance monitoring, and

ensuring the success of data importing jobs. As well, some issues could occur occasionally that

require troubleshooting. For example, a scheduled job for extracting and importing storage

configurations could suddenly stop working for no apparent reason.


Setting Up Data Import for New Storage Equipment

During initial setup of the Storage Configuration Repository, data import and processing were

configured for every piece of existing storage equipment. However, after a few years, new

storage equipment will be acquired to replace the old. Even if the repository was built in-house,

there is no guarantee that anyone will remember how the data import for storage equipment can

be set up after a few years’ time. So, it is very important to make sure every procedure is well

documented.

Future Extensions

After building the Storage Configuration Repository as a foundation, we can continue to

enhance the repository. In this section, we discuss some possible enhancements that can be

made to provide extra value.

Include Performance Data into the Repository

In the above discussion, we focused on importing configuration data. However, it is also

possible to import storage performance data.

At present, a question like, “What is the 95th-prcentile of IOPS that the server produced?” is

difficult to answer as we are not talking about the 95th-percentile of each individual device.

Rather, we need to look at the performance data for the whole server. We cannot just calculate

the 95th-percentile for each storage device and sum them to get the result. Instead, the following

steps are needed:

Look at which storage devices the server is using and extract the corresponding

performance data (IOPS)

Aggregate the IOPS values and get the total IOPS generated by the server at each time

interval (i.e. every 5 minutes)

Use statistical functions to calculate the 95th-percentile of the aggregated IOPS values

It is impossible to do this manually. However, after importing the performance data into the

repository, reports can easily be made to answer questions like this.

Storage administration tools like Unisphere for VMAX can schedule performance data exports.


Correlate Other Non-Storage Related Data

In storage reporting, we may need to summarize storage use for an individual business line,

project, or application. This information is useful for chargeback and management reporting as

well. Non-storage related information can also be included into the repository. Servers that a

particular application is using can be specified and a report generated from the application’s

perspective.

During a storage migration project, migration-related information—e.g. date for migration—can

be included into the repository. For example, total capacity to be migrated can be easily

obtained when there is a change to the migration schedule, i.e. when some servers are re-

assigned to another migration event. This information is extremely useful for providing quick

answers to migration planning.

For manual input, we may need to set up additional data sources for data import, e.g. manual

construction of CSV files containing the mapping between projects and servers. Another way is

to provide a web interface to facilitate editing of these manual inputs and save them to the

repository.

Conclusion

This article discussed the value of having a Storage Configuration Repository and how to

establish it as practice. After reading the article, the audience should be able to get an idea—

along with some technical details—on how they can build their own storage configuration

repository and reporting facilities.


EMC believes the information in this publication is accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION

MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO

THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an

applicable software license.

Documents

CONNECTING THE SCATTERED PIECES FOR STORAGE REPORTING · reporting should be made available to show the interrelationships from server to SAN switches, ... A database connection,