26
EMC Optimized Infrastructure for Smart Metering Best Practice Guide In Partnership with Itron EMC Information Infrastructure Solutions Abstract This white paper provides a summary of how EMC and Itron partnered to increase performance of Itron’s Meter Data Management (MDM) application on EMC storage. This white paper summarizes the testing of three different areas — an EMC ® CLARiiON ® CX4 Model 960 storage system, an Oracle 10gR2 database, and the HP-UX operating system — to achieve a goal of 25,000 meter reads per second. February 2011

EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

Embed Size (px)

Citation preview

Page 1: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering

Best Practice Guide

In Partnership with Itron

EMC Information Infrastructure Solutions

Abstract

This white paper provides a summary of how EMC and Itron partnered to increase performance of Itron’s Meter Data Management (MDM) application on EMC storage. This white paper summarizes the testing of three different areas — an EMC® CLARiiON® CX4 Model 960 storage system, an Oracle 10gR2 database, and the HP-UX operating system — to achieve a goal of 25,000 meter reads per second.

February 2011

Page 2: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 2

Copyright © 2010, 2011 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part Number h7148.1

Page 3: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

3

Table of Contents

Executive summary ........................................................................................................................... 5

The business challenge .................................................................................................................... 6

Introduction........................................................................................................................................ 7 EMC .............................................................................................................................................. 7 Itron ............................................................................................................................................... 7

Technology overview ........................................................................................................................ 8 Smart metering .............................................................................................................................. 8 Meter Data Management .............................................................................................................. 9 EMC and Itron Technology ......................................................................................................... 10

Detailed architecture ....................................................................................................................... 11 Overall configuration ................................................................................................................... 11 High volume import architecture ................................................................................................. 12 MDM physical architecture .......................................................................................................... 12 Oracle database file placement ................................................................................................... 13

Itron/EMC benchmark goals and tools used ................................................................................... 15 The goal ...................................................................................................................................... 15 Tools used ................................................................................................................................... 15

Baseline database results ............................................................................................................... 17

Performance changes ..................................................................................................................... 20 Improving data movement ........................................................................................................... 20 Reducing wait times .................................................................................................................... 20 Final storage configuration .......................................................................................................... 21 Best practices and the payoff ...................................................................................................... 23

Conclusion....................................................................................................................................... 25

References ...................................................................................................................................... 26

Page 4: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 4

Figures Figure 1 Example storage requirements for AMI __________________________________________________ 6 Figure 2 “AMI Stack” __________________________________________________________________________ 8 Figure 3 Flow of meter data to advanced applications _____________________________________________ 9 Figure 4 Itron hardware architecture ____________________________________________________________ 11 Figure 5 MDM physical architecture ____________________________________________________________ 12 Figure 6 MDM baseline performance ___________________________________________________________ 15 Figure 7 Baseline Top 5 Database Wait Events __________________________________________________ 18 Figure 8 IEE database LUNs __________________________________________________________________ 22

Tables Table 1 IEE Database Server ................................................................................................................................. 13 Table 2 Logical Unit (LUN) 04: DATA .................................................................................................................... 14 Table 3 LUN 05: INDEX ........................................................................................................................................... 14 Table 4 Tablespace waits for physical reads ........................................................................................................ 21 Table 5 IEE file system to LUN mapping ............................................................................................................... 23

Page 5: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

5

Executive summary

Utility companies are preparing for an explosion in new data as a result of Advanced

Metering Infrastructure (AMI). As utilities move from monthly meter reads to 15-minute intervals, the amount of data will increase by almost 3,000 times!

Utilities are asking: Where will we store all of this data and how will we manage it?

EMC and Itron provide the components necessary for a high-performing AMI system. Itron provides the meters, the collection system, applications, and database. EMC provides the scalable, reliable, and performant storage where the database will reside and grow over time. In one particular benchmark that Itron and EMC conducted, the goal was to reach 25,000 meter readings per second import from Itron’s OpenWay® Collection Engine head-end system to the Itron Enterprise EditionTM (IEE) Meter Data Management system (MDMS), which is running on the EMC® CLARiiON® CX4 Model 960 storage system.

Itron and EMC successfully met the benchmark goals, but perhaps more importantly, also learned some lessons along the way that can be leveraged no matter the utility customer. This paper will offer a guide detailing those best practices.

Utility companies – from the smallest to those scaling upwards of 5 million meters – now can expect their mission-critical Itron IEE MDMS running on EMC storage to provide the performance they require to successfully collect a significant increase in data, which will be used to improve current methods of operation and enable new initiatives.

Page 6: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 6

The business challenge

Utility companies are preparing for an explosion in new data as a result of Advanced

Metering Infrastructure (AMI) projects. Many utilities will be moving from monthly meter reads to hourly and even15-minute intervals.

Figure 1 Example storage requirements for AMI

AMI can produce a staggering ~3,000-fold increase in data. And this does not even begin to include the rush of data that will stream in from new network-connected devices including synchrophasors and other intelligent equipment.

Utilities are asking: Where will we store all of this data and how will we manage it?

The challenge is not just one of data throughput, but more importantly, one of management. Utilities will need a new architecture to handle the vast increase in data as a result of smart metering.

Page 7: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

7

Introduction

EMC EMC Corporation is the world's leading developer and provider of information

infrastructure technology and solutions that enable organizations of all sizes to transform the way they compete and create value from their information. EMC has served Investor-Owned Utilities, Municipalities, and Cooperatives for over 25 years with solutions that address their information management challenges. More information about EMC's products and services can be found at www.EMC.com/smartgrid.

Itron

Itron is dedicated to delivering end-to-end smart grid and smart distribution solutions to electric, gas, and water utilities around the globe. Itron is the world’s leading provider of smart metering, data collection, and utility software systems, with nearly 8,000 utilities worldwide relying on its technology to optimize the delivery and use of energy and water. Itron’s offerings include electricity, gas, water and heat meters; network communication technology; collection systems and related software applications; and professional services.

Page 8: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 8

Technology overview

Smart metering

Today, meters are mostly analog devices installed on customer premises to measure electric, gas, or water usage and in turn, to bill customers based on those measurements.

“Smart metering” is the transformation to a system of two-way communications over advanced networks, enabling a closer relationship between utility and customer and new programs like demand response and advanced customer portals.

By many accounts, smart metering is considered the gateway to the smart grid.

According to the Electric Power Research Institute (EPRI), smart metering – which is also commonly referred to as Advanced Metering Infrastructure (AMI) – “is comprised of state-of-the-art electronic / digital hardware and software, which combine interval data measurement with continuously available remote communications. These systems enable measurement of detailed, time-based information and frequent collection and transmittal of such information to various parties.”

EPRI continues by saying: “AMI typically refers to the full measurement and collection system that includes meters at the customer side, communication networks between the customer and a service provider…and data reception and management systems that make the information available to the service provider.”i

EMC refers to this system as the “AMI Stack.”

Figure 2 “AMI Stack”

Page 9: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

9

Meter Data Management

The cornerstone of an AMI system is the Meter Data Management System (MDMS).

IDC Energy Insights defines MDMS as follows:

“The MDM is an application for processing meter data collected from any customer type through varying communication methods from many different meter systems. It rationalizes, cleans, and manages data to establish a ‘system of record’ of meter data, which can then be securely used in a variety of billing, analysis, and operational applications.”ii

The most common leverage points for the MDMS include billing and customer portal applications.

Figure 3 Flow of meter data to advanced applications All utilities involved with meter data today have a billing application and most have a customer-facing web portal. The flow of meter data through an MDMS allows for new and advanced functions:

1. Billing applications today allow for standard, monthly non-itemized bills. Increases in meter data and the management of that data via the MDMS will enable new billing models, like Time-of-Use (TOU) and Critical Peak Pricing (CPP).iii

2. Customer portals today are typically unattractive, not user-friendly, and only provide detail at monthly intervals. Advances in portals will enable demand response, which will be an interactive way for utilities to curb demand – in response to a growing supply deficit – and for customers to reduce reliance on paper bills and protect the environment.

iv

Once the data travels from the meters through the collection system – Itron’s OpenWay Collection Engine – to the meter data management application for pre-processing, including validation and estimation of the data, it is placed in the database.

Both the OpenWay database and the IEE MDMS database are housed on EMC storage. Although this paper focuses on the best practices running the IEE MDMS database on EMC storage, we have impressive OpenWay test results that will be released in a forthcoming paper.

Page 10: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 10

EMC and Itron Technology

Itron and EMC provide the components necessary for a high-performing AMI system.

OpenWay CENTRON® Meter is a truly smart device used to collect, process, and transmit vital energy information to utility systems. Rather than simply inserting a network communication card into a standard meter, Itron developed an advanced meter where usage data and other advanced calculations are made within the meter itself, allowing utilities to leverage time-based rates, demand response, home networking, and many other smart grid applications.v

OpenWay Collection Engine provides the interface between the metering system and utility processes such as meter data management, billing, outage management, and load control. Scheduled or interactive remote reading, data normalization, event scheduling, automatic processing, and more are all available through the collection engine.

vi

Itron Enterprise EditionTM (IEE) Meter Data Management system (MDMS) is an enterprise-wide data management solution for interval, register, and event data for residential and commercial and industrial (C&I) customers. It is a scalable, open-architecture system that manages data from many different collection systems. It also provides secure, accurate, reliable data to a wide array of utility billing and analysis systems.”

vii

1. “Pre-Process” = the MDM application servers that receive the data from collection engines and perform additional functions, including: extensive validation, estimation, and editing (VEE); advanced time-of-use, aggregation and calculation services; request brokering among multiple collection systems (legacy or AMR systems); sophisticated export management capability; and auditable change tracking.

The MDM is broken down into two components, as represented in Figure 3 above:

viii

2. “Database” = the standard Relational Database Management system, in this case Oracle (SQL Server is also supported) that is the back end of the MDM, used for long-term storage of register, interval, tamper, and outage and meter event data in fully versioned form. It provides a central repository for integration, access by all business and analytical systems, and users of meter data throughout the utility.

ix

EMC CLARiiON CX4-960 storage system combines five 9s availability with innovative technologies like Fully Automated Storage Tiering (FAST), Flash drives, Virtual Provisioning™, 64-bit operating system, and multi-core processors. The CX4-960 scales up to 1.9 petabytes (PB) to handle future growth and features UltraFlex™ technology, providing multiple protocol options and online-expandable connectivity.

Page 11: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

11

Detailed architecture

Overall configuration

Raleigh, North Carolina is home to Itron’s Research and Development (R&D) facilities for software applications, like OpenWay and IEE MDMS. It is also the location of Itron’s high-scale test lab, where Itron validates and optimizes its systems for the largest AMI metering deployments.

The physical hardware architecture in the Raleigh lab is pictured below, and includes the various application servers and databases, such as OpenWay and IEE, described above. It also includes the storage area network (SAN) and the connections between the application and database servers to EMC storage.

The overall layout appears here for reference, although the paper will focus on the specific architectural considerations of the IEE database and EMC storage.

Figure 4 Itron hardware architecture

Page 12: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 12

High volume import architecture

EMC and Itron developed an architecture to import, validate, and estimate high volumes of meter data into IEE.

The three primary components of the high volume architecture include:

CLARiiON CX4 Model 960 SAN from EMC

Oracle 10gR2 database

HP-UX operating system

Itron uses a multi-tier architecture to process the many different data sources and place them into the database. The Validation, Estimation, and Editing (VEE) servers scrub the data and send it to the database server as external tables (files). The database server then puts these external tables into internal database tables. This architecture makes IEE highly scalable, meeting the needs for managing large volumes of meter data.

MDM physical architecture

The IEE server is UNIX-based, running the HP-UX operating system. This server is connected to the EMC CX4-960 storage system via four FC links. The CLARiiON storage system contains 460 Fibre Channel drives.

Figure 5 MDM physical architecture

Page 13: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

13

More detail on the database server:

Table 1 IEE Database Server IEE Database Server

Processors 8 dual-core 1.6 GHz Itanium processors (16 cores total)

Memory 128 GB

HBAs Four at 4 Gb each

Operating System HP-UX (UNIX) 11.31

File System VXFS

Oracle database file placement

Below are the specifics detailing the Oracle database configuration in the high-scale test lab. Included below are also recommendations for best practices to follow.

Database specifics:

• Oracle Enterprise Edition

• Version 10.2.0.4

• Single Instance database (non-RAC)

The baseline database configuration used a feature called, “Bigfile tablespaces.” Using bigfile tablespaces enables the Database Administrator (DBA) to create very large data files that provide the following benefits:

Using an 8K block size, a bigfile tablespace can have a 32-terabyte data file

Since the data files can be much larger, fewer of them are required

Database management is simplified

When bigfile tablespaces are used, there is a requirement to use Automatic Segment Space Management (ASSM). The requirement of using ASSM has three exceptions: the SYSTEM, temporary, and undo tablespaces. ASSM uses bitmaps to improve space utilization — and it is also self-tuning. In contrast, manual segment management uses linked lists called “freelists” to track free space in the segments.

ASSM is effective with smaller volumes of data. However, in some cases, manually segment management may improve performance.

Page 14: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 14

Table 2 Logical Unit (LUN) 04: DATA Table Name Total Size (MB)

ADMIN_DATA 0

CONFIG_DATA 23

IEE02_DATA 156,531

IEE_DATA 167,036

LOB_DATA 361

LOG_DATA 5,874

MVIEW_DATA 0

OPER_DATA 155

READING_DATA 4,647

TAMPER_DATA 1

TASK_DATA 5 Table 3 LUN 05: INDEX

Table Name Total Size (MB)

ADMIN_INDEX 1

CONFIG_INDEX 86

IEE02_INDEX 44,826

IEE_INDEX 104,383

LOG_INDEX 6,289

MVIEW_INDEX 0

OPER_INDEX 6

READING_INDEX 276

TAMPER_INDEX2 7,637

TASK_INDEX 2

Page 15: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

15

Itron/EMC benchmark goals and tools used

The goal Import, validate, estimate, and store 240,000,000 AMI readings in less than 3 hours.

This is a standard benchmark Itron performs on their AMI systems. The throughput achieved is similar to what is required for 5M two-channel hourly AMI meters, or 10M single-channel hourly meters. A throughput of >25K readings per second is the target.

Past benchmarks have shown that the validation/estimation stage can be accomplished at >100K per second, while storage remains to be the limiting factor in overall throughput. Partnering with EMC has helped Itron attack that area of the system.

In the benchmarking stage of the Itron product release cycle, infrastructure optimization plays a major role. The application code had been previously optimized during unit, system, and solution testing. Now EMC engineers worked alongside Itron test engineers to tune the storage and Database tiers.

The chart below shows the readings per second throughput observed before the system was optimized. Each day, 240M readings are processed and stored. And as the chart shows, the performance deteriorated over time, as the Oracle table partitions filled.

Figure 6 MDM baseline performance

Tools used Various tools were used to study the performance and configuration of the server,

database, and storage system to understand the decline in performance.

Server:

Manual inspection of the operating system configuration

Review of the Oracle-recommended OS settings in supporting the 10gR2 database

Page 16: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 16

Oracle database:

SQL Trace & TKPROF

DBMS Profiler

AWR Reports

Oracle Enterprise Manager (OEM)

EMC CLARiiON CX4-960 storage system:

EMC Navisphere® – provides real-time information about the storage system via a dashboard interface

Navisphere Analyzer – gathers storage-system performance statistics and presents them in various types of charts

The use of the tools provided:

• Validation of key hardware component configurations

• Insight into code efficiency

• A complete performance analysis

Page 17: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

17

Baseline database results

Testing the baseline database configuration showed:

• The UNIX “top” and “sar” utilities were used to monitor processor utilization, showing no processor contention.

• Automatic Workload Repository (AWR) was used to produce reports showing that over time, Logical Input Output Operations per Second (IOPS) dropped, and Physical IOPS remained constant. Over three days, the Logical IOPS decreased by more than half, starting at 170 on day 1 and ending at 70 by day 3. This means that the performance tuning effort had to focus upon opportunities to improve Oracle table and index access patterns as to improve the Logical IOPS response reading over time.

Page 18: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 18

The AWR reports consistently showed the following top 5 database wait events:

Figure 7 Baseline Top 5 Database Wait Events

The list below organizes the Top 5 Database Wait Events by severity and provides an explanation of each. The goal in characterizing the problems is to develop a strategy for improving performance.

DB File Sequential Reads (53%): This event indicates that a buffer is being read into the system global area (SGA) buffer cache and is waiting for the disk to return the requested information. Contrary to the name, this wait event is for a single-block read.

Free Buffer Waits (26%): This event indicates that a free buffer is not available and the database writer must write out dirty buffers to resolve this availability problem.

CPU Time (20%): Is the amount of time the CPU is waiting on other components or processes. For example, the amount of time the CPU waits for the result set to be returned from disk.

Buffer Busy Waits (1%): Indicates that a session is using a block in the buffer cache and is impacting other sessions trying to access the same block.

DB File Scattered Reads (0%): Indicates that a full scan of a table or index is underway. The cause for this wait event is that the blocks from the full scan(s) are

0% 1%

20%

26%

53%

Top 5 Wait Events

db file scattered readbuffer busy waits

CPU time

free buffer waits

db file sequential read

Page 19: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

19

read into different memory locations that are not adjacent to each other.

Taking into account that the database is not CPU-bound and the top 5 wait events center upon disk-based activity, the conclusion is that the focus areas should be:

• Optimizing data movement from the externally attached tables to the internal tables in the database

• Reducing wait times to access data in the database

Page 20: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 20

Performance changes

Improving data movement

As with any performance challenge, the most important question is “where is the bottleneck?” or, more specifically, in this case, “What is the bottleneck in moving data from external to internal tables?” One of the first steps was to look at which tablespaces were experiencing the majority of the physical reads and writes. The findings show that the majority of read and write activity is confined to the READING_INDEX and READING_Y2009_M03 tablespaces, allowing the performance team to narrow the performance tuning effort. By focusing on the objects within these two tablespaces, the team was able to improve performance in short order.

Additionally, one of the indexes related to the READINGGROUPSPC table was experiencing high Interested Transaction List (ITL) waits. An ITL wait occurs when all the slots in the transaction list have been taken and the next session has to wait until a slot becomes available in order to lock a row. Some of the causes of ITL waits include:

• A low setting of the MAXTRANS, which places a hard limit on the number of transactions that can have locks on a block

• The block is so packed that there is no room for the index to grow

• Both of the above

In the process of analyzing the database performance degradation over time as evidenced by Figure 10, the high ITL waits emerged as a possible cause. More specific findings included:

• On day one, tables like READINGGROUPSPC have many free blocks, resulting in the database’s fastest performance

o Over the following days, blocks related to the tables became more congested, causing increases in ITL waits and lowering logical IOPS

• Events like DB File Sequential Read, Free Buffer Waits, and CPU Time grew because of the bottleneck in accessing rows/blocks in these two tablespaces

To improve performance, changes to the settings related to ITL waits include:

• Increase the INITRANS: The INITRANS setting enables the DBA to set aside storage for control information inside of each block.

• Increase the PCTFREE: This setting controls the percentage of space left unused in the block.

Before changing any of these parameters it is important to remember the database is using ASSM. Therefore, in order to set the values for INITRANS and PCTFREE a locally managed tablespace will need to be created, and the tables and indexes will need to be moved into it.

Reducing wait times

An additional architectural consideration was the storage configuration. As the table below shows, testing revealed significant wait time for physical reads of tablespaces,

Page 21: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

21

which was impacting performance.

Table 4 Tablespace waits for physical reads Tablespace Total Time Allocated (MB)

READING_IN 35264334 136706

IEE02_DATA 10709112 56531

IEE_DATA 5635501 186754

READING_Y2 4234921 NA

Investigation of the LUN configuration revealed:

• LUN 4 (DATA) and LUN 5 (INDEX) were sharing the same physical disks

• The sharing of the disks between DATA and INDEX tablespaces was causing excessive I/O demands that the disks couldn’t support

To alleviate the disk contention, a new LUN was created across some slower (400 GB 10k rpm) but relatively underutilized disks. On the new LUN a locally managed tablespace called “READING_DATA_MANUAL” was created and the READINGGROUPSPC table and related indexes were moved to the new tablespace. The creation of a new LUN assured this table was not going to be impacted during load tests by residing on the same physical disks.

Final storage configuration

EMC spends considerable effort researching database and storage system optimization. When provisioning storage for a database, a number of factors must be considered. While there is almost always a trade-off between performance and cost, a few tenets come to mind:

1. High usage tables need to be located on storage that can meet the performance demands.

2. Generally, logs, data, and index structures should be on separate spindles.

3. Understanding the database schema is important; which tables are accessed together as a result of a single query, for example? Tables that are accessed simultaneously should be located on separate sets of spindles to avoid contention (and improve performance).

4. An understanding of the I/O pattern is important as well. Is it random or sequential? Is it small block, or large block? Mostly reads, or mostly writes? If the operations tend to be small block random writes, and there are many of them (high demand), then a RAID 10 configuration would perform better than RAID 5.

5. Spread the load as evenly as possible across all available resources. For example, try to place equal load on both of the storage system’s processors.

Page 22: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 22

With these tenets in mind, the final database storage configuration is illustrated below.

1. The first diagram describes how the LUNs were provisioned on the EMC CX4-960 storage system.

2. The table describes how the IEE database server files systems are mapped to the LUNs.

Figure 8 IEE database LUNs ( ) = Number of Disks A MetaLUN is a type of LUN that is comprised of other LUNs. MetaLUNs are typically used to increase performance by striping the data across the sub-LUNs. This helps for random workloads, as more disks are accessed in parallel.

Page 23: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

23

Table 5 IEE file system to LUN mapping

File System

RAID Type

MetaLUN LUN Names Capacity Number of Disks Disk Size &

Speed

/u01 RAID 5 N/A LRS_WORK1_RG35_LUN_1 1.467TB 5 400GB 10K

/u02 RAID 10 N/A LRS_REDO1_RG81_LUN_1 668GB 10 146GB 15K

/u03 RAID 10 N/A LRS_REDO2_RG83_LUN_1 668GB 10 146GB 15K

/u04 RAID 10 LRS_DATA1_META

LRS_DATA1_RG13_LUN_1 668GB 10

146GB 15k

LRS_DATA1_RG14_LUN_1 668GB 10

LRS_DATA1_RG15_LUN_1 668GB 10

LRS_DATA1_RG19_LUN_1 668GB 10

LRS_DATA1_RG22_LUN_1 668GB 10

/u05 RAID 10 LRS_READ1_META LRS_READ1_RG30_LUN_1 1.833TB 10

400GB 10K LRS_READ1_RG31_LUN_1 1.833TB 10

/u06 RAID 5 N/A LRS_ARCH1_RG33_LUN_1 1.467TB 5 400GB 10K

/u07 RAID 10 N/A LRS_CONF1_RG32_LUN_1 1.833TB 10 400GB 10K

/u08 RAID 5 N/A LRS_APPL1_RG82_LUN_1 584GB 5 146GB 15K

Best practices and the payoff

Database best practices

Within the benchmark, the first changes made were related to the database, including:

• Created a new locally managed tablespace called,“READING_DATA_MANUAL”, which allowed for adjustments to the INITRANS and FREELISTS settings on tables and indexes

• Set the INITRANS on base tables of objects, like the READINGGROUPSPC table, to 2 and the corresponding indexes to 3.

Subsequent database modifications and changes to the storage system were made to further increase performance, including:

• Created a new locally managed tablespace called “READING_DATA_MANUAL” and moved tables with high waits related to physical reads into the new tablespace

• The tablespace READING_DATA_MANUAL was moved from its original location where it was in contention with other tablespaces to a less utilized area.

These changes provided a stabilization of performance across the 6 days of testing. From day 1 through day 6 the business goal of supporting 25,000 meter readings per second was realized. A key observation is that a trade-off was made: Although the database-only changes resulted in higher performance for the first three days, the combined changes resulted in longer-term performance stabilization. Even though, the READING_DATA_MANUAL tablespace was moved from faster physical disks

Page 24: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 24

(15k rpm) to slightly slower disk (10k rpm), the performance improved because there much less disk contention.

As a result of this testing we have identified additional opportunities to optimize the solution including:

1. Storage Tiering to store frequently accessed data on faster disk

2. Make more efficient use of disk space instead of short stroking drives

3. Use various RAID types depending on the read/write intensity of applications Application and lab improvements since the set of benchmark tests described above:

1. Itron is continuously improving its VEE throughput. Since these tests, Itron

further optimized application code to provide consistent performance data to day

2. Itron’s lab test bed has expanded to 250M readings per day, beyond the

original 240M.

Page 25: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide In Partnership with Itron

25

Conclusion

As utility companies are preparing for the massive increase in data flowing in from

smart meters, they will be presented with new challenges. Top of mind will be to develop a strategy for the storage and management of that data on a system that meets the performance requirements of their business.

Itron has delivered a high-performance and scalable meter data management system, and EMC has delivered a high-performance and scalable storage system to meet these requirements.

This paper demonstrates how EMC and Itron work together to successfully meet the throughput and management needs of very large-scale AMI deployments. It also offers some best practice guidelines to follow. No matter the specific dimensions of a particular deployment, these lessons can be applied.

Our utility customers can now be confident their mission-critical IEE MDMS running on EMC storage will provide the performance they require to meet the challenges of AMI.

Page 26: EMC Optimized Infrastructure for Smart Metering · EMC Optimized Infrastructure for . Smart Metering . Best Practice Guide In Partnership with Itron . EMC Information Infrastructure

EMC Optimized Infrastructure for Smart Metering—Best Practice Guide

In Partnership with Itron 26

References

i http://www.ferc.gov/eventcalendar/Files/20070423091846-EPRI%20-%20Advanced%20Metering.pdf ii IDC: “Vendor Assessment: Industry Short List for Meter Data Management – Getting to Scale.”

December 2009. iii http://www.aeic.org/load_research/docs/12_Time-of-Use_and_Critical_Peak_Pricing.pdf iv http://www.oe.energy.gov/demand.htm v http://www.itron.com/pages/products_detail.asp?id=itr_016219.xml vi http://www.itron.com/pages/products_detail.asp?id=itr_017675.xml vii http://www.itron.com/pages/products_detail.asp?id=itr_000300.xml viii Itron: “Itron Enterprise Edition Meter Data Management Connect AMI to the Enterprise: Bridging the

Gap Between AMI and CIS.” ix Itron: “Itron Enterprise Edition Meter Data Management Connect AMI to the Enterprise: Bridging the

Gap Between AMI and CIS.”