Storage Performance Testing - Woody Hutsell ATT593830

8/6/2019 Storage Performance Testing - Woody Hutsell ATT593830

http://slidepdf.com/reader/full/storage-performance-testing-woody-hutsell-att593830 1/35

Storage Performance Testing

Woody Hutsell, Texas Memory Systems



SNIA Legal Notice

The material contained in this tutorial is

copyrighted by the SNIA.Member companies and individuals may use thismaterial in presentations and literature under the


© 2007 Storage Networking Industry Association. All Rights Reserved. 2

Any slide or slides used must be reproduced withoutmodification

The SNIA must be acknowledged as source of any

material used in the body of any document containingmaterial from these presentations.

This presentation is a project of the SNIA

Education Committee.



Abstract


Conducting storage performance tests is essential toselecting storage for tiered storage environments. Someapplications require endless hours of constant data



,

block I/O. The best storage device for one application isalmost never the right storage device for another.

This session will provide an in-depth technical discussion ofstorage performance testing.



Agenda Topics

Why test storage?

Types of storage performance testing:Benchmarking.

A lication simulation.



Application testing.

Production testing.

Common mistakes.



Why Test Storage?

Because it matters to your users and customers.

Slow storage performance means slow response times and longrunning queries.

Because it affects your batch window.

Slow stora e can mean lon er batch or backu windows causin



lower application availability or maintenance windows.

Because it matters to your company’s profitability.

Slow storage can frustrate your customers and waste theinvestment you have made in your server infrastructure.

Inappropriate use of fast storage means wasted dollars spent onperformance.

Because storage vendors do not publicize every relevant

metric for your application and environment.



Types of Storage PerformanceTesting

Benchmarking.

Review published and audited industry benchmarks.Conduct tests with industry standard software.

Conduct tests for data corruption.



Application simulation.Use industry standard software to test a program withconditions similar to a target application.

Application testing.Test an application with sample queries or scripts in aproduction-like environment.

Production testing.



Published Benchmarks

Storage Performance Council

www.StoragePerformance.OrgSPC-1 test simulates an on-line transaction processing (OLTP)environment.

SPC-2 test to simulate large block sequential processing.



Spec-SFSwww.Spec.Org/sfs97r1

A good test for measuring performance of file servers andnetwork attached storage.

TPC-C

www.tpc.org

TPC-C for testing OLTP, TPC-H for decision support and TPC-W

for web e-commerce.



Sample SPC-1 Result

• Shows

peakSPC-1

IOPS.



• Shows

response

timecurve.



Benchmarking Software

IOMeter

Most popular tool among storage vendors.Available free from www.iometer.org.

Primarily a Windows-based tool.



o eBroad OS support.

Available free from www.iozone.org.

Benchmark Factory for Databases by QuestSoftware.

TPC-B, TPC-C, TPC-D (not for publishing results).

Vendor tools.



IOMeter – Disk Targets Tab

Note:•If you leave this

“ ”



Heuristics:•One managerper server.•One worker perprocessor.

,

IOMeter will useall available diskspace.

Can play asignificant role inobservedperformance.



IOMeter – Example Effect of VariedOutstanding I/Os

339,944

250,000

300,000

350,000

400,000

IOPS

25 Outstanding IOs.



125,538

-

50,000

100,000

150,000

200,000

'2k' '4k' '16K' '64K' '256K'

Block Size of Transfer

1 Outstanding IO.



IOMeter – Setting AccessSpecifications

Test storagewith small andlarge blocktransfer request

Try differentread/writemixtures.



sizes.

Try differentsequential vs.random tests.

Usually leave at

default but canbe changed tomatchapplicationbehavior.



IOMeter – Example Effect of VariedBlock Sizes

250,000

300,000

350,000

400,000

IOPS

Small block size = HighIOPS but relatively Lowbandwidth.



-

50,000

100,000

150,000

200,000

'2k' '4k' '16K' '64K' '256K'


IOPS but relatively Highbandwidth.



IOMeter – Example Effect of Randomvs. Sequential

4,626

4,000

5,000

6,000

Sequential Transfers

IOPS



1,413

1,000

2,000

3,000

'2k' '4k' '16K' '64K' '256K'


Random Transfers



IOMeter - Scripting

IOMeter can be used to generate scripts.

Scripts can be set-up to run through a long set ofdata patterns and then record the output to a logfile.



oo or overn g t test ng runs o new pro ucts.Collect these reports and use them to compareproducts.

Monitor a device over time or after majorconfiguration changes to verify performancebaseline.



IOMeter – ScriptingExample

Name your script



test iteration.

This script will incrementthrough Outstanding I/O

levels.



Testing for Data Corruption

Storage devices and storage network

components are almost always reliable inpredictable performance ranges, but thequestion is how do they handle extreme



.

Most benchmarking tools do not automaticallycheck data.

Testing for data corruption usually means testing

with data patterns that challenge components.Need to test extremes of performance.

Need to test extreme data patterns.



Case Study: Server NMIs

• Problem: Storage device was causing server NMI(crash) when other company storage does not.• Weeks were spent testing. Progress was slow because the

problem did not repeat frequently.

• Key test tool: Medusa Labs software was deployed to



.• Resulted in nearly instant NMIs due to their challenging data

patterns (a good thing in this case because it helped diagnosethe problem faster).

• Conclusion: Brand “X” HBA in PCI slot 1 caused NMIs

while the same card in PCI slot 2 did not cause NMI.Brand “Y” HBA worked fine in either PCI slot. Problemonly observed under extreme loads.



Application Simulation Testing

One type of test does not represent all

applications.One type of application does not represent all

uses for a storage product.



Common types of application simulation testing:Test storage latency for messaging or other single-threaded applications.

Test peak storage bandwidth for data acquisition ordata streaming environments.

Test peak storage IOPS for databases.





IOMeter – Simulating Multi-threadedApplications



25

25 OutstandingI/Os simulates amulti-threaded

application.



IOMeter – Simulating DatabaseEnvironments

Small transfer requestsize simulates Match the



8

a a ase rans ers. app cat on s

read/writedistribution.

Databaseactivity is

mostly random.



IOMeter – TestingMixed Access Patterns

Name Your AccessPattern

Set % Access to



application.

O S



IOMeter – Simulating DataStreaming

Big transferrequest size tests Match the



512

pea an w . app cat on s

read/writedistribution.

A mostlysequential

setting is best.



C St d Wi d St



Case Study: Windows StoragePerformance

Windows performance monitor can be used to

monitor storage performance.Capture the following key variables over the

duration of a peak processing period or test run:



Processor: % processor time (total and by processor).Physical disk: average disk queue (total, read andwrite by disk/array).

Physical disk: disk bytes/second (total, read and writeby disk array).

C St d Wi d St




Tips to analyzing Windows Performance Monitor

results:Use the following scaling to ease visual analysis:

Disk queues: 1:1 ratio (default is 100:1).



Processor utilization: 1:1 ratio (default is 1:1).

Disk bytes: 0000001:1 ratio (default is a .0001:1).

Start with “total” fields and then drill down into

“read/write/by disk/by processor” variables.

Alter the line thickness to see your results easier.

Use the slider bars to zoom into trouble spots.

Case Study Windows Storage




More tips to analyzing Windows Performance Monitor

results:

Disk bytes per second should be divided by 1024 to get disk KBand 1024 again to get disk MB.

Where physical disk queues increase is likely at the same point



where you have hit a storage performance limitation.

A system with high processor utilization does not have a storageperformance bottleneck.

Microsoft recommends that physical disk queues greater than 3

(per disk) shows an I/O bottleneck.Processor utilization levels off in places you have physical diskqueues this is an indication that faster storage will improveapplication performance.

Case Study: Windows Storage




Disk Queuesshow pending

requests tostorage.



Disk Bytes PerSecond helpsreveal storage

limitations.

ProcessorTime shows

percentprocessorutilized.

Case Study: Windows Storage




No Processorbottleneck



No diskqueuesLow disk

activity

Monitoring Storage Performance



IOStat results show read and write bytes per

second:

Monitoring Storage Performancewith UNIX

Device: r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util

/dev/sdb 0.00 10619.39 0.00 85570.91 16.12 4636.79 43.52 0.10 101.21

/dev/sdc 0.00 10678.79 0.00 85570.91 16.07 2438.06 22.75 0.10 107.27

avg-cpu: %user %nice %sys %idle

13.04 0.33 68.15 18.48



TOP shows CPU utilization including I/O Wait.load averages: 0.09, 0.04, 0.03 16:31:09

66 processes: 65 sleeping, 1 on cpu

CPU states: 69.2% idle, 18.9% user, 11.9% kernel, 0.0% iowait, 0.0% swap

Memory: 128M real, 4976K free, 53M swap in use, 542M swap free

Monitoring Storage Performance With



Monitoring Storage Performance WithOracle

Elapsed: 68.87 (mins)

Top 5 Wait Events

~~~~~~~~~~~~~~~~~ Wait % Total

Event Waits Time (cs) Wt Time

-------------------------------------------- ------------ ------------ -------

db file sequential read 18,073,422 581,168 59.36

db file scattered read 933,001 267,364 27.31

db file parallel write 25,990 35,898 3.67

SQL*Net message from dblink 181,872 20,372 2.08

latch free 11,936 17,879 1.83

-------------------------------------------------------------

These waitevents areheavily

influenced by



Tablespace

------------------------------

Av Av Av Av Buffer Av Buf

Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)

-------------- ------- ------ ------- ------------ -------- ---------- ------

SESSION_DATA

61 0 20.2 1.2 190,606 94 128,753 56.8

UNDOTBS1

32 0 14.1 1.0 16,517 8 6,083 2.3

.

Tablespacemetrics are agood way to

monitorstorage

performance.



Risk vs. Reward.

Risk: taking an unsupported, well-traveled evaluationunit and putting it in a production environment couldcompromise application availability and expose

Production Testing



.

Reward: sometimes this is the only way to know forcertain that storage performance is acceptable for anapplication.



Typical Mistakes

Testing storage performance with file copy

commands.Comparing storage devices back-to-back without

clearing server cache.



Testing where the data set is so small thebenchmark rarely goes beyond server or storage

cache.

Forgetting to monitor processor utilization duringtesting.

Monitoring the wrong server’s performance.



Q&A / Feedback

Please send any questions or comments on this presentation toSNIA: [email protected]



Many thanks to the following individualsfor their contributions to this tutorial.

SNIA Education Committee

Sarah Worthy Jamon BowenStorage Performance Council Elaine Silber

Chris Lionetti Rob Peglar

Documents

Storage Performance Testing - Woody Hutsell ATT593830