Deconstructing Storage Arrays

Deconstructing Storage Arrays

Timothy E. Denehy, John Bent, Florentina I. Popovici,

Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau

University of Wisconsin, Madison

Gray-box Research• Computer systems becoming more complex

• Transistors• Lines of code

• Each component is becoming more complex• Interactions between subsystems can affect

• Performance• Reliability• Power• Security

Gray-box Research• Interfaces remain the same

• Changes can be difficult and impractical• Support multiple platforms or legacy systems• Commercial acceptance for wide-spread

adoption• Hardware and software phenomenon

• IA-32 instruction set, POSIX OS, SCSI storage

• Problem: lack of information

Gray-box Solution• Treat target system as a gray-box

• General characteristics are known• Extract information from an existing

interface• e.g. determine cache contents

• Exploit information to control system behavior• e.g. access cached data first

Gray-box Information Techniques

• Make assumptions about target system• Observe system inputs and outputs• Statistical methods

• Draw inferences about internal structure• Microbenchmarks and probes

• Parameterize system components• Observe system under controlled input

Gray-box Applications• Gray-box techniques have been used to identify

• Memory hierarchy parameters [Saavedra and Smith]• Processor cycle time [Staelin and McVoy]• Low-level disk characteristics [Worthington et al.]• Buffer cache replacement algorithms [Burnett et al.]• File system data structures [Sivathanu et al.]

• storage array characteristics: Shear

Shear• Software tool that automatically determines

the important properties of a storage array

• Enables file system performance tuning with knowledge of storage array characteristics

• Acts as a management tool to help configure, monitor, and maintain storage arrays

Outline• Introduction• Shear

• Background• Algorithm

• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,

Failure• Conclusion

Shear Goals• Determine storage array characteristics

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI


• Number of disks

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI


• Number of disks• Chunk size

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI


• Number of disks• Chunk size• Layout and redundancy scheme

0 1 2 316171819

4 5 6 7 8 9 1011 1213141520212223 24252627 28293031

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

RAID-0

SCSISCSI



0 1 2 31617181924252627

4 5 6 720212223

4 5 6 720212223RAID-1

SCSISCSI

28293031 28293031

0 1 2 31617181924252627

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031



0 1 2 320212223

P P P P

4 5 6 7 8 9 101112131415

P P P P

P P P P16171819 P P P P32333435 24252627 28293031

36373839 40414243 44454647

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

RAID-5

SCSISCSI

Shear Motivation• Performance

• Tune file systems to array characteristics

• Management• Verify configuration• Detect failure

Shear Techniques• Microbenchmarks and probes

• Controlled, random access read and write patterns

• Measure response time of access patterns• Measure steady-state performance

• Statistical clustering• Automatically classify fast and slow regimes• Identify patterns that utilize only a single disk

Shear Assumptions• Storage array

• Layout follows a repeatable pattern• Composed of homogeneous disks

• System• Able to bypass the file system and buffer

cache• Little traffic from other processes





Shear Algorithm• Pattern size• Chunk size• Layout of chunks to disks• Level of redundancy

Determining the Pattern Size• Find the size of the layout's repeating pattern

• Not always the stripe size

• Choose a hypothetical pattern size• Perform random reads at multiples of that distance

• Repeat for a range of pattern sizes

• Cluster results and identify actual pattern size

Pattern Size Example

RAID-04 Disks

8 KBChunks


Testing2 KB

RAID-04 Disks

8 KBChunks


Testing4 KB

RAID-04 Disks

8 KBChunks


Testing6 KB

RAID-04 Disks

8 KBChunks


Testing8 KB

RAID-04 Disks

8 KBChunks


Testing10 KB

RAID-04 Disks

8 KBChunks


Testing12 KB

RAID-04 Disks

8 KBChunks


Testing14 KB

RAID-04 Disks

8 KBChunks


Testing16 KB

RAID-04 Disks

8 KBChunks


Testing18 KB

RAID-04 Disks

8 KBChunks


Testing20 KB

RAID-04 Disks

8 KBChunks


Testing22 KB

RAID-04 Disks

8 KBChunks


Testing24 KB

RAID-04 Disks

8 KBChunks


Testing26 KB

RAID-04 Disks

8 KBChunks


Testing28 KB

RAID-04 Disks

8 KBChunks


Testing30 KB

RAID-04 Disks

8 KBChunks


Testing32 KB

RAID-04 Disks

8 KBChunks


RAID-04 Disks

8 KBChunks


RAID-04 Disks

8 KBChunks

Actual32 KB

cluster

clustercluster


Determining the Chunk Size• Chunk size

• amount of data contiguously allocated to one disk• Find the boundaries between disks

• Choose a hypothetical boundary offset• Perform random reads on both sides of that offset

• Repeat for all offsets in the pattern size• Cluster results and identify actual chunk size

Chunk Size Example

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing0 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing2 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing4 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing6 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing8 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing10 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing12 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing14 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing16 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

RAID-04 Disks

8 KBChunks

Chunk Size Example

RAID-04 Disks

8 KBChunks

Actual8 KB

cluster

cluster


Determining the Read Layout• Find mapping of chunks to disks

• Choose a pair of chunks in the pattern• Perform random reads to both chunks

• Repeat for all pairs of chunks

• Cluster results and identify chunks on same disk

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks16

25

34

Read Layout Example

07


25

34

Testing { 0, 0 }

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 1 }

07


16

25

34

Testing { 0, 2 }

07


16

25

34

Testing { 0, 3 }

07


16

25

34

Testing { 0, 4 }

07


16

25

34

Testing { 0, 5 }

07


16

25

34

Testing { 0, 6 }

07

Read Layout Example

07


25

34

Testing { 0, 7 }

Read Layout Example

07


25

34

Testing { 1, 1 }

Read Layout Example

07


34

Testing { 1, 2 }

16

Read Layout Example

07


34

Testing { 1, 3 }

16

Read Layout Example

07


34

Testing { 1, 4 }

16

Read Layout Example

07


34

Testing { 1, 5 }

16

Read Layout Example

07


34

Testing { 1, 6 }

16

Read Layout Example

07


34

Testing { 1, 7 }

16

Read Layout Example

07


34

16

Read Layout Example

07


34

Actual{ 0, 7 } { 1, 6 } { 2, 5 } { 3, 4}

16

cluster cluster


Determining Level of Redundancy

• Ratio of read to write bandwidth reveals the type of redundancy in the array

• Expected R/W ratios:• RAID-0: 1 (no redundancy)• RAID-1: 2 (mirroring)• RAID-4: varies (examine write layout)• RAID-5: 4 (parity)

Shear Experience• Shear has been applied to

• Linux software RAID• Poor RAID-5 parity updates

• Adaptec hardware RAID controller• Implements RAID-5 left-asymmetric layout

– RAID-0– RAID-1– Chained

Declustering

– RAID-4– RAID-5– P+Q





RAID-5 Performance• Small writes on RAID-5 are problematic

• Require two reads, parity calculation, two writes

• Writing in full stripes is more efficient

0 1 2 320212223

P P P P

4 5 6 7 8 9 101112131415

P P P P

P P P P16171819 P P P P32333435 24252627 28293031

36373839 40414243 44454647RAID-5

Stripe-aligned Writes• Overcome RAID-5 small write problem• Modified Linux disk scheduler

• Groups writes into full stripes• Aligns writes along stripe boundaries• Approximately 20 lines of code

• Experiment• Hardware RAID-5, 4 disks, 16 KB chunks• Create 100 files of varying sizes

Stripe-aligned Writes Experiment

• Simple modification has a large impact

Detecting MisconfigurationsCo

rrect

RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA

• Software RAID, 4 Disks, 8 KB Chunks• What if one disk is accidentally used

twice?

Detecting MisconfigurationsCo

rrect

Misc

onfig

RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA

Detecting Failures• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks

Detecting Failures• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks

• Disk 5 fails





Conclusion• Gray-box research

• Extract / exploit information from existing interfaces

• Shear• Extracts information

• Automatically determines storage array properties• Exploits information

• File system performance tuning• Storage management

Questions?

http://www.cs.wisc.edu/adsl/

Documents

Deconstructing Storage Arrays