Deconstructing Storage Arrays

Deconstructing Storage Arrays

Timothy E. Denehy, John Bent, Florentina I. Popovici,

Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau

University of Wisconsin, Madison

Gray-box Research• Computer systems becoming more complex

• Transistors• Lines of code

• Each component is becoming more complex• Interactions between subsystems can affect

• Performance• Reliability• Power• Security

Gray-box Research• Interfaces remain the same

• Changes can be difficult and impractical• Support multiple platforms or legacy systems• Commercial acceptance for wide-spread

adoption• Hardware and software phenomenon

• IA-32 instruction set, POSIX OS, SCSI storage

• Problem: lack of information

Gray-box Solution• Treat target system as a gray-box

• General characteristics are known• Extract information from an existing

interface• e.g. determine cache contents

• Exploit information to control system behavior• e.g. access cached data first

Gray-box Information Techniques

• Make assumptions about target system• Observe system inputs and outputs• Statistical methods

• Draw inferences about internal structure• Microbenchmarks and probes

• Parameterize system components• Observe system under controlled input

Gray-box Applications• Gray-box techniques have been used to identify

• Memory hierarchy parameters [Saavedra and Smith]• Processor cycle time [Staelin and McVoy]• Low-level disk characteristics [Worthington et al.]• Buffer cache replacement algorithms [Burnett et al.]• File system data structures [Sivathanu et al.]

• storage array characteristics: Shear

Shear• Software tool that automatically determines

the important properties of a storage array

• Enables file system performance tuning with knowledge of storage array characteristics

• Acts as a management tool to help configure, monitor, and maintain storage arrays

Outline• Introduction• Shear

• Background• Algorithm

• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,

Failure• Conclusion

Shear Goals• Determine storage array characteristics

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI

Shear Goals• Determine storage array characteristics

• Number of disks

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI

Shear Goals• Determine storage array characteristics

• Number of disks• Chunk size

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031 SCSISCSI

Shear Goals• Determine storage array characteristics

• Number of disks• Chunk size• Layout and redundancy scheme

0 1 2 316171819

4 5 6 7 8 9 1011 1213141520212223 24252627 28293031

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

RAID-0

SCSISCSI

Shear Goals• Determine storage array characteristics

• Number of disks• Chunk size• Layout and redundancy scheme

0 1 2 31617181924252627

4 5 6 720212223

4 5 6 720212223RAID-1

SCSISCSI

28293031 28293031

0 1 2 31617181924252627

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

Shear Goals• Determine storage array characteristics

• Number of disks• Chunk size• Layout and redundancy scheme

0 1 2 320212223

P P P P

4 5 6 7 8 9 101112131415

P P P P

P P P P16171819 P P P P32333435 24252627 28293031

36373839 40414243 44454647

0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031

RAID-5

SCSISCSI

Shear Motivation• Performance

• Tune file systems to array characteristics

• Management• Verify configuration• Detect failure

Shear Techniques• Microbenchmarks and probes

• Controlled, random access read and write patterns

• Measure response time of access patterns• Measure steady-state performance

• Statistical clustering• Automatically classify fast and slow regimes• Identify patterns that utilize only a single disk

Shear Assumptions• Storage array

• Layout follows a repeatable pattern• Composed of homogeneous disks

• System• Able to bypass the file system and buffer

cache• Little traffic from other processes

Outline• Introduction• Shear

• Background• Algorithm

• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,

Failure• Conclusion

Shear Algorithm• Pattern size• Chunk size• Layout of chunks to disks• Level of redundancy

Determining the Pattern Size• Find the size of the layout's repeating pattern

• Not always the stripe size

• Choose a hypothetical pattern size• Perform random reads at multiples of that distance

• Repeat for a range of pattern sizes

• Cluster results and identify actual pattern size

Pattern Size Example

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing2 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing4 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing6 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing8 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing10 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing12 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing14 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing16 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing18 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing20 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing22 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing24 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing26 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing28 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing30 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

Testing32 KB

RAID-04 Disks

8 KBChunks

Pattern Size Example

RAID-04 Disks

8 KBChunks

Pattern Size Example

RAID-04 Disks

8 KBChunks

Actual32 KB

cluster

clustercluster

Shear Algorithm• Pattern size• Chunk size• Layout of chunks to disks• Level of redundancy

Determining the Chunk Size• Chunk size

• amount of data contiguously allocated to one disk• Find the boundaries between disks

• Choose a hypothetical boundary offset• Perform random reads on both sides of that offset

• Repeat for all offsets in the pattern size• Cluster results and identify actual chunk size

Chunk Size Example

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing0 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing2 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing4 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing6 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing8 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing10 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing12 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing14 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

Testing16 KB

RAID-04 Disks

8 KBChunks

Chunk Size Example

RAID-04 Disks

8 KBChunks

Chunk Size Example

RAID-04 Disks

8 KBChunks

Actual8 KB

cluster

cluster

Shear Algorithm• Pattern size• Chunk size• Layout of chunks to disks• Level of redundancy

Determining the Read Layout• Find mapping of chunks to disks

• Choose a pair of chunks in the pattern• Perform random reads to both chunks

• Repeat for all pairs of chunks

• Cluster results and identify chunks on same disk

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks16

25

34

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks16

25

34

Testing { 0, 0 }

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 1 }

07

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 2 }

07

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 3 }

07

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 4 }

07

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 5 }

07

Read Layout ExampleRAID-0 ZIG-ZAG 4 Disks

16

25

34

Testing { 0, 6 }

07

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks16

25

34

Testing { 0, 7 }

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks16

25

34

Testing { 1, 1 }

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 2 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 3 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 4 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 5 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 6 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Testing { 1, 7 }

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

16

Read Layout Example

07

RAID-0 ZIG-ZAG 4 Disks25

34

Actual{ 0, 7 } { 1, 6 } { 2, 5 } { 3, 4}

16

cluster cluster

Shear Algorithm• Pattern size• Chunk size• Layout of chunks to disks• Level of redundancy

Determining Level of Redundancy

• Ratio of read to write bandwidth reveals the type of redundancy in the array

• Expected R/W ratios:• RAID-0: 1 (no redundancy)• RAID-1: 2 (mirroring)• RAID-4: varies (examine write layout)• RAID-5: 4 (parity)

Shear Experience• Shear has been applied to

• Linux software RAID• Poor RAID-5 parity updates

• Adaptec hardware RAID controller• Implements RAID-5 left-asymmetric layout

– RAID-0– RAID-1– Chained

Declustering

– RAID-4– RAID-5– P+Q

Outline• Introduction• Shear

• Background• Algorithm

• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,

Failure• Conclusion

RAID-5 Performance• Small writes on RAID-5 are problematic

• Require two reads, parity calculation, two writes

• Writing in full stripes is more efficient

0 1 2 320212223

P P P P

4 5 6 7 8 9 101112131415

P P P P

P P P P16171819 P P P P32333435 24252627 28293031

36373839 40414243 44454647RAID-5

Stripe-aligned Writes• Overcome RAID-5 small write problem• Modified Linux disk scheduler

• Groups writes into full stripes• Aligns writes along stripe boundaries• Approximately 20 lines of code

• Experiment• Hardware RAID-5, 4 disks, 16 KB chunks• Create 100 files of varying sizes

Stripe-aligned Writes Experiment

• Simple modification has a large impact

Detecting MisconfigurationsCo

rrect

RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA

• Software RAID, 4 Disks, 8 KB Chunks• What if one disk is accidentally used

twice?

Detecting MisconfigurationsCo

rrect

Misc

onfig

RAID 5-LS RAID 5-LA RAID 5-RS RAID 5-RA

Detecting Failures• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks

Detecting Failures• Software RAID• RAID-5 LS• 10 disks• 8 KB chunks

• Disk 5 fails

Outline• Introduction• Shear

• Background• Algorithm

• Case Studies• Performance: Stripe-aligned Writes• Management: Detecting Misconfiguration,

Failure• Conclusion

Conclusion• Gray-box research

• Extract / exploit information from existing interfaces

• Shear• Extracts information

• Automatically determines storage array properties• Exploits information

• File system performance tuning• Storage management

Questions?

http://www.cs.wisc.edu/adsl/