Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
High CapacitySpectrum Scale Building Block - 2019
Oliver [email protected]+49 173 31 95 701
2019 Lenovo. All rights reserved.
pro-com DATENSYSTEME GmbHHPC Solution providerGermany
Startpoint
• Recap last GUG
• New Block Design
• Systems
• Performance Numbers
• Best Practices (so far)
2019 Lenovo. All rights reserved.
Last Time on GUG@ISC-2018
• Introduction Lenovo High CapacitySpectrum Scale Storage Block…
– 2 Server
– 4 DS Storage Units
– 1008 Hardrives
– 2 Racks
• Based on a Costumer RFP
• Solid Performance– 15 GB/s Write / 20 GB/s Read per Block
– 3,75 GB/s Write / 5 GB/s Read per Controller
2019 Lenovo. All rights reserved. 3
New Systems 2019
2019 Lenovo. All rights reserved. 4
Update 2019
• Lenovo and Netapp announced Global Strategic Alliance
• Update of our Configuration
• Targets– Increase Density
– Increase Performance
– Same Price per TB Range
2019 Lenovo. All rights reserved.
DE6000-H / DE600s
DE6000-H
• Dual Controller
• RAID 0,1,5,6 / Dynamic Disc Pool
• FSWA Support
– Full Stripe Write Acceleration
• 16-64 GB Cache
• 12GB SAS / 32 GB FC / 25 GbE iSCSI
• Up to 480 Drives
• 2U and 4U Versions
– 24x 2.5“ or 60x 3.5“
DE600s Expansion Chassis
• Dual ESM
• 60x 3.5“
2019 Lenovo. All rights reserved. 6
DE6000-H – FSWA
• Full Stripe Write Acceleration (FSWA)
• Only with classical RAID6
• Massive performance increase
• Depending on Workload
2019 Lenovo. All rights reserved. 7
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
1 MB Data Block per Array
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
FSWA
8
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
256 KB
128 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
9
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
2 MB Data Block
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
10
Q
(Parity)
P
(Parity)
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6 Disk 7 Disk 8
1 MB Data Block
2 MB Data Block
4 MB Data Block
256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB 256 KB
Valid values for the segment size are: 8, 16, 32, 64, 128, 256 and 512 KB
FSWA
11
New Block 2019
2019 Lenovo. All rights reserved. 12
Setup
• How to Connected?– SAS direct attached
• How to Scale– Up to 300 Drives per Disk System
– 1200 Drives per Block
• How to Balance– 16 SAS Ports 12GB / 4x 100 GbE / IB
Ports
– 32 PCI-E Lanes vs. 32 PCI-E Lanes
2019 Lenovo. All rights reserved. 13
1200 drive configuration in two racks
• 2x NSD – Server
– Lenovo ThinkSystem SR650
– 2x Intel Xeon 6242 – 16 Core 2.8 GHz
– 12x 32 GByte Memory
– 2x High Speed Adapter (IB / OPA / 100 GbE)
– 4x 12 GBIT Quad Port SAS Adapter
• 4x Back End Storage
– Lenovo ThinkSystem DE6000-4U
– Dual Controller – 8x 12 GB SAS Ports
– RAID 0,1, 5, 6, 10 – DDP Technology
– Each 300 Drives via 4x DE600s Expansion Unit
2019 Lenovo. All rights reserved.
1410HEB
A2
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
1 2
1 2
<< Storage
Expansion_DE 7
7Y69CTO1
WW
<< Storage
Expansion_DE 3
7Y69CTO1
WW
7Y80CTO1
WW
<< Storage
Expansion_DE 9
7Y69CTO1
WW
<< Storage
Expansion_DE 8
7Y69CTO1
WW
1
7Y69CTO1
WW
<< Storage_DE6000 4
7Y80CTO1
WW
Server_NSD-Server 2
7X06CTO1
WW
1
Co
mb
o P
DU
&1
p, 3
2A
/2
7Y69CTO1
WW
<< Storage
Expansion_DE 12
7Y69CTO1
WW
<< Storage
Expansion_DE 11
7Y69CTO1
WW
1
1
<< Storage
Expansion_DE 4
<< Storage
Expansion_DE 10
<< Storage_DE6000 3
12
Co
mb
o P
DU
&1
p, 3
2A
/2
12
1
A2
U
Rack_Storage-P3-R11-
12 No. MT/M or PN
Co
mb
o P
DU
&1
p, 3
2A
/2
12
7
6
5
4
3
2
13
12
11
10
9
8
19
18
17
16
15
14
25
24
23
22
21
20
31
30
29
28
27
26
37
36
35
34
33
32
1410HEB
A1
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
42
41
40
39
38
<< Storage
Expansion_DE 1
7Y69CTO1
WW
1
1
Co
mb
o P
DU
&1
p, 3
2A
/2
1 2
<< Storage
Expansion_DE 2
7Y69CTO1
WW
<< Storage
Expansion_DE 1
7Y69CTO1
WW
<< Storage_DE6000 1
7Y80CTO1
WW
<< Storage
Expansion_DE 3
7Y69CTO1
WW
1 2
<< Storage_DE6000 2
7Y80CTO1
WW
Server_NSD-Server 1
7X06CTO1
WW
6
7Y69CTO1
WW
<< Storage
Expansion_DE 5
7Y69CTO1
WW
<< Storage
Expansion_DE 4
7Y69CTO1
WW
Co
mb
o P
DU
&1
p, 3
2A
/2
12
1
1
12
12
A1
U
Rack_Storage-P3-R11-
12 No. MT/M or PN
Co
mb
o P
DU
&1
p, 3
2A
/2
<< Storage
Expansion_DE 2
7Y69CTO1
WW
<< Storage
Expansion_DE
6
5
4
3
2
1
12
11
10
9
8
7
18
17
16
15
14
13
24
23
22
21
20
19
30
29
28
27
26
25
36
35
34
33
32
31
42
41
40
39
38
37
14
Benchmarks New vs. Old
2019 Lenovo. All rights reserved. 15
Benchmarks
• Started begin of June 2019 ☺
• Focus– DE6000-H Scaling
- 60,120,180… Drives
– DE4000-H Scaling
– SSD Performance
– Classical Raid FSWA vs. DDP
– Directly mapped LUN to SAS link vs. DMM round-robin
– …
– …
– …
2019 Lenovo. All rights reserved. 16
First Numbers DE6000
• DE6000
– 120 Drives
– RAID 6 – (8+P+Q)
– 12 LUNS
• NSD
– 2x SR650
- 2x 6134
- EDR IB
• Clients
– 24x Clients EDR
• Spectrum Scale– 5.0.3.0
– 16M Blocksize
– 16M IOR Transfer Size
2019 Lenovo. All rights reserved. 17
1
2
3
4
5
6
7
8
9
10
11
12
13
DE6000 - 120 Drives DE6000 - 120 Drives
DE6000-H-120 Drives
Write Read
Predecessor – System DS6200
• DS6200
– 252 Drives
– ADAPT RAID
– 16 LUNS
• NSD
– 2x SR650
- 2x 6142
- 100 GbE
• Clients
– 24x Clients 100 GbE
• Spectrum Scale– 5.0.1.1
– 16M Blocksize
– 16M IOR Transfer Size
2019 Lenovo. All rights reserved. 18
1
2
3
4
5
6
7
8
9
10
11
12
13
DS6200 - 252 Drives DS6200 - 252 Drives
DS6200-252 Drives
Write Read
DS6200 vs. New DE6000-h
2019 Lenovo. All rights reserved. 19
1,75 x higher Write Speed / 2x higher Read Speed
Write Read
1
2
3
4
5
6
7
8
9
10
11
12
13
DE6000 - 120 Drives DE6000 - 120 Drives
DE6000-H-120 Drives
1
2
3
4
5
6
7
8
9
10
11
12
13
DS6200 - 252 Drives DS6200 - 252 Drives
DS6200-252 Drives
Some syntenic numbers
2019 Lenovo. All rights reserved. 20
DE6000
DE4000
Bigger PictureDE6000 vs. Nativ Raid
2019 Lenovo. All rights reserved. 21
Some Nativ Raid Numbers
2019 Lenovo. All rights reserved. 22
Hardware Controller vs. Nativ Raid
• Nativ Raid still King– Performance & Data Integrity
• DE6000– ~70 MB/s per Disk in Write
– ~101 MB/s per Disk in Read
• Nativ Raid– ~85 MB/s per Disk in Write
– ~108 MB/s per Disk in Read
2019 Lenovo. All rights reserved. 23
Write Read
16 MB Blocksize and 16 MB Transfersize
0 2 4 6 8 10 12 14 16 18 20
DS6200 - 252 Drives
DS6200 - 252 Drives
DE6000 - 120 Drives
DE6000 - 120 Drives
DSS-G 201 - 82 Drives
DSS-G 201 - 82 Drives
DSS-G 202 - 166 Drives
DSS-G 202 - 166 Drives
Solution Performance comparison
Best Practices – so far
• Planning and Implementation Guide
– Installation, Configure and Tuning
– Lenovo, Netapp and pro-com
– Available late 2019
– Lenovo Press
• Performance Level
– RAID 6 with FSWA vs. Dynamic Disk Pools
– Performance vs. Data Integrity
• Full Stripe Write Acceleration
– Optimize Segment Size vs. LUNS vs. GPFS Blocksize
• LUNs QTY
– multiple of 8 (to equally spread the LUNs across the Host Ports) - if not possible DMM round-robin
• Linux Disk I/O Settings:– Scheduler
- Noop
– reah_ahead_kb
- 128 (Linux Default)
– nr_request
- 128 (Linux Default)
– max_sectors_kb
- 2048
– …
– …
2019 Lenovo. All rights reserved. 24
2019 Lenovo. All rights reserved.
2019 Lenovo. All rights reserved.