Upload
amazon-web-services
View
1.725
Download
7
Embed Size (px)
Citation preview
PowerPoint Presentation
Darryl Osborne Storage Specialist Solutions ArchitectMarch 28th, 2017Deep Dive on Amazon EBS Elastic Volumes
2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1
A little EBS history2006 EC2 launched with instance storage2008 EBS (Elastic Block Storage) launched on magnetic storage2012 EBS io1 (SSD) & EBS-Optimized instances2014 EBS gp2 (SSD)2014 EBS data volume encryption2015 Larger/faster EBS volumes2015 EBS boot volume encryption2016 EBS st1 (HDD) and sc1 (HDD)2017 EBS Elastic Volumes!
2
Throughput?orIOPSChoosing an EBS volume typeWhat is more important to your workload:
3
Current EBS volume types
General PurposeSSD
gp2
Provisioned IOPSSSDio1
Throughput Optimized HDDst1ColdHDDsc1
SSDHDD
4
EBS volume types: I/O ProvisionedGeneral PurposeSSD
gp2Throughput: 160 MiB/sLatency: Single-digit msCapacity: 1 GiB to 16 TiBBaseline: 3 IOPS per GiB up to 10,000Burst: 3,000 IOPS (for volumes up to 1 TiB)
Great for boot volumes, low-latency applications, and bursty databases
5
EBS volume types: I/O ProvisionedProvisioned IOPSSSDio1Baseline: 100 to 20,000 IOPSThroughput: 320 MiB/sLatency: Single-digit msCapacity: 4 GiB to 16 TiB
Max ratio: 50:1 IOPS to sizeIdeal for critical applications and databases with sustained IOPS
6
EBS volume types: Throughput Provisioned
Throughput Optimized HDDst1
Ideal for large-block, high-throughput sequential workloadsBaseline: 40 MiB/s per TiB up to 500 MiB/sCapacity: 500 GiB to 16 TiBBurst: 250 MiB/s per TiB up to 500 MiB/s
7
Cold HDDsc1
EBS volume types: Throughput ProvisionedBaseline: 12 MiB/s per TiB up to 192 MiB/sCapacity: 500 GiB to 16 TiBBurst: 80 MiB/s per TiB up to 250 MiB/s
Ideal for sequential throughput workloads, such as logging and backup
8
Throughputis more importantSmall, random I/OLarge, sequential I/O
Latency?
i2
gp2
io1
st1
sc1
d2Choosing an EBS volume type
IOPS 65,000> 65,000< 1 msSingle-digit ms
1,250 MiB/sAggregate throughput?> 1,250 MiB/sis more important 10k IOPS> 10k IOPS
Throughput per volume250 MiB/s500 MiB/sIOPS per volume
9
EBS Elastic Volumes
10
EBS Elastic Volumes3
11
Definitions
Original volume(end of the arrow)Target volume(tip of the arrow)
12
What are the new capabilities?
13
What are the new capabilities?3
14
What are the new capabilities?Increase volume size
Change volume type
Increase/decrease Provisioned IOPS3
15
Increase volume size
gp2
gp2
io1
io1
st1
st1
sc1
sc1
16
Increase volume size
gp2
gp2
sc1
sc1
io1
io1
st1
st1
17
Change volume type
gp2
io1sc1
st1
gp2
io1sc1
st1
18
Change volume type
gp2
io1sc1
st1
gp2
io1sc1
st1
19
Change volume type
gp2
io1sc1
st1
gp2
io1sc1
st1
20
Increase/decrease Provisioned IOPS
io1
increase
decrease
21
Combine different modifications
gp2
gp2
io1
io1
sc1
sc1
st1
st1
22
Combine different modifications
gp2
gp2
io1
io1
sc1
sc1
st1
st1
23
Combine different modifications
gp2
gp2
io1
io1
sc1
sc1
st1
st1
24
ConsiderationsVolume modifications must be valid for the target volume type
25
ConsiderationsVolume modifications must be valid for the target volume typeexample400 GiBst1
?400 GiB
gp2
26
EBS volume types: Throughput Provisioned
Throughput Optimized HDDst1Baseline: 40 MiB/s per TiB up to 500 MiB/sCapacity: 500 GiB to 16 TiBBurst: 250 MiB/s per TiB up to 500 MiB/s
Ideal for large-block, high-throughput sequential workloads
27
ConsiderationsVolume modifications must be valid for the target volume typeexample400 GiB500 GiBst1
gp2
28
ConsiderationsVolume modifications must be valid for the target volume typeexample100 GiB1,000 iops?100 GiB10,000 iops
io1io1
29
EBS volume types: I/O ProvisionedProvisioned IOPS SSDio1Baseline: 100 to 20,000 IOPSThroughput: 320 MiB/sLatency: Single-digit msCapacity: 4 GiB to 16 TiB
Max ratio: 50:1 IOPS to sizeIdeal for critical applications and databases with sustained IOPS
30
ConsiderationsVolume modifications must be valid for the target volume typeexample100 GiB1000 iops
200 GiB10,000 iops
io1io1
31
ConsiderationsVolume modifications must be valid for the target volume typeMinimum volume sizes for each volume type
Maximum volume sizes
gp2
io1
st1
sc1
1 GiB4 GiB500 GiB500 GiB16 TiB
32
ConsiderationsVolume modifications must be valid for the target volume typePerf. characteristics for each volume type
Maximum ratio of PIOPS to size for io1 volumes
gp2
io1
st1
sc1
Min 100 iopsMax 3K iopsMin 100 iopsMax 20K iopsMin 12 MiB/s /TiBMax 192 MiB/s /TiBMin 40 MiB/s /TiBMax 500 MiB/s /TiB50:1
33
Where can I use it?
34
Where can I use it?3
35
Where can I use it?Available in all public AWS regions
Current generation volumes
Current generation instances*3* Live volume modifications
36
Available in all public AWS regions
N. VirginiaOhioN. CaliforniaOregonCanadaSo PauloIrelandLondonFrankfurtSydneyTokyoSeoulMumbaiSingaporeBeijing
37
Current generation volumes*
gp2
io1
st1sc1
*Previous generation (Magnetic) volumes are not supported
38
Current generation instances*General purposeComputeoptimized
C3Storage and I/Ooptimized
I3
P2GPUoptimizedMemoryoptimized
C4
M4
D2
X1
G2
R4R3
I2
M3*Some previous generation instances do not support live modifications
39
Caveats
M3Medium=Live Volume Modifications
40
CaveatsM3 Instances other than Medium
M3other thanMedium=No Live Volume Modifications** After each volume modification you MUST stop/start the instance or detach/attach the volume
41
CaveatsAll other instances WITHEBS volumes attached before 11/1/2016
AllEC2=Live Volume Modifications** After a one-time instance stop/start or volume detach/attach
42
What tools do I use?
43
What tools do I use?3
44
What tools do I use?AWS Management Console
Command Line Interface (CLI)
Software Development Kits (SDKs)3
45
AWS Management Console
46
Command Line Interface (CLI)aws ec2 modify-volume[--dry-run | --no-dry-run]--volume-id [--size ] [--volume-type ] [--iops ]
Software Development Kits (SDKs)
48
What can I expect?
49
What can I expect?3
50
What can I expect?No performance impact
No downtime
No over-provisioning3
51
No performance impactDuring the modification there is no performance impact as a result of the process
Your volume performance will be between the original and target volume performance characteristics
52
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.OriginalVolume
TargetVolumePerformance
53
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.
io1
gp2 1 TiB1 TiB example
54
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.
io1
gp23K iops1 TiB1 TiB20K iopsexample
55
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.
io1
gp23K iops1 TiB1 TiB20K iopsexamplePerformance
56
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification. 10 TiB10 TiB examplest1
io1
57
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification. 20K iops10 TiB10 TiB400 iops examplest1
io1
58
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.320 MiB/s20K iops10 TiB10 TiB400 iops400 MiB/sexamplest1
io1
59
No performance impactYour volume performance will be between the original and target volume performance characteristics during modification.320 MiB/s20K iops10 TiB10 TiB400 iops400 MiB/sexamplePerformance
st1
io1
60
How do I modify an EBS volume?
61
How do I modify an EBS volume?3
62
How do I modify an EBS volume?Easy three step process
Modify
Monitor
Extend (only if increasing size)3
63
Step 1: ModifyUsing the EC2 Console
64
Step 1: ModifyUsing the CLIaws ec2 modify-volume [--dry-run | --no-dry-run] --volume-id [--size ] [--volume-type ] [--iops ]
Step 1: Modify
gp23K iops1 TiBexamplevol-05c33eec9820afc20
66
Step 1: Modify
io1
gp23K iops1 TiB5 TiB20K iopsexamplevol-05c33eec9820afc20
67
Step 1: Modifyaws ec2 modify-volume --volume-id vol-05c33eec9820afc20 --size 5000 --volume-type io1 --iops 20000
io1
gp23K iops1 TiB5 TiB20K iopsexamplevol-05c33eec9820afc20
68
Step 1: Modifyaws ec2 modify-volume --volume-id vol-05c33eec9820afc20 --size 5000 --volume-type io1 --iops 20000
io1
gp23K iops1 TiB5 TiB20K iopsexamplevol-05c33eec9820afc20Performance
69
Step 2: MonitorUsing EC2 Console
70
Step 2: MonitorUsing the CLIaws ec2 describe-volumes-modifications [--volume-ids ] [--filters ] [--next-token ] [--max-results ]
Step 2: Monitoraws ec2 describe-volumes-modifications --volume-id vol-05c33eec9820afc20
io1
gp23K iops1 TiB5 TiB20K iopsexamplevol-05c33eec9820afc20Performance
72
Step 2: Monitoraws ec2 describe-volumes-modifications --volume-id vol-05c33eec9820afc20
io1
gp23K iops1 TiB5 TiB20K iopsexamplevol-05c33eec9820afc20Performance
{ "VolumesModifications": [ { "TargetSize": 5000, "TargetVolumeType": "io1", "ModificationState": "optimizing", "VolumeId": "vol-05c33eec9820afc20", "TargetIops": 20000, "StartTime": "2017-03-25T20:32:31.717Z", "Progress": 99, "OriginalVolumeType": "gp2", "OriginalIops": 3000, "OriginalSize": 1000 } ]}
73
Step 2: MonitorUsing Amazon CloudWatch Events event (event-based)
74
Step 2: MonitorUsing Amazon CloudWatch Events event (event-based)
Lambda function
75
Step 2: MonitorUsing Amazon CloudWatch Events event (event-based)
Lambda function
Amazon SNSnotification
76
GitHub Sample Code
awslabs/aws-elastic-volumes
77
Step 2: MonitorModified volume goes through different states:
78
Step 2: MonitorModified volume goes through different states:3
79
Step 2: MonitorModified volume goes through different states:3Modifying
Optimizing
Completed
80
State 1: ModifyingVery short usually just a few seconds
Received modification request Were working on it
81
State 2: OptimizingMajority of the time minutes to hours
Were still working on it
82
State 2: OptimizingVolume type modifications
83
State 2: OptimizingVolume type modifications
Your volume performance will be between the original and target volume performance characteristics during modification.
84
State 2: OptimizingVolume type modifications
OriginalVolume
TargetVolumePerformance
Your volume performance will be between the original and target volume performance characteristics during modification.
85
State 2: OptimizingIncrease/decrease PIOPS
OriginalVolumeTargetVolumePerformance
Your volume performance will be between the original and target volume performance characteristics during modification.
86
State 2: OptimizingIncrease volume size
As soon as you enter the Optimizing state, your volume is the target sizeGo to Step 3 & extend the file system in the OS
gp2
gp2
87
State 3: CompleteSignals the modification is complete
Were done
88
State 3: CompleteSignals the modification is complete
Were done
89
Step 3: Extend (only if increasing size)Using EC2 instance OSLinux:
Determine the file system (if unsure)[ec2-user ~]$ sudo file s /dev/xvd*/dev/xvda1: Linux rev 1.0 ext4 filesystem data ... /dev/xvdf: SGI XFS filesystem data ...
90
Step 3: Extend (only if increasing size)Using EC2 instance OSLinux:
Compare block device size to the file system disk usage[ec2-user ~]$ lsblk[ec2-user ~]$ df -h
91
Step 3: Extend (only if increasing size)Using EC2 instance OSLinux:
For ext2, ext3, ext4 file systems, use resize2fs[ec2-user ~]$ sudo resize2fs device_name
For XFS file systems, use xfs_growfs[ec2-user ~]$ sudo xfs_growfs d mount_point
92
Step 3: Extend (only if increasing size)Using EC2 instance OSWindows:
Run diskmgmt.msc
93
Step 3: Extend (only if increasing size)Using EC2 instance OSWindows:Right click the expanded drive and choose Extend Volume.
94
Step 3: Extend (only if increasing size)Using EC2 instance OSWindows:Enter the number of MB to extend the volume and complete the wizard.
95
How often can I modify volumes?Each volume can be modified once every 6 hours
6hrs.
96
How often can I modify volumes?Each volume can be modified once every 6 hours
An error occurred (IncorrectState) when calling the ModifyVolume operation: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
97
Can I stop a volume modification once started?A modify volume command cannot be stopped
98
Can I modify encrypted volumes?Yes, modifying encrypted volumes is supportedYou cannot change the encryption state of a volume with the modify volume command
99
How much does it cost to modify a volume?There is no cost for modifying the volumeYou start to be charged for the new volume specifications when you issue the modify-volume API call
100
What about permissions?Modifying a volumes type and PIOPS can dramatically impact performanceWe recommend you lockdown permissions to the modify-volume API like you would the delete-volume API
101
Modify Volume Timeline
gp2
io1
102
Modify Volume Timeline
gp2
io1
Modify-volume API
103
Modify Volume Timeline
gp2
io1
Modify-volume APIModifying State
(seconds)
104
Modify Volume Timeline
gp2
io1
Modify-volume APIModifying State
Optimizing State(seconds)(performance between both volumes)Target size available(expand file system)
105
Modify Volume Timeline
gp2
io1
Modify-volume APIModifying State
Completed State
Optimizing State(seconds)(performance between both volumes)Target size available(expand file system)
106
Test: Load During Modify VolumeUnmodified Volume:Volume: New 100GiB gp2 volumeFile System: ext4Instance: m4.2xlargeLoad: fio 50/50 read/write to 10x 8GiB files
Modified Volume:Volume: New 100GiB gp2 volumeFile System: ext4Instance: m4.2xlargeLoad: fio 50/50 read/write to 10x 8GiB filesTest: While the fio script was running, we modified the volume size to 200GiB
Test: Load During Modify VolumeQuestions:
Which is the unmodified?Which is the modified?
Test: Load During Modify VolumeQuestions:
Which is the unmodified?Which is the modified?
Test: Load During Modify Volume
Test: Load During Modify Volume
Test: Load During Modify Volume
Demo
Who is using EBS?
114
InforInfor provides business applications specialized by industry and built for the cloud, with more than 65 million cloud users. Were excited about the announcement of the Elastic Volumes feature for Amazon Elastic Block Store and are looking forward to the flexibility it offers to easily right-size systems, said Jim Plourde, Vice President, Infor Cloud Services. The feature makes it easy to move some of our colder data to sc1 to provide substantial savings on storage costs.
115
Acquia"Elastic volumes make it possible to quickly and efficiently scale EBS volumes for our services in an automated fashion while avoiding mistakes and dataloss. This will enabled us to maintain service uptime, reduce operational maintenance and lower unused data costs while seamlessly scaling our volumes on-demand as needed.
116
HipChatHipChat has a 45+ node elastic search cluster, with 1TB 6k IOPS io1 volumes. These were overprovisioned and they were undertaking an effort to to scale down to 3K IOPS/volume.
Benefit: With Elastic Volumes, HipChat was able to save 4-6 weeks of engineering effort, yet still realized the benefit of a reduction in IOPS spend. This cost reduction was achieved with next to no engineering overhead, and no impact to customers.
117
ReferenceAmazon EBS Resources - https://aws.amazon.com/ebs/developer-resources/
Amazon EBS Web page - https://aws.amazon.com/ebs/
YouTube AWS Channel - https://www.youtube.com/user/AmazonWebServices
Reference Architecture - https://aws.amazon.com/architecture/
qwikLABS - https://aws.qwiklabs.com/
Thank You!https://aws.amazon.com/summits/
Darryl [email protected]&A