EDITOR’S NOTE STUDY THE CLASSES OF AWS STORAGE

NEARLINE STORAGE COMES IN FROM THE COLD

AWS PROVIDES A RANGE OF COLD STORAGE OPTIONS

Sizing Up the AWS Storage OptionsCloud storage can solve an organization’s data problems, but cost, latency requirements, storage classes and other factors need to be taken into account.

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS2

EDITOR’SNOTE

Cloud Storage Questions to Ask

As cloud computing emerged, business decision makers needed to determine just how comfortable they were with putting important data beyond the walls of a data center to which they had physical access. They needed to work through questions like: Is our business ready for the cloud? And is the cloud ready for us?

Some of those same questions remain, even as cloud becomes an increasingly impor-tant part of many businesses. This handbook approaches the issues with a focus on cloud storage.

Cloud expert Ofir Nachmani provides a detailed overview of the multiple storage options available to AWS users. He breaks down how object storage differs from block storage and when an archival storage option might make the most sense.

Those differences are important, Nachmani notes, and sorting through them will likely matter in whether your cloud storage effort

meets performance expectations and falls within your budget. Included in his assessment are points of comparison between storage ser-vices offered by AWS, Google Cloud Platform and Microsoft Azure.

Also, this handbook offers suggestions for how to choose a cloud storage provider. IT ana-lyst Mike Matchett shares pointers on what to look for when evaluating nearline storage options and how to avoid common missteps in selecting a provider.

Lastly, TechTarget’s Stephen Bigelow writes about cold storage in AWS. He offers tips on using the AWS Direct Connect and AWS Import/Export Snowball services and how to do so at minimal expense. n

Phil Sweeney Senior Managing Editor

Data Center and Virtualization GroupTechTarget

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS3

STORAGE CLASSES

Study the Classes of AWS Storage

Without a solid understanding of the storage options AWS offers, it will be difficult to ensure your choice meets application and performance requirements. You’ll need to iden-tify features that strike an appropriate balance between performance and cost. In general, AWS storage can be divided into three classes:

n block device storage (Amazon Elastic Block Store);

n object storage (Amazon Simple Storage Service, or S3); and

n archival storage (Glacier).

Each storage type offers certain capabilities, so it is worth breaking down the distinctions.

AMAZON ELASTIC BLOCK STORE

Amazon’s EBS is a low-latency, persistent stor-age offering. Because it is highly available in

the same availability zone, a component failure doesn’t lead to data loss. Amazon EBS volumes are used with Amazon Elastic Compute Cloud instances.

Amazon EBS volumes are offered with solid state drives (SSD) and hard disk drives (HDD). Within those types are two key options to consider.

SSD offers low-latency results, durability and consistent I/O performance. The performance of SSD volumes is measured by their IOPS (I/O operations per second). To meet further per-formance versus cost requirements, Amazon offers Elastic Block Store SSD volumes in two types:

EBS Provisioned IOPS SSD. These volumes are used for applications that are highly sensitive to latency. For these apps, performance and consistency are crucial. An EBS Provisioned IOPS SSD volume allows you to set a preferred

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS4

STORAGE CLASSES

IOPS level and ensures that you always achieve at least 90% of that level. The size of an EBS Provisioned IOPS volume can fall between 4 GB and 16 TB with up to 20,000 IOPS per volume. Each gigabyte of the volume provides around 30 IOPS. Typical uses of EBS Provisioned IOPS SSD volumes would be I/O intensive databases and critical business applications.

EBS General Purpose SSD. These volumes, referred to as GP2, fit the majority of use cases. Each gigabyte of storage provides up to 3 IOPS, and the size of GP2 volumes can range between 1 GB and 16 TB. Each GP2 volume comes up with baseline performance IOPS, which allows it to burst its performance up to 3,000 IOPS for 30 minutes, ensuring a fast boot-up cycle. The typical uses for EBS General Purpose SSD volumes are system boot volume or root vol-ume, low-latency interactive apps and transac-tional workloads.

HDD volumes, meanwhile, are low-cost, low-latency magnetic volumes. In these, per-formance is measured in megabytes per second rather than IOPS.

Throughput Optimized HDD. These low-latency magnetic storage volumes are used by sequen-tial I/O workloads. The volume size for this can vary from 500 GB to 16 TB with a maxi-mum throughput of 500 megabytes per second (MBps). Uses for EBS Throughput Optimized HDD volumes include big data tasks, log pro-cessing and data warehouses.

Cold HDD. These volumes are designed to handle sequential data that doesn’t need to be instantly available. They are useful for when performance is not a big concern. Cold HDD cannot be used for bootable drives. Volume size varies from 500 GB to 16 TB with a maximum throughput of 250 MBps. Typically, EBS Cold HDD volumes would be used for data that’s accessed infrequently.

AMAZON SIMPLE STORAGE SERVICE

S3 is a highly available, durable, secure object storage service offered by AWS. It can be accessed via API calls from anywhere in the world. There is no upfront storage commit-ment, and S3 and can be integrated with other

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS5

STORAGE CLASSES

Amazon services. Features include cross-region replication, lifecycle rules, event notifications, versioning, encryptions and flexible storage options.

Amazon offers storage options within S3. Amazon S3 Standard provides availability, durability and security with low-latency deliv-

ery for frequently accessed objects. The dura-bility of the objects is set to 9.999999999% and a yearlong availability of 99.99%. It can support encryption of data in transit and at rest. The typical uses for S3 Standard Storage are simple (static) websites, distribution of rich content (such as video) and big data analytics.

Amazon S3 Standard Infrequent Access pro-vides the same performance, availability and durability as S3 Standard, but it also sets up the right combination of cost and performance for objects that may not be used frequently. It works in storage cost and per-gigabyte retrieval

cost. This class applies to objects and can stay in the same bucket along with S3 Standard objects. S3 Standard Infrequent Access is often used for backups and data stored for disaster recovery.

AMAZON GLACIER

Amazon Glacier is a low-cost storage option. It provides durable, secure, long-term storage that’s well-suited for archiving and backup.

Glacier is best deployed where it’s accept-able for data retrieval times to be hours long. The service works well with Amazon S3, where automatic lifecycle policies allow an organiza-tion to store data.

The typical uses for Amazon Glacier are long-term backups and logs. It can also be used to store compliance-related data, which in cer-tain cases needs to be kept for years.

COMPARING STORAGE OPTIONS

AWS isn’t alone in cloud storage. Rival public cloud providers Microsoft Azure and Google Cloud Platform (GCP) also offer multiple

Amazon S3 Standard can support encryption of data in transit and at rest.

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS6

STORAGE CLASSES

storage options. There are some key points of comparison between these services.

With object storage, AWS S3 offers users multiple options and touts availability of 99.99%. Azure’s Blob service offers Blob Zone Redundant Storage, Locally Redundant Storage and Cool Blob options. It assures availability at 99.9%. GCP’s Cloud Storage service offers Standard and Durable Reduced Availability object storage options. It lists availability at 99.9%.

As for block storage, Amazon’s EBS offers SSD and HDD options. Maximum disk size is 16 TB.

Azure Disks is Microsoft’s SSD block stor-age service, for which customers pay based on what they use, regardless of the size pro-visioned. Azure gives users the ability to map shared drives. The number of disks that can be attached to a server depends on VM size. GCP’s Persistent Disks block storage service includes SSD and HDD options. Maximum disk size is 64 TB.

With archive storage, AWS Glacier is a low-cost option ($0.007 per GB per month) with a retrieval time of four hours. GCP’s Nearline

service is low-cost ($0.01 per GB per month), and its retrieval time is measured in seconds. Azure does not have a comparable offering in the archival storage category.

Be sure to look at your organization’s storage selection from multiple perspectives. Specifi-cally, look at the following:

Storage Size. Size plays a vital role in perfor-mance. For block storage options across multiple cloud providers, you can see how IOPS or throughput of a volume or disk linearly increases with the storage size of the disk.

Access Frequency. Each cloud provider has multiple storage options based on your access frequency. If you have data that’s infrequently accessed, you can use different storage classes to save money while achieving the same avail-ability, durability and security.

Performance. If you are looking for predictable, low-latency, consistent performance levels for your applications, you can customize your storage options to fit those requirements.

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS7

STORAGE CLASSES

Availability. Consider your data availability requirements. For example, if you are willing to compromise on availability, you should find a low-cost storage service.

As long as you have your business and appli-cation requirements laid out correctly, you’ll

likely have several good storage options avail-able. These choices will let you juggle perfor-mance, cost, durability and consistency. These storage options can be combined to achieve practical storage solutions for your workloads, allowing you to transform from one storage class to another. —Ofir Nachmani

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS8

NEARLINE STORAGE

Nearline Storage Comes in From the Cold

It was just a few years ago that many CIOs and IT directors were highly skeptical about public nearline cloud storage. Security, gover-nance and cost were big concerns. Bandwidth, migration, access control and provider stabil-ity also impeded the long-predicted journey to the cloud. Most concerns have since been addressed to the point where some new genera-tions of on-premises storage arrays are directly tiering to public cloud storage.

Cloud storage has evolved. It’s now to the point where it can be used in production, not just as a remote backup archive. Network latencies still prevent I/O-hungry data center workloads from using the cloud as primary storage, but hybrid architectures that can move workloads to the cloud using VMs and con-tainers are becoming more the norm than the

exception. Still, many data centers are not yet using nearline cloud storage for purposes other than cold storage.

To nail down good use cases, look at the various tiers of storage each cloud service pro-vider offers. You should consider each tier as a possible plug-in resource in your architecture. Ask vendors how simple—and costly—it is to transfer data between cloud storage services directly, as this can make it easier to shift data around should needs change.

Be sure to understand the costs involved in both storing data over time and access-ing it when necessary. Even though there may be access costs, they might be well within expected budgets. And be sure to pay close attention to access latencies; backups that take hours to recall may not provide satisfactory

Cloud storage has evolved. It’s now to the point where it can be used in production, not just as a remote backup archive.

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS9

NEARLINE STORAGE

levels of business continuity.With AWS and competing providers lower-

ing prices, nearline cloud storage users should position themselves to take advantage of the lowest costs available. It isn’t easy to migrate massive amounts of data from one provider to another, nor is it necessarily cheap. This results in some friction-based lock-in, which is something to be aware of when committing large repositories to any one service provider.

UNITING CLOUD WITH ARRAYS

New storage arrays are going beyond the premises with built-in native cloud tiers. For example, Microsoft’s StorSimple cloud storage service for inactive data is linked with Azure.

If you plan to use nearline cloud storage with an array, there are a number of capabilities to evaluate:

Native cloud tiering. The array can use the public cloud as a colder tier of storage.

Snapshots to the cloud. The array can direct and

recall snapshots to remote systems, including cloud storage targets.

Processing in the cloud. Look for ways in which the cloud-hosted data can still be actively used, such as for analytics, forensics or other cloud-side processing.

Recovery in the cloud. Look for service provid-ers that make it easy to recover storage backed up to the cloud. This enables disaster recovery as a service, in which your DR site is the public cloud.

Content distribution and global access. Cloud storage can also serve to replicate and propa-gate data around the globe for local access or as remote office/branch office primary storage for sharing large design files, videos and follow-the-sun business processing.

WHAT A PROVIDER PROVIDES

You will also want to look for service providers that work closely with your storage-portfolio vendors—or at least the product roadmaps

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS10

NEARLINE STORAGE

anticipating your next storage upgrade. You don’t want to be the one troubleshooting if there are problems between on-premises arrays and cloud storage tiers.

Many providers, such as IBM and Oracle, even offer virtual storage area networks out of their public clouds, not just virtual storage

endpoints or volumes. These virtual SANs can often be used and operated like on-premises storage arrays. This approach helps data center admins directly extend existing storage prac-tices while taking advantage of the cloud infra-structure’s elasticity and utility pricing.

—Mike Matchett

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS11

COLD STORAGE OPTIONS

AWS Provides a Range of Cold Storage Options

Customers can choose from several archival choices for AWS cold storage, a capa-bility that enables an organization to move data to and from the cloud without a significant strain on resources.

Dedicated high-speed network connections, such as AWS Direct Connect, can establish a connection between a data center and an AWS cloud facility. The Ethernet connection bypasses the public internet, avoiding bottle-necks that often reduce bandwidth and lower data transfer performance. AWS Direct Con-nect services are available in 1 gigabit Ethernet (GbE) or 10 GbE speeds; IT teams can aggregate multiple links for additional bandwidth.

THE APPLIANCE OPTION

The AWS Import/Export Snowball is another way to transfer huge volumes of data. It uses a portable out-of-band storage appliance. An

administrator requests a large transfer job, and AWS ships a storage appliance. The admin then connects the appliance to the network and transfers the desired data, which is then encrypted. The company returns the device to AWS, which moves the data to storage using Amazon’s internal network.

The process sounds old-fashioned, but this method can be faster and less expensive than moving data over the internet, depending on a company’s network connection speeds and the amount of data it needs to move. AWS Import/Export Snowball can be a good option for data volumes as low as 60 TB over a 1 GbE con-nection. Also, the appliance might be a better alternative when restoring massive backups from cloud storage to the local data center.

Neither AWS Direct Connect nor AWS Import/Export Snowball directly moves data to Amazon Glacier. These methods transfer data to Amazon Simple Storage Service (S3).

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS12

COLD STORAGE OPTIONS

Admins then move the data offline to a Glacier vault.

Other alternatives to accelerate data transfers between a business and AWS include Amazon Kinesis Firehose, which addresses multiple streaming data sources. Amazon S3 Transfer Acceleration handles recurring storage jobs with incremental changes—usually over long distances—and gateways to cache data locally. Still, AWS Direct Connect and AWS Import/Export Snowball are probably the most desirable options for use with AWS cold storage.

WATCH THOSE COSTS

There are also some cost considerations for using AWS cold storage. For example, each object that Glacier archives requires a company to store some data in S3 to maintain metadata; this can be as much as 40 KB per object. While

that isn’t much, the total can add up for a large number of objects. When an IT team plans to retain those objects in Glacier for months or even years, storing Glacier object metadata in S3 can add unexpected costs to the archives. It can be helpful to compress many small data objects into a single consolidated .tar or .zip file before uploading and moving them to AWS cold storage.

AWS imposes an early deletion fee if an admin deletes or changes an object within three months of creating the archive. There’s no charge for changes made after three months. Amazon charges to move each object to Glacier, which means you may find it expensive to shift a large number of objects.

IT teams can incur other costs when restor-ing data to S3 from Glacier. AWS usually charges for restorations based on the highest rate encountered in GB per hour.

—Stephen J. Bigelow

HOME

EDITOR’S NOTE

STUDY THE CLASSES

OF AWS STORAGE

NEARLINE STORAGE

COMES IN FROM

THE COLD

AWS PROVIDES

A RANGE OF COLD

STORAGE OPTIONS

SIZING UP THE AWS STORAGE OPTIONS13

ABOUT THE

AUTHORS

STEPHEN J. BIGELOW is senior technology editor in TechTarget’s Data Center and Virtualization Media Group. He has more than 20 years of experience in technology writing and is the author of several books.

MIKE MATCHETT is a senior IT analyst and consultant at Taneja Group, which evaluates technologies in the data storage and server markets.

OFIR NACHMANI is a business technology advisor, blogger and lecturer. His IamOnDemand.com blog is a guide for technology startups and developers in the world of cloud computing.

Sizing Up the AWS Storage Options is a SearchAWS.com e-publication.

Margie Semilof | Editorial Director

Phil Sweeney | Senior Managing Editor

Dan Cagen | Associate Features Editor

Linda Koury | Director of Online Design

Megan Cassello | Graphic Designer

Moriah Sargent | Managing Editor, E-Products

Rebecca Kitchens | Publisher rkitchens@techtarget.com

TechTarget 275 Grove Street, Newton, MA 02466

www.techtarget.com

© 2016 TechTarget Inc. No part of this publication may be transmitted or repro-duced in any form or by any means without written permission from the pub-lisher. TechTarget reprints are available through The YGS Group.

About TechTarget: TechTarget publishes media for information technology pro-fessionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and processes crucial to your job. Our live and virtual events give you direct access to independent ex-pert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.

COVER: FOTOLIA

STAY CONNECTED!

Follow @SearchAWS today.