6445 an Introduction Red Hat Storage Architecture

7/31/2019 6445 an Introduction Red Hat Storage Architecture

1/12

www.rh.m

AbstrAct 2

red HAt storAge design goAls 2

ea 2

la sa 2

sa-ou h r Ha sa 4

tecHnicAl differentiAtors 7

sa o 7

o su 7

cm sa oa

sm sak 7

U sa 7

Mua, saka

Ahu 7daa s na fma 8

n maaa h h

ea Hah Ahm 8

r Ha sa a

ama h 9

conclUsion 11

glossAry 11

Red Hat StoRage

An introdUction tored HAt storAge ArcHitectUre


2/12

2 www.rh.m

abStRact

Over the past ten years, enterprises have seen enormous gains in scalability, exibility, and affordability as

they migrated from proprietary, monolithic server architectures to architectures that are virtualized, open

source, standardized, and commoditized.

Unfortunately, storage has not kept pace with computing. The proprietary, monolithic, and scale-up solu-

tions that dominate the storage industry today do not deliver the scalability, exibility, and economics that

modern datacenter and cloud computing environments need in a hyper-growth, virtualized, and increasingly

cloud-based world. The GlusterFS le system was created to address this gap.

Red Hat Storage is scale-out network attached storage (NAS) for private cloud or datacenter, public cloud,

and hybrid cloud environments. It is software-only, open source, and designed to meet unstructured data

storage requirements. It enables enterprises to combine large numbers of commodity storage and compute

resources into a high-performance, virtualized, and centrally managed storage pool. Both capacity andperformance can scale linearly and independently on-demand, from a few terabytes to petabytes and

beyond, using both on-premise commodity hardware and the public cloud compute and storage infrastruc-

ture. By combining commodity economics with a scale-out approach, Red Hat customers can achieve radi-

cally better price and performance in an easily deployed and managed solution that can be congured for

increasingly demanding workloads.

Red Hat Storage is built using the GlusterFS open source project as the foundation. This whitepaper

discusses some of the unique technical aspects of the Red Hat Storage architecture, speaking to those

aspects of the system that are designed to provide linear scale-out of both performance and capacity

without sacricing resiliency.

Red Hat StoRage deSign goalS

Red Hat Storage was designed to achieve several major goals:

elAsticity

Elasticity is the notion that an enterprise should be able to exibly adapt to the growth (or reduction) of data

and to add or remove resources to a storage pool as needed without disrupting the system. Red Hat Storage

was designed to allow enterprises to add or delete users, application data, volumes and storage nodes, etc.,

without disrupting any running functionality within the infrastructure.

lineAr scAling

Linear scaling is a much-abused phrase within the storage industry. It should mean, for example, that

twice the amount of storage systems will deliver twice the realized performancetwice the throughput (as

measured in gigabytes per second) with the same average response time per external le system I/O event(i.e., how long an NFS client will wait for the le server to return the information associated with each NFS

client request).

Similarly, if an organization has acceptable levels of performance, but wants to increase capacity, it should

be able to do so without decreasing performance or getting non-linear returns in capacity.

Red Hat Storage

An introduction to Red Hat Storage architecture


3/12

www.rh.m 3

FigURe 1

Performance and Capacity

Throughput

Unfortunately, most storage systems do not demonstrate linear scaling. This seems somewhat counter-intu-

itive, since it is so easy to purchase another set of disks to double the size of available storage. The caveat indoing so is that the scalability of storage has multiple dimensions, capacity being only one of them.

Adding capacity is only one dimension; the systems managing the disk storage need to scale as well. There

needs to be enough CPU capacity to drive all of the spindles at their peak capacity. The le system must

scale to support the total size. The metadata telling the system where all the les are located must scale at

the same rate disks are added. The network capacity available must scale to meet the increased number

of clients accessing those disks. In short, it is not storage that needs to scale as much as it is the complete

storage system that needs to scale.

Traditional le system models and architectures are unable to scale in this manner and therefore can never

achieve true linear scaling of performance. For traditional distributed systems, each storage node must

always incur the overhead of interacting with one or more other storage nodes for every le operation, and

that overhead subtracts from the scalability simply by adding to the list of tasks and the amount of work to

be done.

Even if those additional tasks could be done with near-zero effort (in the CPU and other system resources

sense of the term), latency problems remain. Latency results from waiting for the responses across the

networks connecting the distributed storage nodes in those traditional system architectures and nearly

always impacts performance. This type of latency increases proportionally relative to the speed and respon-

sivenessor lack ofof the networking connecting the nodes to each other. Attempts to minimize coordina-

tion overhead often result in unacceptable increases in risk.

This is why claims of linear scalability often break down for traditional distributed architectures.

Instead, as illustrated in Figure 1, most traditional systems demonstrate logarithmic scalabilitystorages

useful capacity grows more slowly as it gets larger. This is due to the increased overhead necessary to

maintain data resiliency. Examining the performance of some storage networks reects this limitation as

larger units offer slower aggregate performance than their smaller counterparts.

Red Hat Storage



4/12

4 www.rh.m

scAle-oUt witH red HAt storAge

Red Hat Storage is designed to provide a scale-out architecture for both performance and capacity. Thisimplies that the system should be able to scale up (or down) along multiple dimensions.

By aggregating the disk, CPU, and I/O resources of large numbers of inexpensive systems, an enterprise

should be able to create one very large and high-performing storage pool. If the enterprise wants to add

more capacity to scale out a system, they can do so by adding more inexpensive disks. If the enterprise

wants to gain performance, they can do so by deploying more inexpensive sever nodes.

Red Hat Storages unique architecture is designed to deliver the benets of scale-out (more units means

more capacity, more CPU, and more I/O), while avoiding the corresponding overhead and risk associated

with keeping large numbers of storage nodes in sync.

In practice, both performance and capacity can be scaled out linearly with Red Hat Storage. We do this by

employing three fundamental techniques:

th ma maaa

e u aa ah aa a a

th u aam maxmz ma a a u u ahu

To illustrate how Red Hat Storage scales, Figure 2 shows how a baseline system can be scaled to increase

both performance and capacity. The discussion below uses some illustrative performance and capacity

numbers.

A typical direct-attached Red Hat Storage conguration will have a moderate number of disks attached to

two or more server nodes , which act as NAS heads (or storage nodes). For example, to support a require-

ment for 24 TB of capacity, a deployment might have two servers, each of which contains a quantity of 12

one-terabyte SATA drives. (See Cong A.)

FigURe 2

Red Hat Storage



5/12

www.rh.m 5

If a customer has found that the performance levels are acceptable but wants to increase capacity by 25%,

they could add another four one-terabyte drives to each server and will not generally experience perfor-mance degradation. (i .e., each server would have 16 one-terabyte drives). (See Cong B.) Note that they do

not need to upgrade to larger or more powerful hardware; they simply add eight more inexpensive SATA

drives.

On the other hand, if the customer is happy with 24 TB of capacity but wants to double performance, they

could distribute the drives among four servers, rather than two (i.e., each server would have six one-terabyte

drives, rather than 12). Note that in this case, they are adding two more low-price servers and can simply

redeploy existing drives. (See Cong C.)

If they want to quadruple both performance and capacity, they could distribute among eight servers (i.e.,

each server would have 12 one-terabyte drives). (See Cong D.)

Note that by the time a solution has approximately ten drives, the performance bottleneck has generally

already moved to the network. (See Cong D.)

In order to maximize performance, we can upgrade from a 1-Gigabit Ethernet network to a 10-Gigabit

Ethernet network. Note that performance in this example is more than 25 times what we saw in the baseline.

This is evidenced by an increase in performance from 200 MB/s in the baseline conguration to 5,000 MB/s.

(See Cong E.)

FigURe 3

Red Hat Storage



6/12

6 www.rh.m

As you will note, the power of the scale-out model is that both capacity and performance can scale linearly

to meet requirements. It is not necessary to know what performance levels will be needed two or three years

out. Instead, congurations can be easily adjusted as the need demands.

While the above discussion was using round, theoretical numbers, actual performance tests have proven

this linear scaling. The results , illustrated in Figure 5, show write throughput scaling linearly from 100 MB/s

on one server (e.g. storage node) to 800 MB/s (on eight systems) in a 1 GbE environment. H, a

Innibandnetwork,wehaveseenwritethroughputscalefrom1.5GB/s(onesystem)to12GB/s(eight

systems).

FigURe 4

FigURe 5: lineaR Scaling in Red Hat StoRage 1

1 Results of scalability testing per formed at Red Hat Storage running an Iozone test on eight clients connecting to between one and eight

server nodes. Total capacity was 13 TB. N etwork was 1 GbE. Ser vers were congured with single quad-core Intel Xeon CPUs and eight G B of

RAM. Clients had two GB of RAM.

Red Hat Storage



7/12

www.rh.m 7

We have experience with Red Hat Storage being deployed in a multitude of scale-out scenarios. For example,

Red Hat Storage has been successfully deployed in multi-petabyte archival scenarios, where the goal wasmoderate performance in the


8/12

8 www.rh.m

dAtA stored in nAtive forMAts

With Red Hat Storage, data is stored on disk using native formats (e.g. EXT3, EXT4, XFS). Red Hat Storage

has implemented various self-healing processes for data. As a result, the system is extremely resilient.

Furthermore, les are naturally readable without Red Hat Storage. If a customer chooses to migrate away

from Red Hat Storage, their data is still completely usable without any required modications or data

migration.

no MetAdAtA witH tHe elAstic HAsH AlgoritHM

In a scale-out system, one of the biggest challenges is keeping track of the logical and physical location of

data (location metadata). Most distributed systems solve this problem by creating a separate index with le

names and location metadata. Unfortunately, this creates both a central point of failure and a huge perfor-

mance bottleneck. As traditional systems add more les, more servers, or more disks, the central metadata

server becomes a performance chokepoint. This becomes an even bigger challenge if the workload consists

primarily of small les and the ratio of metadata to data increases.

Unlike other storage systems with a distributed le system, Red Hat Storage does not create, store, or use

a separate index of metadata in any way. Instead, Red Hat Storage places and locates les algorithmically.

All storage node servers in the cluster have the intelligence to locate any piece of data without looking it upin an index or querying another server. All a storage node server needs to do to locate a le is to know the

pathname and lename and apply the algorithm. This fully parallelizes data access and ensures linear perfor-

mance scaling. The performance, availability, and stability advantages of not using metadata are signicant

and, in some cases, dramatic.

FigURe 6: ModUlaR and StacKable aRcHitectURe

Red Hat Storage is a complete storage stack in

userspace

Everything is a volume

Volume=C shared-object library

Stack volumes to create conguration

Customize to match unique workload needs

Open APIs

Volume-to-volume and Platform-to-world

Development is like application programming

Rapid time to market

Third party development

Red Hat Storage



9/12

www.rh.m 9

red HAt storAge globAl nAMespAce tecHnology

While many extol the virtues of their namespace capability as enabling easier management of networkstorage, Red Hat Storage global namespace technology enables an even greater capability and has enabled

innovate IT solutions that are changing the way Red Hat customers leverage cloud technology and legacy

applications.

in tHe privAte cloUd or dAtAcenter

Red Hat Storage global namespace technology enables Red Hat customers who linearly scale their Red Hat

Storage NAS environment to tie together hundreds of storage nodes and associated les into one global

namespace. The result is one common mount point for a large pool of network attached storage. In some

cases, where the Red Hat Storage native access client is used in place of NFSv3, multiple parallel access is

supported by the le system.

In the public cloud, Red Hat Storage global namespace technology enables multiple compute instances and

storage to be congured in a massively scaleable pool of network attached storage. For example, withinthe AWS cloud, Red Hat Storage global namespace enables the pooling of large quantities of Elastic Cloud

Compute (EC2) instances and Elastic Block Storage (EBS) to form NAS in the AWS cloud. EC2 instances

(storage nodes) and EBS can be added non-disruptively resulting in linear scaling of both performance and

capacity. Being able to deploy NAS in the cloud and run POSIX-compliant applications within the cloud using

Red Hat Storage le storeage accelerates cloud adoption and enables new and creative enterprise customer

business solutions.

Red Hat Storage global namespace technology goes one step furtherit enables tying together both private

cloud/datacenters and public cloud NAS resources and enables enterprises to leverage a single mount point

across the private cloud/data center and public cloud environments. This innovative hybrid cloud capability

enables Red Hat customers to leverage the POSIX-compliant aspects of Red Hat Storage and their legacy

applications across vast storage resources.

red HAt storAge AdvAnced topics

elAstic volUMe MAnAgeMent

Given the elastic hashing approach assigns les to logical volumes, a question often arises: How do you

assign logical volumes to physical volumes?

In versions 3.1 and later of Red Hat Storage, volume management is elastic. Storage volumes are

abstracted from the underlying hardware and can grow, shrink, or be migrated across physical storage

nodes as systems as necessary. Storage node servers can be added or removed on-the-y with data auto-

matically rebalanced across the cluster. Data is always online, and there is no application downtime. File

system conguration changes are accepted at runtime and propagated throughout the cluster allowing

changes to be made dynamically as workloads uctuate or for performance tuning.

renAMing or Moving files

If a le is renamed, the hashing algorithm will obviously result in a different value, which will frequently

result in the le being assigned to a different logical volume, which might itself be located in a different

physical location.

Red Hat Storage



10/12

10 www.rh.m

Since les can be large and rewriting and moving les is generally not a real-time operation, Red Hat

Storage solves this problem by creating a pointer at the time a le (or set of les) are renamed. Thus, a clientlooking for a le under the new name would look in a logical volume and be redirected to the old logical

volume location. As background processes result in les ultimately being moved, the pointers are then

removed.

Similarly, if les need to be moved or reassigned (e.g. if a disk becomes hot or degrades in performance),

reassignment decisions can be made in real-time, while the physical migration of les can happen as a back-

ground process.

HigH AvAilAbility

n-a la shu ra

Generally speaking, we recommend the use of mirroring (2, 3, or n-way) to ensure availability. In this

scenario, each storage node servers le data is replicated to another storage node server using synchro-

nous writes. The benets of this strategy are full fault-tolerance; failure of a single storage server iscompletely transparent to Red Hat Storage clients. In addition, reads are spread across all members of the

mirror. Using Red Hat Storage, there can be an unlimited number of storage node members in a mirror.

While the elastic hashing algorithm assigns les to unique logical volumes, Red Hat Storage ensures that

every le is located on at least two different storage system server nodes.

g- a ahu a

For long distance data replication requirements, Red Hat Storage supports Geo-Rep long distance replica-

tion. Customers can congure storage server nodes and Red Hat Storage to asynchronously replication data

over vast geographical distances.

Replicationintheprivatecloud/datacenter,publiccloud,andhybridcloudenvironments

Both n-way synchronous and Geo-Rep asynchronous data replication are supported in the private cloud/

datacenter, public cloud, and hybrid cloud environments.

Within the AWS cloud, Red Hat Storage supports n-way synchronous replication across availability zones and

Geo-Rep asynchronous replication across AWS Regions. In fact, Red Hat Storage is the only way to ensure

high availability for NAS storage within the AWS infrastructure.

While Red Hat Storage offers software-level disk and server redundancy at the storage node server level,

in some cases we also recommend the use of hardware RAID (e.g., RAID 5 or 6) within individual storage

system servers to provide an additional level of protection where required. Red Hat solutions architects can

advise you on the best form of storage level and storage node level data protection and replication strate-

gies for your specic requirements.

Red Hat Storage



11/12

www.rh.m 11

conclUSion

By delivering increased scalability, exibility, affordability, performance, scalability, and ease-of-use in

concert with reduced cost of acquisition and maintenance, Red Hat Storage is a revolutionary step forward

in data management. Multiple advanced architectural design decisions make it possible for Red Hat

Storage to deliver great performance, greater exibility, greater manageability, and greater resilience at a

signicantly reduced overall cost. The complete elimination of location metadata via the use of the Elastic

Hashing Algorithm is at the heart of many of Red Hat Storages fundamental advantages, including its

remarkable resilience, which dramatically reduces the risk of data loss, data corruption, or data becoming

unavailable.

Red Hat Storage can be deployed in the private could or datacenter via Red Hat Storage Software Appliance

(SSA), an ISO image installed on commodity server and storage hardware, resulting in a powerful, turn-key,

massively scalable, and highly available NAS environment. Additionally, Red Hat Storage can be deployed

in the public cloud via Red Hat Virtual Storage Appliance, for example, within the AWS cloud, and deliversall the features and functionally possible in the private cloud/datacenter to the public cloudessentially

massively scalable and highly available NAS in the cloud.

To start a functional trial of Red Hat Storage, visit redhat.com. To speak with a Red Hat representative about

how to solve your particular storage challenges, call +1 (800) 805-5215.

gloSSaRy

bk a: Block special les or block devices correspond to devices through which the system moves

data in the form of blocks. These device nodes often represent addressable devices such as hard disks,

CD-ROM drives, or memory-regions.2 Red Hat Storage supports most POSIX-compliant block level le

systems with extended attributes. Examples include ext3, ext4, and ZFS.

Distributedlesystem:Any le system that allows access to les from multiple hosts sharing via a

computer network.3

Maaa: Data providing information about one or more other pieces of data.4

nama: An abstract container or environment created to hold a logical grouping of unique identiers

or symbols.5 Each Red Hat Storage cluster exposes a single namespace as a POSIX mount point that contains

every le in the cluster.

2 http://en.wikipedia.org/wiki/Device_le_system#Block_devices

3 http://en.wikipedia.org/wiki/Distributed_le_system

4 http://en.wikipedia.org/wiki/Metadata#Metadata_denition

5 http://en.wikipedia.org/wiki/Namespace_%28computer_science%29

Red Hat Storage



12/12

red HAt sAles And inqUiries

nortH AMericA

1888REDHAT1

www.redhat.com

[email protected]

eUrope, Middle eAst

And AfricA

00800 7334 2835

www.europe.redhat.com

[email protected]

AsiA pAcific

+65 6490 4200

www.apac.redhat.com

[email protected]

lAtin AMericA

+54 11 4329 7300

www.latam.redhat.com

[email protected]

Copyright 2011 Red Hat, Inc. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix,and RHCE are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux is the registeredtrademark of Linus Torvalds in the U.S. and other countries .

www.rh.m#8457567_1211

POSIX(PortableOperatingSystemInterface[forUNIX]): A family of related standards specied by the

IEEE to dene the application programming interface (API), along with shell and utilities interfaces for

software compatible with variants of the UNIX operating system. Red Hat Storage exports a fully POSIX-

compliant le system.

RAID(RedundantArrayofInexpensiveDisks): A technology that provides increased storage reliability

through redundancy, combining multiple low-cost, less-reliable disk drives components into a logical unit

where all drives in the array are interdependent.

Ua: Applications running in user space dont directly interact with hardware, instead using the kernel

to moderate access. Userspace applications are generally more portable than applications in kernel space.

Red Hat Storage is a user space application.

n-a a: Local synchronous data replication typically deployed across campus or Amazon Web

Services Availability Zones.

Geo-Rep(Replication):Long-distance replication typically deployed from one private cloud/datacenter

to another, or one cloud region (i.e. AWS Region) to another located further than a 50 mile radius of the

primary data location.

ctdb: CTDB is primarily developed around the concept of having a shared cluster le system across a ll the

nodes in the cluster to provide the features required for building a NAS cluster. http://ctdb.samba.org/

Documents

6445 an Introduction Red Hat Storage Architecture