AB Periasamy | CTO Gluster, Inc.Thu 30 June 2011

GlusterFS ArchitecturePetascale Cloud Filesystem

"userspace filesystem"?

People who think that userspace filesystems are realistic for anything but toys are just misguided. Linus Torvalds

GlusterFS is safe for kids. Misguided Toy Maker

Fast Growing Commercial Deployments

Benchmark - 64 bricks with ib-verbs transport

Servers: 64 bricks clustered storage servers / bricks Clients: Cluster of 220 servers Interconnect: 10 Gbps InfiniBand interconnect; ib0verbs transport protocol Method: 220 clients pounding the storage servers with multiple dd (disk-dump) instances Size: Each clients reads / writes a 1 GB file with 1MB block size. 32 GBs

Typical Gluster Deployment

=N bricks = N x Performance & Capacity

+

+

MS Windows Client

GNU/Linux Client NAS Storage - POSIX Compatible

# gluster peer probe HOSTNAME

# gluster volume info

# gluster volume create VOLNAME [stripe COUNT] [replica COUNT] [transport tcp | rdma] BRICK

# gluster volume delete VOLNAME

# gluster volume add-brick VOLNAME NEW-BRICK ...

# gluster volume rebalance VOLNAME start

GlusterFS Simple Commands

Evolution of GlusterFS

2006-2009 GlusterFS v1.0 v3.0 Distributed Filesystem capabilities with self-healing, synchronous replication, stripe, distribute (global name space)

2010 GlusterFS v3.1Elastic Cloud capabilities

2011 Q2 GlusterFS v3.2GeoGraphic replication, Enhanced monitoring, Directory level quotas (also works as cloud usage billing APIs)

2011 Q3/Q4 Hadoop HDFS drop-in replacement, Unified File and Object Storage (Amazon S3 compatible) and Near CDP.

Gluster Architecture Advantages

Software only

No metadata server

Fully distributed architecture, no bottleneck

Gluster Elastic Hash

High performance global namespaceScale out with linear performance

Hundreds of petabytes

1 GbE, 10GbE

High availabilityReplication to survive hardware failure

Self-healing

Data stored in NFS-like native format

Stackable userspace designNo kernel dependencies, simple install

Match specific workload profiles

Early maturity and rich functionality

Google Storage for EveryoneIntelligence in the SW

Leverage commodity HW

Scale-out elastically

Replication for reliability

Software enables virtualization

GlusterFS FUSE Architecture

VFS

Kernel

Userland

GlibcApplications

FUSEGlusterFS

Kernel

Userland

GlusterFSDisk FS(ZFS / Ext3 / XFS)

TCP, RDMAIB / 10GigE / 1GigE

volume photo-store type storage/posix option directory /data/exportend-volume

volume server type protocol/server option transport-type tcp/server option auth.ip.brick.allow * subvolumes photo-storeend-volume

Volume Specification

work.odscorporate.odpdriver.cbenchmark.pdftest.ogginitcore.cmylogo.xcfdriver.cether.ctest.m4aDistribute Volume

accounts-2007.odsbackup.db.zipaccounts-2006.odsaccounts-2007.odsbackup.db.zipaccounts-2006.dbaccounts-2007.odsbackup.db.zipaccounts-2006.odsReplicated Volume

north-pole-mapdvd1.isokvm-imagenorth-pole-mapdvd1.isokvm-imagenorth-pole-mapdvd1.isokvm-imageStripe Volume

BRICK1

BRICK2

BRICK3

Stripe Volume

Block 0Block 1Block 2Block 3Block 4Block 5Block 6Block 7Brick-A

Brick-B

Brick-C

Brick-D

Thank You

www.gluster.org

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text stylesSecond level

Petascale Cloud Filesystem

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Petascale Cloud Filesystem

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Petascale Cloud Filesystem

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level