NeST: Network Storage

Flexible Commodity Storage AppliancesJohn Bent, Miron Livny, Andrea Arpaci-Dusseauand Remzi Arpaci-Dusseau

TermsAppliance (Merriam-

Webster) b : an instrument or device

designed for a particular use; specifically a household or office device

Storage appliance Storage plus access methods

What storage users wantReliability and availabilityManageability

cost of management > cost of storage itself

“no futz” computingScalabilityPerformance

What storage vendors haveNetApp, EMC, others make storage

appliances (network-attached storage)Manageable

Just plug it in and it works Administrative web interface

Reliable and available Standard RAID techniques

High performance Specialized, thin OS focused on serving files

What storage vendors get,annual revenues

NetApp

$800 million in 2000

EMC

$9 billion in 2000

What’s the problem?False coupling between HW and SW“Playground syndrome”Myth of specialization

H/W and S/W are bundled Hardware decisions are imposed Hard to ride commodity curve

Example:Netapp F720

• $35,000.00, 252 GB• $138 / GB

Maxtor DiamondMax• $279.00, 80 GB• $3.50 / GB

“Playground syndrome”“We have storage appliances . . .

if you use these protocols, if you use these security mechanisms, if you are comfortable with our data

semantics”Non-flexible software entity

Myth of specializationSpecialize for one protocol on one machineSpecialization decreases over time as

Protocols are added Product line expands

Example: Netapp software Generation 1 fit on a single floppy Generation 2 took six Generation 3?

Alternatives?Appliance (Merriam-

Webster) a : a piece of equipment for

adapting a tool or machine to a special purpose

Our game?Flexible, commodity based, software-

only storage appliancesGoal

Find a networked machine “Drop” some software on it Have a ready to use storage appliance

with flexible mechanisms

New worlds, new problemsDiverse hardware, software platforms

Netapp, EMC advantagefewer platforms, control over OS

Our approachAutomate configuration to each host system

• Hardware example - use file system or self-manage• Software example - use either read/write or mmap

Cost of flexibilityKey is design of the software

OutlineIntroductionBuilding flexible storage modules

Big picture Protocol layer Concurrency architecture Storage layer

Motivations for flexible storage appliancesConclusion and current status

NeST structureCleanly separated modules for

communication, transfer and storage Protocol layer

Maps diverse protocols into common control flows Concurrency architectures

Different models to maximize system throughput Storage layer

Provides abstract interface to disks

NeST structure

Protocol Layer

GFTP NeST WiND HTTP NFS

CentralControl

Concurrency ArchitectureEvent driven

Multi-process

Multi-threaded

Transfer request

Storage Layer

Raw disk Local FS RAID

Protocol layerA collection of servers is less than the sum of their parts.

HTTPdNFSd

Operating system Operating system

NeST

HTTPNFS

Consolidate protocolsSingle point of control

Storage quotas and guarantees can be supported across multiple protocols.

Bandwidth can be controlled and quality of service can be guaranteed.

Single administrative interface Set policies Manage user accounts

Protocol layer implementationEach protocol listens on well-defined

portCentral control accepts connectionsProtocol layer reads from connection and

returns generic request objectLike Linux V-nodes

Add new protocol by writing a couple of methods

Protocol layer example,directory list request

FTP

NeSTspeak

Protocol layer

“31: LIST”

“5”

Central controlDirectory list

Storage layer

Directory list

Linked list

Linked list

“ftp, ftp, ftp”

“nest, nest”

Concurrency architectureThree difficult goals

Low latency High bandwidth Multiple simultaneous clients

No single portable solution Provide multiple models to provide solutions on a

range of different platformsMulti-threadedMulti-processEvent driven

Concurrency architecture

Concurrency architectureEvent driven Multi-process Multi-threaded

Central control creates transfer object Socket descriptor from the protocol layer File descriptor from the storage layer

Transfer object passed to concurrency architecture

Concurrency on Linux

Storage layerThree needed areas of flexiblity

File systems interfacesExample: read()/write() or mmap()

Abstract storage modelsRAID, JBOD, etc.

User account administrationCreation and removalQuotas and guarentees for users and groups

File system interfaces on Linux

OutlineIntroductionBuilding flexible storage modulesMotivations for flexible storage

appliancesConclusion and current status

Clients have different needsCommunication protocols Replacement costsData semanticsSecurity and authentication

Communication protocolsThe Esperanto problemToo many protocols to implement them

allToo many clients use proprietary

protocolsStorage must allow pluggable protocols.

Replacement costsInfinite cost to replace first class data.Variable cost to replace cached data

depending on size and distance.Variable cost to replace job output files

depending on computation cost.

Cheap cached files First class data

Cost aware storage can effectively increase its own capacity.

Data semanticsMust stored objects be protected

from read and write dependencies?Is transaction support necessary?Acceptable replies to storage

requests.

Data semantics, exampleProblem

PFS on top of FTP fakes open read may then return file not found

Solution Mechanisms are needed to support flexible

semantics independent of the transfer protocol.

Divorce semantics from the protocol.

Security and authenticationOwnershipPrivacyEncryptionAuthenticationAccess rights

Who, when, how and how much?

Abstinent Promiscuous

Who is allowed to use the storage?Promiscuity and monogamy are easyPolygamy is also easy

Do I know you?

Problem Migrant grid users may need temporary,

preferential storage accessSolution

Provide mechanisms to advertise available storagecreate self-destructing user accounts

Matchmake applications with storage opportunities.

OutlineIntroductionBuilding flexible storage solutionsMotivations for flexible storage

appliancesConclusion

Current status Future work Concluding remarks

Current statusConcurrency architectures are done

Gets, puts, reads and writes perform wellVirtual protocol class interface is built

NeST speak is fully implemented Grid ftp coming soon!!

Simple first implementation of storage reservations and remote quota management is done Venkateshwaran Venkataramani

Future workDiscovery process of client storage

requirements Quality of service guarantees for

bandwidth and storageSupport for transient and

opportunistic users

Concluding remarksReturn storage to the commodity

curve by creating software-only storage appliances

Allow greater storage flexibility for a wide range of application needs