Object-based Storage

Object-based Storage

Long Liu2010-10-23

Outline

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

Outline

Background

• Existing enterprise storage infrastructures are feeling the strain

• the volume of data generated by many network-based applications continues to escalate

BackgroundTwo Technologies:

• Files:– Flexible data sharing– Secure

• Blocks:– High performance– Scalable

Blocks

Files

Comparison

Block Based Disk Object Based Disk

Operations: Read block Write block

Addressing: Block range

Operations:Create object Delete object Read objectWrite object

Addressing: [object, byte range]

Objects can be regarded as the convergenceof two technologies: files and blocks

Comparison

Object Interface

File System

Storage component

ApplicationsApplications

Logical Block Interface

File System

Traditional Storage Object-based Storage

Hard Drive Object-based Storage Device (OSD)

Comparison

File SystemStorage Component

Block I/O Manager

Storage Device

CPU

Applications

System Call Interface

File SystemUser Component

File SystemStorage Component

Block I/O Manager

Storage Device

CPU

Applications

System Call Interface

File SystemUser Component

Object Interface

Block Interface

Two changes :Object-based storage offloads the storage component to the storage deviceThe device interface changes from blocks to objects

(a)Traditional

model

(b)Object storage

model

Motivation

Objects

•Improved device and data sharing – Platform-dependent metadata moved to device

• Improved scalability & security– Devices directly handle client requests– Object security

• Improved performance– Data types can be differentiated at the

device• Improved storage management

– Self-managed, policy-driven storage– Storage devices become more autonomous

Outline

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

Object-based Storage

Object

OSD(Object-based Storage Device)

MDS(Metadata Server)

Object-based Storage

Object

OSD(Object-based Storage Device)

MDS(Metadata Server)

Object

•An object is a logical unit of storage— ID (Identification)— Application data— Metadata which includes block allocation

and length— Attributes that is accessible by users

•Objects have file-like methods— open, close, read, write

Object

•The root object -- The OSD itself•User object -- Created by SCSI commands from the application or client•Collection object -- A group of user objects, such as all .mp3•Partition object -- Containers that share common security and space management characteristics

Object

P1

P2

P3

P4

User Data

AttributesObject ID

Metadata

U1

User Objects (for user data)

Collection Objects

Partition Objects

Root Object(one per device)

OSD

Object-based Storage

Object

OSD(Object-based Storage Device)

MDS(Metadata Server)

Object Storage Devices

Intelligence with its own CPU, Memory, Disk, Network

Redistribution with specific strategy

Objects’ metadata management

File SystemStorage Component

Block I/O Manager

Storage Device

CPU

Applications

System Call Interface

File SystemUser Component

File SystemStorage Component

Block I/O Manager

Storage Device

CPU

Applications

System Call Interface

File SystemUser Component

Object Interface

Interface

Two changes :Object-based storage offloads the storage component to the storage deviceThe device interface changes from blocks to objects

(a)Traditional

model

(b)Object storage

model

Object Storage Devices

Object Storage DevicesExpect wide variety of Object Storage Devices

Disk array subsystem

Stores up to 5 TBs per shelf

2 SATA disks – 240/500 GB Highly integrated, single disk

4 Gbps pershelf to cluster

Orchestrates system activityBalances objects across OSDs

Object-based Storage

Object

OSD(Object-based Storage Device)

MDS(Metadata Server)

Metadata Server

Control the interaction between clients and OSDs

Client

Metadata Servers (MDS)

File System

Applications

Storage component

Object-based Storage Device (OSD)

Metadata Manager

Object Storage Devices (OSDs)

Client

Dataflow of Metadata

Outline

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

T10 OSD Commands(face)

CR

EA

TE

/RE

MO

VE

RE

AD

/WR

ITE

GE

T/S

ET

AT

TR

Object

10001110101100000011101100111011110001111000

..…User Data

Opaque

attributes(stored only)

Shared

attributes

(stored &

processed)

Attribute pages

OSD Target

Interface

OSD Client

OSD Interface

OSD Commands(Interface)

• Basic Protocol– READ– WRITE– CREATE– REMOVE– GET ATTR– SET ATTR

• Specialized– APPEND– CREATE & WRITE– FLUSH– LIST

• Security– Authorization– Integrity– SET KEY– SET MASTER KEY

• Groups– CREATE COLLECTION– REMOVE COLLECTION– LIST COLLECTION

• Management– CREATE PARTITION– REMOVE PARTITION– FLUSH PARTITION– PERFORM SCSI COMMAND– PERFORM TASK MGMT

very basic

space mgmt

attributes

shared secrets

• opaque• internal• shared

Storage Technology Today

• Direct attached storage (DAS)• Fabric Attached Storage (FAS)

-Network Attached Storage (NAS)

-Storage Area Networks (SAN)

1

2

3

4

Direct Attached Storage

BACKUP SERVER

TAPE

RAID

RAID

RAID

RAID

Windows

UNIX

LANWindows

A traditional Direct Attached Storage

model

Fabric Attached Storage

CLIENTS

SERVER

DATA

C/SFabric Attached

Storage

Network Attached Storage

Clients

File I/O

IP network

File server

Block I/OStorage area network

Block storage

This figure illustrates NAS being used to share files among a numberof clients.

The files themselves may be stored on a fast SAN

Storage Area Networks

Clients

Metadata

Data

Block-based storage devices

Storage area network

Management

Servers

This figure illustrates a SAN file system

The files themselves are stored on a fast storage Area to which the clients are also attached.

Security

NETWORK

Management

METADATA SERVERS

Attribute

Capability

CLIENTS

Data

Capability

Attribute

OBJECT-BASED STORAGE DEVICE

Outline

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

Status

• Industrial– Lustre– Panasas

• Academic

Status

• Industrial– Lustre– Panasas

• Academic

Lustre

• First open sourced system with object storage• High-performance parallel file system• Consist of clients, MDS and OST(Object

Storage Targets)

Lustre

NETWORK

MetadataData

Data Management

ManagementMetadata

METADATA SERVERCLUSTER

CLIENTS

OBJECT STORAGE TARGETS(OST)

Panasas

• High-performance file system• Consist of OSD, Panasas File System, MDS

Panasas

Key Object Storage FeaturesIntelligent space management in storage layer

ßMedia geometry aware placement

ßData aware prefetching, caching & recoveryEncapsulation of data and attributes

ßNative object interface, good programming model

ßStorage interpreted attributes for per file properties

Key Object Storage AdvantagesßRobust, shared access by many clients

ßScalable performance via an offloaded data path

ßStrong fine-grained end-to-end security

Panasas

• Clients are from Energy, Government, Finance, Manufacturing and Higher Education

Status

• Industrial– Lustre– Panasas

• Academic

A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Motivation:

• Metadata server cluster maybe the bottleneck• Frequent metadata access and movement• Terrible load balance management

VoDServer

WebServer

DatabaseServer

E-mailServer

FileServer

Storage Network(Fibre Channel)

DataMDS

Cluster

Security

Metadata

Application Server Cluster

Object-based Storage Device Cluster

Object-based Storage System Architecture

A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Application Servers

ApplicationFile Hashing Manager

Mapping Manager

Metadata Server Cluster

Hashing Partition

Logical Partition Manager

Metadata Server Backend

Common Storage Space

Figure 3. Hashing Partition

A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Hashing Partition:

A total solution for – File hashing– Metadata partitioning – Metadata storage

A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Pathname: /Dir1/Dir2/filename

Mapping Manager

Pathname HashingResult (i)

PathnameMetadata

&etc

Pathname HashingResult (i+1)

PathnameMetadata

&etc

Metadata Server Cluster

Logical Partitions

1 4

2

3

Figure 4. Metadata Access Pattern

①.Filename hashing②.Selecting MDS through Mapping Manager③ .Accessing metadata by pathname hashing result④.Returning metadata to application server

Hashing Partition

Mapping Manager

2

1 3Metadata Server Cluster

Logical Partitions

Common Storage Space

4

Figure 5. MDS cluster failover procedure

A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Outline

• Why do we need object based storage?• What is object based storage?• How to take advantage of it?• What's the status of object based storage?• What can we do about it?

From Our Perspective

The EndThank you