Storage technology has enjoyed consid-erable growth since the first disk drivewas introduced nearly 50 years ago.One reason is the slow and steady evo-lution of storage interfaces (SCSI andATA/IDE). The stability of these inter-faces has allowed for continualadvances within both the storagedevices and applications, without fre-quent changes to the standards.

Ultimately, however, the interfacedetermines the functionality supportedby the devices, and current interfacesare now holding system designers back.Storage technology has progressed tothe point that a change in the deviceinterface is needed. Object-based stor-age is designed to address this problem.

Since the first disk drive in 1956,disks have grown by over six orders ofmagnitude in density and by over fourorders in performance. While the stabil-ity of the block-based interfaces hasbenefited systems, it is now becoming alimiting factor for many storage archi-tectures. As storage infrastructuresincrease in both size and complexity,the functions that system designerswant to perform are fundamentally lim-ited by the block interface.

A storage object is a logical collectionof bytes on a storage device, with well-known methods for access, attributesdescribing characteristics of the data, andsecurity policies that prevent unautho-rized access. Unlike blocks, objects areof variable size and can be used to storeentire data items, such as files, databasetables, medical images or multimedia.

Objects can be regarded as the con-vergence of two technologies: files andblocks. Files provide user applicationswith a higher-level storage abstractionthat enables secure data sharing acrossdifferent operating system platforms; butit often comes at the cost of limited per-formance due to file server contention.Blocks offer fast, scalable access toshared data; but, without a file server toauthorize the I/O and maintain themetadata, this direct access comes at thecost of limited security and data sharing.

Objects can provide the advantagesof both files and blocks. Like blocks,objects are a primitive unit of storagethat can be directly accessed on a stor-age device (i.e., without going througha server). This direct access offers per-formance advantages similar to blocks.Like files, objects are accessed using aninterface that abstracts storage applica-tions from the metadata required tostore the object.

History and statusPrimitive forms of object-based stor-

age can be found in the early works(circa 1980) of object-oriented operatingsystems (OS), including the Hydra OSfrom Carnegie Mellon and the iMAX-432 OS from Intel. These operating sys-tems used variable-size objects on diskto store all pageable entities includingprocess state, instructions, and data aswell as files. Although these operationssystems never took off, they wereinstrumental in establishing the funda-mentals of capability-based security (akey component of the object storagearchitecture).

The seminal work on object-basedstorage occurred at Carnegie MellonUniversity’s Parallel Data Lab (PDL) withthe Network-Attached Secure Disks(NASD) project. The architecturefocused on cost-effectively adding pro-cessing power to individual disks,specifically for networking, security andbasic space management functionality.

This research led to a larger industry-sponsored project under the auspices ofthe National Storage Industry Consortium(NSIC). In fact, the NSIC draft continuesto be developed in the Object-basedStorage work group of the StorageNetworking Industry Association (SNIA).

Even as standards develop, theindustry is actively implementing sys-tems using object-based storage tech-nology. IBM is exploring object-basedstorage for their next generation ofStorageTank; the National Laboratoriesand Hewlett-Packard are building thehighly scalable Lustre file system, withobject-based storage as their basic stor-age interface; and smaller companiesand startups (BlueArc, Data Direct, andPanasas) are building devices thatmake use of the Object-based StorageDevice (OSD) protocol. The Venti pro-ject at Bell Labs and Centera from EMChave also used object-based storageconcepts to implement write-oncemedia for the archival of referencedata. Both systems employ the conceptof content addressable storage (CAS) inwhich the id of an object is a unique

hash of the data contents. (Venti actual-ly uses a variable length block.)

Intelligent storage is a hot area of acad-emic research. Carnegie Mellon’s PDLcontinues to explore the use of moreexpressive interfaces between operatingsystems and storage devices. As one suchinterface, objectsenable infor-m a t i o n

exchangebetween the

device and the oper-ating system in order to achieve

better functionality and performance inthe storage system. Researchers at theUniversity of Wisconsin, Madison areexploring an alternative path: semantical-ly-smart disk systems that attempt to learnfile system structures behind existingblock-based interfaces. Many otherresearch groups are beginning to explorestorage-level intelligence as well.

Object-based storageObjects are storage containers with a

file-like interface, effectively representinga convergence of file and block architec-tures. Objects capture the benefits of bothfiles (high-level abstraction that enablescross-platform data sharing as well as pol-icy-based security) and blocks (directaccess and scalability of a switched fabricof devices). However, objects are onlyintended to serve as containers for stor-age applications (e.g., file systems anddatabases) that may implement additionalinterfaces (e.g., locking) and lookupmechanisms (e.g., directories).

An object is variable-length and can beused to store any type of data. A singleobject could even be used to store anentire file system or database; the storageapplication decides what gets stored in anobject. Unlike block I/O, creating objectson a storage device is accomplishedthrough a rich interface similar to a filesystem. And, because objects can growand shrink dynamically, the storage device

APRIL/MAY 2005 0278-6648/05/$20.00 © 2005 IEEE 31

Pushing more functionality into storage

Mike Mesnier, Greg Ganger and Erik Riedel

Object-based storage

©A

RT

VIL

LE

CO

MP

OS

ITE

: MK

C

is responsible for all internalspace management of theobject. (That is, the storagedevice maintains the alloca-tion and free-space metadatastructures, such as UNIXindex-nodes, or inodes andfree-block bitmaps.)

Objects are composed ofdata, user-accessible attributesand device-managed metada-ta. The data is stored in thedata portion of the object andis opaque to the object-basedstorage device. The user-accessible attributes describecharacteristics of the object,some will be opaque and oth-ers will not. For example, aquality of service attributemay describe latency and throughputrequirements for a multimedia object.Lastly, the device-managed metadata isany extra information (e.g., an inode)maintained by the storage device for thepurposes of managing the physical stor-age of the object.

We refer to a device that storesobjects as an object-based storage device(OSD). OSDs can come in many forms,ranging from a single disk drive to astorage controller with an array of dri-ves. OSDs are not limited to random-access or even writeable devices; tapedrives and optical media can be used tostore objects. The difference betweenan OSD and a block-based device is theinterface, not the physical media.

The most immediate effect of object-based storage is the offloading of spacemanagement (i.e., allocation and track-ing of used and free blocks) from stor-age applications (e.g., file systems anddatabases). To illustrate this offloading,consider the traditional file systemarchitecture shown in Fig. 1a.

Block-based file systems can roughlybe divided into two sections: a usercomponent and a storage component.The user component is responsible forpresenting user applications with logi-cal data structures, such as files anddirectories. It also provides an interfacefor accessing these data structures. Thestorage component maps the data struc-tures to the physical storage.

This separation of responsibilitiesmakes it easy to offload management tothe storage device, which is the intendedeffect of object-based storage. In Fig. 1b,the user component of the file systemremains unchanged; however, the stor-age management component is offloaded

(and, therefore, so is the metadata) to thestorage device. Thus, the device interfaceis changed from blocks to objects.

In the traditional block model, themanagement of the block metadata (thestorage component in Fig. 1a) is com-pletely determined by the storage appli-cation. For example, file systems haveunique ways of laying out data andmaintaining on-disk metadata structures.These dependencies make directly shar-ing data blocks among applications dif-ficult, as a priori knowledge of both themetadata structures and on-disk layoutis necessary before accessing the stor-age device. Further, sharing the devicesrequires special coordination among theapplications in order to distribute thetasks associated with space allocation(e.g., by sharing a free-block bitmap).

In offloading metadata to the storagedevice, objects remove the dependencybetween the metadata and the storageapplication. This fact makes data sharingamong different storage applications fea-sible. Storage applications may still main-tain their own indexing information (e.g.,directory metadata) to resolve an objectid from a higher-level name, such as afile or directory name (e.g., the file/foo/bar may be stored in object0xabc123). But, given this id, the objectcan be accessed in a platform-indepen-dent manner. This makes sharing dataconsiderably easier. For example, a back-up application could be handed a list ofobject ids, allowing for a more efficientphysical backup of the device.

Further, with all metadata offloaded,storage applications can now store theirstructures as single objects rather than ascollections of blocks. And, because thedevice can treat objects individually, it is

easy to set security policieson a per-object basis, similarto how files can be protectedby a file server. Flexible secu-rity policies can be set thatwill result in authorization foran entire device, a collectionof objects on the device, asingle object or even byteswithin an object.

The immediate benefits ofobject-based storage aretherefore cross-platform datasharing and application levelsecurity. An additional bene-fit results from the devicesmanaging the physical layoutof the data. As new opportu-nities arise for self-manage-ment, the devices can reorga-

nize data to improve performance,schedule regular backups and recoverfrom failures.

For example, file-level prefetchingwithin a block-based device is precludedby the fact that the device does notknow about files. Indeed, a block-basedstorage device cannot even distinguishbetween a free and a used block. In con-trast, an object-based device could easilyprefetch files (stored as objects) onbehalf of storage applications, or orga-nize files according to the order in whichthey are mostly commonly accessed.

Operating systems must supportobjects if the OSD model is to be wide-ly adopted. Fortunately, the clean sepa-ration between the user and storageoperating system components will facil-itate the change. In particular, object-based file systems will be relativelystraightforward to implement. (A filesystem needs only give up control overblock management.) Also, an operatingsystem’s I/O subsystem can be intro-duced to objects by virtue of a newclass driver, similar to those that alreadyexist for disk and tape. The referencecode from Intel Labs <www.intel.com/labs/storage> shows how both canbe done in Linux.

Data sharing interfaceThe improved data sharing of objects

is a result of both the higher-level inter-face and the attributes describing thedata being stored. The interface toobject-based storage is very similar tothat of a file system. Objects can be cre-ated or deleted, read or written, andeven queried for certain attributes — justlike files are today. File interfaces haveproven easy to understand, straightfor-

32 IEEE POTENTIALS

Block Interface

Block I/O Manager

Object Interface

File SystemUser Component

Applications

System Call Interface

File SystemUser Component

File SystemStorage Component

Block I/O Manager

Storage Device

(a) Traditional Model (b) OSD Model

Storage Device

File SystemStorage Component

Applications

System Call Interface

Fig. 1 Offloading of storage management from the file systemto an object-based storage device (OSD).

ward to standardize (e.g., CommonInternet File System (CIFS), Network FileSystem (NFS), and therefore can beshared among different platforms.

The interface can also be easilyextended with application-specific meth-ods for manipulating data within anobject, a technique referred to as activedisks. For example, a database filtercould be associated with an object, theoutput of which could be returned onsubsequent read operations. Further, anOSD could allow storage applications toestablish sessions with the device toencapsulate application-specific parame-ters such as quality of service or securityguarantees. In short, objects introduce amechanism in the storage device thatallows the device to treat storage appli-cations, and even clients, individually.

AttributesAttributes improve data sharing by

allowing storage applications to share acommon set of information describing thedata (e.g., access times). They are also thekey to giving storage devices an aware-ness of how objects are being accessed.

In most deployments, attributes will atleast contain information analogous to thatcontained in an index-node (inode), theprimary data structure used in many UNIXfile systems. An inode contains file attribut-es such as access time, size, and groupand user information. All the attributes canbe efficiently stored with the object dataand, in certain cases, interpreted by thestorage device. For example, a write oper-ation that updates the size attribute wouldbe reflected on subsequent attributerequests, making the update visible toother storage applications accessing theobject. Clustered applications could there-fore depend on the storage device tomaintain this metadata. They do not haveto delegate this responsibility to an in-band (i.e., on the data path) metadataserver that could hinder performance.

Beyond these file-like attributes, addi-tional information can be made available,such as likely patterns of access to theobject (e.g., sequentially or randomlyaccessed) or even relationships to otherobjects. For example, multimedia files ona storage device may have similar attrib-utes that will cause them to be organizedand efficiently managed as a group.

SecurityStorage security does exist at the

device and fabric level for Storage AreaNetworks (SANs). For example, devicesmay require a secure login and switches

may implement zoning. However,objects make it possible to partition largedevices or fabrics into multiple securitydomains whose access policies are indi-vidually determined by the storage appli-cations. In the object based securityarchitecture, every access is authorized.The authorization is done without com-municating with a central authority thatmight slow the data path. See Fig. 2.

The fundamental building block in anobject-based security system is a crypto-graphically strong capability that containsa tamper-proof description of the rights ofa client. This capability represents thesecurity policy. It is created out-of-band(i.e., off the main data path) by an objectmanager that manages the storagedevices and grants access to clients. Whilein possession of this capability, the clientcan access the storage device. It is the jobof the storage device to validate theintegrity of the capability to ensure thatneither it nor the request has been modi-fied. The OSD therefore provides only themechanism for enforcing security. Theactual policy is set by the storage applica-tion. Separating policy from mechanism iskey to building a scalable security archi-tecture. In particular, not having to main-tain client-specific authentication informa-tion on the device means that the storagedevice will scale independently from thenumber and types of clients in the system.

While an OSD does not question theauthenticity of the client, it does needsome way to validate the integrity of thecapability, or proof that the object man-ager granted access to the client.Providing this guarantee requires thatthe object manager and device share asecret that can be used in creating asecure hash of the contents of the capa-bility. Before granting a client its capabil-ity, the manager will first create a keyedhash of the capability. It uses the secretas the key. It will then return both thesecure hash, referred to as a capabilitykey, along with the capability to the

client. The client is expected to use thiscapability key in creating its own, keyedhash of every request sent to the OSD.This hash protects the command fromundetected modification. This is similarto how the hash of the capability pro-tects the capability from modification.

The request sent to an OSD includesthe command, the client capability anda signature (or digest) of the entirerequest. Upon receipt of a new request,the OSD must first validate the clientdigest. The OSD will create its owndigest of the request and compare thiswith the digest sent by the client.

(Note: It is not immediately obvioushow this is possible when the storagedevice does not possess the capabilitykey used to generate the digest.However, recall that the capability key isjust a hash of the client’s capability(which was sent in the request) and thesecret shared between the object manag-er and the OSD. So the OSD can simplyregenerate the capability key and use thiskey to generate its digest of the request.)

If the digests match, the OSD is guar-anteed that neither the capability norany of the arguments in the requestwere modified. Had either of thesechanged, the digest generated by theOSD would differ from that sent by theclient. The OSD then would reject therequest. All responses sent from theOSD to the client can be protected usinga digest similar to that sent by the client.

IntelligenceWith object-based storage, there is

the potential for storage devices toactively learn important characteristicsof the environments in which theyoperate. Storage devices today arelargely unaware of the users and stor-age applications actually using the stor-age. This is because block-based stor-age devices manage opaque datablocks. With objects, storage devicescan understand some of the relation-

APRIL/MAY 2005 33

Clients

Managers

Network

Cap.

Management

Metadata

Object-based Storage Devices

Data

Secret

Fig. 2 This figure illustratesthe object-based storagesecurity architecture. Objectmanagers grant capabilitiesto clients; clients presentthese capabilities to thedevices on every I/O. Secretsshared between the managerand the storage devices areused to generate a keyedhash of the capability,thereby protecting it frommodification.

ships among the blocks on the device.They can use this information to betterorganize the data and anticipate needs.

Object attributes can contain staticinformation about the object (e.g., cre-ation time), dynamic information that isupdated on each access (e.g., last accesstime), information that is specific to astorage application and uninterpreted bythe device (e.g., filename, group, oruser ids), and information that is specificto a current user (e.g., Quality of Service(QoS) agreement). Attributes may alsocontain hints about the object’s behaviorsuch as the expected read/write ratio,the most likely patterns of access (e.g.,sequential or random), or the expectedlifetime of the object. Having access tosuch attributes enables a storage systemto better organize and serve the data.

Using objects to better manage storageis an active area of academic research.One major question to be addressedrelates to the attributes themselves.Specifically, which attributes are most use-ful in classifying the behavior of objects?

Past research has already shown thatfile attributes play an integral role indetermining file behavior. For example,the name of a file can be used to pre-dict how the file may be accessed.

Many parallels exist for object-basedstorage, as an object’s attributes may alsogive hints as to how it will be used. Forexample, the creation time attribute ofan object may be useful in classifyinghow large the object is likely to be: sup-pose that a database makes regularbackups every morning between 2 and 3am, and stores the backup as a singleobject. A storage device, over time,could learn to recognize this pattern andoptimize accordingly.

In general, if storage devices canbecome aware of the environments inwhich they operate, they could betterallocate and provision resources.Further, with increased knowledge ofthe storage and user applications, stor-age devices could perform application-specific functions, thereby making theSAN a computational resource. Indeed,storage devices are themselves comput-ers, with processors, network connec-tions and memory. Through moreexpressive interfaces, these resourcescan be more effectively exploited.

SummaryWe are now at a point where contin-

ued progress requires a change in thestandard block-based device interface. Theobject interface offers storage that is more

secure and easier to share across plat-forms. What’s more, it performs well there-by eliminating the common trade-offbetween files and blocks. Objects providethe storage device with an awareness ofthe storage application and enable moreintelligence in the device. Replacing blockinterfaces with objects is a step in the evo-lution toward more self-managing systems.

AcknowledgementsIEEE, OSD TWG, reviewers

Read more about it• Storage Networking Industry

Association, Object-based Storage Devices(OSD) work group. http://www.\snia.org/osd.

• G. Almes and G. Robertson. Anextensible file system for HYDRA. ThirdInternational Conference on SoftwareEngineering (ICSE). (Atlanta, GA). IEEEComputer Society, May 1978.

• P. Braam. The Lustre Project. http://projects.clusterfs.com/lustre.

• G. R. Ganger. Blurring the linebetween OSs and storage devices. Technicalreport CMU–CS–01–166. Carnegie MellonUniversity, December 2001.

• G. A. Gibson, D. F. Nagle, K. Amiri,J. Butler, F. W. Chang, H. Gobioff, C.Hardin, E. Riedel, D. Rochberg, and J.Zelenka. A cost-effective, high-bandwidthstorage architecture. Architectural Supportfor Programming Languages andOperating Systems (San Jose, CA, 3–7October 1998). Published as SIGPLANNotices, 33(11):92–103, November 1998.

• H. Gobioff. Security for a high per-formance commodity storage subsystem.PhD thesis, published as TechnicalReport CMU–CS– 99–160. School ofComputer Science, Carnegie MellonUniversity, July 1999.

• IBM. Storage Tank. http:// \www.almaden.ibm.com.

• Intel. Internet SCSI (iSCSI) ReferenceImplementation. http://www. intel.\com/labs/storage.

• M. Mesnier, E. Thereska, D. Ellard, G.Ganger, and M. Seltzer. File classificationin self-* storage systems. IEEE FirstInternational Conference on AutonomicComputing (ICAC-04) (New York, NY,17–18 May 2004). IEEE, 2004.

• F. J. Pollack, K. C. Kahn, and R. M.Wilkinson. The iMAX–432 object filing sys-tem. ACM Symposium on Operating SystemPrinciples (Asilomar, Ca). Published asOperating Systems Review, 15(5):137–147,December 1981.

• S. Quinlan and S. Dorward. Venti: anew approach to archival storage.

Conference on File and StorageTechnologies (Monterey, CA, 28–30January 2002), pages 89–101. USENIXAssociation, 2002.

• E. Riedel, G. Gibson, and C.Faloutsos. Active storage for largescaledata mining and multimedia applications.International Conference on Very LargeDatabases (New York, NY, 24–27 August,1998). Published as Proceedings VLDB,pages 62–73. Morgan KaufmannPublishers Inc., 1998.

• M. Sivathanu, A. C. Arpaci-Dusseau,and R. H. Arpaci-Dusseau. Evolving RPCfor active storage. Architectural Support forProgramming Languages and OperatingSystems (San Jose, CA, 05–09 October2002). Published as Operating systemsreview, 36(5):264–276, 2002.

About the authorsMike Mesnier (mmesnier@ece.cmu.

edu) has been with Intel since 1997. Hisinterests are systems related, primarilyfile and storage technologies. Mikereceived his Masters in computer sci-ence from the University of Illinois andis currently a PhD candidate at CarnegieMellon. Prior to attending CMU, Mikewas a co-chair of the Object-basedStorage Devices work group in theSNIA; he helped lead the industry to aSCSI standard for object-based storage.

Greg Ganger [M] (greg.ganger@cmu.edu) is director of the CMU Parallel DataLab (PDL), academia’s premier storagesystems research center, and an associateprofessor in the ECE department atCarnegie Mellon University. He has broadresearch interests in computer systems,including storage systems, operating sys-tems, security, networking, and distrib-uted systems. He received his Ph.D. incomputer science and engineering fromthe University of Michigan. He is a mem-ber of the ACM.

Erik Riedel (erik.riedel@seagate.com)leads the Interfaces and ArchitectureDepartment at Seagate Research, Pitts-burgh, Pennsylvania. His group focuseson novel storage systems with increasedintelligence for optimized performance,automated management, and content-spe-cific optimizations. Before joining SeagateResearch, he was a researcher at Hewlett-Packard Laboratories. He received hisPh.D. in computer engineering fromCarnegie Mellon University for his workon Active Disks, an extension to NASD.His interests include I/O in a number ofareas, including parallel applications, datamining, databases, file systems, and scien-tific data processing.

34 IEEE POTENTIALS