CSG Storage Survey Results1 CSG Storage Workshop

CSG Storage Survey Results 1

CSG Storage Workshop


Workshop Team

• Bruce Vincent (Stanford)• Scotty Logan (Stanford)• Dennis Cromwell (Indiana)• Ron Thielen (U Chicago)• Kitty Bridges (Michigan)• Additional presenters:

– Cory Snavely (Michigan)– Jim Pepin (USC, en route to Clemson)


Agenda• 1-1:20 Survey Results (Kitty)• 1:20-2:15 Framing the discussion: definitions,

emerging technologies, policy, architecture (Ron)

• 2:15-2:45 Case Study: Research storage (lots of small files, as in genomics) (Jim)

• 2:45-3 BREAK• 3-3:30 Case study: library archive storage

(Cory)• 3:30-4 Case study: research storage (lots of

big files, as in astronomy, physics) (Dennis)• 4-4:45 Case study: tiered storage architecture,

virtualization (Ron, Scotty)


Survey Results


Institutions Responding (19)

• Brown• Carnegie Mellon• Columbia• Duke• Georgetown• Harvard• Indiana• MIT• NYU• Penn State

• Princeton• Stanford• University of California• University of Chicago• University of Colorado• University of Michigan• University of Virginia• University of Wisconsin• Virginia Polytechnic

Institute and State University


File Service Offered

Distributed file service (AFS, DCE, DFS, WAFS)

42%

Microsoft DFS 11%

CIFS 53%

NFS 47%

WebDAV 47%

Xythos 26%

Other 37%


File Service Quotas<100 MB

101-300 MB

301 MB-1 GB

1 GB- 10 GB

>10 GB

Students 4 4 2 8 0

Faculty 2 2 3 8 1

Staff 2 2 3 8 1

Other individuals

2 2 2 6 0

Dept’s 2 2 3 8 2

Courses 2 2 3 5 3

Research groups

2 2 2 5 3


How Many Staff Manage It?

Separate groups? Yes: 74% (14) No: 26% (5)

1 to 3 staff 11

4 or 5 staff 6

6 or more 2


Staffing

Do you have problems recruiting and retaining skilled storage staff?

Yes= 47.4% (9) No= 52.6% (10)


Mitigating these problems?• Adjusted the salary levels for storage staff to reflect the

market.• Hardware and Software Consolidation, automation/tools,

delegation/self-service• By keeping things simple (e.g. no SAN requirements, etc.),

but it is unclear how this will scale.• Standardize on more current mainstream technologies and

move away from technologies with a strong historical legacy but not a significant current relevance such as AFS.

• We're currently using some contractors, and more training for our existing staff, but we're still looking for ideas.

• Take longer to find people who love storage and that is what they want to do as a career.


Mitigating these problems?• Staff members with good storage management skills are

just one of the areas where we worry about finding and keeping talent. It is difficult for us to match corporate compensation, but we find that money is not the ultimate decision maker. Training is important and we are diligent about keeping a training budget. We allocate $1400 per FTE in the organization. This is pooled and allocated as appropriate, so it gives us some flexibility to keep staff skills fresh and to build depth in critical areas. We also like to keep work interesting, which is very important to highly motivated technical staff. In addition, we recognize outstanding work with both small monetary bonuses and publicity. Still, we recently lost our best SAN person.


Client replication technologies, (MS Offline Files & Folders or Apple's

Portable Home Directories?

Yes=12% (2) No=88% (15)


Block storage (11 using, considering)• Considering iSCSI• Don’t provide block storage direct to end-users. We think we

can do it via iSCSI but haven't really done much than ponder the possibility.

• SAN and NAS, as well as DAS• Provide block storage to major campus "services" through EMC

software & the EMC DMX & Clarion storage systems.• Fiber channel primarily iSCSI planned for future• EMC DMX and Clariion• Fibre Channel SAN available to servers hosted in data center• Block storage is only provided in the data center • NAS• NFS, iSCSI and SAN attached.• SAN


Storage in Data Center• EMC Clarion and DMX, HP EVA 8000 and 5000, also use CISCO directors

for the SAN• Network Appliance• EMC, NetApps• Almost exclusively EMC; currently researching other vendor's offerings

(SUN, Hitachi and Apple)• IBM, Sun/Hitachi• EqualLogic for iSCSI ACNC for direct attached• Network Appliance, SUN, Apple, Hitachi, IBM• IBM FastT and Shark HP SANStorage• Network Systems: EMC CX700 Storage Array for Databases, FileServices,

Exchange and Backup Storage Brocade Silkworm Switches Storage TEK L180 Tape Library; Administrative Computing: Storage Tek V2X Storage Array for mainframe, AIX Oracle systems Brocade Silkworm Switches Storage Tek Timberwolf Tape Library

• EMC DMX, Clariion HP EVA Equallogic iscsi


Storage in Data Center• Cisco MDS 9000 series switches, Brocade Silkworm, EMC Clariion, DMX

and Sun• EMC DMX and Clariion storage arrays, EMC NS704G NAS gateway,

Nexsan ATAbeast and SATAbeast storage arrays, IBM and StorageTek tape libraries, Cisco and Qlogic Fibre Channel directors and switches Emulex and Qlogic Fibre Channel HBAs

• NetApp, Dell/EMC SAN, Sun/Hitachi SAN, StorageTek SAN, IBM SAN• EMC DMX, HDS USP Tagmastore, IBM DS4800, Sun 5320 NAS Gateway• IBM, Sun, and Network Appliances for central services• Primary SAN for Enterprise systems will be an Hitachi Tagmastore,

moving from an IBM Shark. We have a mix of Direct attached storage, SAN and NAS for other areas inlcuding EMC and HP.

• Primarily Sun storage (w/ Brocade switches) for Unix platforms. Windows environments typically have direct attach storage on file servers from Dell.

• IBM mid-range SAN disk, Brocade switches, IBM tape robotics with 3592 drives, NetApp filers, Linux and Windows servers, IBM SAN Volume Controller for virtualization


Remote Access

0

2

4

6

8

10

12

native client VPN WebDAV Other


Underlying Tech for Service

0

2

4

6

8

10

12

14

16

File Service Databases WebPublication

Other

Direct attached storage

Network attached storage

Storage attached network (SAN)with iSCSI

Storage attached network (SAN)with Fiber channel


Specialized Storage for HPC?

Yes= 37% (7) No= 63% (12)

NetAppSort of, we provide Network Appliance filer space to our research clusters.EMC DMX2000 and DMX800 via Cisco SAN FabricBlueArc (and other dedicated NFS servers)IBM SANIBM DS4800


If one backend technology for all storage (7 reporting)

• Moving into that direction - virtualized storage - but not there yet. We use the IBM SAN Volume Controller (SVC) on the hDSAN side to virtualize across the EMC and HP storage arrays.

• Network Appliance Filers• Some is on direct attached storage ( a case by case choice by

the user / service) but the vast majority is on EMC storage over a fiber channel SAN

• SAN technology with an EMC Clariion storage array that provides shared access to centralized storage.

• EMC DMX and Clariion FC storage presented via FC McData switches, and CIFS via EMC NS704G NAS gateway. Expect to expand to NFS and CIFS in future.

• HDS Tagmastore USP 1100


Backing it up

TSM 12 (was 13)

TSM and replication 3

Legato 2

Atempo 1

Amanda 1

Veritas 1

Disk-based MAID technology 1


Meets Needs?

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Unacceptable Poor OK Pretty good Excellent


Backing up Mobile Devices?• mobile devices = PDAs - no. mobile devices

= laptop - yes, moving to LiveBackup from Atempo.

• Connected desktop backups• TSM (3)• Support for these devices is provided

through a server that is backed up with our Backup Services. The only supported application is based on MS Exchange and utilizes existing mailboxes already on the servers and being backed up


Desktop Backup

0

1

2

3

4

5

6

7

8

9

connected tsm atempo veritas

We are investigating providing desktop and laptop backup and have made a short list of vendors in this space. Included are: Atempo, Asigra and Iron Mountain


Tiered Solution (17)• TSM (4)

– All of our (3) tiers of storage use the same core back up technology (TSM) which allows individual customizations of timing and overall strategy

– Multiple tiers of disk storage, but predominantly use TSM clients on the hosts for backup. A few exceptions for our large ERP systems where we clone the volumes storing the database and back them up from a dedicated server to minimize the backup window

– TSM, NSR, IBM's "flashcopy" and NetApp's snapshots– IBM Tivoli Storage Manager policy domains, management

groups, and storage groups allow us to implement numerous combinations of storage tiers and timing strategies for different requirements.


Tiered Solutions• Use the traditional Unix methodology of daily, weekly, and

monthly incremental backups.• See detailed slide deck (Walter Wong from CMU)• We haven’t done any tiered data with backups, we do however

utilize different disk solutions based on the service requirements. For Example: Exchange: We use only Fiber Channel 15000 RPM Drives File Servers: We use Raid 5 ATA Drives Backup 2 Disk: We use Raid 3 ATA Drives

• 2 tiered storage based on perf/avail. No backup distinction yet.• Use Veritas Netbackup for large backups that require a small

window to complete because of the technology Vertias provides with support for multiple streams we can complete backups more quickly than we can with TSM.

• All issues we need to consider though!


Separate Archive Service• Operate a ADIC StorNext HSM for research data that sort of fits

this mission.• TSM (5)

– We currently do minimal archiving using a function in TSM and store both to tape and to disk.

– We provide archive via IBM Tivoli Storage Manager for servers and services hosted by IT Services.

– Do not provide a separate archive service and currently use the archive component of both TSM and Legato

• We do some end-of-semesters that are archived for 5 yrs.• Minimal archive service for CIS internal use only, using an archive

to tape with Legato Networker. We see archival strategies as a means to increasing usable storage and will be looking into other application and file system archive solutions in the near future. Mainframe environment: monthly archive of unchanged files to tape.


Library Archives Service (19)

Library doesn’t provide service = 68.4% (13)

Library alone provides service = 26.3% (5)

Library provides a service in partnership with central IT = 5.3% (1)


Considered Outsourcing?• We have a partnership with the Duke Health System. They have a large

SAN for health care data that we leverage for top tier storage.• Twice researched the option and rejected it due both to cost and loss of

local flexibility.• Have looked at outsourcing backup twice. Both times the cost benefit was

not evident. When you assess in detail the breakdown of roles/resp breakdown between customer and provider, there is still a significant resource commitment on the customer side.

• Considered outsourcing our desktop backups. We had issues with lack of support for non-Windows platforms, some support and bandwidth problems, and no way to take a single charge from the vendor and break it out to re-bill our users.

• This topic brings perceived security/privacy concerns to the surface; a cost/benefit analysis will be interesting; outsourcing may fit as a piece of the life cycle management of information; we are just at the early stage of "consideration."

• Considered it but found we could do it cheaper by centralizing all of campus


BC/DR Impact• Increased the storage requirements and increased the importance of

the storage strategy. A driving factor in our desire to explore a more centralized enterprise storage architecture.

• Customers are requesting shorter RPO's and RTO's and in some cases are willing to accept longer RTO's if RPO can be improved. Causing us to more seriously assess replication, quasi single instance stores, virtualization etc. Pushed us toward service mirroring, replication – For top-tier storage, the SAN is in a different location than the ATLs. TSM

and offsite tape storage is also used to provide recovery capabilities.– Working on a project with Virginia Tech to leverage our NLR connections

to hold copies of critical data at their location.– We must modernize and update to new software, new hardware, employ

disk-to-disk-to-tape, and use off-site replication of storage arrays.– Have looked to adopt technologies that support replication and failover in

order to support an enhanced DR strategy.– SAN extension and replication combined with both server and storage

virtualization enable us to address BC and DR requirements in ways that were previously unapproachable.


BC/DR• Post 911 BCP and DRP awareness has increased dramatically but

unfortunately that awareness has not been accompanied by significant increased funding. We are now faced with almost uncontrolled growth in amount of stored data. Since we actively mirror all institutional data the end result of the explosion in data storage has been the consumption of our entire storage budget in "storing". Our costs have grown too fast for us to be able to mount a detailed and effective BCP.

• Currently in the process of re-evaluating our entire backup and storage architectures. Also driven by the new compliance issues around Electronic Evidence and E-Discovery legislation.

• Current disaster recovery strategy is ambiguous. Nightly copies of our backup tapes are sent to Iron Mountain for storage. In the event of a true loss of datacenter disaster we would look for assistance from our vendors and peer institutes to assist us in our recovery. We have taken under advisement the need to have a well documented and institutionally driven business continuity plan. When we do move forward in developing this there should be many driving factors for storage, including requirements for data redundancy, archival, and data accessibility.


Summary of Unsolved Problems

• Funding, funding models, costs• Smart data storage (data de-duplication,

compression, life cycle management)• Multi-platform, with as close to native access as

possible• Replacing current distributed file services (DCE,

DFS)• Virtualization and tiering• More, more, more (and staying ahead of or on par

with demand)


Details on Unsolved Problems

• In the process of implementing a broad array of storage, backup and recovery services. Developing the detailed delivery approaches, the funding/business models and the policy/procedure details will require considerable effort in the coming year.

• Data de-duplication technology will make a huge improvement in our ability to adequately backup our data

• Cost control which equates to storage volume control. We are actively researching the acquisition of an automated archiving solution which would necessarily include storage classification tools / software so we can control both what and where (in terms of tiers and/or archiving) we store as well as simply how much. We are also looking in to adding a 4th tier of storage with lower performance and (hopefully) greatly reduced costs

• A good replacement for what we have in DCE/ DFS now.

• Archival – Desktop Backup – Centralized Storage for Research Computing

• True virtualization of file and storage infrastructure.



• Universal client access with a reasonable authentication / authorization environment doesn't exist. vendor storage implementations are about 10 years behind commonly accepted ideas in storage/filesystem research and they don't seem to be generally aware of that fact. scalable horizontal growth of storage is just starting to get mainstream and usually is appearing with the iSCSI vendors. backup technology is slow to move to CDP ("continuous data protection") and still rooted in old models that don't fully leverage the cheap storage that is available. transparent tiering of data based on usage with policy is barely a glimmer in commercial products. Selling "ILM" solutions is confusing the market place and solutions for properly tagging data is being confused in or buried by marketing babble data scrubbing of failed disk drives in RAID sets doesn't seem to be in the common thought process yet; similarly, disk level encryption is still in its infancy; tape encryption is getting better but that is still slow. key escrow of all this is also a pain. "disks are cheap; storage is expensive" isn't common knowledge so there is unreasonable pressure to provide "enterprise" class storage at commodity pricing. Compare the cost of a 750GB drive ($0.60 per GB) vs. "cheap RAID" ($1.33 per GB) vs. "enterprise class SAN" ($10+ per GB).... and none of these costs include replication for BC/DR, backups, etc (though it does include RAID and hot-spare overheads).



• The need to provide a variety of network file system solutions (cifs,nfs, iscsi) and not a good platform for enabling that type of access easily to the storage infrastructure we have built.

• No single file protocol that works well from on and off campus on Linux, Mac and Windows without extra client software, provides rich access controls and supports our central authentication service. Enterprise scale laptop backup software is still lacking for non-Windows platforms. Key escrow for device and backup encryption is poorly supported by vendors.

• Growth of digitization of information, information life cycle management, encryption, compression

• Petabyte storage solutions for researchers. Trying to find a scalable way to store tens of Petabytes w/o managing multiple storage arrays.

• Adequate funding to keep up with the ever increasing demand within the university community

• Increasing demand for centralized file services from departments. Multi-site redundancy of storage for critical services. An enterprise backup solution that would allow us to provide backup services for department with local storage.


Summary• Growth in data is a huge problem - an unfunded

mandate in higher ed– Federal and other requirements for keeping and protecting

data for longer periods– Unmanaged data is becoming a larger problem - we’re just

keeping ever ything because it is too hard to clean things up

• Inefficiency– Not aligning least used data with least expensive solution– Backing up a lot that doesn’t need backup, not backing up

what does

• Cost and funding models• Technology pieces complicated to knit together into a

solution


NEXT: Setting the Stage & Case Studies

• Setting the stage– Defining terms, regulations, architecture

• Case studies– Library archive– Large data stores

• Small data files• Large data files

– Tiered storage and virtualization– (Case Study on website)

Documents

CSG Storage Survey Results1 CSG Storage Workshop