23
Storage is Cheap” and Other Storage is Cheap” and Other Lies Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re- Use ICPSR, July 31, 2013 “Hard Drives” by Michael Muni CC- BY-NC-ND

“Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Embed Size (px)

Citation preview

Page 1: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

““Storage is Cheap” and Other LiesStorage is Cheap” and Other Lies

Lance Stuchell, University of Michigan LibraryCurating and Managing Research Data for Re-UseICPSR, July 31, 2013

“Hard Drives” by Michael Muni CC-BY-NC-ND

Page 2: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Overview

• Instability of storage hardware costs • Check out David Rosenthal’s blog at

http://blog.dshr.org/

• Costs of storing digitized moving images

• Effects of storage costs on digital preservation, formats, etc.

Page 3: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

“Storage is Cheap”

• Perception is based on “Kryder's Law”• A 2005 Scientific American article about Mark

Kryder, Seagate's Senior VP of Research • Magnetic disk density increases quickly• Disk density closely tied to pricing • 30-year history of disk prices dropping about

40% per year

• Disk costs were affordable & predictable• 30% of total storage costs

Page 4: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

The Party's Over

• Mid 2011: Kryder’s Law slows• Latest projections ~ 20% density growth

• Late 2011: Flooding in Taiwan causes shortfall of 70 million disk drives• Prices remain over 50% higher• Not expected to return to pre-flood levels

until 2014

• Changes in technology David S. H. Rosenthal, “Storage Will Be A Lot Less Free Than It Used To Be,” 2012.http://blog.dshr.org/2012/10/storage-will-be-lot-less-free-than-it.html

Page 5: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

“Optimistically, for the rest of this decade the rapid decrease in cost per bit of storage that

has been a constant of the last three decades will be much slower; it might even stop.”

David S. H. Rosenthal, et al., “The Economics of Long-Term Digital Storage” 2012.http://www.lockss.org/locksswp/wp-content/uploads/2012/09/unesco2012.pdf

Page 6: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

A/V Digitization at MLibrary

Digitization of Special Collections material

Audio:• Uncompressed BWF master files• 1 original audio object ≈ 5 GB storage

Moving Image:• Uncompressed preservation master• Compressed production master• 1 original video object ≈ 40.4 GB Storage

Page 7: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

A/V Storage Estimates

Page 8: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

A/V Storage EstimatesTB/Year

Page 9: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Initial Costs Estimates

Value Storage: $250/TB per year

Page 10: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Initial Costs Estimates

Value Storage:Tape Backups:

$250/TB per year$1,825/TB per year

Page 11: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Initial Costs Estimates

Value Storage:Tape Backups:

$250/TB per year$1,825/TB per year

2018: $245,3252023: $485,200

2013: $32,450

Page 12: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Reaction

Page 13: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Costs Estimates

Value Storage:Tape Backups

(MLib):

$250/TB per yearEquipment costsCheaper most years

Page 14: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Cost Estimates

Page 15: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Cost Estimates

Page 16: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Cost Estimates

Page 17: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Total 11 Year Storage Cost

$569,368

Page 18: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Cost Estimates

Page 19: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Current Cost Estimates

Page 20: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Looking Ahead

• Assuming steady pricing• Instability in per-bit storage

• Will not fall under 20% growth until 2018

• ITS storage costs may not fall at same rate

• Long-term costs of cloud are not known• Amazon costs haven't decreased at HD rate1

• Tape storage for preservation copies?

• Economies of scale• Digital Preservation Network

1Rebecca Pool, “Is cloud storage the answer to preservation?” Research Information, 2/14/2013

Page 21: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Ramifications

• Preservation• Some moving image material is not

retained in uncompressed formats• Still image formats are balance of

preservation and size

• Appraisal and re-appraisal • What are we keeping and for how long?

• Costs of starting up

Page 22: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Storage Takeaways

• HD costs are no longer predictable • Budgets often neglect storage

• Ignoring significant and ongoing costs

• Storage costs are community wide problem• Question of scale• Best practice may not be possible

• Backups can cost more than primary storage

• Archival storage ain’t cheap!

Page 23: “Storage is Cheap” and Other Lies Lance Stuchell, University of Michigan Library Curating and Managing Research Data for Re-Use ICPSR, July 31, 2013 “Hard

Thanks!!