Big Data and the Future of Storage

Big Data and the Future of Storage

Dr. Evangelos Eleftheriou IBM Fellow

© 2014 International Business Machines Corporation 2

Big Data: More than just volume

Volume

Terabytes to exabytes of existing data to store and process

Velocity

Streaming data, milliseconds to seconds to respond

Variety

Structured, unstructured, text & multimedia

Veracity

Uncertainty from inconsistency, ambiguities, etc.


Memory / Storage Stack

Significant advancements in non-volatile memories blur the boundaries between storage and memory by being low-cost, fast, and cost-effective.

101

1

102

103

104

105

106

107

108

109

1010

CPU Operation (1ns)

Fetch data from the L2 Cache (10ns)

Fetch data from DRAM (60ns)

Read / Write SCM (100 – 2000 ns)

Read from a Flash device (20 μs)

Write to a Flash device (1 ms)Read from / Write to disk (5 ms)

Read from / Write to tape (40 s)

Memory

SCM

Storage

Tim

e (n

s)second

minute

hour

day

week

month

year

decade

century

SLOW

FAST


IBM FlashSystem: The Tipping Point

• The first time Flash storage outperformed hard disks in all aspects, including capacity and performance density, cost per I/O Operations Per Second (IOPS) and energy efficiency!"

• With IBM FlashSystem 820 we achieved more than 6 million IOPS running an IBM DB2 workload on IBM Power servers

• 19 kilowatts vs. 4.5 megawatts with high capacity hard disks, 236x better

• Installed in less than 48 Hours, HDD would require two years !


IBM FlashSystem: The Tipping Point


Storage infrastructure in the new era

High IOPS

Ultra-low Latency

Memory-speed Data Processing

Infrastructure Elasticity

Reduced time for deployment

Bottomless Capacity

Optimized TCO

Integrated global infrastructure

Simplicity

Security

Reliability

Pay-as-you-go

Extreme Scalability

Block Interface Novel Interfaces

MicroLatency Storage Global File System

POSIX NFS CIFS Object

GPFS Active!File Management

POSIX CIFS ObjectNFS

ObjectNFSPOSIX CIFS

GPFS

GPFS


Tape Storage: Big Data Needs Tape

• Faster than Hard Disks at streaming!!

• Reliability: read after three decades, against five years for disks!!

• Zero Power Consumption (when idle)!!

• Security: 50 Petabytes on an HDD can be deleted in minutes, tape would take years!!

• Cost: 1 GB of disk storage costs 10 cents, versus 4 cents for tape !!

• The Next Target: the 100 TB cartridgeThe Economist, Magnetic tape to the rescue. 30 Nov 2013


Multi-cloud Storage Tool Kit (live demo in CeBIT booth)

• Can connect to one or more clouds including: IBM Softlayer, AWS, Azure, Rackspace (public, private, or hybrid) !

• Enterprise features, including: encryption, integrity, and resiliency, it’s transparent to GPFS and manages keys/metadata!

• Drag-and-drop usability

© 2014 International Business Machines Corporation© 2014 International Business Machines Corporation !9


A universal, non-volatile memory technology superior to Flash. PCM is very durable and can endure at least 10 million write cycles, compared to current enterprise-class flash at 30,000 cycles or consumer-class flash at 3,000 cycles.

Phase Change Memory (PCM)DRAM Flash PCM Multi-Level

PCM

Timeframe Invented in 1966 Invented in 1980sUsed in smartphones

– wide adoption expected by 2016

2016

Speed

Density

Endurance

Retention

Scaling

Best in class

Good / Adequate

Average / Inadequate

Bad / Worst in class

➢ The qualification is relative and depends on the application!➢ Racetrack memory is not included in this time horizon


Theseus Project: PCM-based PCI-e Prototype Card

Hybrid Storage/Caching subsystem based on PCM

• Design and implementation of a high-performance PCI-e card using PCM!

• Consistent ultra-low latency, high IOPS even for very small operations!

• Integration of PCM and Flash for hybrid use cases: !• Caching!• Tiering!• Persistent Key/Value Store

Technology

Big Data and the Future of Storage