Who Says Data Center Storage Has to be Inefficient? · PRESENTATION TITLE GOES HERE Who Says Data Center Storage Has to be Inefficient? Larry Chisvin. PLX Technology, Inc

PRESENTATION TITLE GOES HERE

Who Says Data Center Storage Has to be Inefficient? Larry Chisvin

PLX Technology, Inc.

2 2014 Data Storage Innovation Conference. © PLX Technology, Inc.. All Rights Reserved.

Agenda

Who is PLX (and why does it matter)? Current usage of PCIe in storage Improving on data center fabrics

How a PCIe fabric works Why it is better

Usage models for PCIe-based storage New data center architectures Development Tools Call to action


Global Enterprise ~155 Employees, Headquartered in USA

Design Center in India

Public Company NASDAQ: PLXT, $100M+ Rev., ~58% GM, Profitable

PLX at a Glance

Market-Leading Customers Widespread use by 1000’s of key accounts

The Market Leader Over 70% Market Share and Growing

Fabless Model TSMC Partner with Designs and Production at 40nm


All Market Leaders Using PLX PCIe Products

Customers Serving Cloud/Data Center

Switches & Routers Control Plane Standard

Servers First Market that

Developed for Switches

Storage Largest Market

Growth & Attach Rate


Storage in the Cloud/Data Center

I/O Fan-out, Redundancy & Backplane Large Switches with PLX added features used throughout Enterprise Storage Systems. Represents most of the storage revenue today in the pie chart below.

Total Storage Doubling Every Two Years

Standard Bus for Market Leaders

Storage Systems & HBAs/CNAs

Storage Largest Market

Growth & Attach Rate

Networking

PC/ Consumer

Embedded

Server

Storage


Rest of System

PCIe Switch

PCIe Switch

Storage SAS/SATA Controller

CPU

Storage Subsystem

Basic Fanout of Storage Systems

Storage Subsystem

Storage Subsystem

Storage Subsystem


Flash Memory

PCIe SSD

Controller

Flash Memory

Flash Memory

Flash Memory

PCIe SSD

Controller

PCIe SSD

Controller

SSD-Based Aggregation

PCIe Switch

Rest of System

PCIe SSD

Controller


Making the Data Center Better

http://www.google.com/url?sa=i&rct=j&q=data+center+with+cable+mess&source=images&cd=&cad=rja&docid=nttzaFZTar7QXM&tbnid=j0hnb7cFrTrzGM:&ved=0CAUQjRw&url=http://community.brocade.com/community/brocadeblogs/data_center/blog/2012/11/27/solutions-solutions-lab-journal--configuring-cable-management&ei=Ac2gUcufCua2igLk6oHYCA&bvm=bv.47008514,d.cGE&psig=AFQjCNHO7LgI-Jh83kx4y4deOeJZ5J9UlQ&ust=1369578961522098


The Rack in the Cloud

Host

Storage

Communications

HBA NIC/LOM

HBA NIC/LOM

HBA NIC/LOM

HBA

NIC/LOM

Host

Host

Host

Host

NIC/LOM


Current Solutions

Host

Storage

Communications

HBA NIC/LOM

HBA NIC/LOM

HBA NIC/LOM

HBA

NIC/LOM

Host

Host

Host

Host

NIC/LOM

Multiple fabrics

“Bridging” Devices on each module

Parts Power Cost

Latency Complexity

More


Problem…or Opportunity?

Host

Storage

Communications

HBA NIC/LOM

HBA NIC/LOM

HBA NIC/LOM

HBA

NIC/LOM

Host

Host

Host

Host

NIC/LOM

Since one major problem here is the need for all of those “bridging” devices…

Then what is the answer?


Is There Something Common?


PCI Express Fabric

ExpressFabric

Redriver

PCI Express

Host

Host Redriver

Redriver

Redriver

Redriver

Redriver

Single Converged

Fabric

Simple, Inexpensive

Redriver

High performance, Low cost,

Low latency Low power Interconnect

Storage

Storage

Comm

Comm


PCIe-Based Fabric Solution

Storage

Host

Storage

Storage

Storage

PCI Express Fabric

Host

Host

Host

Standard capabilities of fabric provided by PCIe

mCPU

Shared I/O Using existing

Hardware and Drivers

Host-Host Using standard

DMA & RDMA

Infrastructure

Management CPU To configure system and manage events


Mixing Data Types on PCIe

Retimer Host

Retimer

Retimer

Storage

Comm

Flash Flash Flash Flash


SSD Controller

SSD Controller

Switch

GPU GPU

GPU

GPU GPU Switch

GPU

ExpressFabric Enterprise SSDs are rapidly

standardizing on PCIe

Hybrid systems can be easily implemented


PCI Express Initiatives

SR-IOV Allows sharing of end points

SRIS (Separate Refclk Independent SSC) Allows each side of link to have its own clock Enables use of general purpose cables

New PCIe Cable spec (OcuLink) Targeted for low cost copper and optical

Signal integrity Better defined repeater & retimer functions


DPC/eDPC

Downstream Port Containment (DPC) Enables the prevention of host time-out or “blue screen”

When triggered by error from an endpoint Take down link to endpoint Send error message/interrupt to the host Reply to read request within time-out window

Completion synthesis for root ports Host software to handle switch responses Important for storage systems

PCIe Switch

I/O Device

I/O SSD Device

I/O Device

Host CPU


Quantifying the Advantages

Replacing existing Ethernet-based racks with PCI Express can offer tangible savings

PCI Express inside the rack Ethernet to rest of data center

Top-of-rack switch similar cost to PCIe Greater reduction if converged data

Remove NIC/LOM Each slot costs $200-$300 Each slot dissipates 5-7w Rack has 24-64 slots

PCIe-base solution uses simple redriver Each redriver can be $2-$5 Power is less than 1w

Rest of Data Center/Cloud


Cost and Power Advantages of PCIe

$- $2.0 $4.0 $6.0 $8.0

$10.0 $12.0 $14.0 $16.0 $18.0

ExpressFabric 10GE SFP+ 10GE LOM

Cost/Gbps ($) 10GE

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

ExpressFabric 10GE SFP+ 10GE LOM

Power/Gbps (w) 10GE


Storage Usage Models for a PCIe Fabric


SSD Adapter Card

Two SSD Controllers NVMe Form-Factor – 2.5” Only ASIC1 (controller) exposed to

Host ASIC1 servicing interrupts from ASIC2

Two or more SSD controllers

aggregated with a switch Adapter Card Form-Factor Host isolation capability One ASIC exposed to Host

Services interrupts from other ASICs

ASIC1

Switch

ASIC2

Switch ASIC ASIC ASIC ASIC ASIC

ASIC ASIC

ASIC


Server Motherboards

Capacity expansion in server enclosures Aggregation through PCIe switch

Server Motherboard

RAM

RAM Switch

NIC

CPU


Hybrid Storage System

RAM

CPU

RAM

Switch

NIC

SAS Drives

SAS HBA

MLC

MLC

SLC

Mixing storage mediums for cost/performance Move data between these mediums

Server Motherboard


IO

Switch PCIe over Cu/Opt

Server

Creating storage expansion box with PCIe Connect to server through PCIe Cu/Op cable No protocol translation overhead

Storage Expansion Boxes

IO Expansion Box


Share an array of SSD modules in a chassis Associate a segment of SSDs to specific server

Servers

PCIe over Cu/Opt

Flash Appliance

IO

Switch

IO Expansion Box

Switch


IO

Switch

IO Expansion Box

PCIe over Cu/Opt

Servers

Build Server Redundancy

Switch

Server Redundancy


IO IO Expansion Box

Share SR-IOV SSD modules in expansion chassis Shared by multiple servers/hosts or blade servers

SR-IOV Sharing of SSDs

PCIe over Cu/Opt Switch

Switch

Servers


Can Form Factors Help?

http://www.google.com/url?sa=i&rct=j&q=data+center+with+cable+mess&source=images&cd=&cad=rja&docid=nttzaFZTar7QXM&tbnid=j0hnb7cFrTrzGM:&ved=0CAUQjRw&url=http://community.brocade.com/community/brocadeblogs/data_center/blog/2012/11/27/solutions-solutions-lab-journal--configuring-cable-management&ei=Ac2gUcufCua2igLk6oHYCA&bvm=bv.47008514,d.cGE&psig=AFQjCNHO7LgI-Jh83kx4y4deOeJZ5J9UlQ&ust=1369578961522098


From Rack to Blade

Rack servers are self-contained systems Traditional architecture in stackable form CPU, memory, I/O, power, cooling, etc. Standard form factors

Blade servers consolidate common subsystems Common power, cooling, external connection Blades offer processing, storage, I/O Systems are vendor-specific

Improvement on rack Allows some savings But non-standard subsystems And doesn’t really address issue completely

29


Next Step - MicroServers

Traditional (brute force) approach to computing limits compute density due to power Open source OS’s and application software allow portability of CPUs and platforms Large scale applications can be effective with smaller compute engines

Web servers Content delivery Analytics

Bottleneck in such applications are memory and I/O bandwidth rather than raw compute power


What is a MicroServer?

Array of low power, low cost processors Less than 45w Moving to less than 10w

Each processor has modest performance Aggregate performance can be powerful

Can execute applications where processing can be easily distributed CPU, memory, & I/O on compact module ARM-based systems are most common

X86 platforms are expected to be popular


Problems with Current Solutions

Cannot adequately handle compute-intensive applications easily

Many applications cannot distribute processing Cannot easily trade off processing, storage & I/O, since the subsystems are self-contained Most systems are proprietary

Need to buy into entire system Similar to blade servers vs rack servers

Definite advantages, but… Limited ability to mix best-in-class subsystems


PCIe-Based MicroServer

µServer

Network of SOCs

µServer

Network of SOCs

PCIe Aggregation

PCIe Aggregation

PCI Express Fabric


Powerful Storage Appliance

µServer

Network of SOCs

µServer

Network of SOCs

PCIe Aggregation

PCIe Aggregation





SSD Controller

SSD Controller

PCIe Aggregation





SSD Controller

SSD Controller

PCIe Aggregation

PCI Express Fabric


Consolidate Complete System

µServer

Network of SOCs

µServer

Network of SOCs Traditional Server Blade

High Perf CPU

Storage Storage

Storage Storage

Comm Comm

Comm Comm

High Perf Server Blade

PCIe Aggregation

PCIe Aggregation

PCI Express Fabric

PCIe Aggregation





SSD Controller

SSD Controller

PCIe Aggregation


Building a Complete Fabric

Development platforms for customer products ExpressNIC

PCIe bus extender Based on PEX 8725 2 QSFP+ ports, x4 PCIe each

Argo Rack-mounted PCIe switch fabric 1U form 32 QSFP+ ports, x4 PCIe


Better Storage Through PCIe

PCIe-based solutions offer substantial benefits to storage systems

Direct connection to vast number of devices Low latency, high bandwidth solution Flexible sharing with existing infrastructure Expand as specification improves

Especially attractive to SSD-based systems


Thank You! For More Info, please visit

www.plxtech.com

Documents

Who Says Data Center Storage Has to be Inefficient? · PRESENTATION TITLE GOES HERE Who Says Data Center Storage Has to be Inefficient? Larry Chisvin. PLX Technology, Inc