Architected for Performance

NVM ExpressTM Infrastructure -Exploring Data Center PCIe® Topologies January 29, 2015

Jonmichael Hands – Product Marketing Manager, Intel Non-Volatile Memory Solutions Group

Peter Onufryk – Sr. Director Product Development – PMC-Sierra

Moderator: Ravi Chari – Storage Technologist – Oracle

View recorded webcast at https://www.brighttalk.com/webcast/12367/141221

2

Legal Disclaimer

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

No computer system can be absolutely secure. Intel does not assume any liability for lost or stolen data or systems or any damages resulting from such losses.

Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.

*Other names and brands may be claimed as the property of others.

Copyright © 2015 Intel Corporation. All rights reserved.

3

Agenda - NVM ExpressTM(NVMeTM) Infrastructure

• What is NVMe?

• NVMe advantages over SATATM

• NVMe driver ecosystem

• PCIe® form factors, cables, and connectors

• Link extension and port expansion for PCIe

• PCIe Solid-State Drive Topologies

• NVMe Management

4

What is

NVM ExpressTM is a standardized

high performance software interface for PCI Express®

Solid-State Drives

Architected from the ground up for SSDs to be more

efficient, scalable, and manageable

NVMe is industry driven to be

extensible for the needs of both the

client and the data center

?

If I had asked people what they wanted, they would have said faster horses- Henry Ford

“

”

5

NVM ExpressTM Community

Promoter GroupLed by 13 elected companies

NVM Express, Inc.Consists of more than 75 companies from across the industry

Technical WorkgroupQueuing interface, NVMe I/O and Admin command set

Management Interface WorkgroupOut-of-band management over PCIe® VDM and SMBus

6

Deployment at scale

Industry standard drivers, software, and management

What NVM ExpressTM

brings to the

DATA CENTER

Works out of the box

In standard operating systems

Lower TCOEfficiency of

protocol, increased storage density,

lower system power

7

NVM ExpressTM(NVMeTM) Advantages over SATATM

PCIe® for scalable performance, flexible form factors, and

industry stability

Increased bandwidth: 1 GB/s per lane – 1-16 lanes per drive

Directly attached to CPU, eliminate HBA cost and overhead

NVMe provides lower latency and increased efficiency:

lower CPU utilization, lower power, lower TCO

Low power features from both PCIe and NVMe

Security from Trusted Computing Group OPAL

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance.Configurations: Intel® S2600CP server, Intel® Xeon® E5-2690v2 x2, 64GB DDR3, Intel® SSD DC P3700 Series 400GB, LSI 9207-8i, Intel® SSD DC S3700, HGST 6GBps SAS

8

NVMeTM Driver Ecosystem

Native / in-box6.5 | 7.0

SLES 11 SP3 SLES 12

Install NVMe driver

ESXi 5.5

13 | 14Windows 8.1

Linux NVMe driver is open source

*Other names and brands may be claimed as the property of others.

9

What do I need to start using an SSD?

Software: NVMeTM driver

Hardware: PCIe® infrastructure

NVMe sits on top of PCIe

10

Form Factors for PCI Express®

Data Center Client

SFF-8639

SATA Express

AIC

2.5inSFF-8639

SATA ExpressTM

M.2

Add in Card

M.2BGA

HD SSD FF

11

Drive Connectors

SATA Signal Pins Power and Control Pins

SATATM

• Keyed only for SATA drives• Separate power and data

Key

Signal Pins (port A) Power and Control Pins

Signal Pins (Dual port, B)

SAS®• Backwards compatible

with SATA• Dual port

SFF-8639• Supports SATA, SAS,

and PCIe® x4 or two x2• PCIe data, reference

clock, and side bandSAS / SATA Power and Control Pins

SASRefClk 0 & Lane 0 Lanes 1-3, SMBus, & Dual Port Enable

Refclk 1, 3.3V Aux, & Resets

12

SATA ExpressTM and SFF-8639 Comparison

SATAe SFF-8639

SATA / SAS® SATA SATA / SAS

PCI Express® x2 x4 or dual x2

Host Mux Yes No

Ref Clock Optional Required

EMI SRIS Shielding

Height 7mm 15mm

MaxPerformance

2 GB/s 4 GB/s

Bottom LineFlexibility

& CostPerformance

Source: Seagate* (with permission)

SFF-8639 designed for data center, SATAe designed for Client

13

M.2 Form Factor Comparison

Host Socket 2 Host Socket 3

Device w/ B&M Slots

M.2 Socket 2

M.2 Socket 3

SATAYes, Shared Yes, Shared

PCIe® x2

PCIe x4 No Yes

Comms Support Yes No

Ref Clock Required Required

Max Performance 2 GB/s 4 GB/s

Bottom Line Flexibility Performance

M.2 Socket 3 is the best option for Data Center PCIe SSDs

14

Cabling Options for Data Center PCIe® SSD Topologies

Reference Clock

PCIe Reset

SMBUS

miniSAS HD cables lightly modified for PCIe are being used due to the robust connector and high volume manufacturing.

Reference Clock

15

Basic PCI Express® SSD Topology – 1 Connector

• SFF-8639 Connector directly attached to board

• Mostly used in small form factors such as compute node, blade, etc. 1

16

Basic PCI Express® SSD Topology – 2 Connector

1 2

miniSAS HD Connector

PCIe® Cable

PCIe 3.0 x4 Enterprise SSDSFF-8639 Connector

External Power

17

Basic PCI Express® SSD Topology – 3 Connector

Motherboard

1

miniSAS HD Connector

PCIe® Cable

miniSAS HD Connector

3Backplane

SSD Drive Carrier

SFF-8639 Connector

2

18

Port Expansion Devices - Switches

Use Switches to expand number of PCIe® SSDs

SwitchPCIe 3.0 x8 link

PCIe SSD

x4 link

Intel CPUPCIe SSD

x4 link

PCIe SSD

x4 link

PCIe SSD

x4 link

Port A

Port B

Port C

Port D

19

Link Extension Devices – Switches and Retimers

Use Link Extension Devices for longer topologies

RetimerPCIe 3.0 x4 link

PCIe SSD

x4 link

Switch

PCIe SSD

x4 link

Intel CPU

x4 link

Port A

Port B

PCIe® 3.0 x4 link

20

PCI Express® (PCIe®) Switches and Retimers

PCIe Switches

• Use for link extension and/or port expansion

• Hot-plug and error isolation

• High performance peer-to-peer transfers

• Extra software features

Retimers

• Mostly transparent to software

• Retimers should be more common in PCIe 4.0

Link Extension Devices

• Use when channel has > -20db loss: at 8GT/s PCIe 3.0

Retimer vs. Re-driver

• Repeater: A Retimer or a Re-driver

• Re-driver: Analog and not protocol aware

Retimer: Physical Layer protocol aware, software transparent, Extension Device. Forms two separate electrical sub-links. Executes equalization procedure on

each sub-linkRecommend using only switches or retimers for link extension of PCIe

21

High Function Switches

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSDNVMe

SSD

NVMe

SSD

PCIe Switch

NVMe

SSD

NVMe

SSD

PCIe Switch

Host Host

Inerconnect

PCIe PCIe

22

Complex PCI Express® Topology – 4 Connector

PCIe x16 slot

PCIe® Cable

Cabled Add in card with Link

Extension

miniSAS HD for PCIe

Backplane

SSD Drive Carrier

1

2

3

4 SFF-8639 Connector

23

Complex PCI Express® Topology – 5 Connector

PCIe x16 slot

PCIe Cable

Cabled Add in card with Link

Extension

miniSAS HD for PCIe

Backplane

SSD Drive Carrier

1

24

SFF-8639 ConnectorPCIe® x16 Riser

3

5

24

PCI Express® cabling for future topologies - OCuLink*

Category OCuLink

Standard Based PCI-SIG®

PCIe® Lanes X4

Layout Smaller footprint

Signal Integrity Similar on loss dominated channels

PCIe 4.0 ready 16GT/s target

Clock, power Supports SRIS and 3.3/5V power

Production Availability

Mid 2015

12.85mm

2.83mm

Source:

OCuLink internal cables and connectors

25

OCuLink* Provides Flexible Data Center Topologies

Board to board connections

Cabled add in card

Backplane

SFF-8639 Connector

PCIe® SSD

26

NVMeTM Storage Device Management

Server Caching Server Storage External Storage

Root Complex

PCIe/PCIe RAID

NVMe NVMe NVMe

x16

x4

Root Complex

NVMe

PCIe Switch

NVMe NVMe NVMe

x16

x4

Controller A Controller BRoot Complex

PCIe Switch

x16

Root Complex

PCIe Switch

x16

SAS SAS

NVMeNVMeNVMeNVMe

SASHDD

Example Pre-boot Management

Inventory, Power Budgeting, Configuration, Firmware Update

Example Out-of-Band Management During System Operation

Health Monitoring, Power/Thermal Management, Firmware Update, Configuration

27

Driver vs. Out-of-Band Management

NVM Subsystem

NVMe SSD

PCIe Port

SM

Bu

s/I2C

NVMe

Driver

Applications Management

Controller

Operating SystemPlatform

Management

PCIePCIe VDM

SMBus/I2C

28

Management Interface

Protocol Layering

Management Applications (e.g., Remote Console)

SMBus/I2C PCIe

MCTP overSMBus/I2C Binding

MCTP overPCIe VDM Binding

Management Component Transport Protocol (MCTP)

NVMe Management Interface

Management Controller(BMC or Host Processor)

Management Applications (e.g., Remote Console)

PhysicalLayer

TransportLayer

ProtocolLayer

ApplicationLayer

NVMe SSD

30

NVMeTM Technical Overview• Supports deep queues of 64K commands per queue, up to 64K queues

• Supports MSI-X and interrupt steering, enables even performance scaling

• Streamlined & simple command set (13 required commands), optional features to address target segments

• Built for the future, ready for next gen NVM

31

Fully Exploiting Next Gen NVM

With Next Gen NVM, the NVM is no longer the bottleneck

App to SSD read latency for 4KB transfer at Queue Depth of 1

0 10 20 30 40 50 60 70 80 90 100 110 120

Future NVM PCIe x4 Gen3

NAND MLC PCIe x4 Gen3 ONFI3

NAND MLC SATA 3 ONFI3

NAND MLC SATA 3 ONFI2

us

NVM Tread NVM xfer Misc SSD Link Xfer Platform + adapter Software

32

NVMeTM Development History

2011 2012 2013 2014

• Host Memory Buffer• Replay Protected Area• Active/Idle Power and RTD3• Temperature Thresholds• Namespace Management• Controller Memory Buffer• Live Firmware Update• Atomicity Enhancements

NVMe 1.2 – Q4 2014

• Multi-Path IO• Namespace Sharing• Reservations• Autonomous Power

Transition• Scatter Gather Lists

NVMe 1.1 – Oct 2012

2015

• Queuing Interface• Command Set• End-to-End Protection• Security• PRPs

NVMe 1.0 – Mar 2011

Architected for Performance