IP-Based NVMe Development Platform€¦ · 07-08-2018 · KCU105 – Kintex Ultrascale Fidus...

IP-Based NVMe Development Platform

Mickael GuyardProduct Marketing Director (IP-Maker)

Flash Memory Summit 2018 Santa Clara, CA 1

Agenda

The need for NVMe IPs NVMe device platform NVMe host platform Use cases and applications

Flash Memory Summit 2018 Santa Clara, CA 2

Part 1 – The need for NVMe IPs

Flash Memory Summit 2018 Santa Clara, CA 3

NVMe the new universal interface

The new universal interface for storage First specification released in 2011 13 board members 90 companies with NVMe-based products (G2M

research report) But not only…

Flash Memory Summit 2018 Santa Clara, CA 4

NVMe applications

Storage: PCIe SSD Cache: PCIe MRAM and NVRAM Processing accelerator

Flash Memory Summit 2018 Santa Clara, CA 5

Heterogenous architecture

Flash Memory Summit 2018 Santa Clara, CA 6

FPGA in the data centers

Flash Memory Summit 2018 Santa Clara, CA 7

The need for NVMe IPs

Flash Memory Summit 2018 Santa Clara, CA 8

Massive usage of FPGA in data centers NVMe as a universal interface New architectures

=>NVMe IPs for FPGA are needed Both device and host

IP-Maker IPs

Flash Memory Summit 2018 Santa Clara, CA 9

Part 2 – NVMe Device Platform

Flash Memory Summit 2018 Santa Clara, CA 10

NVMe protocol

Flash Memory Summit 2018 Santa Clara, CA 11

3- NVMe command submission fetch

2- NVMe command ready

CPU

1-Host driverSetup PCIe

Phy + CtrlPCIe rootcomplex

NVMe IP

DDRDMA4- Data transfer DDR

5- NVMe command executed

NVMe IPs

Flash Memory Summit 2018 Santa Clara, CA 12

Key Features1.3 NVM Express specificationAutomatic NVMe commandUp to 65536 I/O queuesQueue arbitrationAll mandatory commands / log managementLegacy interrupt/MSI/MSI-XAXI/Avalon interface Up to 32 Read DMA channels + 32 write DMA channelsScalable data buswidth (64/128/256 bits)Available for PCIe Gen1/2/3

Full hardware

Hardware + SoftwareFor more flexibility, such as vendor

specific commands

HW/SW architecture

Automatic command processing => Low latency

Multi-channel DMA => IOPS acceleration

Flash Memory Summit 2018 Santa Clara, CA 13

Validated platforms

Flash Memory Summit 2018 Santa Clara, CA 14

VC709 – Virtex7

KCU105 – Kintex UltrascaleNallatech 250S+ - Kintex Ultrascale+

Fidus Sidewinder – Zynq Ultrascale+

Reference design

Flash Memory Summit 2018 Santa Clara, CA 15

PCIe Interface

IPM-NVMe

DMADMADMADMA

AutomaticCommandProcessing

Bridge Bridge

DDR3 Controll

er

FPGA

PCIe hard IP DDR controller soft IPHost

CPU

DDR

NVMe driver

Device

DDR3

Performances

Setup• Hardware : reference design• Standard NVMe driver• Use of standard benchmark tool for storage: FIO

• Latency• IOPS

Flash Memory Summit 2018 Santa Clara, CA 16

Latency

QD=1, IO=4kB

Flash Memory Summit 2018 Santa Clara, CA 17

IOPS

Gen3x4, QD8, 4kB IO, random R/W• 700kIOPS

High IOPS at low queue depth Scalable data path : up to Gen3x16, Gen4 x8

Flash Memory Summit 2018 Santa Clara, CA 18

Part 3 – NVMe Host Platform

Flash Memory Summit 2018 Santa Clara, CA 19

NVMe Host IP overview

Flash Memory Summit 2018 Santa Clara, CA 20

Memory orFIFO

PCIe

Roo

tPo

ort

Automatic initengine(state

machine)User InterfaceData to Transfer NVMe SSD

NVMe Command Manager

Control

Data transferengine

AXI/Avalon

AXI/Avalon

IP-Maker IP

3rd party IP

Theory of operation

Flash Memory Summit 2018 Santa Clara, CA 21

Data transfer request

Init with a state machinePCIe rootport/endpoint settingsNVMe device/host configuration

NVMe commandssetup by the host

Data transfer

Memory orFIFO

PCIe

Roo

tPo

ort

Automatic initengine(state

machine)User InterfaceData to Transfer NVMe SSD

NVMe Command Manager

Control

Data transferengine

0

1

2 3

Different configurations

Flash Memory Summit 2018 Santa Clara, CA 22

Single port, up to 128 queues N* root ports, 1 queue

*depending on FPGA interfaces

Multiroot

Flash Memory Summit 2018 Santa Clara, CA 23

Key FeaturesNVM Express CompliantAutomatic NVMe Command managementAutomatic PCIe/NVMe initMulti rootport supportSingle I/O queueSingle NamespaceVendor specific commandsUp to PCIe Gen 3x8

Advanced

Flash Memory Summit 2018 Santa Clara, CA 24

Key FeaturesNVM Express CompliantAutomatic NVMe Command managementAutomatic PCIe/NVMe init128 I/O queuesVendor specific commandsSingle NamespaceUp to PCIe Gen 3x8

Open Channel support

• Full submission command controlall vendor specific commands are possiblesThe complete control is possible.

• Full completion control

Flash Memory Summit 2018 Santa Clara, CA 25

Validated platforms

Flash Memory Summit 2018 Santa Clara, CA 26

KCU105 – Kintex UltrascaleNallatech 250S+ - Kintex Ultrascale+

Fidus Sidewinder – Zynq Ultrascale+

Reference design

Host IP configuration• 1 root port• 1 queue version

Embedded test bench

Flash Memory Summit 2018 Santa Clara, CA 27

Ultrascale Xilinx FPGA

Samsung NVMe 960EVOGen3x 4 PCIe interface

Performance

Performance• Write : 2.2 GB/s• Read : 3.2GB/s

Flash Memory Summit 2018 Santa Clara, CA 28

Part 4 – Applications

Flash Memory Summit 2018 Santa Clara, CA 29

Use cases and applications

PCIe Flash NVRAM Emerging NVM Smart SSD HBA NVME2NVMe

Flash Memory Summit 2018 Santa Clara, CA 30

NVRAM Reference design

Flash Memory Summit 2018 Santa Clara, CA 31

NVMe device ref design close to a end-product Detected as a NVMe device by the driver

Just need to add « non-volatile » feature

PCIe NVRAM

Using NVDIMM-N like technology Or using directly a NVDIMM-N

Flash Memory Summit 2018 Santa Clara, CA 32

PCIe NVRAM

Specification Up to 32GB 1.5MIOPS on Gen3x8 10us latency

Flash Memory Summit 2018 Santa Clara, CA 33

Flash Controller IP

Flash Memory Summit 2018 Santa Clara, CA 34

Key FeaturesONFI 3/4 CompliantSLC / MLC / TLC2 ChannelsSupported modes

Async, DDR, DDR2, DDR3AXI/Avalon interfaceConfigurable ECC

BCHLDPC

Evolution

• MRAM support

Flash Memory Summit 2018 Santa Clara, CA 35

MR

AM

NVMe to NVMe HBA

For NVMe SSD aggregation:• Better performance and reliability

Flash Memory Summit 2018 Santa Clara, CA 36

HBACPU

NVMe to NVMe HBA

• 2.5 x86 cores full time at 3GHz required to sustain 750kIOPS on each SSD• 10 cores total!

• =>Need of hardware accelerator enginesFlash Memory Summit 2018 Santa Clara, CA 37

HBA:10 x86 cores?CPU

NVMe to NVMe HBA

• Let’s use both NVMe device and host IPs

Flash Memory Summit 2018 Santa Clara, CA 38

NVMe Device IP

NVMe Host IP

CPU

NVMe to NVMe

Flash Memory Summit 2018 Santa Clara, CA 39

PCIe Interface NVMe Device

DDR3 Controller

PCIeInterfaceNVMe Host

PCIeInterfaceNVMe Host

PCIeInterfaceNVMe Host

PCIeInterfaceNVMe Host

Namespacemanagement

software

4 x Gen3x4

Gen3x 16or

Gen4x4

Namespace management

Many configurations• Basic capacity aggregation: one namespace across the 4

SSDs• Asymmetric : one namespace on one SSD and 10

namespaces on the 3 other SSDs.• Multi namespaces with different characteristic (encryption,

compression…) seen only as one storage SSD.• Raid 1 storage totally transparent for the host software.

Flash Memory Summit 2018 Santa Clara, CA 40

Evolution

Path to computational storage Advanced computing accelerators can be added

such as key-value store, search engine and deep learning

Flash Memory Summit 2018 Santa Clara, CA 41

Thanks

Contactmickael.guyard@ip-maker.com

Visit IP-Maker booth #710NVMe live demo!

Flash Memory Summit 2018 Santa Clara, CA 42