NVMe Management Interface

Technology and Testing

9/14/2017 1

Teledyne LeCroyPROTOCOL SOLUTIONS GROUP

Agenda

NVMe-MI Protocol Analysis & Test Tools

MCTP and NVMe-MI Introduction

NVMe-MI Conformance Test

NVMe and NVMe-MI Test Tools

9/14/2017 2

NVMe Protocol Analysis and Test Product Line

9/14/2017 3

Summit T3-16

Gen3x16

Summit T3-8

Gen3x8

Summit T34

Gen3x4

Summit T28

Gen2x8

Summit T24

Gen2x4

Summit Z3-16 and Test Platform

Gen3x16

AnalysisTraffic

Generation

Teledyne LeCroy Provides Protocol Testing Expertise

Teledyne LeCroy PCIe equipment are the backbone of

the following industry events

PCI SIG Workshops

University of New Hampshire NVMe Plugfests

Austin Test Labs

Teledyne LeCroy’s Austin Labs is the premier third-party

test and validation center for servers, storage, and network

devices.

9/14/2017 4

NVMe-MI Introduction and

Testing

9/14/2017 5

NVMe Management Interface

What is the NVMe Management Interface?

A programming interface that allows out-of-band management of an

NVM Subsystem

9/14/2017 6

Management

Applications (e.g.,

Remote Console)

SMBus/I2C PCIe VDM

MCTP over

SMBus/I2C Binding

MCTP over

PCIe Binding

Management Component Transport Protocol (MCTP)

NVMe Management Interface

Management Controller

(BMC or Host Processor)

Management Applications (e.g., Remote Console)

Physical

Layer

Transport

Layer

Protocol

Layer

Application

Layer

Management

Applications (e.g.,

Remote Console)

NVMe-MI

NVMe-MI has the following key capabilities: Discover devices that are present

and learn capabilities of each device

A standard format for Vital Product Data and defined mechanisms to read/write VPD contents

Store data about the host environment; enabling a Management Controller to query the data later

Health and Temperature monitoring.

Multiple Command Slots to prevent a long latency command from blocking monitoring operations

Processor and operating system agnostic

9/14/2017 7

NVMe Management Interface (NVMe-MI)

The NVMe Management Interface is used to

Send Command Messages which consist of standard NVMe Admin Commands that target a Controller within the NVM Subsystem

Send commands that provide access to the PCI Express configuration, I/O, and memory spaces of a Controller in the NVM Subsystem

Send Management Interface specific commands for inventorying, configuring and monitoring of the NVM Subsystem.

Uses the concept of a “Management Endpoint”

Management Endpoints may support the same commands regardless of the Physical medium used.

9/14/2017 8

Example: Temperature monitoring

Fan Control can be based on temperature reported by a specific drive

BMC can monitor temperature of individual devices out-of-band

Servers may contain many dozens of drives

Management Interface can be used to identify individual drives

Failures can be detected and reported remotely through networked

infrastructure of BMCs

Management Interface simplifies greatly the identification and fault

finding within a large system

9/14/2017 9

NVMe Single and Dual Port SSD

Single port device has one PCIe connection

Dual port device has two PCIe connections on the device

SMBus/I2C link can be used for Management Interface

9/14/2017 10

Topology Overview

9/14/2017 11

Host Operating System

NVMe Drive

BMC Operating System

Host

PCIe Connector

SMBus connection

(Part of PCIe connector)

NVM Subsystem for Single Port Device

9/14/2017 12

NVM Subsystem for Dual Port Device

Dual Port Device has multiple Management Endpoints

9/14/2017 13

VPD – Vital Product Data

NVMe-MI defines a standard

format for VPD

NVMe-MI makes VPD contents

accessible out-of-band

Can be stored in NVM

Subsystem or on external device

(EEPROM) or both

Is accessible by any port

supporting NVMe-MI

Accessed using VPD read and

write MI commands

9/14/2017 14

In-Band vs Out-of-Band Management

NVMe driver communicates to NVMe controllers over

PCIe according to the NVMe Specification

Baseboard Management Controller (BMC) runs on its

own OS on it own processor independent from host OS

and driver

Two Out Of Band paths:

PCIe Vendor Defined Message (VDM) TLPs

SMBus connection (on PCIe/U.2 connector)

Important Note: PCIe VDMs are completely separate

from in-band PCIe traffic though they share the same

physical connection

9/14/2017 15

FMS 2015

MCTP/NVMe Management

Interface

MCTP

9/14/2017 16

NVMe-MI

Specification is moving to 1.0a

Incorporating changes since 1.0

First Plugfest planned for May 22nd 2017 at UNH

9/14/2017 17

MCTP/NVMe Management Interface Overview

NVMe-MI involves various different specifications

NVMe-MI Specification

NVMe-MI to MCTP Binding Specification

MCTP Base Specification

MCTP to SMBus binding Specification

MCTP to PCIe Vendor Defined Message binding Specification

PCI Express Base Specification

I2C/SMBus Specification

189/14/2017

MCTP/NVMe-Management Interface Overview

9/14/2017 19

NVMe Management Interface Messages

Management Interface Request Messages and Response

Messages are transported as MCTP messages with the

Message Type set to NVM Express Management

Messages over MCTP

See the MCTP IDs and Codes specification

All Command Messages originate with the Management

Controller and result in a Response Message from the

Management Endpoint.

9/14/2017 20

MCTP Initialization and Discovery

MCTP discovery occurs over the course of several discrete,

ordered steps:

Bus enumeration

Bus address assignment

MCTP capability discovery

Endpoint ID assignment

Distribution and use of routing information

9/14/2017 21

Byte ordering

MCTP specifications use big endian byte ordering while NVM

Express specifications use little endian byte ordering.

All figures in MCTP specification are illustrated with little

endian byte ordering. Note that this pictorial representation

does not change the order that bytes are sent out on the

physical layer.

9/14/2017 22

MCTP Bus Enumeration

The actions taken in this step are specific to a given

medium

Because enumeration of devices on the physical bus is

medium-specific, this information is provided in the

transport binding specification for the medium.

9/14/2017 23

NVMe-MI Conformance Testing

9/14/2017 24

NVMe-MI Conformance Test Specification

9/14/2017 25

NVMe-MI Conformance Testing

Tests are split into groups:

1. MCTP Base tests

2. MCTP Control Message Tests

3. MCTP Commands

4. NVMe Error Handling

5. NVMe Management Interface Tests

6. NVMe-MI Message Processing Tests

7. Control Primitives Tests

8. Management Interface Commands

9. NVMe Admin Command Set Tests

10. Management Enhancement Tests

11. Vital Product Data Tests

12. Management Endpoint Reset Tests

9/14/2017 26

Mandatory and Informational tests

Tests in the Test specification are marked as Mandatory

or Informational

Mandatory means the test is required for Integrators List

testing

Informational means the test will be run but the result will

not count towards requirements for the integrators list

Informational tests may become mandatory in the future

9/14/2017 27

In-band vs. Out of band testing

NVMe-MI is independent of Physical Transport

Tests apply to both (PCIe VDM) or SMBus

NVMe testing is separate from NVMe-MI

9/14/2017 28

Mandatory Test Level

NVMe-MI Conformance Testing – Mandatory tests

Tests are split into groups:

1. MCTP Base tests

2. MCTP Control Message Tests

3. MCTP Commands

4. NVMe Error Handling

5. NVMe Management Interface Tests

6. NVMe-MI Message Processing Tests

7. Control Primitives Tests

8. Management Interface Commands

9. NVMe Admin Command Set Tests

10. Management Enhancement Tests

11. Vital Product Data Tests

12. Management Endpoint Reset Tests

9/14/2017 29

Mandatory Tests (as of Plugfest #7)

9/14/2017 30

Test 4.1 - NVMe-MI Invalid Opcode – Mandatory Test

Test 4.2 - NVMe-MI Reserved Identifier – Mandatory Test

Test 4.3 - NVMe-MI Health Status Change – Mandatory Test

Test 4.4 - NVMe-MI Reserved Configuration Identifier – Mandatory Test

Test 4.5 - NVMe-MI MAXRENT Error – Mandatory Test

Test 4.6 - NVMe-MI Reserved Data Structure Type – Mandatory Test

Test 4.8 - NVMe-MI Invalid VPD Write Status – Mandatory test

Test 4.9 - NVMe-MI Invalid Parameter Status – Mandatory Test

Test 5.1 – NVMe-MI Message Type – Mandatory Test

Test 5.2 – NVMe-MI Message IC – Mandatory Test

Test 5.4 – NVMe-MI Command Slot – Mandatory Test

Test 5.6 – NVMe-MI MCTP packet padding – Mandatory Test

Test 5.7 – NVMe-MI Message Integrity Check – Mandatory Test

Test 6.1 – NVMe-MI Reserved Fields – Mandatory Test

Test 6.2 – NVMe-MI Error Response Code – Mandatory Test

Test 7.1 – NVMe-MI Response Tag – Mandatory Test

Test 7.2 – NVMe-MI Response Message – Mandatory Test

Test 7.4 – NVMe-MI Response Message Replay – Mandatory Test

Test 7.5 – NVMe-MI Response Replay Offset (RRO) – Mandatory Test

Test 8.1 – NVMe-MI Response Header – Mandatory Test

Test 8.2 – NVMe-MI Configuration Set – Mandatory Test

Test 8.3 – NVMe-MI Config Get Response – Mandatory Test

Test 8.4 – NVMe-MI Health Status Poll – Mandatory Test

Test 8.5 – NVMe-MI Controller Health Status Poll – Mandatory Test

Test 8.6 – NVMe-MI Read Data Structure – Mandatory Test

Test 8.7.1 – NVMe-MI Verify NVMSSI Data Length – Mandatory Test

Test 8.7.2 – NVMe-MI PortInfo Data Length – Mandatory Test

Test 8.7.3 – NVMe-MI CtrlrList Data Length – Mandatory Test

Test 8.7.4 – NVMe-MI CtrlrInfo Data Length – Mandatory Test

Test 8.7.5 – NVMe-MI OptCmds Data length – Mandatory Test

MCTP/NVMe-MI Conformance Process

Set up equipment

Z3-16 Exerciser (with SMBus support)

T34 Analyzer (using SMBus iPass cable)

Test platform

NVMe-MI Capable DUT

Select Test Cases

Power on DUT

Run Test Cases

Log File Created

Trace File Created

9/14/2017 31

Summit T34 Analyzer

Testing of different form factors

Testing of U.2 and M.2 Form Factors done using adapters

Protocol is basically the same regardless of form factor

32February 6th 2017

Dual Port U.2

Dual Port PCIe/NVMe devices exist now.

A Dual Port device has 2 separate independent PCIe links

It requires 2 analyzers and a separate Interposer to debug

Compliance Testing requires each port to be tested

separately

33

Single Port/Port A Port B

February 6th 2017

U.2 Dual Port Analysis Configuration

Host

Analyzer #1

PCIe

SwitchNVMe Drive

U.2(SFF8639)(TAP Only)

U.2(Dual)

Interposer

Analyzer #2

Sync

PC

U.2(SFF8639)

Dual port view

34

PCIe Gen.3 x4

(PCIe External Cable 3.0 )

9/14/2017

Running NVMe-MI Conformance Tests

Select Desired Tests in the Link Expert Menu

9/14/2017 35

Running NVMe-MI Conformance Tests

9/14/2017 36

Select “Run All”

Results

9/14/2017 37

Log Files

Result

Analyzing MCTP Trace files

Set view to MCTP Message Level/Command Level

Navigate to SMBus packets

What you should see is…

What this is doing/saying is…

There is SMBus traffic that is MCTP

Maybe what happens/should happen next

May have SMBus traffic that is not MCTP

May have MCTP traffic that is not NVMe-MI

9/14/2017 38

MCTP Decodes

MCTP Messages can be captured from SMBus and PCIe

Simultaneously

9/14/2017 39

Example MCTP Trace

Analyzer decodes SMBus traffic into MCTP messages and NVMe-MI

commands

9/14/2017 40

MCTP

NVME-MI

Triggering on NVMe-MI events

Set trigger to include SMBus

Set buffer to desired size

Set MCTP Trigger conditions

Record…

Search for….

SMBus Traffic

PCIe VDM Traffic

MCTP Decode in trace

9/14/2017 41

MCTP Triggers

9/14/2017 42

MCTP Packet Creation in Z3-16 Trainer Script

9/14/2017 43

Creating NVMe-MI Exerciser scripts

Create a new Trainer Script

Creates MCTP Messages

SMBus

VDM packets

Choose the fields for the packet

Save Script

Look at created packet

Run script

9/14/2017 44

Example: SMBus

Example: PCIe VDM

NVMe/NVMe-MI

Test Tools

9/14/2017 45

Debugging in Storage System Environment

9/14/2017 46

PCIe Storage Protocol Analyzer Teledyne LeCroy Summit T34

Protocol Analyzer Deep Buffer Memory (up to 64GB)

NVMe, SATA Express and SCSI Express protocol decoding, measurement, and performance analysis

SMBus Support

PCIe Storage Optimized Interposers 90 degree

Standard

M.2

U.2(SFF-8639) 5”

U.2(SFF-8639) 12”

PCIe External Cable

U.2(SFF-8639) 5”

90° Rack Mount

Standard

Interposer

M.2

InterposerSFF-8639 Dual12” InterposerPCIe External Cable 3.0

Interposer

New SMBus Products

47

Summit Z3-16 Exerciser

with SMBus SupportTest Platform

with SMBus Support PCIe G3x4/x8 Interposers

with SMBus Support

SMBus Trace

9/14/2017 48

SMBus

Packets

SMBus

Packets

G3x4/x8 Interposers with SMBus Support

Supports data rates

from 2.5 to 8 GT/s

x4 and x8 Interposers

NVMe MCTP/NVMe-

MI in-band/out-of-band

CLK REQ# and SRIS

support

Requires new

PE015UCA-X cable for

SMBus traffic

9/14/2017 49

PCIe M.2 Interposer with SMBus Support

Supports data rates from 2.5 to 8 GT/s

Linkwidth: x4

NVMe MCTP/NVMe-MI in-band/out-of-band

CLK REQ# and SRIS support

Supports M.2 SSD sizes of 30 mm, 42mm, 60mm, 80mm and 110mm (all x 30mm width)

Requires new PE015UCA-X cable for SMBus traffic

Individual interposer for “B” and “M” key SSD devices.

9/14/2017 50

PCIe U.2 Single and Dual Interposers with SMBus Support

Supports data rates from 2.5 to 8 GT/s

Linkwidth: Single: x4, Dual:x2

NVMe MCTP/NVMe-MI in-band/out-of-band

CLK REQ# and SRIS support

Requires new PE015UCA-X cable for SMBus traffic

Supports U.2 form factor SSDs.

5” and 12” Interposers available.

9/14/2017 51

PCIe External Cable 3.0 Interposer

Supports data rates from 2.5 to 8 GT/s

Up to x8 link widths

Supports PCI Express External Cabling Specification Revision 3.0

NVMe MCTP/NVMe-MI in-band/out-of-band

CLK REQ# and SRIS support

Power supply

Requires new PE015UCA-X cable for SMBus traffic

52

Gen 3 Test Platform

Supports data rates from 2.5 to 8 GT/s

Two x16 lane width CEM connectors(host/device)

NVMe MCTP/NVMe-MI in-band/out-of-band

CLK REQ# and SRIS support

Internal Interposer

Power supply

Requires new PE015UCA-X cable for SMBus traffic

9/14/2017 53

Summit Z3-16 Exerciser with SMBus Support

9/14/2017 54

Supports x1 – x16 @ 2.5/5/8Gbps

Connectivity Host Emulation Platform

Device Emulation Card

2GB Scripting Memory

Link Training and Status State Machine

(LTSSM) testing.

PCIe Error Scripting/Generation

Script Level Traffic Generation

Auto Convert Traces into Scripts

Point and Click Script Editor

Support Automated Test Scripts

SMBus Generation

PTM Messaging

Scripting Structure Support

NVMe / NVMe-MI

SATA Express

SCSI Express NVMe MCTP/NVMe-MI in-band/out-of-band

NVMe Compliance and Interoperability Testing

UNH-IOL(University Of New

Hampshire IOL) and the NVM

Express Promoters Group are

collaborating to create an interop

and conformance test program

centered at UNH-IOL.

They will be creating a multi-vendor

test bed to help products prove

interoperability, and conformance

test services to prove that products

follow the NVMe specification

correctly.

NVMe SSD ComplianceTester

SSD

Controller

(DUT)

Protocol

Analyzer

Exerciser

Running Compliance Tests

Test

Platform

Live Demonstration NVMe-MI Compliance Testing

9/14/2017 56

Q&A

9/14/2017 57

References

NVM Express Specification v1.2b

NVMe-MI Specification v1.0

PCI Express Base Specification v3.1

PCI Express CEM Specification v.3.0

MCTP Base Specification v1.2.1

MCTP SMBus/I2C Transport Binding Specification v1.0.0

MCTP PCIe VDM Transport Binding Specificaion v1.0.2

589/14/2017

Contact Teledyne LeCroy PSG

Fax(Sales&Service): 845-578-5985

Email Sales: psgsales@teledynelecroy.com

Phone Support: 1-800-553-2769

www.teledynelecroy.com

Summit Z416

Exerciser

SummitT416

Analyzer

More Information - Datasheet

9/14/2017 60

Thank You!

9/14/2017 61