Low Latency Server Virtualization UiUsing R idIORapidIO

Low Latency Server Virtualization U i R idIOUsing RapidIO

November 29-30, 2011 – Server Design Summit

Mohammad AkhteMohammad Akhter

Devashish Paul

Integrated Device Technology

©2010 Integrated Device Technology, Inc.

The Analog and Digital Company™

Application-Optimized Mixed-Signal Solutions

AnalogThe World Leader in Timing,

Serial Switching and Interfaces• Maximum system performance

• Lowest overall Bill-of-Materials• Shortest time-to-market

System

● in Silicon Timing● in Memory Interfaces System

Expertise● in Memory Interfaces● in DisplayPort™ ● in RapidIO® Switching ● in PCIe® Solutions

Analog

● in PCIe Solutions● in Business PC Audio● in VME Bus Solutions● in PowerPC® Host Bridges

• Communications

• ComputingAnalog● in PowerPC Host Bridges

• Consumer

CONFIDENTIALPAGE 2 CONFIDENTIALwww.IDT.com

Re-thinking the System to Architect a better solution

Agenda●RapidIO Success●RapidIO Success

●RapidIO Enables Disruptive Server Architecture

●Server IO Virtualization●Server IO Virtualization

●IO Virtualization Flows

●Servers with RapidIO●Servers with RapidIO

●IO Virtualization using RapidIO

●SummarySummary


RapidIO Success●Lowest Latency Interconnect ~ 100 ns●Lowest Latency Interconnect ~ 100 ns

●Shipped millions of devices

●Scalable System●Scalable System● Inherently scales to large system with thousands of nodes● Scale Compute Nodes, Storage blades, Aggregate Chassis to large

systemsy

●Addresses Critical Interconnect Problems in multiple markets● Military, Video, Computing, Wireless Industry

●Large Eco-System

Silicon Partners and Suppliers


RapidIO Enables Disruptive Server Architectures● Supports both x86 and ARM based Architectures● Supports both x86 and ARM based Architectures

● Eliminates need for large ToR Switches● Large Servers x86 CPU with RapidIO● Micro-servers using ARM CPU with power efficient low cost RapidIO Interconnect

● Supports Thousands of Nodes and flows● Supports Thousands of Nodes and flows

● Best-in-class end-to-end latency● Switch latency around 100 ns

● Supports Secured Virtualization

High PerformanceLow Latency

RapidIO based Serverpp

● Supports any kind of topology● Start, Mesh, Dual-star, Hypercube, Torus etc.. RapidIO

Switch Hub

RapidIOCPS-1848

48 laneswitch

Compute Node RapidIO Switch

x86CPU

PCIe2 –S-RIO 2

x86CPU

PCIe2 –S-RIO 2

x86CPU

PCIe2 –S-RIO 2

x86CPU

PCIe2 –S-RIO 2

RapidIOCPS-1848


CPU CPU CPU CPU

RapidIO Based Compute Node

48 laneswitch

RapidIO in Server

Lowest latency 100 nsLowest latency 100 nsScales to 1000’s of nodes

8W per switch< 2 W per “NIC”

13x13 mm PCIe to S-RIO “NIC”

RapidIO Switch Hub

RapidIOCPS-1848

48 laneswitchswitch

Compute Node RapidIO Switch

PCIe2 –S-RIO 2

PCIe2 –S-RIO 2

PCIe2 –S-RIO 2

PCIe2 –S-RIO 2

RapidIOx86CPU

x86CPU

x86CPU

x86CPU

RapidIO Based Compute Node

RapidIOCPS-1848

48 laneswitch


13x13 mm PCIe to S-RIO “NIC”For < 2W

p p

Server IO Virtualization●What is Server IO Virtualization●What is Server IO Virtualization

● Logical sharing of physical IO resources between server and data center resources

●Why care?● On-demand efficient allocation of IO● Match CPU and OS processing requirements to IO● Allow better utilization of CPU● Allow many nodes in the system to access IO resources


Server IO Virtualization

OS

APP

APP

APP

Kernel/Driver Driver/OS

Applications

Core 0 Core Nory

ory

Kernel/Driver Driver/OS

CPU 0

Mem

Memo

IO HubStorage

LPC

Hardware

Compute Node

I/O Device


Server IO Virtualization

Guest OS

APP

APP

APP

HypervisorGuest OS

APP

APP

APP

Driver/OS

Applications

Core 0 Core Nmo

ryor

y

( )Hypervisor

(Kernel/Driver)/

CPU 0

Mem

Mem

IO HubStorage

LPC

Hardware

Compute Node

I/O Device


IO Virtualization - Flows

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

APP

Core 0 Core N

mory

ory

Guest OS

( )Hypervisor

(Kernel/Driver)

Guest OS

Core 0 Core Nory ry

Guest OS

( )Hypervisor

(Kernel/Driver)

Guest OS

CPU 0

IO H b

Mem

Memo

Storage

CPU 0

Core 0 Core N

Memo

Memo

Storage

Compute Node

IO Hub

I/O Device

LPC

Compute N d

IO Hub

I/O Device

LPC

LAN

Node Node

SAN


LAN

IO Virtualization - Hypervisor Bypass

● Communication through Hypervisor is expensive● Higher CPU overhead● Higher latency

Guest OS

APP

APP

APP

APP

APP

APP

Hypervisor(Kernel/Driver)

Guest OSAP

P AP

P AP

P AP

P

APP

APP

● Early days IO virtualization was done entirely by software

● Virtualized enabled IO Card bypasses Hypervisor

CPU 0

Core 0 Core N

Memo

ryMe

mory

Hypervisor● Data path goes directly from OS to IO Card● IO device creates virtualized view of distinct

resources and address space to OS● Control/Management functions may still flow

IO HubStorage

LPC

Control/Management functions may still flow through Hypervisor Compute

NodeI/O Device


IO Virtualization using RDMA

Guest OS

APP

APP

APP

APP

APP

APP

( )Hypervisor

(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

Guest OS

APP

APP

APP

APP

APP

APP

(Kernel/Driver)Hypervisor

(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

CPU 0

Core 0 Core N

Memo

ryMe

mory

CPU 0

Core 0 Core N

Memo

ryMe

moryRDMAP

DDPMPATCP

IO Hub

Storage

LPC

Compute

IO Hub

I/O Device

Storage

LPC

TCP

Compute Node

I/O Device Compute Node

I/O Device

Ethernet Switch

Today’s I/O NIC have too many steps for RDMA! Higher latency and Power


Higher latency and Power Higher cost and CPU overhead

IO Virtualization using RapidIO

Guest OS

APP

APP

APP

APP

APP

APP

( )Hypervisor

(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

Guest OS

APP

APP

APP

APP

APP

APP


(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

CPU 0

Core 0 Core N

Memo

ryMe

mory

CPU 0

Core 0 Core N

Memo

ryMe

mory

IO Hub

Storage

LPC

Compute

IO Hub

I/O Device

Storage

LPC

Tsi721Compute Node


I/O DeviceTsi721 Tsi721

S-RIO Switch

PCIe to S-RIO bridge and Switch Simplies IO Virtualization Improves RDMA without overhead – built in DMA/Messaging Reduces latency


y Supports large number of nodes Improves IO and CPU utilization

RapidIO: Low power Servers

Guest OS

APP

APP

APP

APP

APP

APP

( )Hypervisor

(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

Guest OS

APP

APP

APP

APP

APP

APP


(Kernel/Driver)

Guest OS

APP

APP

APP

APP

APP

APP

CPU 0

Core 0 Core N

Memo

ryMe

mory

CPU 0

Core 0 Core N

Memo

ryMe

mory

IO Hub

Storage

LPC

Compute

IO Hub

I/O Device

Storage

LPC

Tsi721Compute Node


I/O DeviceTsi721 Tsi721

S-RIO Switch

PCIe to S-RIO bridge replaces your NIC for <2W


g p yin a 13x13 package!!

RapidIO: Layered Protocol

Gl b ll

System Integration

Logical Specification

Globally SharedMemory

Message Passing

Logical I/O Data Streaming

ardw

are Flow

ControlSystem Bring up

Transport Specification

Transport

men

ted

in H

a

Multicast

Error

Interop

Bring up

x8, x16 ParallelPhysical

SpecificationFuturePhysical Specs

x1, x2, x4, x8, x16 Serial

Impl

em

Error Mgmt

p


IO Virtualization using RapidIO

● Low latency Memory to Memory cut-through transactions AP

P AP

P AP

P AP

P

APP

APP

APP

APP

APP

APP

APP

APP

transactions● Bypass Hypervisor for data transfer

● I/O virtualization in RapidIO● 1000’s of Nodes with 16 bit deviceID

VNIC

Guest OS

VM

VMM VMM ManagementAPI

VNIC

Guest OS

VM

● 1000 s of Nodes with 16 bit deviceID● Message Passing Logical Layer

● Map Mailbox, Letter, xmbox to IOV needs● Direct IO Logical Layer

M d ith 66 bit /d ti ti

Device Driver

VMM VMM ManagementAPI

Hypervisor

● Many nodes with 66-bit source/destination address and TID

● Data Streaming Logical Layer● 1000’s of flows with streamID, xtype, xh CPU 0

Core 0 Core N

Memo

ryMe

mory

IO Hub

Storage

LPC


Compute Node

RapidIO

x86 Server with RapidIO Gen2 (Today)

PCIe to S-RIO bridge and Gen2 S-RIO Switch brings x86 CPU into RapidIO based Servers

Multiple links each 20G


pMultiple storage, networking, computing and chassis linksScalable Virtualized System

Native RapidIO Gen2 in ARM Server

● Native RapidIO with ARM● RapidIO Gen2 Switch

M l i l li k h 20G● Multiple links each 20G• Multiple storage links• Multiple compute node links• Multiple networking links

● Lowest power● Lowest latency● Lowest footprint

S d I/O Vi t li ti● Secured I/O Virtualization with RDMA


x86 Server with RapidIO 10xN (Future)

10xN RapidIO-PCIe Bridge brings x86 into RapidIO based Servers Support Large Servers Live VM migration Lowest Switch Latency Modular Scalable Architecture

b d h l


Upto 160 Gbaud per port with 10.3125 G per lane

ARM Server with Native S-RIO 10xN

Native on-chip RapidIO Interconnect Lowest Latency Lowest Power consumption


p Smallest Footprint Secured Virtualized Architecture

Example S-RIO based Server Cluster

Compute Node withpRapidIO NIC

S l bl A hit t ith R idIO S it h d NIC


Scalable Architecture with RapidIO Switch and NICAny kind of Topology

Summary

R idIO Si lifi Vi t li ti●RapidIO Simplifies Virtualization● Low overhead RDMA● Live Virtual Machine migration

1000’ f d d fl● 1000’s of nodes and flows● 100’s of virtual NICs

●Lowest Latency Interconnect in the industryy y●~100 ns switch latency

●Green Solution● 8W per switch and <2W per NIC

●Enables large Data Center for Cloud computing● Low Power Micro servers with ARM and NATIVE RapidIO● Low Power Micro servers with ARM and NATIVE RapidIO● Macro servers with x86 and PCIe to RapidIO NIC

RapidIO Simplifies IO VirtualizationRapidIO Simplifies IO Virtualization


p pLowest Latency, Power, and Costp p

Lowest Latency, Power, and Cost

ContactVisit Serial RapidIO Products at www idt comVisit Serial RapidIO Products at www.idt.com

● Product Data Sheets● Reference Designs● RapidIO Trade Association● RapidIO Trade Association

● http://www.rapidio.org/home/news/membernews● RapidIO Overview

● http://www.eetimes.com/design/signal-processing-p g g p gdsp/4017736/Backplane-tutorial-RapidIO-PCIe-and-Ethernet

● RapidIO versus Ethernet● http://advancedtca-systems.com/comparing-ethernet-rapidio/

Integrated Device Technology6024 Silver Creek Valley Road


6024 Silver Creek Valley RoadSan Jose, CA 95138

Documents

Low Latency Server Virtualization UiUsing R idIORapidIO