TROIKA University 111 Technology Positioning Building a better Data Center Mike Dutch 2000-Dec-13 Copyright 2000 Troika Networks, Inc

TROIKA University 111

Technology PositioningBuilding a better Data Center

Mike Dutch 2000-Dec-13

Copyright 2000 Troika Networks, Inc.


Technology PositioningBuilding a better Data Center

This presentation is the property of Troika Networks, Inc. and should not be distributed outside the company

without explicit authorization.

Mike Dutch 2000-Dec-13

Copyright 2000 Troika Networks, Inc.


Networking Standards

OSI Reference Model (ISO 7498:1984)

TCP/IP Protocol Suite (IETF)

Ethernet (IEEE 802.3, ISO 8802-3)

Gigabit Ethernet (802.3z, 802.3ab, 802.3ae)

Fibre Channel (ANSI FC-PI, FC-FS, 10GFC)

InfiniBand (IBTA)


OSI Reference Model

Application

Presentation

Session

Transport

Network

Physical1

2

3

4

5

6

7Layer

Logical Link Control

Medium Access Control

Routing

Reliability

End to end communication

Transformation

Function

Framing

Signaling

OS Interfaces

Dat

a L

ink

Bridges

Repeaters

Routers

Gateways

Devices

Switches


TCP/IP Protocol Suite

Application

Presentation

Session

Transport

Network

Data-Link

Physical

Application

Transport

Internet

NetworkInterface

OSI Model DARPA Model

TCP

IP

802

.1

TCP/IP Protocol Suite

HT

TP

FT

P

Te

lne

t

DN

S

RIP

SN

MP

802

.3

802

.5

X.2

5

FR

AT

M

UDP

IGMPARP

ICMP

Eth

ern

et80

2.3

LAN WAN


Other network technologies

Frame Relay

X.25

TCP/IP

SDLC

Source: The Basic Guide to Frame Relay Networking, published by the Frame Relay Forum (www.frforum.com)

Lightweight WAN

Heavyweight WAN

Internet

IBM SNA


Fibre Channel Standard

Application

Presentation

Session

Transport

Network

Data-Link

Physical

OSI Model

FC-4

FC-3

FC-2

FC-1

FC-0

Functional Levels

Link Trans., Speed Neg., EnDec

Framing, Flow Control, CoS

Physical Components, SerDes

Upper Level Protocol Mappings

Common Services, Link ServicesNod

e

2048 4464244

CRC EOFSOF HDR PayloadHDR

…

Sequence

…

Sequence Exc

hang

e

FC-2 Frame

Channels Networks OtherP

ort

(OX

_ID

, R

X_I

D)

(SEQ_ID)

(SEQ_CNT)

HuntGroups

Signaling Protocol

Transmission Protocol

Interface / Media

ULPs / FC-4 Mappings


FC-2 Frame Fields

A frame is transmitted in the following byte order: A0..A3, B0..B35, C0..C3

Header

Link_ControlLink_Data:- Basic- Extended- FC-4Device_DataVideo_Data

Depends onRouting bits:

- Command- Unsolicited / Solicited- Category

Routing(bits 31-28)

Information(bits 27-24)

OptionalHeaders

fields uniquely identify a frame

Priority bitsClass 2 and 3Preference bit


FC-0 Nomenclature

FC-PH-2

FC-PI


Fibre ChannelClass of Service

Class of Service

Data DeliveryFlow

ControlExample Use

1

1-to-1

Reliable and dedicated(connection oriented)

Node Large files,

Full-bandwidth service

2

n-to-n

Reliable and multiplexed(connectionless)

Node,

Buffer

Clustering, Networking, OLTP

3

n-to-n

Datagram and multiplexed Buffer Storage, Networking, Broadcast, Multicast

4

n-to-n

Reliable and Guaranteed with QoS

Node,

Buffer

Real-time systemsFractional-bandwidth service

6

1-to-n

Reliable and dedicated, Multicast Server(uni-directional)

Node Video Distribution,

Data Acquisition

Source: FCIA, Fibre Channel, Connection to the Future, ISBN 1-878707-45-0, second edition, 1998Not to be confused with Quality of Connection (QoC) classes introduced in a Strategic Research profile.


Gigabit Ethernet

FC-1

FC-0

TCP UDP

IP

Applications

Layer 1

Layer 2

TCP/IP Suite(Upper Layers)

802.3 MAC

802.2 LLC

802.3z

802.3ab

1000Base-T1000Base-CX, SX, LX

WiringCloset

BuildingBackbones

CampusBackbones

25m

100m

220-275m

500-550m

5km

CX

T

SX(62.5u MM)

SX(50u MM)

LX (9u SM)

1000Base- Distance

CRCLengthPreamble LLCDestination Source Data

IEEE 802.3 Frame

46-1500 42668


InfiniBand Architecture

OSILayer

Upper LayerProtocols

Transport

Network

Link Encoding

MAC

1

2

3

4

5-7Consumer Transactions

InfiniBand Messages (QP)

Inter-Subnet Routing (GRH)

SwitchRouter

Subnet Routing (LRH)

Flow Control

Signaling

End IdlesStart Packet

Imm Data ICRCGRH xETH DataBTH VCRCLRH

PhysicalSymbols

QP

Consumer

WQE CQE

Transport

Port

Packet CA

Link


An InfiniBand Subnet

Switch

CPU

CPU

Chipset HCA Switch

TCA

Target

Memory

Processor Node

I/O Unit

Link

Router

SubnetManager

Consumer …

Channel Adapter

Port Port…Channel Adapter

Port Port……

Verbs

ConsumerConsumer

OS Message and Data Service

Processor Node

Infin

iBa

nd


InfiniBand

Link Width Designation

Laser Transmitter wavelength (nm),and connector

Optic Fiber (micron)Distance (meters, feet)

1X-SX 850,

dual LC

62.5/125 μm

multimode

2 – 125,

6.56 – 410.1

50/125 μm

multimode (beige)

2 – 250,

6.56 – 820.2

1X-LX 1310,

dual LC

9/125 μm

singlemode (blue)

2 – 10,000,

6.56 – 32,808

4X-SX 850,

single MPO (MTP)

62.5/125 multimode 2 – 75,

6.56 – 246

50/125 multimode 2 – 125,

6.56 – 410.1

12X-SX 850,

dual MPO (MTP)

62.5/125

multimode

2 – 75,

6.56 – 246

50/125

multimode

2 – 125,

6.56 – 410.1

InfiniBand Optical Links

T R


Application

Presentation

Session

Transport

Network

Data-Link

Physical

OSI Model

ApplicationLayers

IP

ATM

SONET/SDH

DWDM

De-factostandard

IP

ATM

DWDM

IP

DWDM

SONET/SDH

IP over ATM

IP

DWDMIP over Fiber

Potentialarchitectures

Packet over SONET

(POS)

ATM = Asynchronous Transfer ModeSONET = Synchronous Optical NetworkSDH = Synchronous Digital HierarchyDWDM = Dense Wave Division Multiplexing

Wide Area Data Networking


Packet Switching

FrameSwitching

Application

Presentation

Session

Transport

Network

Data-Link

Physical

TCP, UDP

IP, ARP,ICMP, IGMP

802.2 LLC802.3 MAC

IEEE 802.x

HTTP, FTP,DNS SNMP

RPC/XDR

Sockets (1)

SPX, NCP,SPP/PEP

IPX, IDP

QP

GRH

LRH

iSCSI, iFCP,FCIP, SEP

Direct (3)NetBIOS (2),XNS Courier

NFS, XNS, RFS, NW

SONET/SDHDWDM

FC-3FC-2

FCP IPFC FC-VI others

FC

-4

ATM

OSI

DAFS

SCTP, STP

InfiniBand IP Storage

SCSI

FibreChannel

NetworkFile Systems

Internet

CellSwitching

CircuitSwitching

SMB, NFSP,Streams

FC-1FC-0

1X, 4X, 12X

SNADS DIA, DDM

PresentationServices

Data FlowControl

TransmissionControl

Path Control

ANSI FDDITR MAC

FDDI ProtocolFDDI MDI

SNA

Primary Block Stream Changer Multimedia Controller Enclosure Object

SPI SBP FCP SSA-S3P SST SRP

Interconnects(4)

Com

man

d S

ets

Pro

toco

ls

SCSI

7

6

5

4

3

2

1

Summary of Interfaces


Transport Interfaces

Virtual Architecture (VI) Architecture

Sockets and NetBIOS

Windows Sockets Direct Path / SDP

Direct Access (Fast) Sockets


VI Architecture

Application

OS Communication InterfaceSockets, MPI, Cluster, Other

VI User Agent

VI Network Adapter

VI Kernel Agent

User Mode

Kernel Mode

VI Provider

VI Consumer

RECEIVE

SEND

VI

RECEIVE

SEND

VICOMPETION

CQ

Send/Receive/RDMA Read/RDMA Write

Open/Connect/Register Memory


Application Interfaces

TCP

IP IGMPARP

ICMP

Network Interface Layer

UDP

WindowsSockets

NetBT

WinSockApplication

NetBIOSApplication

Session andDatagram Services

Stream andDatagram Sockets


WinSock Direct

TransportDriver

Interface

ServiceProviderInterface

TCP/IP

SAN (VI)

WinSockProvider

Base

WinSockProvider

NDIS

Switch

WinSock

User mode

Kernel mode


Direct Access Sockets (Fast Sockets)

Application

Sockets API

FSockApp

Instance

FSockBroker

VI NIC

OS

Node

Application

Sockets API

FSockApp

Instance

FSockBroker

VI NIC

OS

Node

Data Flow

FSock Flow

App Flow


Storage Protocols

Block-level I/O – SAN Channel Protocols IP Storage

File-level I/O - NAS NFS CIFS DAFS

Object-level I/O

DB/File System

Block Storage

ApplicationSAN

NAS

SAN groups the file systemwith the application

NAS groups the file systemwith the storage


SNIA Storage Model

File/record subsystemFile/record subsystem

Database(dbms)

File system(FS)

Sto

rage

dom

ain

Application

Se

rvic

es

su

bs

yst

emS

erv

ice

s s

ub

sy

stem

Dis

cove

ry,

mon

itorin

gD

isco

very

, m

onito

ring

Res

ourc

e m

gmt,

con

figur

atio

nR

esou

rce

mgm

t, c

onfig

urat

ion

Sec

urity

, bi

lling

Sec

urity

, bi

lling

Red

unda

ncy

man

agem

ent

Red

unda

ncy

man

agem

ent

Hig

h av

aila

bilit

yH

igh

avai

labi

lity

Cap

acity

pla

nnin

gC

apac

ity p

lann

ing

Block subsystemBlock subsystem

Storage devices (disks, …)Storage devices (disks, …)

Block aggregationBlock aggregation

Device-basedblock-aggregation

SAN-basedblock-aggregation

Host-basedblock-aggregation


Sample Architectures

Hos

t. w

ith L

VMan

d so

ftwar

e R

AID

1. Direct-attach

File

/re

cord

su

bs

yst

em

Blo

cks

ub

sy

ste

m

Device block-aggregation

SAN block-aggregation

Host block-aggregation

NASserver

4. NAS serverHost3. NAS head

NAS head

Host

LANH

ost.

with

LVM

Disk array

SAN

2. SAN-attachApplication


Small Computer Systems Interface


SCSI Request/Response

Initiator

ApplicationClient

Target

Logical Unit

TaskManager

DeviceServerDevice Service

Task Management

Source: “iSCSI Technical Overview”, by Mike Mesnier, Intel, March 2001


iSCSI

SCSI Application

iSCSI ProtocolServices

SCSI DeviceServer

iSCSI ProtocolServices

Initiator I/O System Target I/O System

SCSI CDB

iSCSI ProtocolLayer

TCP/IP

EthernetData link +Physical

Data link +Physical

SCSIApplication

Layer

iSCSI PDU

TCP segmentsin IP

datagrams

EthernetFrame

SCSI ApplicationProtocol

iSCSI Protocol

Ethernet

Protocol ServiceInterface

iSCSI TransportInterface

TCP/IPTCP/IPTCP/IP

TCP/IPTCP/IP

TCP/IP TCP/IP Protocol

iSCSI session

Source: “iSCSI Technical Overview”, by Mike Mesnier, Intel, March 2001

Challenges /Proposals

• TCP/IP overhead• Multiple data copies• Out of order delivery

• TCP/IP offload• Large buffers• Stream Markers• RDMA (WARP/SCTP)

TCP Header

4820

iSCSI Header802.3 Header

2014

IP Header iSCSI Data

46

802.3 Trailer


Connecting FC Fabrics

FibreFibreChannelChannel

SANSAN

FibreFibreChannelChannel

SANSAN

FCIPTunnel

FCIPTunnel

IP Network

FCServer

FCJBOD

FCServer

FC TapeLibrary

FCJBOD

FCServer

FCServer

FC TapeLibrary

FCIP creates a single fabric by connecting E_Ports through gateways

IP Tunneling


Connecting Storage Devices

IP Network

iFCPGateway

iFCPGateway

iFCPGateway

iFCPGateway

FCServer

FCServer

FC TapeLibrary

FCServer

FCServer

FC TapeLibrary

iSCSI attaches iSCSI storageiFCP attaches FC storage

iSCSIServer

iSCSIStorage

FCJBOD

FCJBOD


Distributed File Systems

Network File System (NFS)

Server Message Block (SMB) Common Internet File System (CIFS) Samba

Many others (e.g. XNS, RFS, FTP)


Server Message Block


Direct Access File System

A file access and management protocol optimized for local file sharing and clustering

Designed to use memory-to-memory data movement technologies like VI and InfiniBand

TROIKA holds leadership positions

OS InterfaceLibrary

VI ProviderLibrary

Application

Adaptation Library orTransparency Library

DAFS ClientLibrary

File SystemSwitch

DAFS ClientFile System

VI ProviderDriver

OS InterfaceLibrary

Application

Example Deployment ModelsLocal File-Sharing Environment


DAFS Protocol

Session Management• Operation Channel• Back-Control Channel• RDMA Read Channel• Security

File System Operations• Data Transfer

• Inline (Send/Receive)• Direct (RDMA)

• NFSv4-derived operations• Batch write operations• Client read-ahead into server prefetch buffer• Locking (Persist, Autorecover)• Chaining

Failure Recover• “Exactly Once” Semantics• Server Response Cache• Server Failover• File System Migration• Fencing


DAFS Provider Architecture


Positioning Technology

Understand the business goals

Select technologies designed to solve the technical challenges

Simplify to balance risk of deployment with cost of operation


Applications Driving Network Growth

Applications Data Types / Size Traffic Implications

• Internet (public access)• Extranet (partner access)• Intranet (company access)

• Data (e-mail, browsers)• Audio/Video• High transaction rates• Large files to 100 MB

• Large files increase required bandwidth• Low transmission latency• Many data streams

• Scientific Modeling (3D)• Engineering (distr. teams)• Publications (digital printing)• Medical Data Transfer

• Data• 100s of MBs to GBs

• Large files increase required bandwidth

• Data warehousing• Network backup

• Data (queries, reports)• Updated regularly• GBs to terabytes

• Large files increase required bandwidth• Transmitted during fixed and limited time period

• Video conferencing• Interactive whiteboardiing

• Constant data stream•1.5 to 3.5 Mbps at desktop

• Class of service reservation (predicatable latency)• Many data streams

Reference: Gigabit Ethernet Alliance White Paper, 1998


Network Positioning

Fibre Channel Well-suited for Storage Area Networks TROIKA enables converged networks

Gigabit Ethernet Designed for Ethernet compatibility TROIKA monitoring use for IP Storage

InfiniBand Designed for processor communication TROIKA considering product development


Interface Positioning

Virtual Interface (VI) Designed for high-performance I/O Evolving to support InfiniBand

WinSock Direct / Sockets Direct Protocol Designed for transparent sockets acceleration on

Windows 2000 Datacenter Server – SoIB basis Direct Access Sockets (DASockets)

Designed for sockets acceleration on Unix and Windows platforms (not as transparent)

Direct Access File System (DAFS) Designed for high-performance local file sharing


Building a Better Data Center

Storage NetworkCluster NetworkClient Network

Rest of World Server Farms and Appliances Block and File Storage

Block I/O – SCSI (FCP, SRP, iSCSI)File I/O – NFS, CIFS, DAFS, SDP

Intra-server – PCI, PCI-X, IBInter-server – Proprietary, Ethernet, FC, IB

Local – EthernetRemote – ATM, SONET, DWDM

InfiniBand

Fibre Channel

Gigabit Ethernet

Router

PCI-X, IB10/100, GbE

LAN

MANWAN

FC, GbE, IB

Affordable, Manageable, Available, and Secure


Questions & Answers

Name the technologies, interfaces,and protocols we discussed.

Are the technologies converging or balkanizing?

Describe how to deploy these technologies to build a better data center.

Documents

TROIKA University 111 Technology Positioning Building a better Data Center Mike Dutch 2000-Dec-13 Copyright 2000 Troika Networks, Inc