Hosted by

10Gig Emergence in the Data Center

Marc Staimer, CDS, Dragon Slayer Consultingmarcstaimer@earthlink.net

Hosted by

Agenda

10Gig “101”

Applications

Value Prop

Issues

Market Forecast

Conclusions

Hosted by

10Gig “101”

What is 10Gig?

Why should I care?

10Gig applications?

What is the value proposition?

When will it matter?

Hosted by

What is 10Gig really?

Usually refers to the usable bandwidth• 10Gbps

Ethernet, Fibre Channel, & SONET OC192

10x 1Gig

12.5Gbps total including overhead

• InfiniBand (IBA) is slightly lower

10Gbps is total and 8Gbps is net a.k.a. as 4X

Hosted by

Why should I care?

BW increasing > than ability to use it

My server I/O can’t use it

And my backbones are being swamped

Hosted by

10Gig applications

Switch Trunking

Server-to-storage fan-out

HPCC

DBMS clustering

Shared I/O eliminating bus contention

Hosted by

Switch TrunkingEthernet, FC, & IBA

• Ethernet: 10/100/1000 to the edge, 10Gig Core

• FC: 1,2,4 Gig to edge, 10Gig Core

• IBA: 4X (10Gig) edge, 12X (30Gig) Core

1Gig Ethernet

10Gig Ethernet10Gig SONET

10Gig iSCSI

1Gig Ethernet

10Gig Ethernet

Hosted by

Source: Fibre Channel Industry Association

FC Throughput Gain Example

Speed Throughput MByte/s

(full duplex)

Line Rate

(Gbaud)

1 GFC 200 1.0625

2 GFC 400 2.125

4 GFC 800 4.25

10 GFC 2,400 10.5 or 3.1875

Hosted by

10Gig-4/2Gig FC Switches

Server-Storage Fan-Out Results

FC definitely, GigE (RDMA) maybe• Server-storage pt fan-out increases from 8:1 to 48:1

COMPACT

Pow e r

COMPACT

Power

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Power

COMPACT

Pow e r

COMPACT

Power

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Power

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

COMPACT

Pow e r

40 (1U) IA Application

Servers10Gig FC SAN

Storage

Hosted by

HPCC

InfiniBand • Potential 10GigE (RDMA) down the road

• Key node-to-node issues

Very low latency (minimal fabric hops)

Very high bandwidth

Hosted by

128 (4X) IBA ports in 12U

HPCC Illustrated 128 node 4X IBA Fabric

Vertical rack space• 12U

10Gig connection• Copper

Full bi-sectional bandwidth• 10Gig/port

Max node-to-node hops• 3 Switches, 5 ASICs

Latency• Memory-to-memory = 6s

~ List pricing• < $1k/port

Hosted by

DBMS Clustering Increasing DBMS performance

• IBA (primary focus)

• GigE

Value Prop

• Lower latency

> IOPS

• Higher throughput

• Fewer connections

< complexity

< mgt

IPoIB, uDAPL, SDP, SRP, &

FCP, over IBA

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

Oracle RAC or DB2 Cluster

FC SAN Storage

Hosted by

10Gig Shared I/O: Eliminating Bus Contention

4X IBA HCA on PCI-X bus• Provides I/O for

TCP/IP to Ethernet FCP to Fibre Channel iSCSI to Ethernet

• Shares 10Gig pipes Transparent to apps < cables < costs < complexity

Potentially doable on• FC or Ethernet w/RDMA

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

PowerEdge2450

Lintel/Wintel servers

FC SAN Storage

Hosted by

Blade Server “Fan-in”

Boot OS from external storage

• More blades per storage device

• High activity at startup

SAN Storage

10Gig

1/2/4 Gig

Hosted by

10Gig Issues

I/O Infrastructure

Timing, Availability, & Cost

Copper vs. Optical

Compatibility

MSAs

Hosted by

10Gig I/O Infrastructure

Current I/O Buses cannot handle 10Gig throughput

• PCI = Max 4Gbps

• PCI-X = Max 8Gbps

• Each additional bus card cuts the max in half

Future I/O infrastructure is slipping to the right

• PCI-X 2.0 = Max 26.4Gbps

• PCI Express 1.0 = Max 128Gbps

• Servers utilizing new I/O not available until late 04 early 05

• Storage utilizing new I/O not available until late 04 early 05

Hosted by

10Gig Timing, Availability & Costs

Infrastructure

• Optics

~ $1.2K to $5K/port

• Ethernet Switches

~ $29K/port

Decreasing ~28%/yr

• ~ $8K by 2007

• FC Switches

~ $1.5K/port (w/o optics)

• ~ $.5K/port by 2007

• IBA Switches

~ $1K/port (w/o optics)

Adapters

• 10Gig Ethernet NICs

~ $6K

Timing: 2005

• 10Gig FC HBAs & Target ASICs

~ $5K

Timing: 2005

• 4X IBA HCAs

Timing: Now

12X (30Gig)

• Late 04 early 05

Hosted by

Gartner 10Gig Market Forecasts

Ethernet Switches 2003 2004 2005 2006 2007 CAGR

Ports 1,800 5,000 15,200 51,800 185,000 218%Revenue (K) 52,200$ 97,500$ 218,272$ 554,260$ 1,426,350$ 129%Price/Port 29,000$ 19,500$ 14,360$ 10,700$ 7,710$ -28%

FC Switches 2003 2004 2005 2006 2007 CAGRPorts - 49,000 393,000 1,317,000 2,303,000 261%

Revenue (K) -$ 68,600$ 321,300$ 903,300$ 1,209,700$ 160%Price/Port -$ 1,400$ 818$ 686$ 525$ -28%

IBA Switches 2003 2004 2005 2006 2007 CAGRPorts 2,450.0 81,375 189,177 402,325 531,650 284%

Revenue (K) 3,063$ 81,375$ 160,800$ 261,511$ 239,243$ 197%Price/Port 1,250$ 1,000$ 850$ 650$ 450$ -23%

*Note: IBA numbers are calculated from the Gartner/Dataquest forecast

Hosted by

10Gig Copper vs. Optical

Copper

• Low cost

• Limited distance

~ 15 meters

• Not Cat 5 or 6 compatible

• Cat 7 work going on

~ 100 meters

NOTE: Meter = 3.28 feet

Optical

• High cost

• Multi-mode (common)

Distance limited

• 300 to 550 meters

• Designed for single mode

Dark fiber

• 10 Km

• 40 Km

• Up to 64 Km

Hosted by

10Gig Compatibility Question

Is 10Gig backwards compatible?• Ethernet

A. Yes

B. No

• Fibre Channel

A. Yes

B. No

Hosted by

10Gig Compatibility

10Gig Ethernet & FC

• Not backwards compatible

It’s the optics

And the encoding

• 8B/10B

• 64/66

Hosted by

10Gig Definitions

XAUI• 10Gig attachment unit interface

XMGII• 10Gig media independent interface

Transponder• Module containing

Optical transmitter & receiver

& mux that changes line rate

MSA• Multi-source agreement

802.3ak• 10Gig over copper

RDMA• Remote direct memory access

RDDP• Remote direct data placement

RDMA on TCP/IP & GigE

QuickTime™ and a decompressor are needed to see this picture.

Hosted by

10Gig MSAsTransponder MSAs

• Xenpak Intel, Agilent, Infineon, JDSU, Picolight

• XPAK Agilent, Intel, Picolight

• X2 Agilent, JDSU, Mitsubishi, OpNext, Optillion

• IBPAK Agilent, Infineon, InfiniCon, Mindspeed,

Molex, OCP, Picolight, SUN, Tyco, W. L. Gore

Transceiver MSA• XFP: 10Gig serial transceiver

JDSU

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

Xenpak

XPAK

X2

XFP

Hosted by

IBPAK

4x Copper Cable

MPO Optical Cable

Dual MPO Optical Cable

12x OpticalPluggable Module

4x OpticalPluggable Module

4x CopperPluggable Module

Module Holder

106 ckt Z-Axis Connector

Hosted by

10Gig Value Prop

Reduced Infrastructure Costs• < Cabling

• < Connections

• < Complexity

• < Management

Increase Performance• > Throughput

• < Latency

Hosted by

10Gig Market Emergence

IBA: Now• Mainstream 2004

Ethernet: Now• Mainstream 2005

Fibre Channel: 2004• Mainstream 2005/2006

Hosted by

10Gig Conclusions

It’s coming

There are real cost justifiable applications

1st applications are in 2004

Mainstream market = 2005/2006

Hockey stick = 2007

Hosted by

Questions