Patrick Crowley and Jon Turnerand John DeHart, Mart Haitjema Fred Kuhns, Jyoti Parwatikar, Ritun Patney, Charlie Wiseman, Mike Wilson, Ken Wong, Dave ZarComputer Science & EngineeringWashington University

www.arl.wustl.edu

Prototype Deployment of Internet Scale Overlay Hosting

2

Project Objectives

Deploy five experimental overlay hosting platforms » located at Internet 2 PoPs» compatible with PlanetLab, moving to GENI control framework» performance characteristics suitable for service deployment

• integrated system architecture with multiple server blades• shared NP-based server blades for fast-path packet processing

Demonstrate multiple applications

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

Control Processor

General Purpose Processing Engines

Network Processing EngineLine Card

Chassis Switch

External Switch

3

Role in Cluster B•Overlay Hosting Platform•Deployed at 5 locations• Serve as backbone linking cluster partners (and others)•Demonstrate high quality service delivery on overlays•Managed using experiment support tools

4

Target Internet 2 Deployment

23

2

22

33

5

Hosting Platform Details

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

PLOS

. . .

General PurposeProcessing Engine

filter

. . .

. . .

filter

Line Card

lookupparse headerformat

... ...

queues

...

Network Processing Engine

6

Application Framework Fastpath/slowpath

» fastpath mapped onto NPE» slice manager in vServer

on GPE

Configurable elements» code option – determines

how packets processed by parse, header format

» logical interfaces• may be public or tunnel• guaranteed bandwidth

» TCAM filters» Queues

• length, bandwidth

Slice manager can configure fastpath using provided library» or manually, using command line interface

Parse Lookup

Filters

Control Interface

HeaderFormat

QueueManagerFast Path

...

...

...

outp

ut

inte

rface

s

input

inte

rface

s

SliceManager

(in vServer)

GPE

Remote Login Interface

exce

pti

on p

ack

ets

& in-b

and c

ontr

ol

exce

pti

on p

ack

ets

& in-b

and c

ontr

ol

7

Creating a SlicePLC

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

SRM

8

Starting up a Slice

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

SLM

Logging into slice

SFTP connection for downloading code

Requires Network Address

Translationdatapath detects new

connection, LC control processor

adds filters

9

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

LC

SRM

RMP

Configuring an External Port

●Request external port number on specific interface thru Resource Manager Proxy (RMP)

●RMP relays request to System Resource Manager (SRM)

●SRM configures LC filters for interface

●Arriving packets directed to slice, which is listening on socket

10

Setting Up a Fast Path

●Request fastpath through RMP●SRM allocates fastpath●Specify logical interfaces and

interface bandwidths●Specify #of filters, queues, binding

of queues to interfaces, queue lengths and bandwidths

●Configure fastpath filters

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

LC

SRM

RMP

lookupparse headerformat

queues

...

11

23

2

22

33

Planned Wide-Area OpenFlow

CPnetFPGA

NOX lookup

pars

e

hdrF

mt

queue

WashU

Princeton

Stanford

GaTech

Texas

NOX

to SPP

12

NP Blade

SPI Switch

xScaleinput

. . .ME Mem

Int

SRAM SDRAM

ME

ME

output

Exte

rnal

Inte

rface

sSw

itch

Inte

rface

TCAM

TCAMInt

NP A

NP B

Loc. Mem.

TC . . .TC

ALU

thre

ad

conte

xts

13

Line Card Datapath

Lookup(2 ME)

RxIn(2 ME)

TxIn(2 ME)

QueueManager

(4 ME)

KeyExtract(2 ME)

HdrFormat(1 ME)

TCAM

DRAM

SRAMSRAM

exte

rnal in

terf

ace

s

swit

ch inte

rface

ingress side

Lookup(2 ME)

KeyExtract(1 ME)

RxEg(2 ME)

TxEg(2 ME)

QueueManager

(4 ME)

HdrFormat(1 ME)

DRAM

SRAM SRAM

egress side

FlowStats(2 ME)

SRAM

Filter/route and rate-control traffic Network Address Translation for outgoing flows Record traffic statistics for all outgoing flows

14

NPE Datapath (version 1)

Parse and Header Format include slice-specific code » parse extracts header fields to form lookup key» Hdr Format does any required post-lookup processing

Lookup uses opaque key for TCAM lookup Multiple static code options can be supported

» multiple slices per code option» each has own interfaces, filters, queues and private memory

SRAM

Lookup(1 ME)

Rx(2 ME)

Tx(2 ME)

QueueManager

(4 ME)

Parse(1 ME)

HdrFormat(1 ME)

TCAM

DRAM

SRAMSRAM

Substr.Decap(1 ME)

SRAM

15

NPE Datapath (Version 2)

Use both NPs, enabling 10 Gb/s throughput Integrated Decap,Parse,Lookup uses MEs more efficiently Multicast supported by substrate

SRAM

Tx(2 ME)

TCAM

Decap, Parse, Lookup, AddShim(8 MEs)

SRAM

from

sw

itch

Rx(2 ME)

SRAM

to s

wit

ch

SRAM

QueueManager(4 MEs)

Rx(2 ME)

Tx(2 ME)

Lookup& Copy(2 ME)

HdrFmt(4 MEs)

SPI Switch

SRAM

16

Project Plan Highlights Initial deployment

» first two (maybe three) nodes deployed by mid 2009» shooting for v2 of NPE software» PlanetLab control with local resource allocation

GENI-compatible control software» implement component manager interface» resource allocation using rspecs/tickets

Working with users» online and hands-on tutorials» collaborating with users on new code options

Completing deployment» final nodes deployed in late 2010» complete support for netFPGA

17

Looking AheadBad news

» slow market for ATCA means high cost, limited support» Intel dropped IXP and Radisys discontinuing IXP blades

Good news» ATCA market now projected to grow rapidly and become

more cost-competitive (10x growth over 3 years)» new NPs and NP blades

•Netronome 3200 – IXP successor with 40 microengines•Cavium Octeon, RMI XLR732 – MIPS-based, uses cache

» can also assemble systems from commodity components•10 GbE switches now at $400-500 per port•conventional rack-mount servers with 8-16 processor cores•NPs and FPGAs on lower cost PCI-express cards