Winter 2004EE384x Handout 11 EE384x: Packet Switch Architectures Handout 1: Logistics and Introduction Professor Balaji Prabhakar [email protected]

Winter 2004 EE384x Handout 1 1

EE384x: Packet Switch Architectures

Handout 1: Logistics and Introduction

Professor Balaji [email protected]

Professor Nick [email protected]


Outline

This two course sequence is about the theory and practice of designing packet switches and Internet routers.

1. Introduction: What is a packet switch? The evolution of Internet routers, their basic architectural components, and some example architectures.

2. Part I: Output Queued Switches (Emphasis on Deterministic Analysis) OQ as the simplest and ideal architecture.Output queueing and shared-memory switches. Packet arrival processes: ()-constrained arrivals, leaky buckets, Bernoulli arrivals, bursty arrivals, adversaries.Providing bandwidth and delay guarantees, scheduling, fairness, Fair-Queueing, Generalized Processor Sharing and Deficit Round Robin.Practical difficulties: When output queued switches are impractical.Memory bandwidth and capacity scaling. Some approaches: Emulating output queued switches. Parallel packet buffers as standalone shared memory, with design examples. Routers with a single stage of buffering and constraint sets, Parallel Shared Memory Routers, Distributed Shared Memory Routers, and Parallel Packet Switches. Output link scheduling in a Distributed Shared Memory router. Combined input and output queued (CIOQ) switches, stable marriage matchings.


Outline

3. Part II: Input Queued Switches (Emphasis on Probabilistic Analysis). What is an input-queued (IQ) switch?Definition of IQ switch with single FCFS queue. Switching fabrics, crossbars. Head of line blocking. The balls and bins model. Proof of Karol's 2-sqrt(2) (58%) result. Virtual output queues and crossbar schedulers. Bipartite Matchings: Maximum Sized Matchings, Maximum Weight Matchings, maximal matchings. Definitions of 100% throughput. When traffic is uniform: simple RR and random matchings. When traffic matrix is known: Birkhoff- von Neuman decomposition. When traffic is not known: heuristics. PIM, iSLIP, WFA.

4. Fundamentals (Review Sessions): Introduction to probability, Poisson process, Discrete and Continuous-time Markov chains. Basic queueing theory: M/M/1, M/G/1, Little’s result, PASTA.


Outline EE384y

1. Part II: Input Queued Switches (Continued).Intro to Lyapunov functions, proof that max weight matching gives 100% throughput.Some case studies: The Tiny Tera architecture. The Cisco GSR 12000.

2. Part III: Other Switch ArchitecturesBuffered crossbars. Scaling crossbars and parallelism. Multistage switches: Clos networks, 2-stage switches (random and deterministic).

3. Part IV: Other Switch FunctionsAddress Lookup: Exact matches, longest prefix matches, performance metrics, hardware and software solutions. Packet Classification: For firewalls, QoS, and policy-based routing; graphical description and examples of 2-D classification, examples of classifiers, theoretical and practical considerations.

4. Special topics.5. Project presentations.

5 EE384x Handout 1 Winter 2004

Some logisticsWeb page: http://www.stanford.edu/class/ee384x

Office hours: Balaji Prabhakar – TBA on web pageNick McKeown – Wednesday 4.00pm – 5.00pm

Course assistant:Theresa Wan – [email protected]

Gates 351; Tel: (650) 725 9077

TAs:Rui Zhang-Shen – [email protected] Keshavarzian- [email protected] hours: See class web page.

Grades:You need to sign up with “eeclass” on the EE384x web page.


More LogisticsPrerequisite

EE284/CS244A and familiarity with probability.

Useful Stats 116 (or EE178, EE278) and CS161

Papers URLs to all the papers are on the eeclass web page.

Grading (40%) 5 Problem sets (10%) Several surprise quizzes (20%) In-class midterm exam (Tuesday February 17) (30%) Final exam (Wednesday March 17)

SITN Students Same schedule as in-class students Fax your assignment to us at: (702) 977 0556.

All deadlines are hard!


An IntroductionThe class starts here!

Background What is a router? Why do we need faster routers? Why are they hard to build?

Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.


What is Routing?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5F

R3E

R3D

Next HopDestination

D


What is Routing?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5F

R3E

R3D

Next HopDestination

D

16 3241

Data

Options (if any)

Destination Address

Source Address

Header ChecksumProtocolTTL

Fragment OffsetFlagsFragment ID

Total Packet LengthT.ServiceHLenVer

20

byte

s


What is Routing?

A

B

C

R1

R2

R3

R4 D

E

FR5


Points of Presence (POPs)

A

B

C

POP1

POP3POP2

POP4 D

E

F

POP5

POP6 POP7POP8


Where High Performance Routers are Used

R10 R11

R4

R13

R9

R5

R2R1 R6

R3 R7

R12

R16R15

R14

R8

(2.5 Gb/s)

(2.5 Gb/s)(2.5 Gb/s)

(2.5 Gb/s)


What a Router Looks LikeCisco GSR 12416 Juniper M160

6ft

19”

2ft

Capacity: 160Gb/sPower: 4.2kW

3ft

2.5ft

19”

Capacity: 80Gb/sPower: 2.6kW


Basic Architectural Components

of an IP Router

Control Plane

Datapathper-packet processing

SwitchingForwarding

Table

Routing Table

Routing Protocols


Per-packet processing in an IP Router

1. Accept packet arriving on an incoming link.

2. Lookup packet destination address in the forwarding table, to identify outgoing port(s).

3. Manipulate packet header: e.g., decrement TTL, update header checksum.

4. Send packet to the outgoing port(s).5. Buffer packet in the queue (在 output

queue).6. Transmit packet onto outgoing link.


Generic Router Architecture

LookupIP Address

UpdateHeader

Header ProcessingData Hdr Data Hdr

~1M prefixesOff-chip DRAM

AddressTable

AddressTable

IP Address Next Hop

QueuePacket

BufferMemoryBuffer

Memory~1M packetsOff-chip DRAM



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

Data Hdr

Data Hdr

Data Hdr

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr


Why do we Need Faster Routers?

1. To prevent routers becoming the bottleneck in the Internet.

2. To increase POP capacity, and to reduce cost, size and power.


0,1

1

10

100

1000

10000

1985 1990 1995 2000

Spec

95In

t CPU

resu

ltsWhy we Need Faster Routers

1: To prevent routers from being the bottleneck

0,1

1

10

100

1000

10000

1985 1990 1995 2000

Fib

er

Ca

pa

cit

y (

Gb

it/s

)

TDM DWDM

Packet processing Power Link Speed

2x / 18 months 2x / 7 months

Source: SPEC95Int & David Miller, Stanford.


POP with smaller routers

Why we Need Faster Routers 2: To reduce cost, power & complexity of

POPsPOP with large routers

Ports: Price >$100k, Power > 400W. It is common for 50-60% of ports to be for interconnection.


Why are Fast Routers Difficult to Make?

1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with Moore’s Law.



Speed of Commercial DRAM


0.001

0.01

0.1

1

10

100

1000

1980 1983 1986 1989 1992 1995 1998 2001

Acc

ess

Tim

e (n

s)

Moore’s Law2x / 18 months

1.1x / 18 months




2. Moore’s Law is too slow: Routers need to improve faster than Moore’s Law.


Router Performance Exceeds Moore’s Law

Growth in capacity of commercial routers: Capacity 1992 ~ 2Gb/s Capacity 1995 ~ 10Gb/s Capacity 1998 ~ 40Gb/s Capacity 2001 ~ 160Gb/s Capacity 2003 ~ 640Gb/s

Average growth rate: 2x / 18 months.


Outline




RouteTableCPU Buffer

Memory

LineInterface

MAC

LineInterface

MAC

LineInterface

MAC

Typically <0.5Gb/s aggregate capacity

First Generation RoutersShared Backplane

Line Interface

CPU

Memory


Second Generation RoutersRouteTableCPU

LineCard

BufferMemory

LineCard

MAC

BufferMemory

LineCard

MAC

BufferMemory

FwdingCache

FwdingCache

FwdingCache

MAC

BufferMemory

Typically <5Gb/s aggregate capacity


Third Generation Routers

LineCard

MAC

LocalBuffer

Memory

CPUCard

LineCard

MAC

LocalBuffer

Memory

Switched Backplane

Line Interface

CPUMem

ory FwdingTable

RoutingTable

FwdingTable

Typically <50Gb/s aggregate capacity


Fourth Generation Routers/Switches

Optics inside a router for the first time

Switch Core Linecards

Optical links

100sof metres

0.3 - 10Tb/s routers in development


Outline





LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable


IP Address Lookup

Why it’s thought to be hard:1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 150,000 entries today,

and growing. 3. The lookup must be fast: about 30ns for a 10Gb/s

line. 批注：那么假设一个 Packet的平均大小为 300bit


IP Lookups find Longest Prefixes

128.9.16.0/21128.9.172.0/21

128.9.176.0/24

0 232-1

128.9.0.0/16142.12.0.0/1965.0.0.0/8

128.9.16.14

Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.


IP Address Lookup



line.


Address Tables are Large

Source: http://www.cidr-report.org/


IP Address Lookup



line.


Lookups Must be Fast

12540Gb/s2003

31.2510Gb/s2001

7.812.5Gb/s1999

1.94622Mb/s1997

40B packets (Mpkt/s)

LineYear


Outline





LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory


Fast Packet Buffers

Example: 40Gb/s packet bufferSize = RTT*BW = 10Gb; 40 byte packets批注： SRAM速度快，但容量小； DRAM则相反

Write Rate, R

1 packetevery 8 ns

Read Rate, R

1 packetevery 8 ns

BufferManager

BufferMemory

Use SRAM?+ fast enough random access time, but

- too low density to store 10Gb of data.

Use SRAM?+ fast enough random access time, but

- too low density to store 10Gb of data.

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).


Outline





LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

N times line rate

N times line rate



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

Data Hdr

Data Hdr

Data Hdr

Scheduler

Winter 2004 EE384x Handout 1 440% 20% 40% 60% 80% 100%

Load

Del

ayA Router with Input Queues

The best that any queueing system can

achieve.

Winter 2004 EE384x Handout 1 450% 20% 40% 60% 80% 100%Load

Del

ayA Router with Input Queues

Head of Line Blocking


achieve.

2 2 58%


Head of Line Blocking


Virtual Output Queues批注： VOQ消除了 IQ 的 HOL问题

Winter 2004 EE384x Handout 1 480% 20% 40% 60% 80% 100%

Load

Del

ayA Router with Virtual Output

Queues


achieve.


Maximum Weight Matching

A1(n)

N N

LNN(n)

A1N(n)

A11(n)

L11(n)

1 1

AN(n)

ANN(n)

AN1(n)

D1(n)

DN(n)

L11(n)

LN1(n)

“ Request” Graph Bipartite Match

S*(n)

MaximumWeight Match

*

( )( ) argmax( ( ) ( ))T

S nS n L n S n


Outline of Proof

( 1) ( 1 .

( )

) ( ) (

[ ( )

) ( )

]

( )

We can show that:

Hence, if is large enough, there is an expected

single-step downward drif t in occupancy.

and 100

T TE L n L n L n L n |

L n

E L n

L n c L n

%throughput is achieved.


There are now many ways to achieve 100% throughput…

52

Scheduling algorithms to achieve 100% throughput

1. Basic switch model.2. When traffi c is uniform (Many algorithms…)3. When traffi c is non-unif orm, but traffi c matrix is known.

• Technique: Birkhoff -von Neumann decomposition.

4. When matrix is not known.• Technique: Lyapunov f unction.

5. When algorithm is pipelined, or inf ormation is incomplete.• Technique: Lyapunov f unction.

6. When algorithm does not complete.• Technique: Randomized algorithm.

7. When there is speedup.• Technique: Fluid model.

8. When there is no algorithm.• Technique: 2-stage load-balancing switch.• Technique: Parallel Packet Switch.


The Evolution of Switching

Theory:

Practice:

InputQueueing

(IQ)

InputQueueing

(IQ)

InputQueueing

(IQ)

InputQueueing

(IQ)

58% [Karol, 1987]

IQ + VOQ,Maximum weight matching

IQ + VOQ,Maximum weight matching

IQ + VOQ,Sub-maximal size matching

e.g. PIM, iSLIP.

IQ + VOQ,Sub-maximal size matching

e.g. PIM, iSLIP.

100% [M et al., 1995]

Different weight functions,incomplete information, pipelining.

Different weight functions,incomplete information, pipelining.

Randomized algorithmsRandomized algorithms

100% [Tassiulas, 1998]

100% [Various]

Various heuristics, distributed algorithms,

and amounts of speedup

Various heuristics, distributed algorithms,

and amounts of speedup

IQ + VOQ,Maximal size matching,

Speedup of two.

IQ + VOQ,Maximal size matching,

Speedup of two.

100% [Dai & Prabhakar, 2000]


Current Internet Router TechnologySummary

There are three potential bottlenecks: Address lookup, Packet buffering, and Switching. （问题：这是什么意思？与前 2 者有何不同）

Techniques exist today for: 10+Tb/s Internet routers, with 40Gb/s linecards.

Documents

Winter 2004EE384x Handout 11 EE384x: Packet Switch Architectures Handout 1: Logistics and Introduction Professor Balaji Prabhakar [email protected]