CSE CSE 123123Computer NetworksComputer Networks

Fall Fall 20092009

Lecture Lecture 13: 13: IP Router IP Router DesignDesigngg

and and QoSQoS

Many portions courtesy Nick McKeown

Miscellaneous stuffMiscellaneous stuffMiscellaneous stuffMiscellaneous stuff Good news

I gave up on project #3 (too sick to get it out the door) Bad news

I’m out on Wednesday I m out on Wednesday Good news

Geoff Voelker has agreed to fill in (network security)B d Bad news You still have to go to his lecture (there will be at least one

final question based on it) Good news

I’ll post a sample final by Wednesday

Overview for todayOverview for todayOverview for todayOverview for today How routers are built How routers are built

Router basics Interconnection architecture

» Input Queuing» Output Queuing» Virtual output Queuing» Virtual output Queuing» Scheduling

Future bottlenecks Quality of service basics

What’s in a router?What’s in a router?What s in a router?What s in a router? Physical components Physical components

One or more input interfaces that receive packets One or more output interfaces that transmit p

packets A chassis (box + power) to hold it all

Functions Forward packets

D k t ( ti it Q S) Drop packets (congestion, security, QoS) Delay packets (QoS) Transform packets? (Encapsulation Tunneling) Transform packets? (Encapsulation, Tunneling)

What a router does:What a router does:the normal casethe normal casethe normal casethe normal case

1. Receive incoming packet from link input interfaceg p p2. Lookup packet destination in forwarding table

(destination, output port(s))3 V lid t h k d t ttl d t h k3. Validate checksum, decrement ttl, update checksum 4. Buffer packet in input queue5. Send packet to output interface (interfaces?)5. Send packet to output interface (interfaces?)6. Buffer packet in output queue7. Send packet to output interface link

What a router looks like?What a router looks like?What a router looks like?What a router looks like?

Cisco 2500 Linksys DEFSR81

1.75”

19” 8”

Capacity: <10Mbps Capacity: <10MbpsCapacity: <10Mbps Capacity: <10Mbps

What a router looks like (2) What a router looks like (2) What a router looks like (2) What a router looks like (2) Cisco GSR 12416 Juniper M160

1919””1919””

Capacity: Capacity: 160Gb/s160Gb/sP 4 2kWP 4 2kW

Capacity: Capacity: 80Gb/s80Gb/sP 2 6kWP 2 6kW

6ft6ftPower: 4.2kWPower: 4.2kW

3ft3ft

Power: 2.6kWPower: 2.6kW

Fall, 2005 CSE 123A -- Lecture 14 – IP Router Design 7

2ft2ft 2.5ft2.5ft

Alcatel 7670 RSP Juniper TX8/T640

TX8

Avici TSRAvici TSR

Capacity: Capacity: nTbnTb/s/sp yp yPower: 10s of kW (~100’s of homes)Power: 10s of kW (~100’s of homes)

HighHigh--performance routersperformance routersHighHigh--performance routersperformance routers Geared to core and distribution service needs

Requirements: high speed & high density Why do we care?

Moore’s Law slower than link speed growth (and BW demand)

» OC48c (2.5Gbps), common, 128ns/packet( p ), , p» OC192c (10Gbps), in backbone deployment, 33ns/packet» OC768c (40Gbps), emerging, 8ns/packet

Need high density/low power for POP deployments Need high density/low power for POP deployments» Points-of-Presence (POP) – places where a network

service provider provides dense connectivity» $20-100k & 2-400W per port 50% ports frequently for» $20 100k & 2 400W per port, 50% ports frequently for

internal connectivity (why?)

Functional architecture Functional architecture Functional architecture Functional architecture

Reservation/Reservation/Admission

Control

Routing ProtocolsFirewall Control Plane

• ComplexP t l ti

Routing Table

ClassificationRules

• Per-control action• May be slow

ForwardingT bl D t lTable Data plane

• Simple• Per-packet

PacketClassification Switching Output

Scheduling • Must be fast

Interconnect architectureInterconnect architectureInterconnect architectureInterconnect architecture Input & output p p

connected via switch fabricKi d f i h f b i

Input Output Kinds of switch fabric

Bus Crossbar

p pSwitch

Crossbar Shared Memory

How to deal with transient contention? Input queuing

O t t i Output queuing Combination

First Generation RoutersFirst Generation RoutersFirst Generation RoutersFirst Generation Routers

CPU RouteTable Buffer

Memory•Single CPU andshared memory;

Shared Bus(s)

•All classification by main CPU

Line Card Line Card Line Card

MAC MAC MAC

Second Generation RoutersSecond Generation RoutersSecond Generation RoutersSecond Generation Routers

RouteCPU RouteTable

Di t DMAShared Bus(s)

Direct DMA oncache hit

Line CardBuffers

Line CardBuffers

Line CardBuffers

Cache ofrecent routes

MAC

ForwardingCache

MAC

ForwardingCache

MAC

ForwardingCache

Third Generation RoutersThird Generation RoutersThird Generation RoutersThird Generation RoutersSwitch Fabric Sh d i t tSwitch Fabric •Shared interconnect

(frequently crossbar)•Centralized scheduler•Full forwarding table

i li din line card•Fixed cells

Line CardBuffers

CPU CardCPU

Line CardBuffers

MAC

ForwardingTable

RoutingTable

MAC

ForwardingTable

Output queuingOutput queuingOutput queuingOutput queuing Output interfaces Output interfaces

buffer packets Input OutputSwitch

Pro Simple algorithmsp g Single congestion point

Con N inputs may send to

the same outputR i d f N Requires speedup of N

Input queuingInput queuingInput queuingInput queuing Input interfaces buffer Input interfaces buffer

packets Pro

Input OutputSwitch

Single congestion point Simple to design algorithms

Con Must implement flow control Low utilization due to Head-

of-Line (HoL) Blocking » Utilization limited to 2» Utilization limited to 2-

2^.5=58%

HeadHead--ofof--Line BlockingLine BlockingHeadHead--ofof--Line BlockingLine Blocking

Virtual Output QueuesVirtual Output QueuesVirtual Output QueuesVirtual Output Queues

Fall, 2005 CSE 123A -- Lecture 14 – IP Router Design

IQ + Virtual Output QueuingIQ + Virtual Output QueuingIQ + Virtual Output QueuingIQ + Virtual Output Queuing Input interfaces buffer p

packets in per-output virtual queuesP

Input OutputSwitch

Pro Solves blocking problem

Con Con More resources per port Complex arbiter at switchp Still limited by input/output

contention (scheduler)RR: 1/e = 63% RR: 1/e = 63%

Switch schedulingSwitch schedulingSwitch schedulingSwitch scheduling Problem

Match inputs and outputs Resolve contentions, no packet drops Maximize throughput Do it in constant time…

If traffic is uniformly distributed its easy If traffic is uniformly distributed its easy Lots of algorithms (approximate matching)

Recent result (Dai et al, 2000)Recent result (Dai et al, 2000) Maximal size matching + speedup of two

guarantees 100% utilization for most traffic assumptionsassumptions

Modern highModern high--performance routerperformance routerModern highModern high--performance routerperformance router

IQ + VoQ + OQ IQ VoQ OQ Speedup of 2 Central scheduler

Input OutputSwitch

Fixed-sized internal cells Pro

Can achieve utilization of 1 Can scale to > Tb/s

Con Multiple congestion points Complexity

Next bottlenecksNext bottlenecksNext bottlenecksNext bottlenecks Buffering at high speed Buffering at high speed

SRAM density too low for BW*D of 40Gbps link DRAM too slow

SRAM t h f DRAM SRAM memory management as cache for DRAM

Scheduler overheadHard to do central scheduler much over 1Tbps Hard to do central scheduler much over 1Tbps

Multi-stage load-balanced switches High density (100’s-1000’s of line cards) High density (100 s-1000 s of line cards)

Physical distance to support density; electrical links degrade Optical links; optical cross connect (MEMs,tunable lasers)

Time to market, Power/Heat

New topic: Quality of serviceNew topic: Quality of serviceNew topic: Quality of serviceNew topic: Quality of service So far, we’ve assumed all traffic is equal -- So far, we ve assumed all traffic is equal

best effort delivery Not always best model. Why?y y

Application demands» I want low-delay low-loss for phone service

Market differentiation» I want to sell better service for more money

Bandwidth management Bandwidth management» Don’t let BitTorrent eat up all UCSD bandwidth

Original MotivationOriginal Motivation::multimedia applicationsmultimedia applicationsmultimedia applicationsmultimedia applications

Basic idea Basic idea Sample signal, packetize, transmit Repeat in reverse at receiver p

Network Requirements (@ given load) Delay Jitter (variation in delay) Packet loss Exact parameters a function of interactivity

demands, buffer capacity, retransmission time and loss toleranceloss tolerance

However… as a rule they want more

Different kinds of demandsDifferent kinds of demandsDifferent kinds of demands…Different kinds of demands…

ElasticElastic

Util

ity

tility

Hard real-timeBandwidth

ty

Delay-adaptive

Ut

Bandwidth

Util

it Bandwidth

Bandwidth

What to do?What to do?What to do?What to do? Want to treat some traffic differently/better Want to treat some traffic differently/better

than others Issues

How to identify the more important traffic? How much better do we want to treat it? How do we actually treat it better?

Packet classificationPacket classificationPacket classificationPacket classification Router classifies based on packet header Router classifies based on packet header

Aggregates» From particular network (IP src address)» For particular protocol (e.g., port 80 traffic)

Individual network flows» 5-tuple (src dst src port dst port protocol)» 5-tuple (src, dst, src port, dst port, protocol)

Special header field that indicates traffic “type”

Kinds of service Kinds of service classesclassesKinds of service Kinds of service classesclasses Best-effort

Vanilla IP Differentiated services

Bronze, Silver, Gold, etc… (effectively priorities, up to some amount of bandwidth per time)

E g best service up to 10Mbps then best effort E.g., best service up to 10Mbps, then best effort Predicted service (soft real-time)

Network guarantees good performance on averageg g p g Application promises only send as fast as negotiated

Guaranteed service (hard real-time) Network guarantees good performance always Application promises only send as fast as negotiated

How to specify service?How to specify service?How to specify service?How to specify service? Kind of service (service class) Specify “flowspec” for data flow limits

Tspec: describes the flow’s traffic characteristics» Average bandwidth + burstiness (contract with ISP)

Rspec: describes the service requested from the network (e g delay target)network (e.g., delay target)

Interface can be interactive (ask network) orvia business interface (ask salesman)via business interface (ask salesman) Can say no If yes, then use scheduling mechanisms in routers

(not FIFO anymore) to deliver

New elements in the routerNew elements in the routerNew elements in the routerNew elements in the router

an

e

Routing Routing

MessagesQoS Control

messagesSignaling

Admission Control?

Co

ntr

ol P

lP

lan

e

Data InData Out

Dat

a P

SchedulerClassifierDest Lookup

Forwarding Table Per Flow QoS Table

Data OutSchedulerClassifierDest Lookup

Token bucket traffic modelToken bucket traffic modelToken bucket traffic modelToken bucket traffic model Two parametersp

Token bucket size: b (maximum number of tokens) Token bucket rate: r (rate at which new tokens are

provided)provided) Need k tokens to send a packet of k bytes Applications Applications

Shaping (delay packets until k tokens appear)» Long term rate is limited to r, short term bursts to b» Over some interval T, traffic is limited to b+r*T

Policing (drop packets if insufficient tokens) Buffer management (mark packets and xmit if > k) Buffer management (mark packets and xmit if > k)

» Drop marked (i.e. out-of-spec) packets during congestion

Example: Traffic ShapingExample: Traffic ShapingExample: Traffic ShapingExample: Traffic Shaping

Shape packets according to user profile Shape packets according to user profile Output limited to average of r bps and bursts

of br bps

User Profile

Packet

Queue,Drop on overflow

b bits (token bucket)

Wait fortoken

Packetinput

Packetoutput

Example: Traffic PolicingExample: Traffic PolicingExample: Traffic PolicingExample: Traffic Policing

Drop packets that don’t meet user profile Drop packets that don t meet user profile Output limited to average of r bps and bursts

of b

r bps

b bitsUser Profile(token bucket)

Test iftoken

tokenPacketinput

Packetoutput

No token, drop

Example: Buffer ManagementExample: Buffer ManagementExample: Buffer ManagementExample: Buffer Management

Mark packets according to user profile Mark packets according to user profile During congestion, drop unmarked pkts first

r bps

No tokenb bits

User Profile(token bucket)

Test iftoken

Markpacket

tokenPacketinput

Packetoutput

SchedulingSchedulingSchedulingScheduling Give each “flow” (or traffic class) own queue

(at least theoretically) Weighted fair queuing

Proportional share scheduling Schedule round-robins among queues in proportion

to some weight parameterto some weight parameter Interesting result: Parekh & Gallager

If input traffic is shared to rate r burst b and If input traffic is shared to rate r, burst b and All routers schedule WFQ with weight = fraction of

link capacity that is r Guaranteed bound on delay of b/r! Wow… why don’t we do this?

How to extend across a How to extend across a network?network?network?network?

Integrated services Integrated services Motivated by need for end-to-end guarantees On-line negotiation of per-flow requirementsg p q End-to-end per-router negotiation of resources Complex

Differentiated services Motivated by economics (multi-tier pricing)

N fl t t No per-flow state Not end-to-end and not guaranteed services Simple Simple

Integrated Services Integrated Services ExampleExampleExampleExample

Example: guarantee 1MBps and < 100 ms delay to a flow

SenderReceiver

p g p y

Integrated Services Integrated Services ExampleExampleExampleExample Allocate resources - perform per-flow admission control

Yes you may

SenderReceiver

May I?

Yes, you may.

Integrated Services Integrated Services ExampleExampleExampleExample Install per-flow state

SenderReceiver

Integrated Services Integrated Services ExampleExample Install per flow state

ExampleExample

SenderReceiver

Integrated Services Integrated Services Example: Data PathExample: Data PathExample: Data PathExample: Data Path Per-flow classification

SenderReceiver

Integrated Services Integrated Services Example: Data PathExample: Data PathExample: Data PathExample: Data Path Per-flow buffer management

SenderReceiver

g

Integrated Services Integrated Services ExampleExampleExampleExample• Per-flow scheduling

SenderReceiver

DiffServ architectureDiffServ architectureDiffServ architectureDiffServ architecture• Edge router

Shape & police traffic• Shape & police traffic• Mark “class” of traffic in DS header field (e.g., gold service)

• Core router• Schedule aggregates according to marks in header• Schedule aggregates according to marks in header• Drop lower-class traffic first during congestion

• Network Engineer• Makes it all work outMakes it all work out

A

B

Domain

B

Edge router Core router

Aside: DS Aside: DS header fieldheader fieldAside: DS Aside: DS header fieldheader field0 5 7

DS 6 bits stolen from ToS field(other two for ECN)

length

0 15 16 31

ver HL TOSM D

20

identificationMF offset

TTL protocol header checksum

DF

RES

bytesTTL protocol eade c ec su

source address

destination address

Quick discussionQuick discussionQuick discussionQuick discussion These days most routers support most of These days most routers support most of

these capabilities WFQ scheduling, priority queues, Diffserve

marking, RSVP signaling, etc Integrated services largely unsuccessful

How to make economics work inter-ISP? Popular applications fairly elastic (Skype) and

average bandwidth pretty goodaverage bandwidth pretty good Complex

Diffserve somewhat successful within ISPs Diffserve somewhat successful within ISPs

In conclusionIn conclusionIn conclusion…In conclusion… Thanks, you’ve been a great class Thanks, you ve been a great class

I’ll see you at the final I ll see you at the final