Ken Wong Jon Turner and Prashanth Pappu Washington University [email protected] Distributed Queueing Gigabit Kits (June 2002)

Ken WongJon Turner and Prashanth Pappu

Washington [email protected]

Distributed QueueingGigabit Kits (June 2002)

2 -Ken Wong, June 2002

Distributed Queueing

Switch Fabric

TI TI TITI TI

I O I O I OI O I OI O

TI

ControlProcessor

Routing

Sched.

Routing

Sched.

Routing

Sched.

Routing

Sched.

Routing

Sched.

Routing

Sched.queueper output

periodic queuelength reports

Scheduler paces eachqueue according to

backlog share


What is Distributed Queueing? Performs like an output queueing system

» I.e., Maximize output link utilization»Without the need for a switch with speed-up of N

Goals of the DQ Algorithm»Avoid switch fabric congestion (and therefore cell

loss)»Avoid output underflow (maximize output link

utilization) Topics

»The STRESS experiment»Discrete-event simulation results»SPC-only prototype measurements»Current work


Stress Test

can vary number of inputs and outputs used, and length of

“phases”

K = 4 Sources R = 5 Phases

70 Mbps

70 Mbps

70 Mbps

70 Mbps

S x L = 2 x 70

= 140 Mbps

L = 70 Mbps

L = 70 Mbps


0

500

1000

1500

2000

2500

3000

3500

0

100

200

300

400

500

600

700

800

900

1000

1100

DQ Period

Bac

klo

g (K

B) Output 1

2

3

4

1=>31=>21=>1

Stress Test Simulation - Backlog


Basic Ideas

To Avoid Output Underflow» Input-side backlog B(i,j) indicates need for switch

bandwidth »Output-side backlog B(j) indicates less need for

switch bandwidth (back pressure) Apportion switch capacity based on relative

input-side backlogs to avoid switch congestion and output underflow


Basic DQ Algorithm Goal: Avoid switch congestion and output queue

underflow. Let hi(i,j) be input i’s share of input-side backlog to

output j.» Avoid switch congestion by sending from input i to output j at

rate LShi(i,j)» where L is external link rate and S is switch speedup

Let lo(i,j) be input i’s share of total backlog for output j.» Avoid underflow of queue at output j by sending from input i to

output j at rate Llo(i,j) » This works if L(lo(i,1)+···+lo(i,n)) LS for all i

Let wt(i,j) be the ratio of lo(i,j) to lo(i,1)+···+lo(i,n). Let rate(i,j)=LS min ( wt(i,j), hi(i,j) ). Note: Algorithm avoids congestion and output underflow

for large enough S.


0

500

1000

1500

2000

2500

3000

3500

0

100

200

300

400

500

600

700

800

900

1000

1100

DQ Period

Bac

klo

g (K

B) Output 1

2

3

4

1=>31=>21=>1

Stress Test Simulation - Backlog


0

20

40

60

80

100

120

1400

100

200

300

400

500

600

700

800

900

1000

1100

DQ Period

Min

Rat

e fr

om In 1

(Mb/s

)

to 12

34

5

External Link Rate: 70 Mb/s

Stress Test Simulation - Min Rates


0

20

40

60

80

100

120

1400

200

400

600

800

1000

1200

DQ Period

Act

ual

Rat

es

External link rate: 70 Mb/ s

to 1

2

3

4

5

Stress Test Simulation - Actual Rates


SPC Code

DQ cell (40 of 48 bytes for N=8 ports)»Port number»Output backlog B(i)»N input backlogs B(i,j)

SPC kernel at each port (Every D = 500 usec)»Multicast 1 DQ cell containing its backlogs to all ports

using vpi/vci 0/61»Read incoming DQ cells from all ports»Call dq_set_pace()

dq_set_pace»Compute rate(i,j) for all j»Set APIC pacing rates for each j


Stress Test Measurement Results


Improving Basic Algorithm Allocated rates will oscillate when backlogs are

near 0»Use artificial minimum backlog in hi(i,j) and artificial

minimum rate in lo(i,j) Does not always make full use of available

input bandwidth»Does not reallocate bandwidth that is lost when

queues are “output limited”»Extend algorithm to reallocate “excess” rate

Allocate rates in decreasing order of largest lo(i,j)/hi(i,j) I.e., rate(i,j) can “donate” excess rate to remaining rates


Current/Future Work “Design and Evaluation of a High-Performance

Dynamically Extensible Router,” DANCE Conference, May 2002.

FCFS property across input streams SPC

»Output-limited rate redistribution algorithm»Dynamic configuration of DQ parameters and

algorithm INFOCOM 2003 paper (Pappu, Turner, Wong)

»Handles unequal link speed case FPGA implementation Extension to fair queueing Extension to reserved bandwidth flows


Summary Fluid model simulator (C++) Discrete-event simulator (C++)

»Study alternative DQ algorithms»Understand operating characteristics in controlled

environment»Pre-testing of SPC (integer) algorithm

SPC-only prototype»Basically works»Great monitoring tools (Java GUI, DQ-cell capture,

Other utilities)»Great traffic generator (AAL5Generator)

FPX version coming Lots of fun


Backlog Shares Let hi(i,j) be input i’s share of input-side

backlog to output j:

Let lo(i,j) be input i’s share of total backlog for output j:

),(

),(),(

jB

jiBjihi

),()(

),(),(

jBjB

jiBjilo


Allocated Rates r(i,j)

where

)),(),,((min),( jiwtjihiLSjirate

),(

),(),(

ilo

jilojiwt


Artificial Backlogs & Rates Let hi(i,j) be input i’s share of input-side

backlog to output j:

Let lo(i,j) be input i’s share of total backlog for output j:

),(

),(),(

jBqN

jiBqjihi

N

jBjB

jiBjilo

/1

,),()(

),(),(


Rate Redistribution Algorithm Revised rate allocation at input i:

R = SLrepeat n times

Let j be unassigned queue with largest ratio lo(i,j)/hi(i,j)Let wt(i,j) = lo(i,j)/(sum of lo(i,q) for unassigned queues q)rate(i,j) = min{Rwt(i,j), SLhi(i,j)}R = R - rate(i,j)

Documents

Ken Wong Jon Turner and Prashanth Pappu Washington University [email protected] Distributed Queueing Gigabit Kits (June 2002)