Router Buffer Sizing and Reliability Challenges in Multicast

Aditya Akella

02/28

TCP Performance

• Can TCP saturate a link?• Congestion control

– Increase utilization until… link becomes congested– React by decreasing window by 50%– Window is proportional to rate * RTT

• Doesn’t this mean that the network oscillates between 50 and 100% utilization?– Average utilization = 75%??– No…this is *not* right!

Single TCP FlowRouter without buffers

Summary Unbuffered Link

t

W Minimum window for full utilization

• The router can’t fully utilize the link– If the window is too small, link is not full– If the link is full, next window increase causes drop– With no buffer it still achieves 75% utilization

TCP Performance

• In the real world, router queues play important role– Window is proportional to rate * RTT

• But, RTT changes as well the window

– Window to fill links = propagation RTT * bottleneck bandwidth

• If window is larger, packets sit in queue on bottleneck link

TCP Performance• If we have a large router queue can get 100%

utilization– But, router queues can cause large delays

• How big does the queue need to be?– Windows vary from W W/2

• Must make sure that link is always full• W/2 > RTT * BW• W = RTT * BW + Qsize• Therefore, Qsize > RTT * BW

– Ensures 100% utilization– Delay?

• Varies between RTT and 2 * RTT

Single TCP FlowRouter with large enough buffers for full link utilization

Example

• 10Gb/s linecard– Requires 300Mbytes of buffering.– Read and write 40 byte packet every 32ns.

• Memory technologies– DRAM: require 4 devices, but too slow. – SRAM: require 80 devices, 1kW, $2000.

• Problem gets harder at 40Gb/s– Hence RLDRAM, FCRAM, etc.

• Rule-of-thumb makes sense for one flow– Typical backbone link has > 20,000 flows– Does the rule-of-thumb still hold?

If flows are synchronized

• Aggregate window has same dynamics• Therefore buffer occupancy has same dynamics• Rule-of-thumb still holds.

2maxW

t

max

2

W

maxW

maxW

If flows are not synchronized

ProbabilityDistribution

B

0

Buffer Size

W

Central Limit Theorem

• CLT tells us that the more variables (Congestion Windows of Flows) we have, the narrower the Gaussian (Fluctuation of sum of windows)

– Width of Gaussian decreases with

– Buffer size should also decreases with

n

CT

n

BB n

21

n

1

n

1

Loss Recovery in Multicast

• Sender-reliable– Wait for ACKs from all receivers. Re-send on

timeout or selective ACK– Per receiver state in sender not scalable– ACK implosion

• Receiver-reliable– Receiver NACKs (resend request) lost packet– Does not provide 100% reliability– NACK implosion

R1

Implosion

S

R3 R4

R2

21

R1

S

R3 R4

R2

Packet 1 is lost All 4 receivers request a resend

Resend request

Retransmission

• Re-transmitter– Options: sender, other receivers

• How to retransmit– Unicast, multicast, scoped multicast,

retransmission group, …

• Problem: Exposure

R1

Exposure

S

R3 R4

R2

21

R1

S

R3 R4

R2

Packet 1 does not reach R1;Receiver 1 requests a resend

Packet 1 resent to all 4 receivers

1

1

Resend request Resent packet

Ideal Recovery Model

S

R3 R4

R2

2

1

S

R3 R4

R2

Packet 1 reaches R1 but is lost before reaching other Receivers

Only one receiver sends NACK to the nearest S or R with packet

Resend request

1 1Resent packet

Repair sent only to those that need packet

R1 R1

Scalable Reliable Multicast

• Originally designed for wb (whiteboard)

• Receiver-reliable– NACK-based

• Every member may multicast NACK or retransmission

R1

SRM Request Suppression

S

R3

R2

21

R1

S

R3

R2

Packet 1 is lost; R1 requests resend to Source and Receivers

Packet 1 is resent; R2 and R3 no longer have to request a resend

1

X

XDelay varies by distance

X

Resend request Resent packet

Deterministic Suppression

d

d

d

d

3d

Time

data

nack repair

d

4d

d

2d

3d

= Sender

= Repairer

= Requestor

Delay = C1dS,R

SRM Star Topology

S

R2

21

R3

Packet 1 is lost; All Receivers request resends

Packet 1 is resent to all Receivers

X

R4

Delay is same length

S

R2

1

R3 R4

Resend request Resent packet

SRM: Stochastic Suppression

datad

d

d

d

Time

NACK

repair

2d

session msg

0

1

2

3

Delay = U[0,D2] dS,R

= Sender

= Repairer

= Requestor

SRM (Summary)

• NACK/Retransmission suppression– Delay before sending– Delay based on RTT estimation– Deterministic + Stochastic components

• Periodic session messages– Full reliability– Estimation of distance matrix among

members