Testing TCP Westwood+ over Transatlantic Links at 10 Gigabit/Second rate

1

Testing TCP Westwood+ over Transatlantic Testing TCP Westwood+ over Transatlantic Links at 10 Gigabit/Second rateLinks at 10 Gigabit/Second rate

SaverioSaverio Mascolo Mascolo

Dipartimento di Elettrotecnica ed Dipartimento di Elettrotecnica ed ElettronicaElettronica

Politecnico di BariPolitecnico di Bari

Via Orabona 4, Bari, ItalyVia Orabona 4, Bari, Italy

PFLDNET 05, Feb. 3, 2005, Lyon

Giuseppe RacanelliGiuseppe Racanelli

Summer student Summer student at at

CERN IT DIVISIONCERN IT DIVISION

2 Saverio Mascolo – PFLDNET’05

MotivationMotivation

Recent introduction of Recent introduction of 10 Gigabit Routers and 10 Gigabit Routers and 10 Gigabit Ethernet cards makes of great 10 Gigabit Ethernet cards makes of great importanceimportance the issue of designing and testing the issue of designing and testing new protocols capable of efficient utilization of new protocols capable of efficient utilization of 10 gigabit Internet paths10 gigabit Internet paths


OutlineOutline

Brief summary of problems of TCP Brief summary of problems of TCP over Gigabit netsover Gigabit nets

Brief dBrief description of Westwood+ TCPescription of Westwood+ TCP Performance evaluation of Performance evaluation of

Westwood+Westwood+ over the DataTAG at over the DataTAG at CERN IT divisionCERN IT division


Standard TCP troughputStandard TCP troughput

The The long-term throughput long-term throughput T T of of standard TCPstandard TCP can be approximated can be approximated asas

wwhichhich sets a fundamental limitation sets a fundamental limitation for the TCP for the TCP

pRR

wT

2.1


In other terms…In other terms…

TTo fill a high speed path with o fill a high speed path with bandwidth B it is necessary to open bandwidth B it is necessary to open a congestion windowa congestion window

pRBw

2.1


Required packet lossRequired packet loss

which requires a packet loss which requires a packet loss probabilityprobability

i.e., i.e., to obtain full link utilization, a to obtain full link utilization, a lower and lower lower and lower pp is required with is required with increasing B.increasing B.

22.1

RBp


From S. Floyd draft on From S. Floyd draft on HS-TCPHS-TCP AA Standard TCP connection with 1500- Standard TCP connection with 1500-

byte packets and a 100 ms round-trip byte packets and a 100 ms round-trip time would require an average time would require an average congestion window of 83,333 segments congestion window of 83,333 segments to achieve a steady-state throughput of to achieve a steady-state throughput of 10 Gbps in the presence of a packet 10 Gbps in the presence of a packet drop rate of at most one loss event every drop rate of at most one loss event every 5,000,000 packets. The average packet 5,000,000 packets. The average packet drop rate of at most drop rate of at most 2*10^-102*10^-10, which is , which is needed for full link utilization in this needed for full link utilization in this scenario, corresponds to a bit error rate scenario, corresponds to a bit error rate of at mostof at most 2*10^-14 2*10^-14 , which is , which is unrealistic for current networksunrealistic for current networks..


Reasons to investigate Reasons to investigate Westwood+Westwood+

For these considerations, the main For these considerations, the main ideaidea of Westwood+, which consists of Westwood+, which consists of shrinking the control windows of shrinking the control windows after congestion by taking into after congestion by taking into account an estimate of the available account an estimate of the available bandwidth, is valuable of bandwidth, is valuable of investigation in the context of very investigation in the context of very high speed networks. high speed networks.


WESTWOODWESTWOOD++ TCP TCP

key idea of Westwood+: use the stream of ack packets to get an e2e estimate of the available bandwidth to be used for setting cwnd and ssthresh after congestion (whereas standard TCP implements a “blind” by half window decrease)


TCP WestwoodTCP Westwood++

Congestion Avoidance

Slow start

cwnd

time

Timeoutssthresh

BWE*RTTmin

Adaptive decrease cwnd=ssthr=BWE*RTTmin

Westwood Adaptive decrease vs (New) Reno blind by ½ window shrinking

E2E bandwidth estimationE2E bandwidth estimation

The rate of returning ACKS is exploited to The rate of returning ACKS is exploited to estimate the “best-effort” available bandwidthestimate the “best-effort” available bandwidth

ACKs

packets

Filter

RECEIVERSENDER

Bandwidthestimate

ACKs

packets

Network


Warning…Warning…

ACKs reach the TCP sender ACKs reach the TCP sender compressedcompressed

Bandwidth samplesBandwidth samples

contain high frequency components contain high frequency components that cannot be filtered out by a that cannot be filtered out by a discrete-time filter due to aliasingdiscrete-time filter due to aliasing

1

jj

jj tt

db

timealinterarriv 1 ACKtt jj


An anti-aliasing filter in An anti-aliasing filter in packet networkspacket networks

j

jj

db

RTT Lastj

RTTlast in the edacknowledg dataalld j

Antialiased samples


We are currently using the standard We are currently using the standard exponential filter exponential filter

kkk bbb )1(ˆˆ1

kRTTkd

kb


Summary on bandwidth Summary on bandwidth estimateestimate

Westwood TCP: one bandwidth Westwood TCP: one bandwidth sample computed for each ACK sample computed for each ACK (Mobicom 01)=>> Bandwdith (Mobicom 01)=>> Bandwdith overestiamte (when ACK overestiamte (when ACK compression)compression)

Westwood+ TCP: one bandwidth Westwood+ TCP: one bandwidth sample for each RTT (see sample for each RTT (see ACM CCR, ACM CCR, April 04April 04))

KnownKnown Advantages Advantages of of Westwood+ TCPWestwood+ TCP

higher throughput over wireless higher throughput over wireless linkslinks because losses due to because losses due to unreliable links do not provoke unreliable links do not provoke overshrinking of the congestion overshrinking of the congestion windowwindow

Improved fairness wrt to Reno Improved fairness wrt to Reno (Reno (Reno throughput is proportional to 1/RTT throughput is proportional to 1/RTT whereas Westwood throughput is whereas Westwood throughput is proportional to 1/sqrt(RTT) )proportional to 1/sqrt(RTT) )


Pseudo code of Pseudo code of Westwood+Westwood+

a)a) On ACK reception:On ACK reception:-cwnd is increased accordingly to the Reno -cwnd is increased accordingly to the Reno

algorithm;algorithm;-an estimate BWE of the available bandwdith -an estimate BWE of the available bandwdith

is is computed;computed; b)b) When 3 DUPACKs are received:When 3 DUPACKs are received:

ssthresh =max(2, (BWE* RTTmin) / seg_size); ssthresh =max(2, (BWE* RTTmin) / seg_size); cwnd = ssthresh; cwnd = ssthresh;

c)c) When coarse timeout expires:When coarse timeout expires:ssthresh = max(2,(BWE* RTTmin) / seg_size); ssthresh = max(2,(BWE* RTTmin) / seg_size); cwnd = 1;cwnd = 1;


Experimental testbedExperimental testbed


Single Stream TestsSingle Stream Tests• congestion window and slow start threshold of a single congestion window and slow start threshold of a single

TCP TCP NNewReno stream over a 10GbpsewReno stream over a 10Gbps.. At t=180s, due to a At t=180s, due to a lossloss,, cwndcwnd reduces from 2.5*10^8 bytes to 2.7*10^7 reduces from 2.5*10^8 bytes to 2.7*10^7 bytes and the TCP enters the congestion avoidance phase. bytes and the TCP enters the congestion avoidance phase.

0,00E+00

5,00E+07

1,00E+08

1,50E+08

2,00E+08

2,50E+08

3,00E+08

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

cwn

d/t

hre

sho

ld (

byt

e)

cwnd threshold


Instantaneous and mean Instantaneous and mean throughput of NewReno TCPthroughput of NewReno TCP

• it is around 1.8Gbps, which is less than one fifth of the it is around 1.8Gbps, which is less than one fifth of the channel capacity.channel capacity.

0,0E+00

1,0E+09

2,0E+09

3,0E+09

4,0E+09

5,0E+09

6,0E+09

7,0E+09

8,0E+09

9,0E+09

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

thro

ug

hp

ut

(bit

/s)

throughput average


cwndcwnd and and ssthreshssthresh dynamics dynamics obtained in the same scenario obtained in the same scenario

using Westwood+ TCPusing Westwood+ TCP cwndcwnd after congestion reduces from 2.5*10^8 bytes to after congestion reduces from 2.5*10^8 bytes to

2.3*10^8 bytes, which is remarkably larger than the 2.3*10^8 bytes, which is remarkably larger than the corresponding value obtained using New Reno.corresponding value obtained using New Reno.

0,0E+00

5,0E+07

1,0E+08

1,5E+08

2,0E+08

2,5E+08

3,0E+08

3,5E+08

4,0E+08

4,5E+08

5,0E+08

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

cwn

d/t

hre

sh (

byt

e)

cwnd threshold


Instantaneous and mean throughput of Westwood+ TCP

the achieved throughput is now around 7 Gbps

0,0E+00

1,0E+09

2,0E+09

3,0E+09

4,0E+09

5,0E+09

6,0E+09

7,0E+09

8,0E+09

9,0E+09

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

thro

ug

hp

ut

(bit

/s)

throughput average


Cwnd and ssthresh of Westwood+ Cwnd and ssthresh of Westwood+ TCPTCP

an UDP stream at 5Gbpsan UDP stream at 5Gbps is injected for few is injected for few secondsseconds; ; the slow start threshold is set to the slow start threshold is set to 3.5*10^7 bytes after congestion and, again, it 3.5*10^7 bytes after congestion and, again, it takes a long time for the TCP in congestion phase takes a long time for the TCP in congestion phase to grab all the bandwidth available after the UDP to grab all the bandwidth available after the UDP is turned off. is turned off.

0,0E+00

5,0E+07

1,0E+08

1,5E+08

2,0E+08

2,5E+08

3,0E+08

3,5E+08

4,0E+08

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

cwn

d/t

hre

sh (

byt

e)

cwnd threshold


Throughput of Westwood+Throughput of Westwood+ UDP active for a whileUDP active for a while: : around one tenth of the around one tenth of the

available bandwidth (i.e. 1.2 Gbps) is achieved.available bandwidth (i.e. 1.2 Gbps) is achieved.

0,0E+00

1,0E+09

2,0E+09

3,0E+09

4,0E+09

5,0E+09

6,0E+09

7,0E+09

8,0E+09

0 100 200 300 400 500 600 700 800 900 1000

time (sec)

thro

ug

hp

ut

(bit

/s)

throughput average


TCP Westwood+ with a TCP Westwood+ with a modified probing phase à modified probing phase à

la Scalable TCPla Scalable TCP

on ACK reception;on ACK reception;

If If ssthreshssthresh < = < = cwndcwnd < < windowwindow__thresholdthreshold

cwnd=cwnd+1/cwnd;cwnd=cwnd+1/cwnd;

If If cwndcwnd>> window window__thresholdthreshold

cwnd=cwnd+0.04cwnd=cwnd+0.04

By increasing cwnd of 0.04 on every ack reception, By increasing cwnd of 0.04 on every ack reception, cwndcwnd increases of one twenty-fifth per RTT, i.e., the growth is increases of one twenty-fifth per RTT, i.e., the growth is greater with larger windows. greater with larger windows.


cwnd and ssthresh of Westwood+ cwnd and ssthresh of Westwood+ TCP using the modified congestion TCP using the modified congestion

avoidanceavoidance UDP UDP active for a whileactive for a while

0,0E+00

5,0E+07

1,0E+08

1,5E+08

2,0E+08

2,5E+08

3,0E+08

3,5E+08

4,0E+08

0 200 400 600 800 1000

time (sec)

cwn

d/t

hre

sh (

byt

e)

cwnd threshold


Westwood+ TCP using a modified Westwood+ TCP using a modified probing phaseprobing phase

In this case, even though the setting of the threshold is In this case, even though the setting of the threshold is below the network capacity, the congestion window below the network capacity, the congestion window quickly increases and provides good results in terms of quickly increases and provides good results in terms of average throughput, which jumpsaverage throughput, which jumps to 6.2 Gbps to 6.2 Gbps

0,0E+00

1,0E+09

2,0E+09

3,0E+09

4,0E+09

5,0E+09

6,0E+09

7,0E+09

8,0E+09

0 200 400 600 800 1000

time (sec)

thro

ug

hp

ut

(bit

/s)

throughput average


Multiple Stream TestsMultiple Stream Tests TThe testbed he testbed is theis the 10Gbps 10Gbps connection going from Geneva to connection going from Geneva to

ChicagoChicago, where, where thethe link between the Cisco router 7606 at link between the Cisco router 7606 at Geneva and the Extreme router s01gva is aGeneva and the Extreme router s01gva is att 1 Gbps link. 1 Gbps link.

TTo investigate fairness in bandwidth utilization o investigate fairness in bandwidth utilization we consider 3we consider 3 flows sharflows sharinging the bottleneck. the bottleneck.


CwndCwnd of 3 NewReno flows of 3 NewReno flowsNew Reno flows exhibit the classic “sawtooth” oscillatoryNew Reno flows exhibit the classic “sawtooth” oscillatory

behaviour of the behaviour of the cwndcwnd due to the by half window reduction. due to the by half window reduction.

0,0E+00

2,0E+06

4,0E+06

6,0E+06

8,0E+06

1,0E+07

1,2E+07

1,4E+07

1,6E+07

1,8E+07

2,0E+07

0 200 400 600 800 1000

time (sec)

cwn

d (

byt

e)


CCwndwnd of of 3 3 Westwood+Westwood+ flows flows REMARK: REMARK: oscillation free behavior oscillation free behavior (the congestion window is (the congestion window is

kept around the same value of 5*10^06 byte during all the test)kept around the same value of 5*10^06 byte during all the test)

0,0E+00

5,0E+06

1,0E+07

1,5E+07

2,0E+07

2,5E+07

3,0E+07

0 200 400 600 800 1000

time (sec)

cw

nd

(b

yte

)


Throughput Throughput -- 3 3 NNew Reno flowsew Reno flows

the average per-connection throughput in the case the average per-connection throughput in the case of New Reno is 270 Mbpsof New Reno is 270 Mbps

0,0E+00

1,0E+08

2,0E+08

3,0E+08

4,0E+08

5,0E+08

6,0E+08

7,0E+08

8,0E+08

9,0E+08

0 200 400 600 800 1000

time (sec)

thro

ug

hp

ut

(bit

/s)


Throughput Throughput -- 3 Westwood+ 3 Westwood+ streamsstreams

the average per-connection throughput in the case of the average per-connection throughput in the case of Westwood+ is 320 Mbps.Westwood+ is 320 Mbps.

0,0E+00

1,0E+08

2,0E+08

3,0E+08

4,0E+08

5,0E+08

6,0E+08

7,0E+08

8,0E+08

9,0E+08

1,0E+09

0 200 400 600 800 1000

time (sec)

thro

ug

hp

ut

(bit

/s)


FairnessFairness

ToTo provide a mathematical evaluation of the provide a mathematical evaluation of the fairness, we plot the dynamics of the Jain fairness, we plot the dynamics of the Jain fairness index defined as below:fairness index defined as below:

where where bbii((tt) is the instantaneous throughput of ) is the instantaneous throughput of the the ith ith connection and M is the number of connection and M is the number of connections sharing the bottleneck. The Jain connections sharing the bottleneck. The Jain fairness index belongs to the interval [0,1] fairness index belongs to the interval [0,1] and increases with fairness up to the value and increases with fairness up to the value of one.of one.

M

i i

M

iFI

tbM

tbtJ

2

2

1

)(

)()(


Jain Fairness Index of Jain Fairness Index of 33 NewReno flowsNewReno flows


Jain Fairness Index Jain Fairness Index of of 3 3 Westwood+ flowsWestwood+ flows


ConclusionsConclusions

The setting of the cwnd and ssthresh a là The setting of the cwnd and ssthresh a là Westwood provides improvement in Westwood provides improvement in throughput and fairness wrt NewReno throughput and fairness wrt NewReno TCP also in the context of gigabit TCP also in the context of gigabit networks.networks.

We plan to make much more experimentsWe plan to make much more experiments We plan to blend the Westwood+ We plan to blend the Westwood+

features with more aggresive probing features with more aggresive probing phases such as the ones of Scalable TCP phases such as the ones of Scalable TCP or HS-TCPor HS-TCP


ACKNOWLEDGMENTSACKNOWLEDGMENTS

We thank Olivier Martin at the IT We thank Olivier Martin at the IT division of CERN and all the CS division of CERN and all the CS group, namely: Sylvain Ravot, Paolo group, namely: Sylvain Ravot, Paolo Moroni, Edoardo Martelli and Dan Moroni, Edoardo Martelli and Dan Nae (from Caltech) for their great Nae (from Caltech) for their great support and for allowing us to support and for allowing us to collect measurements reported in collect measurements reported in this paper.this paper.


Thanks for the attentionThanks for the attention and and Questions?Questions?

Documents

Testing TCP Westwood+ over Transatlantic Links at 10 Gigabit/Second rate