Embed Size (px)
Citation preview
FAST TCP in Linux
Cheng Jin David Wei
http://netlab.caltech.edu/FAST/WANinLab/nsfvisit
netlab.caltech.edu
Outline
Overview of FAST TCP. Implementation Details. SC2002 Experiment Results. FAST Evaluation and WAN-in-Lab.
netlab.caltech.edu
FAST vs. Linux TCP
Distance = 10,037 km; Delay = 180 ms; MTU = 1500 B; Duration: 3600 sLinux TCP Experiments: Jan 28-29, 2003
1333173.182Linux TCPtxqlen=100
3909319.352Linux TCPtxqlen=10000
78 1851.86 1Linux TCPtxqlen=100
753
387
111
Transfer(GB)
1,79718.032FAST
19.11.2002
9259.281FAST
19.11.2002
2662.671Linux TCPtxqlen=10000
Throughput(Mbps)
BmpsPeta
Flows
netlab.caltech.edu
Aggregate Throughput
Linux TCP Linux TCP FAST
Average utilization
19%
27%
92%FAST Standard MTU Utilization averaged over 1hr
txq=100 txq=10000
95%
16%
48%
Linux TCP Linux TCP FAST
2G
1G
netlab.caltech.edu
Summary of Changes
RTT estimation: fine-grain timer. Fast convergence to equilibrium. Delay monitoring in equilibrium. Pacing: reducing burstiness.
netlab.caltech.edu
FAST TCP Flow Chart
Slow Start
Fast Convergence
Equilibrium
Loss Recovery
NormalRecovery
Time-out
netlab.caltech.edu
RTT Estimation
Measure queueing delay. Kernel timestamp with s resolution. Use SACK to increase the number of RTT
samples during recovery. Exponential averaging of RTT samples to
increase robustness.
netlab.caltech.edu
Fast Convergence
Rapidly increase or decrease cwnd toward equilibrium.
Monitor the per-ack queueing delay to avoid overshoot.
netlab.caltech.edu
Equilibrium
Vegas-like cwnd adjustment in large time-scale -- per RTT.
Small step-size to maintain stability in equilibrium.
Per-ack delay monitoring to enable timely detection of changes in equilibrium.
netlab.caltech.edu
Pacing
What do we pace? Increment to cwnd.
Time-Driven vs. event-driven. Trade-off between complexity and
performance. Timer resolution is important.
netlab.caltech.edu
Time-Based Pacing
cwnd increments are scheduled at fixed intervals.
data ack data
netlab.caltech.edu
Event-Based Pacing
Detect sufficiently large gap between consecutive bursts and delay cwnd increment until the end of each such burst.
SCinet Caltech-SLAC experiments
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
Experiment
Sunnyvale Baltimore
Chicago
Geneva
3000km 1000km
70
00
km
C. Jin, D. Wei, S. LowFAST Team and Partners
Internet: distributed feedbacksystem Rf (s)
Rb’(s)
x
p
TCP AQM
Theory
FAST TCP Standard MTU Peak window = 14,255 pkts Throughput averaged over > 1hr 925 Mbps single flow/GE card
9.28 petabit-meter/sec 1.89 times LSR
8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR
21TB in 6 hours with 10 flows
Implementation Sender-side modification Delay based
Highlights
1 2
1
2
7
9
10G
enev
a-Sunnyv
ale
Baltim
ore-
Sunn
yval
eFA
ST
I2 L
SR
#flows
netlab.caltech.edu
Network
(Sylvain Ravot, caltech/CERN)
netlab.caltech.edu
FAST BMPS
Internet2Land Speed
Record
FAST
1 2
1
2
7
9
10
Geneva-S
unnyvale
Baltim
ore-S
unnyvale
#flows
FAST Standard MTU Throughput averaged over > 1hr
netlab.caltech.edu
Aggregate Throughput
1 flow 2 flows 7 flows 9 flows 10 flows
Average utilization
95%
92%
90%
90%
88%FAST Standard MTU Utilization averaged over > 1hr
1hr 1hr 6hr 1.1hr 6hr
netlab.caltech.edu
Caltech-SLAC Entry
Rapid recoveryafter possiblehardware glitch
Power glitchReboot
100-200Mbps ACK traffic
SCinet Caltech-SLAC experiments
netlab.caltech.edu/FAST
SC2002 Baltimore, Nov 2002
PrototypeC. Jin, D. Wei
TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)
Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.
Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.
Navratil, J. Williams StarLight: T. deFanti, L. Winkler TeraGrid: L. Winkler
Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF
Acknowledgments
netlab.caltech.edu
Evaluating FAST
End-to-End monitoring doesn’t tell the whole story.
Existing network emulation (dummynet) is not always enough.
Better optimization if we can look inside and understand the real network.
netlab.caltech.edu
Dummynet and Real Testbed
netlab.caltech.edu
Dummynet Issues
Not running on a real-time OS -- imprecise timing.
Lack of priority scheduling of dummynet events.
Bandwidth fluctuates significantly with workload.
Much work needed to customize dummynet for protocol testing.
netlab.caltech.edu
10 GbE Experiment
Long-distance testing of Intel 10GbE cards.
Sylvain Ravot (Caltech) achieved 2.3 Gbps using single stream with jumbo frame and stock Linux TCP.
Tested HSTCP, Scalable TCP, FAST, and stock TCP under Linux.
1500B MTU: 1.3 Gbps SNV -> CHI; 9000B MTU: 2.3 Gbps SNV -> GVA
netlab.caltech.edu
TCP Loss Mystery
Frequent packet loss with 1500-byte MTU. None with larger MTUs.
Packet loss even when cwnd is capped at 300 - 500 packets.
Routers have large queue size of 4000 packets.
Packets captured at both sender and receiver using tcpdump.
netlab.caltech.edu
How Did the Loss Happen?
loss detected
netlab.caltech.edu
How Can WAN-in-Lab Help?
We will know exactly where packets are lost.
We will also know the sequence of events (packet arrivals) that lead to loss.
We can either fix the problem in the network if any, or improve the protocol.
netlab.caltech.edu
Conclusion
FAST improves the end-to-end performance of TCP.
Many issues are still to be understood and resolved.
WAN-in-Lab can help make FAST a better protocol.
