Layer 4 TCP Performancein carrier transport networksThroughput is not always equal to bandwidth
Roland Stooss
JDSU Deutschland GmbHSenior ConsultantData/IP Analysis SolutionsMühleweg 5, D-72800 Eningen u. A.Tel. +49 (0)7121/86 2138Email: [email protected]
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks2
Network
Datalink
Physical1
23
OSI Model: Responsibility and recommended tests
Customer’sIT departmentresponsibility
NetworkProvider’sresponsibility
Sharedresponsibility
Transport4
Application
Presentation
Session5
6
7HTTP, FTP, Email, etc.
TCP
IP
Ethernet
RFC2544and/or
Y.1564
RFC6349
Network
Datalink
Physical1
23
Transport4
Application
Presentation
Session5
6
7
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks3
Throughput is not always equal to available bandwid th
Layer 2/3 throughput (bandwidth) test
� Across test link� 100 % available
Layer 4 throughput test (upload)� Back-to-back� 100 % available
Layer 4 throughput test (upload)
� Across test link� 40 % throughput -> Why?
Layer 4/TCP …”really” outside of the responsibility of the carrier.
100%
100%
40% ?
40% ?
LAN
Typical customer complains• Bad application responds times• Total throughput is not like total
available bandwidth -> Why?
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks4
Upload
TCP throughput – the limiting factors … (1)
Window Size
PerformanceBandwidthPerformance
Retransmission Timer
Latency Error Rate
Sender can use full bandwidth and transmit at 100 % .� Thus the transmit buffer is quickly used up.Latency is too large.� Therefore the ACKs of the receiver are coming too la te.� Sender may already start sending unnecessary retran smissions.Cause� Window Size does not fit to bandwidth and latency.� Only one TCP session with high available bandwidth.
Window Size
Bandwidth Latency
… even if there is no JAM in the network
Max. Segment Size
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks5
Upload
Error rateLatencyBandwidth
Window Size
PerformancePerformance
Retransmission TimerPolicing/Shaping
MTU
TCP throughput – the limiting factors … (2)
Window Size
PerformancePerformance
Retransmission Timer Window SizeResponsibility with carrier Responsibility with customer
Error rateBandbreiteBandbreiteBandwidth LaufzeitLaufzeitLatency
Policing/Shaping
MTU
� MTU: Customer configured too small MTU or carrier has packet size limitations because of used encapsulation method (MPLS)
� Bandwidth: Customer may ordered too low BW or carrier doesn’t deliver BW� Latency: Customer didn’t fix RTT in SLA. Carrier selected long path
� Policing/Shaping: At the critical rate limitation interfaces (in customers or carriers network) Basic policing is extreme critical for TCP performance.
Shared responsibility:
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks6
Why does a TCP traffic get less throughput?
� Performance of TCP client and server and Round Trip Time– As higher the performance and as lower the RTT as faster the Slow Start
TCP Traffic rate
t
Available BandwidthRate limit by policing600
1000
� High performance TCP client and low RTT [1]:� Session could go above available BW. It always runs in extreme congestion. Final
averaged throughput is very low� High performance TCP client and large RTT [2]:
� Session can’t go above available BW, therefore no congestion and stable traffic rate.� Multiple TCP sessions with possible cumulated TCP traffic above BW may not reach
the BW, because of congestion effect.
[1]
[2]
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks7
Where might be the network problems for TCP?
LAN CE PE LANCE
� At carrier access with rate limit function– Most critical point, especially when Traffic Policy instead of Traffic Shaping is used
� At customers LAN to WAN CE interface− Total LAN rates are cumulated higher than WAN access interface (1 GigE)
Access
1GigE SDH/OTN �Fixed BW
Metro/Core
600Mb
� At carrier metro/core SDH/OTN– This is mainly an end-to-end fixed and 100% available bandwidth– No further congestion situations (loss, delay) -> fine for TCP traffic
PEIP/MPLSEth/VLAN �Dynamic BW
� At carrier metro/core IP/MPLS or Eth(VLAN)– This is packet based switching with dynamic and possible over booked BW– Congestion situations can occur (loss events and increasing delay)-> can be stress for TCP sessions. Probably less li ke with policing at the access.
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks8
Carrier Access – When is their rate limit?
� Multiple services with different CIRs has to be con sidered as point of rate limit like single service with rate limit.
� Recommended test application to detect TCP performa nce problems-> RFC6349 Frame Work for TCP Throughput testing
� Not really helpful – RFC2544 and/or Y.1564
Total BW 100%, but each service has its “CIR” rate limit
Single service rate limit
� The carrier access port is limiting the accepted traffic to the available transport rate.
� The carrier access port is limiting the accepted traffic per service to their available service rate (CIR)
� Total access bandwidth might be 100%.
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks9
Turn-up Related Problem RFC2544 Y.1564 RFC 6349
Single Service, Layer 2/3 SLA Issues (loss, jitter, etc.)
N/A
Multi-service, Layer 2/3 SLA Issues (service prioritization, loss, jitter, etc.)
N/A
Demonstrate the effect of End customer TCP Window size on throughput (CPE issue)
Inadequate device buffers to handle bursty applications
Policing effects to TCPperformance
Lack of TCP Testing is a Service Activation Gap
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks10
Application aware turn -up testing workflow
Application level testing
RFC-6349 / TrueSpeed™
Application level testing
RFC-6349 / TrueSpeed™
Experience Your Network Like Your
Customer Do
Class of Service verification
ITU-T Y.1564SAMComplete
Class of Service verification
ITU-T Y.1564SAMComplete
Prove L2/3 KPIs for differentiated
services
Traditional L2/3 Turn-up
RFC-2544
Traditional L2/3 Turn-up
RFC-2544
Verify basic performance for single services
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks11
Quality of testing =
Reliability + Repeatabilityindependent of individual skill level
Automated test routines help ensure quality testing
TrueSpeed� automated RFC-6349 test in <5 min
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks12
TCP Performance Tests – What has to be tested?� Can the maximum packet size be used?
– Path MTU (IP level) should be 1500 bytes = 1460 bytes MSS on TCP level
� What is the best Round Trip Time (RTT)?– Measured with low load to avoid influence of filled buffers
� Show influence of TCP Window Size– Run 4 TCP sessions with different WS values and compare actual measured versus
ideal TCP throughput.– This helps to explain the TCP WS influence and will already show critical problems.
� Run a typical TCP session below the Bottleneck Band width– Test duration based on bytes to transfer instead of fixed test time.– Compare actual versus ideal rate and transfer time – Graphical Results to correlate possible TCP performance issues due to
retransmissions and/or congestive RTT.
� Run multiple TCP sessions to force congestion situa tion– This test will conduct multiple TCP connection transfers to test whether the link fairly
shares (traffic shaping) or unfairly shares (traffic policing) the bandwidth
(MTU: Maximum Transfer Unit; MSS: Maximum Segment S ize)
New RFC 6349
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks13
Why does TCP traffic not reach the full bandwidth?
Few main factors if rate limit is in place:� TCP load is mainly based on large file transfers an d less short sessions
– Short sessions are not all active at the same time, they “better” share the bandwidth– With large file transfers a bad throughput is more likely recognized.
� How is the performance of TCP client/server and how is Round Trip Time– High-performance servers or “old & weak” PCs– As higher the performance and as lower the RTT as faster the Slow Start
� Possible “total” TCP rates are higher then available transport rate.– The TCP session(s) may run into congestion– As larger the difference as more critical (E.g. 100Mbps interface to 2 Mbps transport)
� TCP window size is larger then ingress buffer of th e network port– During slow start the buffer is quickly filled and results in packet loss– If there is also large RTT the sessions runs into Dup Ack situation
� Policing function at the network access port– Is the throughput rate done in slots of available bandwidth
Why do customer only sometimes complain about not getting the committed throughput?
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks14
TCP performance and congestion in the metro/core
AccessCore
� Congestion in the core is different to congestion ( policy based rate limitation) at customer access− This is packet based switching with dynamic and possible over booked BW.
Congestion may cause loss events and increasing delay.
� Recommended test application to detect TCP performa nce problems-> RFC6349 Frame Work for TCP Throughput testing-> Layer 2/3 traffic test with focus on packet dela y and delay variation− PDV shows short burst situations and PD shows longer high load and
overload situations, which may result in loss.
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks15
Carrier Access – Where and how to limit the rate?� Who is managing the bandwidth limitation – customer or/and carrier?� Essential criteria: Access rate = or > as transpor t rate
NothingPolicingShapingPolicingShaping
NothingNothingPolicingPolicingNothing
CPE CE
Customer ProviderN x Access rate Access rate Transport rate
� Access rate even as transport rate
NothingPolicingNothingShapingShaping
PolicingPolicingShapingShapingPolicing
� Access rate larger as transport rate
The above examples are only exemplary and not compl ete. They shall only show the complexity
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks16
Why layer 4 TCP full state test stream emulation?
� Customer requires proof of the transfer capacity wi th „real“ layer 4 TCP load (According RFC 6349)– Mostly as second step after failed self-test with laptop file transfer. – Quite meaningfully, if the connection goes through a packet oriented (Ethernet, PBB,
MPLS) core network.
– UDP displaces TCP traffic. This is not detected with pure L3 or state less L4 load.– Low packet loss events can extremely reduce the TCP throughput if there is also
high RTT.– To determine the best TCP window size configuration for max. throughput. – Performance tests across full state fire walls.
� When layer 4 TCP state less load is enough?– When the transport link has fixed bandwidth and no rate limit at the access
(Fiber, optical channel, SDH/OTN)
– Only RTT and TCP Windowsize is influencing the final throughput
© 2011 JDSU. All rights reserved. Layer 4 TCP Performance in carrier transport networ ks17
Estimated, ideal max TCP throughput
� The test application is to supply additional inform ation, which points the protocol specific throughput limits out.– What is the max. throughput for a TCP session, which can be expected?– Depends on the standard TCP window size values (8/16/32/64 kbyte)
� The table shows the max. throughput rates, dependent on the TCP window size, RTT and number of TCP sessions.
� Note: Calculation is based on ideal network situations and is done with statically test load
Calculation based on ideal network situations