1. High-resolution Timer-basedPacket Pacing Mechanismon the
Linux Operating SystemRyousei Takano, Tomohiro Kudoh,Yuetsu Kodama,
Fumihiro OkazakiInformation Technology Research Institute,National
Institute of Advanced IndustrialScience and Technology
(AIST)20102010b1026NS
2. 2 c V 3. NW5_G)2N5C @8BTCP incast@ MPI All-to-all MapReduce
3::1 23N 4. _G)2N5 ):_G={ l):_Gla5j4V1 23BWBW / 3BWm123BW / 3 5. mV
*ij I_G1/2 jc1 GbpsMTU 1500B2410 GbpsMTU 9000B14.4 5 6. PSPacer
_G2) XV ):_Ga5TUlV6BufferOverflowlflrSwitch/RouterPSPacer|a5TUo 7.
VeV V3ecce FPGANPV Chelsio T210PSPacerPSPacer/HT7 8. 8PSPacer: 3e
c*iC OS110 .%K0 .) 1U10 Gbps0.8 *i:7.2 us 7.2 us9000B (byte)0 9K
18K 27KPU9000B 9. 9PSPacere PAUSEIEEE 802.3x *i) - j/ V s/ (PC V
10. e*1. lj CPUlPCII: 10 GbE32bit/33MHz PCI ( 133MB/s)GbE2.
Ethernet M: VpPAUSE3. Yk Bondingtap 8Yk:Yj10 11. 11 c V 12.
LinuxcccTicks (Jiffies) c 1/HZ?V M%=V c n+: ? %1000 1001 1002 1003
100412 13. 3e$@c.%YCPU ce c Linux kernel 2.6.31 c1/16 OS| 14. CPU
Yk NICZ TCP Segmentation Offload13 15. 14PSPacer/HTV QdiscV $ R
Linux) Iproute2 (tc(8))SocketbufferProtocol stackDevice
DriverenqueuedequeuePSPacer/HTByte clockschedulerSocket
LayerInterfacequeuesClassifierNetlinksocket I/F 16.
15!U+sD{[[!~VjcU+g 17. ;el CPUcc(PSPacer/HT)(PSPacer)16 18. 17 c V
19. V aF_G 100 Mbps+L5V^' burstiness Yt H CPU { Y
PSPacerPSPacer/HTHTB (Hierarchical Token Bucket)18 20. HTB:
Hierarchical Token Bucket LinuxQdisc CBQClass based queuing]_G*i:
c) Linux kernel 2.6.31 c1/16 g PSPacer/HT: HTB: l2t (length to
time)f 256c19 21. Myri-10G Myri-10G20VJ PC A CPU: Quad-core Xeon
(E5430) x 2 NIC: Myricom Myri-10G (PCIe x 8) MTU: 9000 byte Memory:
8GB DDR2-667 OS: Ubuntu 9.10 serversender receiver Linux kernel
2.6.31-10 + myri10ge driver 1.5.1 sysctl:
net.core.netdev_max_backlog 25000 net.core.rmem_max 16777216
net.core.wmem_max 16777216 net.ipv4.tcp_rmem 4096 65536 16777216
net.ipv4.tcp_wmem 4096 87380 16777216 net.ipv4.tcp_no_metrics_save
1GtrcNET-10 22. 21GtrcNET NFPGAv : GtrcNET-1: GbE (GBIC) x 4ports +
16MBytes Memory/port GtrcNET-10: 10GbE (XENPAK) x 3ports + 1GBytes
Memory /port V _GU66VLAN6 dw o
*ihttp://projects.itri.aist.go.jp/gnet/ 23.
aF_G*iIperf5V_GGtrcNET-10U ^'N+473 Mbps (+9.5%)+36 Kbps (0.0%)-287
Mbps (-5.7%)HTB: OzPSPacer: 22PSPacer/HT: V 24. Burstiness
Ytburstiness N7 5 Gbps70burstinessNmax. burstinessPSPacer
7PSPacer/HT 9HTB 8c1 3923cl,2NnTSO2m 25. c%d.ULdU10 1burstiness =
53 burstiness = 9(1) .10 2.1c%c3d24 26. CPU1_GPSPacer PSPacer/HT
HTB1_G/50 Mbps 1_G/50 Mbps 1_G/50 Mbps1 Gbps 0.66 0.71 0.842 Gbps
1.80 1.60 1.834 Gbps 3.74 3.66 3.928 Gbps 7.67 8.35 8.88_GNc%25 27.
CPU{_GPSPacer PSPacer/HT HTB1_G/50 Mbps 1_G/50 Mbps 1_G/50 Mbps1
Gbps 0.66 1.04 0.71 0.91 0.84 0.822 Gbps 1.80 2.16 1.60 2.44 1.83
1.884 Gbps 3.74 4.78 3.66 8.19 3.92 4.498 Gbps 7.67 11.19 8.35
17.04 8.88 25.55{Kc%26 28. Vl CPUccCPUxE27 29. 28 c V 30. cu e*A
10GbEJX: {9_GICPU h {CPU, HTBx29 31. 30PSPacer/HTGNU
GPL#http://www.gridmpi.org/pspacer.jsp}SS120800083y ~qNEDOQ4qITo
32. TCP Segmentation Offload MTUN `TSO64 KB31TSO7.250.42(5 Gbps/MTU
9000B)(5 Gbps/MTU 9000B)
