View
55
Download
0
Category
Tags:
Preview:
DESCRIPTION
FABRIC 4 Gigabit Work & VLBI-UDP Performance and Stability. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks”. Multi-Gigabit Trials on GEANT Collaboration with Dante. What is inside G É ANT2 What is the collaboration interesting? 10 Gigabit Ethernet - PowerPoint PPT Presentation
Citation preview
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester1
FABRIC 4 Gigabit Work
&
VLBI-UDP Performance and Stability.
Richard Hughes-Jones The University of Manchester
www.hep.man.ac.uk/~rich/ then “Talks”
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester2
What is inside GÉANT2 What is the collaboration interesting?
10 Gigabit Ethernet UDP memory-2-memory flows TCP flows with allocated Bandwidth
Options using GÉANT Development Network 10 Gbit SDH Network
Options Using the GÉANT LightPath Service PoP Location for Network tests
Multi-Gigabit Trials on GEANT
Collaboration with Dante.
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester3
GÉANT2 Topology
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester4
GÉANT2: The Convergence Solution
NREN AccessNREN Access
ExistingIP Router
ExistingIP Router
GÉANT2POP B
GÉANT2POP A
Managed Lambda’s
1626 LM
1626 LM
L2Matrix
L2
TDM Matrix
TDM
1678 MCC
1678 MCCDar
k F
iber
EXPReS PC10 GE
EXPReS PC10 GE
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester5
What do we want to do? Set up 4 Gigabit Lightpath Between GÉANT PoPs
Collaboration with Dante PCs in their PoPs with 10 Gigabit NICs we FABRIC provide the SX XFPs, Dante the MCC ports
VLBI Tests: UDP Performance
Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems
Dante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP when using BW limited
Lightpaths
Need A Collaboration Agreement
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester6
Generic solution:
Rate adaptive mapping into VC4-nv with flow ctrl. and LCAS, n=1..64;
up to 4 ports per boardFull rate option:
Ethernet full rate up to VC4-68v, flow ctrl. and LCAS for n<68
Optical Transport Hierarchy option:
Mapping into OpticalDataUnit1/2 with flow ctrl.
O/EData
Mapper
4x 10GE LAN board in 1678MCC
DataMapper
MAC
MACO/E
O/E
BKPdrData
Mapper
DataMapper
MAC
MACO/E
Data Mapper Device - GFP-F into VC-4-nv - GFP-F into ODU1/2
4x10GE LAN interface ports
Tomatrix
10GE-LAN interface board in 1678 MCCDante interested in MCC performance
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester7
Options Using the GÉANT Development Network
10 Gigabit SDH backbone Alkatel 1678 MCC Node location:
London Amsterdam Paris Prague Frankfurt
Can do traffic routingso make long rtt paths
Available Feb 07 Less Pressure for
long term tests
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester8
Options Using the GÉANT LightPaths Set up 4 Gigabit Lightpath Between GÉANT PoPs
Collaboration with Dante PCs in Dante PoPs
10 Gigabit SDH backbone Alkatel 1678 MCC Node location:
Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna
Can do traffic routingso make long rtt paths
Ideal: London Copenhagen
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester9
Test PCs Have Arrived
Boston/Supermicro X7DBE Two Dual Core Intel Xeon
Woodcrest 5130 2 GHz Independent 1.33GHz FSBuses
530 MHz FD Memory (serial)
Chipsets: Intel 5000P MCH – PCIe & MemoryESB2 – PCI-X GE etc.
PCI 3 8 lane PCIe buses 3* 133 MHz PCI-X
2 Gigabit Ethernet SATA
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester10
Lab Tests 10 Gigabit Ethernet 10 Gigabit Test Lab set up in Manchester
Cisco 7600 Cross Campus λ <1ms Server quality PCs Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased
B2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing
Installed Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) on the Intel dual-core PCs (Stephen Kershaw)
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester11
iBOB
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester12
Prototype iBOB with two sampler boards attached
FPGA based signal processing board from UC Berkeley
See Ralph’s Talk this afternoon
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester13
Bryan Anderson
Station
Board
RAM
iBOB10GE
VSI
10GE CX4
systembasedDisk
VSI or headstack
CX4 - fibre media converter
iBOB block diagram
10 Gigabit Ethernet now available UDP/IP module exists Use for Demonstration of FPGA driven
IP networking Link iBoB to PC NIC – basic tests Test over GÉANT 4 Gigbit flow Onsala - Jodrell
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester14
vlbi_udp
Multiple stream tests over the Academic network
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester15
vlbi_udp Flow Diagram
Ring buffer
Input thread
Ring buffer
Send thread
Disk Memory
Receive thread
Output thread
Disk Memory
From Simon Casey
UDP Data
TCP Control
Control thread Control thread
Vlbi_control
vlbi_send vlbi_recv
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester16
Jodrell BankUK
DwingelooDWDM link
MedicinaItaly Torun
Poland
e-VLBI at the GÉANT2 Launch Jun 2005
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester17
Geant2 Launch event, June 2005
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester18
UDP Performance: 3 Flows on GÉANT Throughput: 5 Hour run Jodrell: JIVE
2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s
Medicina (Bologna): JIVE 800 MHz PIII – mark623 1.2 GHz PIII 330 Mbit/s limited by sending PC
Torun: JIVE 2.4 GHz dual Xeon – mark575 1.2 GHz PIII 245-325 Mbit/s limited by security policing (>400Mbit/s 20 Mbit/s) ?
Throughput: 50 min period Period is ~17 min
BW 14Jun05
0
200
400
600
800
1000
0 500 1000 1500 2000Time 10s steps
Rec
v w
ire r
ate
Mbi
t/s
JodrellMedicinaTorun
BW 14Jun05
0
200
400
600
800
1000
200 250 300 350 400 450 500Time 10s steps
Rec
v w
ire r
ate
Mbi
t/s
JodrellMedicinaTorun
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester19
UDP Performance: Flows on GÉANT Packet Loss & Re-ordering
Jodrell: 2.0 GHz Xeon Loss 0 – 12% Reordering significant
Torun: 2.4 GHz Xeon Loss 6 - 12% Reordering in-significant
Torun 14Jun04
0
1
2
3
4
5
0 500 1000 1500 2000Time 10s
num
re-
orde
red
020000
400006000080000100000
120000140000
num
lost
re-ordered
num_lost
jbgig1-jivegig1_14Jun05
0
500
1000
1500
2000
0 500 1000 1500 2000Time 10s
num
re-
orde
red
0
50000
100000
150000
num
lost
re-ordered
num_lost
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester20
Instability: 2 streams into JIVE, March 2006From Simon Casey
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester21
Dedicated
Gbit link
EVN-NREN
OnsalaSweden
Gbit link
Jodrell BankUK
DwingelooDWDM linkMedicina
Italy
Chalmers University
of Technolog
y, Gothenbur
g
TorunPoland
Gbit link
Metsähovi
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester22
Any Questions?
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester23
Backup Slides
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester24
From PoS to EthernetConnect. Communicate. Collaborate
•More Economical Architecture
•Highest Overall Network Availability
•Flexibility (VLAN management)
•Highest Network Performance (Latency)
Router
IP Links
1/10 Gigabit Ethernet
VC-4-nv Channels
L2Matrix
TDM Matrix
1678 MCCTransport Node
VLANs
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester25
10 Gigabit Ethernet: UDP Throughput
1500 byte MTU gives ~ 2 Gbit/s Used 16144 byte MTU max user length 16080 DataTAG Supermicro PCs Dual 2.2 GHz Xenon CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate throughput of 2.9 Gbit/s
CERN OpenLab HP Itanium PCs Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz PCI-X mmrbc 4096 bytes wire rate of 5.7 Gbit/s
SLAC Dell PCs giving a Dual 3.0 GHz Xenon CPU FSB 533 MHz PCI-X mmrbc 4096 bytes wire rate of 5.4 Gbit/s
an-al 10GE Xsum 512kbuf MTU16114 27Oct03
0
1000
2000
3000
4000
5000
6000
0 5 10 15 20 25 30 35 40Spacing between frames us
Rec
v W
ire
rate
Mb
its/
s
16080 bytes 14000 bytes 12000 bytes 10000 bytes 9000 bytes 8000 bytes 7000 bytes 6000 bytes 5000 bytes 4000 bytes 3000 bytes 2000 bytes 1472 bytes
EVN-NREN Meeting, Zaandan, 31 Oct 2006, R. Hughes-Jones Manchester26
10 Gigabit Ethernet: Tuning PCI-X
16080 byte packets every 200 µs Intel PRO/10GbE LR Adapter PCI-X bus occupancy vs mmrbc
Measured times Times based on PCI-X times from
the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s
mmrbc1024 bytes
mmrbc2048 bytes
mmrbc4096 bytes5.7Gbit/s
mmrbc512 bytes
CSR Access
PCI-X Sequence
Data Transfer
Interrupt & CSR UpdateKernel 2.6.1#17 HP Itanium Intel10GE Feb04
0
2
4
6
8
10
0 1000 2000 3000 4000 5000Max Memory Read Byte Count
PC
I-X
Tra
nsfe
r tim
e
us
measured Rate Gbit/srate from expected time Gbit/s Max throughput PCI-X
DataTAG Xeon 2.2 GHz
0
2
4
6
8
10
0 1000 2000 3000 4000 5000Max Memory Read Byte Count
PC
I-X
Tra
nsfe
r tim
e
us
measured Rate Gbit/srate from expected time Gbit/s Max throughput PCI-X
Recommended