Upload
odele
View
43
Download
0
Embed Size (px)
DESCRIPTION
4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit Development of Real Time eVLBI at Jodrell Bank Observatory 7 th International eVLBI Workshop, Shanghai, 16-17 th June 2008. Richard Hughes-Jones DANTE Jonathan Hargreaves JBO. The 4 Gig Path for EXPReS Onsala – Jodrell. Multi: - PowerPoint PPT Presentation
Citation preview
Connect. Communicate. Collaborate
4 Gigabit Onsala - Jodrell Lightpath for e-VLBI
The iNetTest Unit
Development of Real Time eVLBI at Jodrell Bank Observatory
7th International eVLBI Workshop, Shanghai, 16-17th June 2008
Richard Hughes-Jones DANTE Jonathan Hargreaves JBO
Connect. Communicate. Collaborate
The 4 Gig Path for EXPReSOnsala – Jodrell
Multi:• Domain• Vendor• Technology
Connect. Communicate. Collaborate
First Test Path
4 Gig TDM/SDHLighpathover GÉANT2+
10 Gigabit Ethernet Lambdaover NORDUnet
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
UDP Throughput Stoc-Lon• Alcatel Metro Core Connect MCC
Flow control OFF• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• Max throughput 4.05 Gbit/s
• Packet loss as expected
• Sending host, 3 CPUs idle• For <8 µs packets,
1 CPU is >90% in kernel mode
• Receiving host 3 CPUs idle• For <20 µs packets,
1 CPU is ~30% in kernel mode
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 5 10 15 20 25 30 35 40Spacing between frames us
Recv W
ire r
ate
Mb
it/s
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytes
Stoc2-1_10Jan08
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
ac
ke
t lo
ss
1000 bytes
1472 bytes 2000 bytes
3000 bytes 4000 bytes
5000 bytes 6000 bytes
7000 bytes 8972 bytes
8000 bytes
Stoc2-1_10Jan08
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40
Spacing between frames us
%c
pu
1 k
ern
el
sn
d
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytes
Stoc2-1_10Jan08
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40
Spacing between frames us
% c
pu
1 k
ern
el
rec
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytes
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
UDP Jitter Stoc-Lon• Alcatel Metro Core Connect MCC
Flow control OFF• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=0 Coalescence OFF
• 8972 byte packets• 16.5 μs spacing
• Peak in packet spacing at 16 μsas expected
• Other peaks at 19 and 51 µs
8972B w=16.5 sto-man_rx-usecs0_5Apr08
0
100000
200000
300000
400000
500000
600000
700000
0 20 40 60 80 100 120 140Latency us
N(t
)8972B w=16.5 sto-man_rx-usecs0_5Apr08
1
10
100
1000
10000
100000
1000000
0 20 40 60 80 100 120 140Latency us
N(t
)
Connect. Communicate. Collaborate
exp1-Stoc2_10Jan08.xls
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
ac
ke
t lo
ss
1000 bytes
1472 bytes 2000 bytes
3000 bytes 4000 bytes
5000 bytes 6000 bytes
7000 bytes 8972 bytes
8000 bytes
Connect. Communicate. Collaborate
UDP Throughput Lon-Stoc• Alcatel Metro Core Connect MCC
Flow control ON• Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• Max throughput 4.05 Gbit/s
• No Packet loss – flow control
• Sending host, 3 CPUs idle• For <18 µs packets,
1 CPU is >60% in kernel mode
• Receiving host 3 CPUs idle• For <20 µs packets,
1 CPU is ~30% in kernel mode
exp1-Stoc2_10Jan08.xls
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40Spacing between frames us
%c
pu
1 k
ern
el
sn
d
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytesexp1-Stoc2_10Jan08.xls
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% c
pu
1 k
ern
el
rec
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytes
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 5 10 15 20 25 30 35 40Spacing between frames us
Recv W
ire r
ate
Mb
it/s
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8972 bytes
8000 bytes
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
TCP reno Throughput Lon-Stoc• Alcatel Metro Core Connect MCC
Flow control ON• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• TCP buffer 32 MByte 2*BDP
• iperf Ave throughput 3.85 Gbit/s• iperf Max throughput 4.02 Gbit/s
• 3 re-transmitsDue to “other reductions”
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
TCP reno Throughput Stoc-Lon• MCC NO Flow control• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• TCP buffer 32 MByte 2*BDP
• iperf Ave throughput 192 Mbit/s• iperf Max throughput 465 Mbit/s
• TCP Cwnd to 0• Many re-transmits•
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
Alcatel MCC Buffer size
• Classic Bottleneck• 10 Gbit/s input 4 Gbit/s output• Use udpmon to send a stream of spaced UDP packets• Measure packet number of first lost frame as function of w packet spacing
)R*w(PNQ out1lostlen )/Q(R*wP/QN/1 lenoutlen1lost
Slope gives buffer size ~57 kBytes
Connect. Communicate. Collaborate
Use UDP to emulate TCP slowstart
• udpmon sends bursts of spaced packets:– 32 packets– Jumbo 8000 bytes– back2back– 4 ms between bursts
• PathLon-Ams_FF-Prague-Paris-Lon
• Rtt 55.5 ms
• See 13 packets then loose 1 in 3
• Confirm the TCP problem!
0
5
10
15
20
25
30
0 10 20 30 40 50 60
packet number
1-w
ay t
ime
Connect. Communicate. Collaborate
iNetTest: iBoB FPGA with 10 GE
Connect. Communicate. Collaborate
iNetTest: Control & Operation
10/100 Ethernet
10Gb Port 0
iNetTest iBOB 1
Send and receive packets of selected length and spacing
Count packets transmitted and received, calculate histograms of arrival times
Control PC
IP control of multiple iBOBs
10Gb Port 1
10Gb Port 0
10Gb Port 1
Ten Gb Network
iNetTest iBOB n
Expansion to more than two iBOBs
iNetTest iBOB 2
Send and receive packets of selected length and spacing
Count packets transmitted and received, calculate histograms of arrival times
Connect. Communicate. Collaborate
iNetTest: Simulink design
Jonathan HargreavesJBO
Network test parameter CSRs
10GE MAC core
Inter-packet timehistogram
Transmit & receive event time log
Connect. Communicate. CollaborateiNetTest: Details• iNetTest FPGA runs with a 200 MHz (5 ns) clocks • Two iNetTest units can be controlled over ethernet IP from a PC• iNetTest Ethernet IP address and MAC preset in code
– last digit selected by jumper• Ten GE IP address, Port and Gateway can be user configured• Automatic or manual ARPing. ARP tables for each port can be examined• iNetTest responds to and generates PING• UDP packets can be sent by firmware between two iNetTests
or between iNetTest and PCs• User selects the number of packets to send, their length,
and the time between them• iBOBs count received and transmitted packets, and store up to 2048 time
stamped events per port• iBOBs generate histograms of the arrival time distribution for received
packets
Connect. Communicate. Collaborate
UDP Throughput vs. Packet Spacing
• PC• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R CX4
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• UDP Packets• Max throughput 9.4 Gbit/s
• iBoB• Packet 8234
Data: 8192+ Header: 42• 100 MHz clock• Max rate 6.6 Gbit/s• See 6.44Gbit/s• 200 MHz clock - Linespeed
Connect. Communicate. Collaborate
PC to iNetTest Packet Jitter
• Kernel 2.6.20-web100_pktd-plus
• Myricom 10G-PCIE-8A-R Fibre
• Packet separation 100 µs
• iBoB bins 2320 ns
• Width ± 34 µs
• Similar to PC-PC but no extra peaks
• Structure caused by PC
8900 bytes w=100 exp1-2_rxcoal0_16May07
1
10
100
1000
10000
100000
0 50 100 150 200 250 300
Latency us
N(t
)
esleaPC-ibob_ipg_24Oct07
1
10
100
1000
10000
100000
1000000
10000000
0 50 100 150 200 250
Time between frames us
N(t
)
The PC – PC plots look like:
Connect. Communicate. Collaborate
iNetTest – iBOB to iBOB Results
• 10 GE 3m CX4 cable link• 8192 byte packets• 1 million transmitted at line rate between two iNetTest units.• 5 ns bins
• B2B 10 ns FWHM• • With Switch:
– Fujutsu XG2000 20 ns– HP 6400cl 35 ns long tail 1
10
100
1000
10000
100000
1000000
6570
6575
6580
6585
6590
6595
6600
6605
6610
6615
6620
6625
6630
6635
6640
6645
6650
6655
6660
6665
6670
6675
6680
iBOB-iBOB
via Fujitsu XG2000
via HP6400cl flow ctrl ON
via HP6400cl flow ctrl OFF
Number of
Packets
Time between packet arrivals (ns)
Connect. Communicate. Collaborate
Back to the 4 Gig Path
Connect. Communicate. Collaborate
Stockholm – Manc. with TSS
4 Gig TDM/SDHLighpathover GÉANT2+
4 Gigabit Ethernet Path over TSSNORDUnet
4 Gig on EthernetOptical TransmissionJANET & NNW
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
UDP Throughput Stoc-Man with TSS• Alcatel TSS – 10GE – MCC
• Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=75 Coalescence ON
• MTU 9000 bytes
• Max throughput 3.78 Gbit/s(Previously was 4.05 Gbit/s)
• Packet loss 10 % at 4.096 Gbit/s for 8192 Bytes
• Man-Stoc:No packet loss @ 4.096 Gbit/s
sto-gig6_rx75_TSS_21May08
0
1000
2000
3000
4000
5000
6000
7000
0 5 10 15 20 25 30 35 40Spacing between frames us
Rec
v W
ire
rate
Mb
it/s
1000 bytes
1472 bytes
2000 bytes
3000 bytes
4000 bytes
5000 bytes
6000 bytes
7000 bytes
8000 bytes
8972 bytes
sto-gig6_rx75_TSS_21May08
0
20
40
60
80
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
acke
t lo
ss
1000 bytes
1472 bytes 2000 bytes
3000 bytes 4000 bytes
5000 bytes 6000 bytes
7000 bytes 8000 bytes
8972 bytes
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
Packet loss Stoc-Man with TSS• Alcatel TSS – 10GE – MCC
• Send 1M packets from PC in Stockholm
• 8192 bytes
• All cross the TSS cloud• All enter Alcatel MCC
as 10 GE frames
• Packet loss in MCC Card • ~10% at 4096 Mbit/s
• Classic “bottleneck” performance
• But WHY???
0
2
4
6
8
10
12
0 1000 2000 3000 4000 5000
Offered Rate Mbit/s
% P
acke
t L
oss
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
UDP Jitter Man-Stoc with TSS• Alcatel MCC – 10GE – TSS – PC • Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=0 Coalescence OFF
• 8192 byte packets• Alcatel MCC is TDM 28 VC-4
Can only send packets at 16 µsNot at 10 Gbit/s
• Peak packet spacing ~ 6 µs ie packets arriving at 10 Gbit/s
• Suggests packet bunching in the network
8192B w=16 gig6-sto_rx0_19Jun08
050000
100000150000200000
250000300000
350000400000
0 20 40 60 80 100 120 140Latency us
N(t
)
8192B w=16 gig6-sto_rx0_19Jun08
1
10
100
1000
10000
100000
1000000
0 20 40 60 80 100 120 140Latency us
N(t
)8972B w=16.5 sto-man_rx-usecs0_5Apr08
0
100000
200000
300000
400000
500000
600000
700000
0 20 40 60 80 100 120 140Latency us
N(t
)
Sto-Man no TSS16 µs
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
UDP 1-way delay Man-Stoc with TSS• Alcatel MCC – 10GE – TSS – PC • Kernel 2.6.20-web100_pktd• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=0 Coalescence OFF
• 8192 byte packets•
• Structure similar to interrupt coalescence but this is OFF
• As function of recv timesee gaps of ~ 130 uspackets at ~ 10Gbit/s
• Again suggests packet bunching in the network
W16 gig6-sto_rx0_19Jun08
18600
18650
18700
18750
18800
18850
18900
18950
19000
0 50 100 150 200Packet No.
1-w
ay d
elay
us
W16 gig6-sto_rx0_19Jun08
1860018650
1870018750
1880018850
1890018950
19000
75580000 75585000 75590000 75595000 75600000receive time us.
1-w
ay d
elay
us
Connect. Communicate. Collaborate
Last week: Onsala – JodrelliNetTest - iNetTest
4 Gig TDM/SDHLighpathover GÉANT2+
4 Gigabit Ethernet Path over TSSNORDUnet
4 Gig on EthernetOptical TransmissionJANET & NNW
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
iNetTest: Onsala JBOUDP Packet Jitter
• 1M packets• 8192 byte packets• 16 µs spacing 4.096 Gbit/s• • 10 % packet loss
• Peaks 14 & 15.7 µs• Tail to 130 µs
• Cyclic variations spacing3.25 µs
• 45 packets at 6.5 µs (line rate)
0
50000
100000
150000
200000
250000
0 50 100 150Packet to Packet arrival time (us)
Num
ber
of P
acke
ts
1
10
100
1000
10000
100000
1000000
0 50 100 150Packet to Packet arrival time (us)
Num
ber
of P
acke
ts
Connect. Communicate. Collaborate
Development of Real Time eVLBI at JBO
eMERLIN Import – Onsala to Jodrell eMERLIN Export – Jodrell to JIVE
Connect. Communicate. Collaborate
Overview of eVLBI at JBO
eMERLIN CORRELATOR
JBO JIVE
4Gbps light path
OnsalaJBOCX4 4GbpsOr fibre if > 15m
CX4 4Gbps iBOB 5SwitchiBOB 4 Switch ADCStation Board VSI
VSI
4Gbps light
path
CX4 1Gbps
CX4 1Gbps
CX4 1Gbps
CX4 1Gbps
iBOB 0 SwitchStation Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
Switch
iBOB 1Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
iBOB 2Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
iBOB 3Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
VLBI Mk V b receiv
ers
Connect. Communicate. Collaborate
Onsala to JBO: Transmit eMERLIN Import
ZDOK0
iADC
2 channels
1024MSPS per channel
8 bit sampling
10GB PORT
. 4GB/s out
ASSEMBLE PACKETS
Divide data into 8192 byte packets plus 16 byte MkVC compatible header
Alternatively generate test packets as per iNetTest
Count packets transmitted
DATA Left Pol
ADCCLK
ONEPPS
Convert data to 2 bit
Measure signal power on both channels
Generate real time clock based on ADC clock
Pass data & sync from ADC clock to iBOB 200MHz clock domain
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
Measured RMS Signal Power
Read and Set Real Time Clock
Set up test mode parameters
Control Data Flow ON/OFF
Set Network Source and Destination
DATA Right Pol
iBOB
Connect. Communicate. Collaborate
Onsala to JBO: Receive
ZDOK0
10GB PORT
. 4GB/s in
RE-ORDER PACKETS
Store data in SRAM location according to its time stamp and sequence number
Regenerate Sync and Data Valid signals
FIFO
16k deep
Smooth out bunching due to network delay variations
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
iBOB
STREAM OUT
Clock data out of SRAM on 128MHz correlator clock
Stream the left and right polarisations to the Station Board VSI chips
ZDOK1
VSI Chip 0
Route left polarisation data to input stage of Station Board
Send correlator clock to iBOB
VSI Chip 1
Route right polarisation data to input stage of Station Board
Station Board
SRAM0
512 packets L pol. data
SRAM1
512 packets R pol. data
DATA 32 bit
DATA 32 bit
128MHz clock
Connect. Communicate. CollaborateOnsala to JBO - Status
• iADC and transmit iBOB tested with ADC clocked at 1GS/s in diagnostic mode – ADC is set to output a fixed bit pattern
• iBOB receiver and VSI chip coded in Simulink and VHDL
Next Steps
• Test iADC and transmit iBOB with sinusoidal test signal• Generated deliberately out-of-order packets and bursts of
greater than 4Gbps to test receiver buffering
Connect. Communicate. Collaborate
JBO to JIVE: Transmit – OvervieweMERLIN Export
ZDOK0
10GB PORT
. 1 Gb/s out
REMOVE DELAY MODEL
(COURSE)
Use 72 bit wide bt 512k deep SRAM to remove eMERLIN delay model with 16 microsecond resolution
FORMAT
Generate 10000 byte MkVb frames
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
iBOB
ZDOK1
VSI Chip 0
Process left polarisation data
Detail on next slide
VSI Chip 1
Process right polarisation data
Detail on next slide
Station Board
DATA 32 bit
DATA 32 bit
CLOCK
VALID
PDATA
CLOCK
VALID
PACKETISE
Divide each MkVb frame into two 5008 byte packets (incl header)
Add Mk5C header to each packet
Connect. Communicate. Collaborate
JBO to JIVE: Transmit -VSI Chip Detail
VSI Chip 0 - Process left polarisation data
Monitor & Control Bus (MCB)
Station Board
DATA 32 bit
CLOCK
VALID
PDATA
Select a 128MHz
band from the
filter bank
Subtract Merlin Delay (Fine)
Variable coefficient FIR filter changes delay in steps of 1/16th of original sample period
Reclock for delays up to 16ns
Mixer 0
FIRLow pass filter
Decimate by 8/16128 taps
Convert 4/8 bit data to 2 bit VLBI
Mixer 1
Mixer 8/16
Stream data out to iBOB over VSI
cable
MCB InterfaceLoad band selection, delay model and filter
parameters
0
1
15
Mixer Bank
Select 8 or 16 VLBI bands from
128MHz input
Remove eMERLIN offset
(a multiple of 10kHz)
Connect. Communicate. CollaborateJBO to JIVE: Status
• VSI chip DSP functions currently under development and simulation
• Transmit iBOB re-uses packetising code from iNetTest
• Next Steps• VSI chip MCB interface – based on Verilog code provided
by Dave Fort at Penticton• Transmit iBOB needs coarse delay and MkVb formatting
Connect. Communicate. Collaborate
Summary
• UDP on a 28 VC-4, 4.2 Gigabit lightpath provides the performance required for EXPReS
• TCP performance poor due to small Ethernet buffer size on the equipment connecting 10GE to the 4 Gig Lightpath
• Ethernet flow-control with short tails helps TCP
• The network behaves correctly.• Care needed on choice of PC for 10 Gigabit.• FPGA iNetTest solutions work very well.• Onsala – Jodrell 4 Gigabit path now in place.• Packet loss and packet spacing need more understanding. • VLBI FPGA application progressing well.
Connect. Communicate. Collaborate
Any Questions ?
Connect. Communicate. CollaborateFor More Information
• www.geant2.net• www.dante.net• For latest news and factsheets http://www.geant2.net/media
• For research activities http://www.geant2.net/research• http://expres-eu.org/ [note: only one “s”]• http://www.jive.nl/ • Contact information:
Richard Hughes-Jones, DANTE, [email protected] Hargreaves, JBO, [email protected]
• EXPReS is made possible through the support of the European Commission (DG-INFSO), Sixth Framework Programme, Contract #02664
Connect. Communicate. Collaborate
GÉANT2 Topology
November 2006
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
TCP BIC Throughput Lon-Stoc• Alcatel Metro Core Connect MCC
Flow control ON• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• TCP buffer 32 MByte 2*BDP
• iperf Ave throughput 3.98 Gbit/s• iperf Max throughput 4.02 Gbit/s
• Several re-transmits•
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
TCP BIC Throughput Stoc-Lon• MCC NO Flow control• 27 VC-4• RTT 31 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• TCP buffer 32 MByte 2*BDP
• iperf Ave throughput 71.6 Mbit/s• iperf Max throughput 192 Mbit/s
• TCP Cwnd rapidly opens• Many more re-transmits•
Connect. Communicate. CollaborateConnect. Communicate. Collaborate
TCP BIC Throughput Lon-StocBut Faster and Longer
• MCC Flow control• 48 VC-4• RTT 60 ms• Kernel 2.6.20-web100_pktd-plus• Myricom 10G-PCIE-8A-R Fibre
– rx-usecs=25 Coalescence ON
• MTU 9000 bytes• TCP buffer 59 MByte 1 * BDP• txquelen 1300• …
• iperf Ave throughput 6.5 Gbit/s• iperf Max throughput ~6.9 Gbit/s
• Some glitches & re-transmits•