JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University

JIVE VLBI Network Meeting 28 Jan 2004R. Hughes-Jones Manchester

Brief Report on Tests Related to the e-VLBI Project

Richard Hughes-JonesThe University of Manchester

DataGrid WP7 – Dante Tests on the GÉANT CoreEnd-2-End Measurements from the 4th Year VLBI Project at ManchesterDiskPack-2-Memory Throughput and PCI Activity in a Mark5 PCUpdate on 1 and 10 Gigabit Ethernet NICs in the PC


DataGrid WP7 – Dante Tests on the GÉANT Core Set-up

Supermicro PC in: London GEANT PoP Amsterdam GEANT PoP

Smartbits in: London GEANT PoP Frankfurt GEANT PoP

Long link UK-SE-DE2-IT-CH-FR-BE-NL

Short Link UK-FR-BE-NL


Tests GÉANT Core: UDP throughput UDP Throughput London-Amsterdam Available BW to packet on wire Then 1/t Wire rate 998 Mbit/s

for packets > 1400 bytes

Packet Loss None for large packets

Dips in BW lined to packet loss SysKonnect NIC int. per packet CPU load important

uk-nl_20tg4-hs-w100_01Oct03

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40Spacing between frames us

Rec

v W

ire r

ate

Mbi

ts/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

0102030405060708090

100


% P

acke

t los

s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes


Tests GÉANT Core: Packet re-ordering Effect of Packet size London-Amsterdam Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%

Re-order Distribution

Packet re-order uk-nl 10,000 BE sent wait 10 us 01 Oct 03

0

5

10

15

20

25

30

0 500 1000 1500Packet size bytes

Out

of o

rder

% 0

10

20

30

1400 1401 1402 1403 1404Packet size bytes

Ou

t o

f o

rde

r %

Packet re-order uk-nl 10,000 sent wait 10 us

0

100

200

300

400

500

0 1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets 1400 bytes

1401 bytes

1402 bytes


Tests GÉANT Core: Packet re-ordering Effect of LBE background Amsterdam-London BE Test flow Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%

Re-order Distributions:

UDP 1472 bytes NL-UK-lbexxx_7nov03

02468

101214161820

2 2.2 2.4 2.6 2.8 3 3.2Total Offered Rate Gbit/s

% O

ut o

f ord

er

hstcpStandard TCP line speed90% line speed

Packet re-order 1472 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us

020000400006000080000

100000120000140000160000180000200000

1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets

0 % lbe

10 % lbe

20 % lbe

30 % lbe

40 % lbe

50 % lbe

60 % lbe

70 % lbe

80 % lbe

Packet re-order 1400 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us

0500

100015002000250030003500400045005000

1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets

0 % lbe

10 % lbe

20 % lbe

30 % lbe

40 % lbe

50 % lbe

60 % lbe

70 % lbe

80 % lbe


VLBI Project: Test Topology

SuperJANET4

Jodrell

Manchester

SURFnet

JIVEDwingaloo

Adam MathewsSteve O’TooleUniv of Manchester


Gnt5-DwMk5 11Nov03-1472 bytes

0

2

4

6

8

10

12


% P

acket

loss

Gnt5-DwMk5

DwMk5-Gnt5

Manchester to Dwingeloo 2.0G Hz Xeon 1.2 GHz PIII

Re-ordering vs Offered Load

VLBI Project: Throughput

Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes

0

200

400

600

800

1000

1200


Recv W

ire r

ate

Mbits/s

Gnt5-DwMk5

DwMk5-Gnt5

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l S

ende

r

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l R

ecei

ver


1472 byte Packets man -> JIVE FWHM 22 µs (B2B 3 µs )

VLBI Project: Jitter & 1-way Delay

1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03

0

2000

4000

6000

8000

10000

0 20 40 60 80 100 120 140

Jitter us

N(t

)

1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03

1

10

100

1000

10000

0 20 40 60 80 100 120 140

Jitter us

N(t

)

1472 bytes w12 Gnt5-DwMk5 21Oct03

0

2000

4000

6000

8000

10000

12000

2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000Packet No.

1-w

ay

de

lay

us

1472 bytes w12 Gnt5-DwMk5 21Oct03

0

2000

4000

6000

8000

10000

12000

0 1000 2000 3000 4000 5000Packet No.

1-w

ay d

elay

us

1-way Delay – note the packet loss (points with 0 –way delay)


Measure the time between lost packets in the time series of packets sent.

Lost 1410 in 0.6s Is it a Poisson process? Assume Poisson is

stationary λ(t) = λ

Use Prob. Density Function:

P(t) = λ e-λt

Mean λ = 2360 / s[426 µs]

Plot log: slope -0.0028expect -0.0024

Could be additional process involved

VLBI Project: Packet Loss Distributionpacket loss distribution 12b bin=12us

0

10

20

30

40

50

60

70

80

12 72 132

192

252

312

372

432

492

552

612

672

732

792

852

912

972

Time between lost frames (us)

Num

ber

in B

in

Measured

Poisson

packet loss distribution 12b

y = 41.832e-0.0028x

y = 39.762e-0.0024x

1

10

100

0 500 1000 1500 2000

Time between frames (us)

Num

ber

in B

in


Divide time series of packets into 1000 slices of 50 packets

Total lost packets 1410 Average number / slice = 1.4

Calc Poisson Probability P(n, µ) = µ n e -µ

n!

Curves close but not exact Could be more than 1 process

VLBI Project: Packet Loss – Is it Poisson?

0

50

100

150

200

250

300

350

400

0 5 10 15n num lost in sub-sample

N(n

)

run12b

1

1.3

1.4

1.8


Aggregated Variance Method Divide time series length N into

blocks of size m Calc mean of each section Xm(k)

k= 1 … N/m Calc variance VXm of these Xm(k) Vary m size of the blocks

Plot on log-log & fit slope β Hurst parameter H

β = 2H -2 Measure:

β = -0.355 which gives H 0.822 H =1 no long range dependence

VLBI Project: Packet Loss – Long Range Effects?

y = -0.355x + 2.8826

0

0.5

1

1.5

2

2.5

3

3.5

4

0 0.5 1 1.5 2 2.5 3sub-sample size Log10( m )

Ag

gri

ga

te-v

ari

an

ce L

og

10

( X

(m)

)


Traffic Flows Manchester – NetNorthWest - SuperJANET Access links

Two 1 Gbit/s

Access links:SJ4 to GÉANT GÉANT to SurfNet


Throughput & PCI transactions on the Mark5 PC:

Read /Write

n bytes

Wait timetime

Mark5 uses Supermicro P3TDLE 1.2 GHz PIII Mem bus 133/100 MHz 2 *64bit 66 MHz PCI 4 32bit 33 MHz PCI

SuperStor

NIC

Input Card

IDEDiscPack

Ethernet

Logic AnalyserDisplay


PCI Activity: Read 1 data block 0 wait time

Data block 131,072 bytes Read consists of:

Setup CSRs Data movement Update CSRs

Data block contains PCI bursts 4096 bytes long

For 0 wait between reads: Data block 608µs long Then 655µs gap

PCI transfer rate 830 Mbit/s Read_sstor rate 778 Mbit/s

(97 Mbyte/s)

CSR Access

PCI Burst

Data Block


PCI Activity: Read Multiple data blocks 0 wait Read 999424 bytes Each Data block:

Setup CSRs Data movement Update CSRs

For 0 wait between reads: Data blocks ~600µs long

take ~6 ms Then 744µs gap

PCI transfer rate 1188Mbit/s(148.5 Mbytes/s)

Read_sstor rate 778 Mbit/s (97 Mbyte/s)

PCI bus occupancy: 68.44% Concern about Ethernet Traffic

64 bit 33 MHz PCI needs ~ 82% for 930 Mbit/s Expect ~360 Mbit/s


PCI Activity: Read Throughput

Flat then 1/t dependance ~ 860 Mbit/s for Read blocks

>= 262144 bytes

CPU load ~20% Concern about CPU load

needed to drive Gigabit link


10 GigEthernet: Throughput 1500 byte MTU gives ~ 2 Gbit/s Used 16144 byte MTU max user length 16080 DataTAG Supermicro PCs Dual 2.2 GHz Xenon CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate throughput of 2.9 Gbit/s

CERN OpenLab HP Itanium PCs Dual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate of 5.7 Gbit/s

SLAC Dell PCs giving a Dual 3.0 GHz Xenon CPU FSB 533 MHz PCI-X mmrbc 4096 bytes wire rate of 5.4 Gbit/s

an-al 10GE Xsum 512kbuf MTU16114 27Oct03

0

1000

2000

3000

4000

5000

6000


Rec

v W

ire

rate

Mb

its/

s

16080 bytes 14000 bytes 12000 bytes 10000 bytes 9000 bytes 8000 bytes 7000 bytes 6000 bytes 5000 bytes 4000 bytes 3000 bytes 2000 bytes 1472 bytes


10 GigEthernet: PCI-X Activity

Supermicro P4DP8-2G motherboard

Packet transmissionMemory to NIC

Packet receptionNIC to Memory

CSR Access

Interrupt

PCI-X segment 512 bytes

Transfer of 16114 bytes


10 GigEthernet: Tuning PCI-X

mmrbc 1024 bytes

mmrbc 2048 bytes

mmrbc 4096 bytes

Transfer of 16114 bytes

mmrbc 512 bytes

PCI-X segment 2048 bytes

CSR Access


10 GigEthernet at SC2003 BW Challenge (Phoenix) Three Server systems with 10 GigEthernet NICs Used the DataTAG altAIMD stack 9000 byte MTU Streams From SLAC/FNAL booth in Phoenix to:

Pal Alto PAIX

Chicago Starlight Amsterdam SARA

10 Gbits/s throughput from SC2003 to PAIX

0

1

2

3

4

5

6

7

8

9

10

11/19/0315:59

11/19/0316:13

11/19/0316:27

11/19/0316:42

11/19/0316:56

11/19/0317:11

11/19/0317:25 Date & Time

Throughput

Gbits/s

Router to LA/PAIXPhoenix-PAIX HS-TCPPhoenix-PAIX Scalable-TCPPhoenix-PAIX Scalable-TCP #2

10 Gbits/s throughput from SC2003 to Chicago & Amsterdam

0

1

2

3

4

5

6

7

8

9

10

11/19/0315:59

11/19/0316:13

11/19/0316:27

11/19/0316:42

11/19/0316:56

11/19/0317:11

11/19/0317:25 Date & Time

Throughput

Gbits/s

Router traffic to Abilele

Phoenix-Chicago

Phoenix-Amsterdam



Traffic QoS Classes on GÉANT Backbone

Normal Traffic

Normal Traffic +

Less Than Best Effort 2.0 Gbit/s

Normal Traffic +

Radio Astronomy Data 500 Mbit/s

Normal Traffic +

Radio Astronomy Data +

Less Than Best Effort 2.0 Gbit/s

Max Throughput on 2.5 G PoS


Some Measurements made during ER2002

No LBE

0

2

4

6

8

10

12

14

16

18

20

0 20 40 60 80 100 120 140 160 180 200Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost

With 1.8Gbit LBE

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

0 20 40 60 80 100 120 140 160Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost

Documents

JIVE VLBI Network Meeting 28 Jan 2004 R. Hughes-Jones Manchester Brief Report on Tests Related to the e-VLBI Project Richard Hughes-Jones The University