High Rate Internet Data Transfer for eVLBI

Ralph Spencer, Richard Hughes-Jones and Simon Casey

The University of ManchesterAll Hands Sept 2005

Radio Astronomy• The study of celestial objects at <1 mm to >1m

wavelength.• Sensitivity for continuum sources• B=bandwidth, integration time.• High resolution achieved by interferometers.

Some radio emitting X-ray binary stars in our own galaxy:

B/1

GRS 1915+105 MERLIN

SS433MERLIN and European VLBI

Cygnus X-1VLBA

GRS 1915+105: 15 solar mass BH in an X-ray binary: MERLIN observations

receding

600 mas = 6000 A.U. at 10 kpc

Very Long Baseline Interferometry

How interferometry works

• Interferometer: signals from each telescope brought together coherently for correlation (multiplication)

• Resolution /D << that possible from a single telescope, e.g the pimples on a golf ball at 3000 miles at the highest resolution

• Output of the correlator is one component of the Fourier transform of the sky brightness

Dcos

X

D

Earth-Rotation Synthesis

.

The baseline between each pair of telescopes forms an ellipse as the Earth rotates

The Aperture Plane (u,v)

• Need ~ 12 hours for full synthesis, not necessarily collecting data for all that time.

• Telescope data correlated in pairs: N(N-1)/2 baselines

• NBTrade-off between B and t for sensitivity ~sqrt(Bt)

• Data are calibrated, Fourier inverted and then deconvolved to give the images.

EVN

MERLIN

The European VLBI NetworkEVN

• Detailed radio imaging uses antenna networks over 100s-1000s km

• At faintest levels, sky teems with galaxies being formed

• Currently use disk recording at 512Mb/s (MkV)

• real-time connection allows greater – response– reliability– sensitivity

– Need Internet eVLBI

WesterborkNetherlands

Dedicated

Gbit link

EVN-NREN

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM link

CambridgeUK

MERLIN

MedicinaItaly

Chalmers University

of Technolo

gy, Gothenbu

rg

TorunPoland

Gbit link

eVLBI Milestones

• January 2004: Disk buffered eVLBI session:– Three telescopes at 128Mb/s for first eVLBI

image– On – Wb fringes at 256Mb/s

• April 2004: Three-telescope, real-time eVLBI session.– Fringes at 64Mb/s– First real-time EVN image - 32Mb/s.

• September 2004: Four telescope real-time eVLBI– Fringes to Torun and Arecibo– First EVN, eVLBI Science session

• January 2005: First “dedicated light-path” eVLBI– Australia to JIVE via Canarie: data from Huygens

descent transferred at ~450Mb/s

• 20 December 20 2004• connection of JBO to Manchester by 2 x 1 GE• eVLBI tests between Poland Sweden UK and Netherlands at 256 Mb/s

• February 2005• TCP and UDP memory – memory tests at rates up to

450 Mb/s (TCP) and 650 Mb/s (UDP)• Tests showed inconsistencies between Red Hat

kernels, rates of 128 Mb/s only obtained on 10 Feb• Haystack (US) – Onsala (Sweden) runs at 256 Mb/s

• Regular tests with VLBI data every ~6 weeks • 128 Mpbs OK, 256 Mpbs often,• But not 512 Mbps – WHY NOT?

• Correlator can cope with large error rates• up to 2 %• but need high throughput for sensitivity• implications for protocols

Gnt5-DwMk5 11Nov03-1472 bytes

0

2

4

6

8

10

12

0 5 10 15 20 25 30 35 40Spacing between frames us

% P

acket

loss

Gnt5-DwMk5

DwMk5-Gnt5

Throughput vs packet spacing Manchester: 2.0G Hz Xeon Dwingeloo: 1.2 GHz PIII Near wire rate, 950 Mbps UDPmon

Packet loss

CPU Kernel Load sender

CPU Kernel Load receiver 4th Year project

Adam Mathews Steve O’Toole

Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes

0

200

400

600

800

1000

1200

0 5 10 15 20 25 30 35 40Spacing between frames us

Recv W

ire r

ate

Mbits/s

Gnt5-DwMk5

DwMk5-Gnt5

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100

0 5 10 15 20 25 30 35 40Spacing between frames us

% K

erne

l S

ende

r

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100

0 5 10 15 20 25 30 35 40Spacing between frames us

% K

erne

l R

ecei

ver

UDP Throughput Oct-Nov 2003 Manchester-Dwingeloo Production

Throughput (JbGig1 - Mark608 @ JIVE via production)

500

600

700

800

900

1000

25/0

1/20

0500

:00

25/0

1/20

0512

:00

26/0

1/20

0500

:00

26/0

1/20

0512

:00

27/0

1/20

0500

:00

Time of day

Wir

e r

ate

(M

b/s

)

12us

18us

20us

% Packets lost (JbGig1 - Mark608 @ JIVE via production)

0

0.5

1

1.5

2

25/0

1/20

0500

:00

25/0

1/20

0512

:00

26/0

1/20

0500

:00

26/0

1/20

0512

:00

27/0

1/20

0500

:00

Time of day

% L

os

t

12us

18us

20us

January 2005 repeated UDP over 2 days

ESLEA

• Packet loss will cause low throughput in TCP/IP

• Congestion will result in routers drooping packets: use Switched Light Paths!

• Tests with MB-NG network Jan-• JBO connected to JIVE via UKLight

in June (thanks to John Graham, UKERNA)

• Comparison tests between UKLight connections JBO-JIVE and production (SJ4-Geant)

UKLight Switched light path

Tests on the UKLight switched light-path Manchester : Dwingeloo

• Throughput as a function of inter-packet spacing

• Packet loss • Maximum size packets can

reach full line rates with no loss, and there was no re-ordering (plot not shown).

gig03-jiveg1_UKL_25Jun05

0100200300400500600700800900

1000

0 10 20 30 40Spacing between frames us

Rec

v W

ire r

ate

Mbi

t/s

50 bytes

100 bytes

200 bytes

400 bytes

600 bytes

800 bytes

1000 bytes

1200 bytes

1400 bytes

1472 bytes

gig03-jiveg1_UKL_25Jun05

0.0001

0.001

0.01

0.1

1

10

100

0 10 20 30 40Spacing between frames us

% P

acke

t lo

ss

50 bytes

100 bytes 200 bytes

400 bytes 600 bytes

800 bytes 1000 bytes

1200 bytes 1400 bytes

1472 bytes

Tests on the production network Manchester : Dwingeloo.

• Throughput

• Small (0.2%) packet loss was seen

• Re-ordering of packets was significant

gig6-jivegig1_31May05

0.0001

0.001

0.01

0.1

1

10

100

0 10 20 30 40Spacing between frames us

% P

acket

loss

50 bytes

100 bytes 200 bytes

400 bytes 600 bytes

800 bytes 1000 bytes

1200 bytes 1400 bytes

1472 bytes

UKLight using MkV terminals

The GÉANT2 Launch June 2005

Jodrell BankUK

DwingelooDWDM link

MedicinaItaly Torun

Poland

e-VLBI at the GÉANT2 Launch Jun 2005

UDP Performance: 3 Flows on GÉANT• Throughput: 5 Hour run 1500 byte MTU

• Jodrell: JIVE2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s

• Medicina (Bologna): JIVE 800 MHz PIII – Mk5 (623) 1.2 GHz PIII 330 Mbit/s limited by sending PC

• Torun: JIVE 2.4 GHz dual Xeon – Mk5 (575) 1.2 GHz PIII

245-325 Mbit/s limited by security policing

(>600Mbit/s 20 Mbit/s) ?

• Throughput: 50 min period• Period is ~17 min

BW 14Jun05

0

200

400

600

800

1000

0 500 1000 1500 2000Time 10s steps

Rec

v w

ire ra

te M

bit/s

JodrellMedicinaTorun

BW 14Jun05

0

200

400

600

800

1000

200 250 300 350 400 450 500Time 10s steps

Rec

v w

ire ra

te M

bit/s

JodrellMedicinaTorun

18 Hour Flows on UKLightJodrell – JIVE, 26 June 2005

• Throughput:• Jodrell: JIVE

2.4 GHz dual Xeon – 2.4 GHz dual Xeon

960-980 Mbit/s

• Traffic through SURFnet

• Packet Loss– Only 3 groups with 10-150 lost

packets each– No packets lost the rest of the

time

• Packet re-ordering– None

man03-jivegig1_26Jun05

0

200

400

600

800

1000

0 1000 2000 3000 4000 5000 6000 7000

Time 10s steps

Rec

v w

ire r

ate

Mbi

t/s

w10

man03-jivegig1_26Jun05

900910920930940950

960970980990

1000

5000 5050 5100 5150 5200

Time 10s

Recv w

ire r

ate

Mbit/s w10

man03-jivegig1_26Jun05

1

10

100

1000

0 1000 2000 3000 4000 5000 6000 7000

Time 10s steps

Packet

Loss

w10

Conclusion and Future directions• eVLBI needs to get to rates of 512 Mpbs – not there yet.

• Rates of 256 Mpbs on production networks possible now using TCP/IP in good conditions, but not higher rates.

• G2 launch showed continuous long term flows as in VLBI have different characteristics to those in short term tests (Iperf, UDPmon) : more investigation needed – collaborating with DANTE and SURFnet

• Packet loss is a killer for TCP – new protocols needed (ESLEA) - VSI-E standard

• The End Host Problem– The performance of Motherboards, NICs, RAID controllers and

Disks matter: Mk V units (Conduant)– MkV-B 2 Gbps data recording in 2 yrs?

• Next generation correlator 10 Gbps in 5-10 yrs

Thanks to

• Paul Burgess JBO• Steve Parsley, Arpad Szomoru, Cormac Reynolds JIVE• Nicola Pezzi, John Graham, Colin Greenwood, Peter

Clarke ESLEA• EVN observatories staff