eLBA – Current Developments

Chris Phillips

eVLBI Project Scientist

16 June 2008

ATCA

MopraParkes

Hobart

Ceduna

Tidbinbilla

Sydney

ASKAP

The LBA

LBADR – LBA Data Recorder

• Cousin of MRO/EVN-PC• Commodity PC with VSIB input card• Primarily record onto Apple Xserve RAID

• Control software highly modified from original MRO

• Mark5b emulation mode• eVLBI with TCP or UDP• Very flexible

• Data written to normal Linux filesystem• Realtime sampler statistics• Flexible realtime fringe checking

CSIRO. eVLBI-Aus

LBA status

• Using DiFX software correlator for all LBA operations

• All disk correlation has been done at Swinburne University of Technology (Melbourne)

• eVLBI correlated at Parkes• Disk correlation will transfer to Curtin University (Perth) by August 2008

CSIRO. eVLBI-Aus

DiFX

• All LBA correlation now runs on DiFX• Distributed FX

• MPI parallelization on Beowulf style cluster• Written by Adam Deller at Swinburne University of Technology

• Active development also from Walter Brisken• Available free of charge for scientific research

• Supports LBADR, Mark5a, Mark5b, Mark5c• Crucial part of VLBA sensitivity upgrade• Supports mixed mode correlation• Easy to add support for new formats

CSIRO. eVLBI-Aus

DiFX: Architecture

• Based on MPI• Written in C++• Uses IPP libraries extensively

• 2 types of processes• I/O processes (DataStream)• Compute processes (Core)

• Incoming data stream is time divided and copied round robin from DataStream to Core

• Asynchronous sends mitigate swarming effect

• Supports disk and eVLBI operation• TCP & UDP, filesystem and native Mark5

CSIRO. eVLBI-Aus

APSR Cluster

• ATNF Parkes Swinburne Recorder• Backend for Pulsar processing at Parkes

• 18 node Dell cluster• Dual processor quad core

• VLBI front end I/O cluster PAMHELA• 5 node, dual processor dual core

• 4 (Intel) NICs for network flexibility• Job control from PAMHELA, APSR used as slave

CSIRO. eVLBI-Aus

Image Credit: Shaun Amy

PAMHELA APSRT

eles

cope

Con

nect

ions

Case Study: How not to implement a cluster

• PAMHELA running 32bit Debian Etch• APSR running 64bit CentOS• No shared filesystem • MPI/DiFX does not like mixed 32 and 64bit binaries

• DataStream sending 4 times more data than receiving

• Many issues getting MPI• Mpich would not scale to use full cluster• Openmpi seemed to get confused by multiple

interfaces

CSIRO. eVLBI-Aus

ATNF Observatory Network

• Provided by AARNet• Pair of dark fiber, carrying multiple 10 Gbps SDH

• 2 x 1 Gbps connection to each observatory• Used for commodity traffic (web, email, ftp, ssh, observing etc) and eVLBI

• Each link effectively limited to 512 Mbps for eVLBI traffic

• Sydney to observatory can be run at line rate using bog standard PCs

• 989 Mbps user data (evlbi software)• 20 Mbps typical for scp

CSIRO. eVLBI-Aus

ATCA

Mopra

Parkes

Sydney

To Swinburne

To Seattle 10 Gbps

10 Gbps AARNET3To Perth

Current eVLBI status

• eVLBI being offered via normal “call for proposals” and actively encouraged

• 3x512 Mbps ATCA-Parkes-Mopra• 2x1 Gbps ATCA-Parkes (or Parkes-Mopra)

• Implemented as 2x 2x512 Mbps

• Exclusively layer 2 • Hand crafted traffic engineering for “routing”,

using multiple VLANS

• Connection to Hobart problematic• Effectively single 155 Mbps shared IP network

• Cannot sustain 64 Mbps TCP reliably

CSIRO. eVLBI-Aus

http://noc.atnf.csiro.au/

http://noc.atnf.csiro.au

Network Upgrades

• 1 Gbps link to Swinburne imminent…• Multi-gigabit access to 10 Gbps AARNet3 backbone in progress

• Implementing VPLS “Layer2 over Layer3”• eVLBI to Curtin• New cluster for ATCA

• 3x1 Gbps using distributed approach

• Still waiting on upgraded bandwidth to Hobart

CSIRO. eVLBI-Aus

EXPRES-Oz

• AARNet and ATNF Partners on EXPReS project

• Demonstrate realtime eVLBI from Australia to JIVE

• AARNet provided 1 Gbps light paths from Parkes, Mopra and ATCA to JIVE

• CENIC, Pacific Wave, CANARIE, SurfNet

• Science observation of SN1987a at 1.6 GHz• 11hr observation at 512 Mbps sustained• Using Mark5b UDP• 340msec RTT

CSIRO. eVLBI-Aus

Image Credit: Paul Boven,

Image created by Paul BovenSatellite image: Blue Marble Next Generation, courtesy of NASA Visibible Earth

Contours ATCA 9 GHz super resolved image (0.4” FWHM)

e-APT Demo

• Connecting Shanghai and Kashima to Parkes at 512 Mbps

• AARNet have provisioned 3x 622 Mbps circuits

• Mixture of SDH and layer 2/3• AARNet, CENIC, JGN2, CSTNet, Pacific Wave,

HKOEP• Dedicated Mopra-Parkes link• Presented as gigabit Ethernet

CSIRO. eVLBI-Aus

e-APT Demo: Implementation

• ATNF telescopes using LBADR data format and TCP

• Kashima and Shanghai using JIVE UDP format

• Raw Mark5 data stream with 64bit sequence #

• Shanghai using jive5a control on Mark5a• Kashima written realtime Mark5b converter and also generate JIVE UDP packets

• 1500 MTU used exclusively

CSIRO. eVLBI-Aus

Image created by Paul BovenSatellite image: Blue Marble Next Generation, courtesy of NASA Visibible Earth

CSIRO. eVLBI-Aus

Internet2: Wave of the Future

• Winners of inaugural award!• 10 Gbps connection on Internet2 (continental USA) for 1 year

• Possible usage:• eVLBI to Haystack• eVLBI to JIVE

• Details still being worked through

CSIRO. eVLBI-Aus

Contact UsPhone: 1300 363 400 or +61 3 9545 2176

Email: enquiries@csiro.au Web: www.csiro.au

Thank you

ATNFChris PhillipseVLBI Project Scientist

Phone: +61 2 93724608Email: Chris.Phillips@csiro.auWeb: www.atnf.csiro.au/vlbi