Goals of a New VLBI Data System

• Low cost

• Based primarily on unmodified COTS components

• Modular, easily upgradeable

• Robust operation, low maintenance cost

• Easy transportability

• Conformance to VLBI Standard Interface specification (VSI)

• Compatibility with existing VLBI systems

• Flexibility to support e-VLBI

• Minimum of 1 Gbps data rate

• 24-hour unattended operation at 1 Gbps

-2

-1

0

1

2

3

4

5

6

7

1980 1985 1990 1995 2000 2005 2010

Year

log

($/G

B)

Disc Drive Street Prices

Computer Tape Media

1998 IBMDisc Projection

Mark IV/VLBA/K4 Media

S2, S3 Media

Disc industryProjections

1998 NSICDisc Projection

LTO Media Projection

c

Tape vs. Disc Price Comparison

Advantages of Magnetic Discs

• Readily available consumer product

• Standard electrical interface

• Technology improvements independent of electrical interface

• Self contained; don’t have to buy expensive tape drives

• Rapid random access to any data

• Essentially instant synchronization on playback to correlator (no media-wasting early starts needed)

• No headstacks to wear out or replace – ever!Virtually maintenance free.

Mark 5 VLBI Demonstration System – March 2001

3 months start to finish!

Disk-based VLBI Data Systems • Europe – PC/EVN

– Developed primarily by Metsahovi group

– Supports partial implementation of VSI-H specification

– Uses 5 PC’s to support recording and playback at 1 Gbps

– Has been used for several European experiments

• Japan – K5– Developed by CRL

– Uses 4 PC’s to support recording and playback at 512 Mbps

– Several experimental units deployed in Japan

• Australia – Swinburne group– Based on Apple Xserve RAID array interface

– Designed for pulsar work; plan to use for VLBI at 128 Mbps

• U.S. – Mark 5A– Developed at Haystack Observatory with support of international consortium (BKG,

EVN, KVN,MPI, NASA, NRAO & USNO)– 1 Gbps in single chassis

– Chosen by IVS and EVN to replace Mark 4 and VLBA tape systems

– ~30 systems now deployed; expect 60-70 by end 2003

Mark 5A VLBI Data System

Mark 5A Characteristics

• 1 Gbps recording and playback

• Direct ‘plug-compatible’ replacement for VLBA or Mark 4 tape drive

• ‘8-pack’ disk modules for ease of disk handling

• Packaged in single 5U chassis

• Supports e-VLBI

• Supported by NASA Field System

• Supported by Mark 4 correlators (JIVE, MPI, USNO, Haystack)

• Available commercially from Conduant Corp for ~$16K(compared to Mark 4/VLBA tape drive for ~$150K)

• Plan intercompatible support for ATA serial disks

• Plan inexpensive ‘expansion chassis’

• Mark 5B (VSI-compatible) system under development;will allow VLBA to support 1 Gbps; Mark 5A can be upgraded with interface card replacement

Mark 5A Experience

• Daily UT1 ‘Intensive’ observations Wettzell-Hawaii have been exclusively Mark 5 for ~10 months – almost no problems

• 15-day geodesy experiment in Oct 02 successfully recorded and processed from Westford antenna; processed one disk set with two missing disks!

• Several astronomy experiments, including recent mm-VLBI experiment have successfully used Mark 5A

• Have been some teething problems, particularly learning to properly manage disks, but believed under control

Disk-Media Status

• Hard disk price vs capacity/performance will continue to drop rapidly- Now ~$1.00/GB, expected to drop to ~$0.50/GB by ~2005

(Mark 4/VLBA tape is ~$2.00/GB)

• 200 GB disks now available – 27 hours @ 256 Mbps unattended

• (comparable to ~5 VLBA tapes)

• 320 GB disks expected soon – 22 hours @ 512 Mbps unattended(comparable to ~9 VLBA tapes)

• 700 GB disks expected ~2005 – 24 hours @ 1 Gbps unattended(comparable to ~19 VLBA tapes)

Strawman Plan for VLBA36 Mark 5 systems - ~$600K

Rate(Mbps)

Total (TB)

Disk($/GB)

Other($/GB)

~Cost

2003 256 825 1 0.15 1.0M$

2004 512 825 0.75 0.15 +0.8M$

2005 1024 1750 0.5 0.10 +1.0M$

Total 3300 2.8M$

Storage requirements:24 hours @ 1 Gbps = 11TB300 station-days @ 1 Gbps = 3300TB

VLBA correlator:• Should be able to keep up with 512 Mbps for up to 20 stations• Can process 1024 Mbps for up to 10 stations

Total cost over 3 years ~3.4M$

Media costs

e-VLBI: The Next Step

• Bandwidth growth potential for higher sensitivity– VLBI sensitivity (SNR) proportional to square root of Bandwidth

resulting in a large increase in number of observable objects(only alternative is bigger antennas – hugely expensive)

– e-VLBI bandwidth potential growth far exceeds recording capability(10 Gbps per station is possible today; 100 Gbps in near future!)

• Rapid processing turnaround– Astronomy

• Ability to study transient phenomena with feedback to steer observations

– Geodesy• Higher-precision measurements for geophysical investigations

• Better Earth-orientation predictions, particularly UT1, important for military and civilian navigation

Practical Advantages of ‘e-VLBI’

• Increased Reliability– remove recording equipment out of field

– remote performance monitor & control capability in near real-time

• Lower Cost– Automated Operation Possible

• eliminates manual handling and shipping of storage media

– Real-time or near-real-time Processing• forestalls growth of storage-capacity requirements with bandwidth growth

– Reduce and perhaps eliminate expensive recording-media pool(millions of $’s!)

• Avoid unexpected media-shipping interruptions and losses

Japan has led the way in e-VLBI!

• 1998-2001: Keystone project– 4 antennas around Tokyo area connected in real-time at 256 Mbps

• 2000: Gbps ftp e-VLBI demonstration

• 2001: ATM-based 1 Gbps real-time VLBI

• 2002: 2 Gbps VLBI demonstration

Connecting the Global VLBI Array in the New Era of High-Speed Networks

With the world increasingly wired for high-speed data communications, the prospects for routine global electronic transmission of VLBI data (dubbed ‘e-VLBI’) become brighter every day. Not only will e-VLBI help eliminate costly and complex recording equipment, but it should eventually lead to data rates and volumes unattainable by traditional recording equipment. This will lead to improved sensitivity, allowing new science to be explored at lower costs.

This international two-day meeting at MIT Haystack Observatory is being organized to explore the current state of high-speed astronomy data transmission, concentrating on e-VLBI, but recognizing the synergy with other geodesy/astronomy applications requiring real-time or near-real-time high-speed data transmissions. Among the topics to be discussed:

8-9 April 2002

MIT Haystack Observatory, Westford, MA USA

Sponsored by:

International VLBI Service (IVS)Global VLBI Working Group (GVWG)

MIT Haystack Observatory

Registration is required for attendance Registration closes 10 March 2002

Registration is limited to 80, so register early!For complete information see the meeting web site at:

http://web.haystack.mit.edu/e-vlbi/meeting.html

Important Dates

• 3 December 2001- First announcement• 1 February 2002 – Second announcement• 24 February 2002 – Deadline for abstract submission• 1 March 2002 – Final announcement• 10 March 2002 – Deadline for registration• 22 March 2002 – Deadline for hotel reservations

• International networking facilities – now and future• User requirements for high-speed networking• Reports on current and projected e-VLBI and related efforts• Networking protocols for real-time data transmission• Public vs. dedicated networks• Establishing international standards for e-VLBI data transfer

Program Committee

• Alan Whitney – Haystack (USA)• Yasuhiro Koyama – CRL (Japan)• Steve Parsley – JIVE (Europe)• Jon Romney – NRAO (USA)

e-VLBI Development at Haystack Observatory

• Phase 1: Develop eVLBI-compatible data system– Mark 5 system

• Phase 2: Demonstrate 1 Gbps e-VLBI using Bossnet(w/ DARPA and NASA support)

– ~700km link between Haystack Observatory and NASA/GSFC

– First e-VLBI experiment achieved ~788Mbps transfer rate

• Phase 3: Develop adaptive network protocol(newly awarded NSF grant to Haystack Observatory; collaboration with MIT Lab for Computer Science and MIT Lincoln Laboratory);

– New IP-based protocol tailored to operate in shared-network ‘background’ to efficiently using available bandwidth

– Demonstrate on national and international networks

• Phase 4: Extend e-VLBI to national and global VLBI community– ‘Last mile’ problem remains a serious challenge

Mark 5 VLBI Disk-Based Data System (Phase 1)

Haystack

Westford

NASA/GSFCUSNO

(correlator)

(correlator)

Bossnet 1 Gbps e-VLBI demonstration experiment(Phase 2)

Initial experiment

Future

Performance test results – Haystack/GGAO

Average sustained rate >900 Mbps over 10 hours

Correlation results of e-VLBI test observation

Mk4 Fringe Plot 4C39_25.pufxhs, 277-1815, EZWESTFORD - GGAO7108, fgroup X, pol RR

9

34.50.0e+00590.412

2.373-52.2

-0.007122

0.000000

0.005924

8932.99001.000

29922002:277

181500.00182500.00182000.00

2002:279:010613

2003:006:213921

09h27m 3.0139s+39Ê02'20.852"

Fringe quality

SNRPFDIntg.timeAmpPhaseSbdelay (us)

Mbdelay (us)

Fr. rate (Hz)

Ref freq (MHz)

AP (sec)

Exp.Exper #Yr:dayStartStopFRTCorr. date:

Fourfit date:

Position (J2000)

International e-VLBI experiments

• Westford, MA to Kashima, Japan - experiments in Oct 02 and Mar 03– Files exchanged over Abilene/GEMnet networks

• Nominal speed expected to be ~20 Mbps; best achieved so far ~11 Mbps

– Correlation on Mark 4 correlator at Haystack and PC Software correlator at Kashima; nominal fringes obtained

– Further experiments are scheduled; network tuning is in progress

• Kauai, Hawaii to Wettzell, Germany (in progress)– Daily experiments of ~100GB are ideal candidate for early e-VLBI

– Data will be transferred to Haystack Observatory for processing(OC-3 speeds are possible)

– Network links are now being brought up

New IP Protocols for e-VLBI (Phase 3)

• Based on observed usage statistics of networks such as Abilene, it is clear there is much unused capacity

• New protocols are being developed to utilize networks in ‘background’ mode for applications such as e-VLBI– Will ‘scavenge’ and use ‘secondary’ bandwidth– Will give priority to ‘normal’ users– Requires a new ‘end-point adaptive strategy’– Should substantially reduce cost of e-VLBI fiber bandwidth

• Work being carried out under NSF sponsorship by MIT Haystack Observatory in collaboration with MIT Laboratory for Computer Science and MIT Lincoln Laboratory– 3-year program; will demonstrate e-VLBI connections both nationally and

internationally

Typical bit-rate statistics on Abilene network

Conclusion: Average network usage is only a few % of capacity

Usage >20Mbps less than 1% of the time

100 500 Mbps

0.1

1.0

0.01

0.001

Typical distribution of heavy traffic on Abilene

Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes

secs200 400 1000

1.0

0.9

0.8

0.7

<10% of ‘bulk’ transfers exceed ~100 secs

Extend to national and global community (Phase 4)

• Many possibilities for international connections– Surfnet – U.S. to Europe at 2.5 Gbps

– IEEAF – U.S. to Europe at 10 Gbps

– TransPAC – U.S. to Japan at 655 Mbps (upgrade to 1.2 Gbps planned)

– GEMnet – currently ~20 Mbps, planning to upgrade to 2.2 Gbps

– Super-Sinet – 2.5 Gbps Japan-to-U.S.

– AMPATH – possible connections to telescopes in Chile and Brazil

• Not well covered– Australia

– Africa

– China

– Large parts of South America

But there is a significant problem – ‘Last-mile’ costs

• Most of the world’s telescopes are not well connected• Electronic and electro-optic costs are dropping rapidly

– GigE switch: 2001 - $15K 2003 - $1.2K– GigE transceivers 2001 - $750 2003 - $180– CWDM transceivers $400-800 for 50-100km reach!

• Direct fiber cost is relatively low– $60/fiber-km in 80-fiber bundle• If you can buy or lease existing fiber, there is no better time!• But – fiber installation cost is still tall pole

– Europe: >$20/m (or any populous wide-area)– U.S.: >$10m (in simplest desert environment)

• The upside: there is developing a lot of momentum and support from the greater networking community to get the job done!

Also desperately needed:– Modern digital filter banks to replace aging and obsolete analog BBC’s!

10 Gbps

0

UKUKUKUK

FRFRFRFR

CHCHCHCH

SESESESE

ITITITIT RORORORO

HRHRHRHR

EEEEEEEE LVLVLVLV LTLTLTLT

IEIEIEIENLNLNLNL

BEBEBEBE

DEDEDEDE

PLPLPLPL

CZCZCZCZ

HUHUHUHUATATATAT

SISISISI

SKSKSKSK

LULULULU

ESESESES

PTPTPTPT

GRGRGRGR

BGBGBGBG

CYCYCYCY ILILILIL622622 3434

3434

4545

622622

622622

155155155155

155155

155155

3434

3434

155155622622

155155 3434 4545

10 Gbps10 Gbps 2.5 Gbps2.5 Gbps

SE - PoP for Nordunet

GÉANT:The connectivity at 10 Gbps

622 Mbps +10 Gbps

Transoceanic donations to IEEAF (in red)

Conclusions• Modern PC and networking technology has led to a revolution in VLBI data technology• Transition to all disks will be rapid due to technical, economic and practical advantages• e-VLBI is riding an unprecedented wave of global network connectivity and networking

community enthusiasm• There is no better time to lease or buy installed fiber than now!• Gradual transition from disks to disk/e-VLBI to all e-VLBI is likely• 10-100 Gbps/antenna is technically possible with e-VLBI; can VLBI correlators keep up?

• Can e-VLBI survive the long-term networking costs?

• There is momentum gathering in the networking community to provide national and international ultra-high-speed networking as a critical ‘enabling infrastructure’ for U.S. and international science; the astronomy community needs to makes its voice heard loud and clear!

But big hurdles remain…..

EVN Future Vision