!"#$%&'&()(*"+,-)./("+01,(2"3-)"4.*5-"!1,)-6-20"

Online archive

Staging archive

ICRAR, Perth

Staging & processing

Fornax, Perth

Long-term archive

PBStore, Perth

MIT, USA

Mirrored archive

AARNet

UI

Research

VO

Web

Scripting

1/10Gbps

10Gbps

1 Gbps

10Gbps

10Gbps

~ 400 MB/s

Catalog VO-Table

VO-Table

Data distribution, placement, scheduling

Online processing

MRO 320Gb/s

**Volume Title**ASP Conference Series, Vol. **Volume Number****Author**c�**Copyright Year** Astronomical Society of the Pacific

The MWA Archive - A Multi-tiered Dataflow and Storage System

Chen Wu1, Andreas Wicenec1, Dave Pallot2, and Alessio Checcucci1

1ICRAR/University of Western Australia2ICRAR/Curtin University

Abstract. The Murchison Widefield Array (MWA) is a next-generation radio tele-scope, generating visibility data products continuously at about 400 MB/s. The entireMWA archive consists of dataflow and storage sub-systems distributed across threetiers.

1. MWA Data Products

The Murchison Widefield Array (MWA) is optimized for the observation of extremelywide fields at frequencies between 80-300 MHz. The complete array will provideabout 2000 m2 of collecting area at 150 MHz with 30.72 MHz instantaneous band-width at 40 kHz spectral resolution and a field of view between 200 and 2500 squaredegrees with a time resolution of 0.5 seconds. The initial MWA commissioning willinstall 32 antennae referred to as “tiles”. Each tile is a 4 by 4 grid of dual-linear-polarization, wide-beamwidth antenna elements. Each data sample is a complex num-ber representing the amplitude and phase of a signal at a particular point in time withina specific frequency channel. For each channel at each time step, the correlator carriesout (N × (N + 1)/2) × (2 × 2) = 2N(N + 1) pair-wise cross/auto-correlation, where Ndenotes the number of tiles, and 2 × 2 indicates that polarised samples of orthogonaldirections are cross-correlated. For example, under the N = 32-tile configuration dur-ing commissioning, each correlator produces 2, 112 visibilities per frequency channelper time step. Since a “visibility” is a complex number (2 × 32 bits = 8 Bytes), theaggregated data rate amounts to 2N(N + 1)× 8× 1/T ×M per second, where T denotesthe time resolution (i.e. 0.5 seconds). M is the total number of frequency channels:30.72 MHz bandwidth / 40 kHz spectral resolution = 768. Hence correlators under thefull 128-tile configuration will produce visibility data at 387 MB/s. This requires 24on-site GPU-based software correlators, each processing 32 channels, and producingfrequency-split visibilities at a rate of about 16 MB/s.

During the 32-Tile phase of the MWA commissioning, two GPU machines withsoftware correlators have been deployed at MRO. For the purpose of commissioning,the spectral resolution is set to 10kHz with a 1-second time resolution. Therefore, eachone of the two correlators shares 30.72 MHz / 10kHz / 2 = 1,536 frequency channels,generating visibilities at the rate of 24.75 MB/s. Visibilities are currently stored, alongwith several observation-specific metdata, as FITS image or binary table extensions. Inaddition to visibilities, it is envisaged that images produced by the real-time imagingpipeline (Ord et al. (2010)) running on those GPU boxes will also be archived online.

1

NG/AMS Servers

NG/AMS Servers

Data Capture (Online Processing – Tier 0)

DataCaptureMgr

DataHandler DataHandler

In-memory Buffer Staging Area

NGAS Client

Local NGAS Servers

Data Producer

Remote NGAS Servers

Wide Area Network

Data Network

•  Plugin-based •  Multi-threaded •  Throughput-oriented •  Fault-tolerant •  Admission control •  File handling

Archive DB

A Single Process

M&C Interface

Metadata Network

M&C Subsystem

HDD Ramdisk SSD

Web / Query / VO Interface

-200

0

200

400

600

800

1000

1200

1400

16 24 32 40 48 56 64 72

Tota

l arc

hive

thro

ughp

ut -

MB/

s

Simulated data rate per client - MB/s

Archiving simulation throughput

A - 1Gbps bandwidth for commissioningB - 12 clients / 4 servers on 6 / 2 Fornax nodes

C - aggregated data producing rate for 12 clientsD - 12 clients / 4 servers on 6 iDataplex / 1 SupermicroE - 12 clients / 2 servers on 6 iDataplex / 1 Supermicro

F - 24 clients / 4 servers on 6 / 1 Fornax nodesG - 24 clients / 4 servers on 24 / 2 Fornax nodesH - 24 clients / 4 servers on 12 / 2 Fornax nodes

I - 24 clients / 4 servers on 28 Fornax nodesJ - aggregated data producing rate for 24 clients

!"#$%&

'"()&'%*+,-&

.+/(0&.1"1%&2,(3%&

4",0&2(56&2,(3%&

7"8%&2,(3%&

!"#$%&!

!"#"$%&&'(($)"*'+$! ,('+$%&&'(($-".'+/!

!"#"$0&1'234'+!

0#5+"6'$78'+"+&1*!

'&(&$)&!

*+,-./&!

01/&!

21"3!

'&%+.&/&!

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

*4$(35&! !"#"$-4"&'9'/#$:/68/'!

662177$4#!

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

!"

#$$#

!%"&

&'&()%)!)")*)&)))+),)')(+%+!+"+*

;'('"+&1$<5&3(!

66'&89&)%!

Calibration

Imaging

Quality Control

Source Extraction

Calibration

Imaging

Quality Control

Source Extraction

Offline (re)processing on Fornax (Tier 1) - DIRP

! !

!"#$%&'()*+,-.(/#012,-0,3#-(45-#52-6

!""#!""$ !"%& !"%'

()*+,$ ()*+,#

!"%-!"%. !"%% !$""

7"82$(9":-+

;".<3,-(9":-+

=-5->"<.-$,(9":-+

!/01#

!/01$ !)11$ !)11#

73+,#-(9":-+

!)112 !)113

!)11& !)11' !)11. !)11-

(045!)6,47879, (+:9,19 /),+;)6

?@(9":-+

!)6,478<,

A-%:(9":-

B$C2$2D%$: EF(G,1-#$-,

+)$ +)#

;"<*(9":-+

EHFG,1-#$-,G&,-#$%>

!""#$%%&'()*+,-')."'(/.012/'(3%3415

7TB per node

Calibration

Imaging

Quality Control

Source Extraction

Data-Aware DIRP Job Scheduler

Long-Term Archive

Staging Archive

78("#$)9851-6":5*(;(%*"!)).0"<#:!="51"."6(>,'/(6().&-6").*5-",(%(19-?(@"/(6().&6/").A"*.,."9-6&6$-$1%0".,"B"CDE1F"78("(6&)("#:!".)985G("9-6151,1"-3"*.,.H-A".6*"1,-)./("1$I'101,(21"*51,)5I$,(*".9)-11",8)(("&()1F"""!,"75()"J@"<#4K"'"#$)9851-6"4.*5-"KI1()G.,-)0@":(1,()6"!$1,).%5.=@"1,.&-6"I(.2"*.,."51"?)-9(11(*"-6%56(@"?)-*$956/"LG515I5%5,0M"1,)(.21".,".6".//)(/.,(*").,("-3"NOJJ"#DE1F"78("G515I5%5,0"*.,."51"561,.6,%0"9.?,$)(*"56",8("-6%56(".)985G("3.95%5,0F""!,"75()"P"<Q(),8@"RJJ"S2"1-$,8"-3"#4K=@",8("1,./56/".)985G("56/(1,1"G515I5%5,0"1?%5,1"3)-2"75()"J"<,8)-$/8";I)('-?&9":!T",).613()=".6*"8.6*%(1",8(2"-G()",-"I-,8"%-6/',()2".)985G56/".6*"-U56("?)-9(1156/F"!"85().)9859.%"1,-)./("2.6./(2(6,"<V+#="101,(2"51"*(?%-0(*@"A8598"%(G()./(1"-$)")(1(.)98"56"*.,.".99(11"?.W()6".6.%0151",-"25/).,("*.,.".9)-11",8("1,-)./("85(.)980F"!"*.,.'56,(615G("CQX'9%$1,()"<Y-)6.>="51"$1(*"3-)"-U56("*.,."?)-9(1156/@"A8598".%1-"198(*$%(1"*.,."2-G(2(6,F""!,"75()"Z"<#[7@"X+!=@",8("25))-)(*".)985G("3.95%5,0"1$I19)5I(1",-"1?(95;9"*.,."?)-*$9,1"A5,8"75()"P@".6*"9-6&6$-$1%0"56/(1,1"$?*.,(*"*.,."1,(.21"-3")(%(G.6,"*.,.",0?(1"-6".")(/$%.)"I.151F"""

{chen.wu, andreas.wicenec}@uwa.edu.au

!"#"$%#&'(&)"#*)&+$