EURO-VO DCA GRID and the VO

EURO-VO: GRID and VO

Lofar Information System DesignOmegaCEN

Kapteyn Institute TARGET- Computing Center

University Groningen

Garching, 10 April 2008

Lofar Information System DesignOmegaCEN

Kapteyn Institute TARGET- Computing Center

University Groningen

Garching, 10 April 2008

Edwin A. ValentijnEdwin A. Valentijn

EURO-VO DCA GRID and the VOEURO-VO DCA

GRID and the VO


LOFAR @ (20-)30-80 MHz and 120-240 MHz

LOFAR @ (20-)30-80 MHz and 120-240 MHz

- EoR

- transients

- Survey

- astroparticles

- EoR

- transients

- Survey

- astroparticles



E-Lofar: from experiment to observatory

E-Lofar: from experiment to observatory


LOFAR in shortLOFAR in short• Large distributed radio telescope

– In NL 2008: initial roll out 20 stations – In NL 2009: 18 central (24 Gbps) + 18 remote (2 Gbps)

phased array antenna stations each ~4 soccerfields size– Full scale aperture synthesis array, extends 100 km

• Two main bands– High Band ~ 7,500 tiles 120 – 240 MHz – Low Band ~ 7,500 dipoles 20 – 80 MHz

• Digital Radio 40 MHz processing bandwidth– Extreme tunability in time/space/frequency– Large instantaneous sky coverage

• Fibre network, Software Correlator• Run as a broad common-user observatory

– Making the transition now from “project” to “observatory”

• Large distributed radio telescope– In NL 2008: initial roll out 20 stations – In NL 2009: 18 central (24 Gbps) + 18 remote (2 Gbps)

phased array antenna stations each ~4 soccerfields size– Full scale aperture synthesis array, extends 100 km

• Two main bands– High Band ~ 7,500 tiles 120 – 240 MHz – Low Band ~ 7,500 dipoles 20 – 80 MHz

• Digital Radio 40 MHz processing bandwidth– Extreme tunability in time/space/frequency– Large instantaneous sky coverage

• Fibre network, Software Correlator• Run as a broad common-user observatory

– Making the transition now from “project” to “observatory”

E-LOFAR ParticipantsE-LOFAR Participants

• GLOW:-- Effelsberg station delivered-- Garching, Potsdam, Tautenburg stations ordered--Jülich station federal funding proposal submitted

• LOFAR-UK:– Chilbolton, Cambridge, Jodrell, Edinburgh– 1 station funded, site to be chosen--3 more stations

• Sweden:– Onsala station funded

• + Poland, Italy, Austria, Bulgaria, Ireland, France

• GLOW:-- Effelsberg station delivered-- Garching, Potsdam, Tautenburg stations ordered--Jülich station federal funding proposal submitted

• LOFAR-UK:– Chilbolton, Cambridge, Jodrell, Edinburgh– 1 station funded, site to be chosen--3 more stations

• Sweden:– Onsala station funded

• + Poland, Italy, Austria, Bulgaria, Ireland, France


Wires in the fieldWires in the field



Station based processing

Station based processing

–Receiver–Filter–Beamformer

– Bu

ffer


– Bu

ffer


– Bu

ffer


– Bu

ffer

–10 GbE

–1 GbE

–Used for 1 beam & station cross correlation

Input data rate: ~ 460 Gbps

Output data rate: ~ 2 Gbps

Processing capacity: ~ 1.5 Tmul/s

Storage capacity: 96 Gbyte

Lofar: Data StreamLofar: Data Stream

Gb

E s

witc

h

Gb

Esw

itch

Gb

Es

witc

hG

bEsw

itch

Gb

Esw

itch

Gb

Esw

itch

Auxiliary processingsection

Sto

rag

e sy

ste

m

Off-line processing/user software

Off-lineStorageOn-Line

Long term Archiving& Data Export

Gb

Esw

itch

On-Line pipeline processingBlue Gene/L

WAN



Lofar StorageLofar Storage

Lofar Long term (>2weeks)archive and user enabling

ADD

Lofar Long term (>2weeks)archive and user enabling

ADD


STREAM of MetaDataSTREAM of MetaData

• Design: data driven vs. compute driven• from the field to central processor• Distributed – p2p / federated

– Communities– Dataservers – own + EGEE GRID– Compute servers – own+ EGEE GRID

• AstroWISE• EURO - VO

• Design: data driven vs. compute driven• from the field to central processor• Distributed – p2p / federated

– Communities– Dataservers – own + EGEE GRID– Compute servers – own+ EGEE GRID

• AstroWISE• EURO - VO



LoWISE designLoWISE design

Lofar-WISE: Data StorageLofar-WISE: Data Storage

–Groningen

–SARA AmsterdamEURO-VO: GRID and VO

• IBM – GPFS filesystem• Connect to AstroWISE

– Storage table

• IBM – GPFS filesystem• Connect to AstroWISE

– Storage table


LoWise- conceptsLoWise- concepts• All Tier 1s on an equal footing• Centrally operated GPFS distributed filesystem• File retrieval

– Db viewer (web)– Prompt (db query)– Pathname – data server– gLite – Grid ftp

• Single distributed db- with all Meta and source data– Valid --Quality -- ready for publication flags

• Visibility of data– Owner –Project –Archive --EURO-VO

• Authentication / Assigned resources– AstroWISE Oracle system

• All Tier 1s on an equal footing• Centrally operated GPFS distributed filesystem• File retrieval

– Db viewer (web)– Prompt (db query)– Pathname – data server– gLite – Grid ftp

• Single distributed db- with all Meta and source data– Valid --Quality -- ready for publication flags

• Visibility of data– Owner –Project –Archive --EURO-VO

• Authentication / Assigned resources– AstroWISE Oracle system


Processing - optionsProcessing - options

• Tier 0 – CP• Tier 1 – re-processing

– Local clusters– EGEE GRID

• Tier 2– On Tier 1– Local anarchy

• Tier 0 – CP• Tier 1 – re-processing

– Local clusters– EGEE GRID

• Tier 2– On Tier 1– Local anarchy



AstroWISE-UpAstroWISE-Up


–VST - Virtual Survey Telescope–VST - Virtual Survey Telescope


– www.astro-wise.org– www.astro-wise.org


Astro-WISE: ServicesAstro-WISE: Services


TARGET diagram TARGET diagram



QC - calibration scientist monitoring

QC - calibration scientist monitoring


Extreme data lineage Extreme data lineage

Astro-WISE: Grid Computing

Astro-WISE: Grid Computing

–selection of Target in Astro-Wise

–storage element defined a priori –EGEE computing EURO-VO: GRID and VO


LoWise and Euro-VOLoWise and Euro-VO



GRID- EuroVO –Awe visualization

GRID- EuroVO –Awe visualization


– AI – Cultural Heritage – AI – Cultural Heritage


http://omegate.astro.rug.nl:9001/Monk

http://omegate.astro.rug.nl:9001/Monk

Documents

EURO-VO DCA GRID and the VO