Birmingham/Warwick AstroGrid Science Workshop Meeting Goals Silvia Dalla Oct 10-11, 2006

Birmingham/Warwick AstroGrid Science Workshop

Meeting Goals

Silvia Dalla

Oct 10-11, 2006

Workshop goals

• Aim to work with you, the early users of AstroGrid, to demonstrate its capabilities and the potential of the Virtual Observatory (VO)

• Introduce you to the elements of the system and how to do science with it

• Look at a science problem in your research area and use VO tools to tackle it

Workshop goals (contd)

• Collect your feedback

What you like/dislike

What needs to be improved

Which data you need

Which applications you want to see running in AG

Workshop Plan

• Today: background talks on AstroGrid and how the system works (am). Initial workgroup set up (pm)

• Tomorrow: workgroups work on science cases. Report and feedback Session (pm)

• Don’t miss the dinner (Today – 7:00 pm The Chamon)

What you can do with AstroGrid

AstroGrid Science Team

Silvia Dalla, Eduardo Gonzalez-Solares, Jonathan Tedds, Anita Richards and Nicholas Walton

Outline

1. Background: Virtual Observatories (VOs) and AstroGrid

2. Example of non-VO solution to a data retrieval and processing task: making a movie of solar images

3. The AstroGrid solution

4. Introduction to AstroGrid components

What is the VO?

www.encyclopedia.com: VO = collection of integrated astronomical data archives and software tools that utilize computer networks to create an environment in which research can be conducted

Thanks to Ohishi Masatoshi (NAOJ) for the pointer…

VOs worldwide

• Similar efforts now in 15 countries:– UK, USA, Canada, France,

Germany, Italy, Holland, Japan, Australia, India, China, Russia, Hungary, South Korea, ESO, Spain

• Active collaboration among projects– Standards, common demos– International VO roadmap being

developed– Regular telecons over 10

timezones

Formal collaboration:International Virtual Observatory Alliance (IVOA)

AstroGrid: the UK’s VO

• Consortium of several UK Universities, funded by PPARC to build a VO for the UK.

• Part of the Euro-VO www.euro-vo.org

• Astrogrid release 2006.3: www.astrogrid.org/launch

• Data discovery, access to data and applications.

AstroGrid Goals

• Enable science by:

– Improving the quality, easy, speed of on-line astronomy

– Making integration and comparison of data from diverse sources transparent

– Removing data access barriers to multiwavelength analysis

– Enabling access and manipulation of large datasets

The need for a VO

• Data volume doubles every year

• Increase of size and multiplex capabilities of new instruments– WFCAM: 100Gb/night (100Tb/yr)– VISTA: 300Gb/night (300Tb/yr)– e-MERLIN/ALMA: 1Tb/day– SDO: 1 Tb/day

• We will have Petabytes of data by 2010

The need for a VO

• Hubble UDF

• Million second exposure

• 6000x6000 pix

• 11.5 sq. arcmin

• 10,000 galaxies

The need for a VO

• SWIRE ELAIS N1

• 9 sq. degrees (~3000 UDF)

• (moon ~0.2 sq. deg.)

• ~ 600,000 objects

The need for a VO

• Solar Dynamics Observer

• Launch 2008

• >1 TB/day

• Over 5 years: 1.8 PB

Making a solar movie – non VO

Web interface to database of SOHO/EIT observations

Making a solar movie – non VO

Download imagesto local machine

Making a solar movie – non VO

IDL SolarSoft requiredto calibrate images andmake a movie. Routinesavailable, however eachuser rewrites code calling them to produce a movie.

Making a solar movie – non VO: weaknesses

• If user wishes to make a movie for a different time period, the above steps need to be repeated – by hand

• Similarly for astronomy data if one is interested in many objects

• The entire archive is not ‘visible’ to the user – only the downloaded subset can be processed

• Scripting eg with Python, Perl etc is possible though code is different for different archives – not easy for the ‘general’ user / for many datasets. This limits feasibility of multi-wavelength, multi-instrument work

Making a solar movie – with AG

Solar Movie Maker science workflow. Other science workflows available:Redshift Maker, Colour Cutter, Cone Search, SWIRE images

Making a solar movie – with AG: under the hood

• Access to database of observations – via AG DSA (DataSet Access) software

• Requests are sent using ADQL (Astronomy Data Query Language), similar to SQL

Database of observations(at archive)

DSAAG Workbench/workflow engine

Input: ADQL query

Output: table of observationssatisfying the query, in VOTable format

Making a solar movie – with AG: under the hood: CEA

CEAAG workbench/workflow engine

Input: CEA applicationinput parameters

Output: whatever the outputof the application is, delivered to user’s Myspace

• Capability to call a set of processing routines that do operations on data, or a model (apps in any programming language can be wrapped) – CEA Application

Application. Eg software that

processes data, model etc

(on apps server)

Making a solar movie – with AG: under the hood: Workflow

• Workflow capability so that queries to datasets and calls to applications can be managed

• Jobs are run remotely and asynchronously

• Queries and workflows can be re-used and shared

• AstroGrid is currently the only VO project with a workflow capability

AstroGrid: Workbench

• Workbench gives access to AG services

• Data discovery: AstroScope and HelioScope

• Task Launcher – send a query to a database or launch a application

AstroGrid: Workbench- contd

• Run a ready made science workflow

• Build your own workflow

• View your files in MySpace

AstroGrid: MySpace

• Virtual disk space where you can store results, temporary files, and new things like query files and workflow files, so you can adjust and re-run jobs on a later day.

• Visible from any computer.

Finding Information : the Registry

• How do you find the data you require?

• How do you decide which resource (data, application, information, disk, …) to use?

• The registries are the yellow pages for astronomical resources

• All VO registries harvesting each other: thus querying any one returns full list of globally held resources.

VOTable: an interchange format

• XML standard for the interchange of data represented as a set of tables

• A table is an unordered set of rows, as specified in the table metadata

• Each row is a sequence of table cells, each of them containing a primitive data type or an array of such primitives

• It bridges two ways to express structured data: XML and FITS

• Use of Universal Column Descriptors (UCDs) to express the content of each parameter.

<?xml version="1.0"?><VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/VOTable/v1.1"> <COOSYS ID="J2000" equinox="J2000." epoch="J2000." system="eq_FK5"/> <RESOURCE name="myFavouriteGalaxies"> <TABLE name="results"> <DESCRIPTION>Velocities and Distance estimations</DESCRIPTION> <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel" unit="m" value="3.6"/> <FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Name" ID="col3" ucd="meta.id;meta.main" datatype="char" arraysize="8*"/> <FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int" width="5" unit="km/s"/> <FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc" datatype="int" width="3" unit="km/s"/> <FIELD name="R" ID="col6" ucd="phys.distance" datatype="float" width="4" precision="1" unit="Mpc"> <DESCRIPTION>Distance of Galaxy, assuming H=75km/s/Mpc</DESCRIPTION> </FIELD> <DATA> <TABLEDATA> <TR> <TD>010.68</TD><TD>+41.27</TD><TD>N 224</TD><TD>-297</TD><TD>5</TD><TD>0.7</TD> </TR> <TR> <TD>287.43</TD><TD>-63.85</TD><TD>N 6744</TD><TD>839</TD><TD>6</TD><TD>10.4</TD> </TR> <TR> <TD>023.48</TD><TD>+30.66</TD><TD>N 598</TD><TD>-182</TD><TD>3</TD><TD>0.7</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE></VOTABLE>

Table Metadata

Table data(XML, FITS)

VOTableExample

Visualising results: Topcat

Visualising results: Topcat

Astrogrid Help Pages

Linked from workshop pages

Summary

• Workshop is an opportunity to see AG in action

• Current capability can be of use to a diverse range of science problems

• New cases – perhaps emerging from this workshop – may demand new science services: input is welcome from you!