AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick

AstroGrid: The UK’s Virtual Observatory

Dr Dugan Witherick – Astrophysics Group, UCL

Wednesday 5th December 2007The University of Warwick

AstroGrid – The UK’s Virtual Observatory

Outline of this Seminar

• Describe the different types and volume of data that astronomers use

• Discuss the concept of Virtual Observatories and how they might solve some of these problems

• Introduce AstroGrid as the UK’s Virtual Observatory


The Types and Volume of Data in Astronomy


Ground Based Observatories

Copyright RASC Calgary Centre


Space Observatories

Copyright NASA

Copyright ESA

Copyright ESA/D. Ducros


Data Products Provided by Observatories

For many years astronomers have been provided the raw data products and were required to reduce the data to useable forms themselves. Increasingly, images and spectra are provided in a reduced form as they are reduced automatically through a pipeline.


Astronomical Surveys

• Sloan Digital Sky Survey or SDSS– Aim to map 25% of the sky– Image (through 5 coloured filters) around 100 million

objects– Take spectra of around 1 million objects

• Dark Energy Survey or DES– Plan to collect data on 300 million galaxies– 520 Megapixel camera installed on the existing 4-metre

Blanco Telescope in Chile– Survey to begin in 2009


How many bytes?• Multimission Archive

– Nearly 50 Tb, over half is from the HST (Aug 2007)– Approx. 24 Tb served from Jan-Aug 2007

• European Southern Observatory Archive– Approx. 65 Tb (Aug 2007)– Over 12 Tb (public data) served every year

• Sloan Digitized Sky Survey– 40 Tb when complete

• Visible and Infra-red Survey Telescope for Astronomy (VISTA)– 315 Gb per typical night, est. 150 Tb per year


How many astronomers are accessing these data?

• American Astronomical Society– Approximately 6500 members (source AAS

Nov 2007)

• Royal Astronomical Society– Approximately 3000 members (source RAS

Nov 2007)


Accessing Data

• Digital Audio Tape (DAT) and Linear-Tape Open (LTO)

• CDs, DVDs and increasingly Blu-Ray and HD DVD

• USB Flash Drives/Sticks

• Online– Common Gateway Interface (CGI)– Web Services– HTML Forms


Online Access - Multimission Archive


Online Archive - ESO Archive


Online Access - SDSS Skyserver


Virtual Observatories and how they can help


What are Virtual Observatories (VOs)?

“The VO vision can be summed up as the desire to make all archives speak the same language – all searchable and analyzable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.”

September 2003

Professor Andy Lawrence FRSEInstitute for AstronomyRoyal Observatory Edinburgh


The Virtual Observatory Aim

• The astronomer should be able to easily search all astronomical data worldwide from anywhere– A single user interface which can find/search all

available data

• It should be possible to view and analyse the data using a common set of tools– The returned data products should use common

formats


Creating a Virtual Observatory - An International Effort

The International Virtual Observatory Alliance

"facilitate the international coordination andcollaboration necessary for the development anddeployment of the tools, systems and organizationalstructures necessary to enable the internationalutilization of astronomical archives as an integrated

and interoperating virtual observatory."


As a result of the IVOA we have

• A common Resource Metadata

• A common data querying language– Astronomical Data Query Language (ADQL)– Based on Structured Query Language (SQL)

• Common table format– Virtual Observatory Table (.vot)


Identifying these resources

• International Virtual Observatory Resource Name or IVORN– Very similar to a Uniform Resource Location (URL)– Ivo://community.authority.id/resource/etc…

• Example 1 - ivo://nasa.heasarc/hstpaec– HST planned and archived exposures catalogue– Provided by NASA’s High Energy Astrophysics Science

Archive Research Center

• Example 2 - ivo://wfau.roe.ac.uk/sdssdr5-dsa– SDSS Data Release 5– Provided by Wide Field Astronomy Unit at the Royal

Observatory, Edinburgh


AstroGrid


What is AstroGrid?

AstroGrid is a UK government funded, open source project

designed to create a working Virtual Observatory (VO) for UK

and International astronomers. Working closely with other VO

projects worldwide through the International Virtual

Observatory Alliance (IVOA), internationally recognised

interface standards are emerging which promote true

scientifc interoperability of astronomical data and processing

resources worldwide. AstroGrid is a leading member of this

community.


Who’s Involved?

And others………

http://www.astrogrid.org/


How does it all work?

• Server side applications– Dataset Access (DSA)– Common Execution Architecture (CEA)– Community and VOSpace– Registry

• Client side application - VO Desktop– Graphical User Interface– Scripted access to the VO– Client-side workflow (Taverna)


Dataset Access - the DSA

• Provides access to catalogues

• Server application connects to Relational Database e.g. MySql, Postgres

• IVOA cone interface (positional search)

• ADQL query interface – More complex queries, similar to SQL


Common Execution Architecture - the CEA

• Provides a method for computation code to be executed– Any command-line application, although may need to

be “wrapped” in additional code

– HTTP Post/Get applications

• Limited success with executing jobs on the National Grid Service

• May look to running jobs on the UCL Condor Pool


Community and VOSpace

• Community– Used to give users an identity in VOSpace– A method of authentication for running queries/jobs

• VOSpace– Personal file storage– Attached to community identity– Can execute jobs and save results online– Can access data from any location– ivo://uk.ac.ucl.star/DuganWitherick#atable.vot


The Registry

• Perhaps most important server application• Allows astronomer or service to be able to locate,

get details of, and make use of, any resource located anywhere in the IVO space

• Two types:– Full registry - searches (or harvests) other registries

and maintains the entries in a list which is searchable by users or other services

– Publishing registry - tells full registries what is available under that authority


How everything communicates - an example

Client

Registry

DSA Server

CEA Server

VO Space

Invoke

Monitor

Notification

Read Results

Resolve

Read Inputs

Write Results


Client Side Application - The VO Desktop

• Application developed in Java• Currently in alpha form but beta

released end of month• User is presented with a single

button “Start”• When running, provides XML-RPC

interface to the Virtual Observatory


The All-VOScope


PLatform for Astronomical Tool InterConnection - PLASTIC• Communication protocol for client side virtual

observatory tool• Essentially allows tools to transmit and receive

data and instructions from other tools• Tools can instruct each other (e.g.):

– To load virtual observatory tables– Highlight subsets of rows– Load an image of a particular area of sky

• For more information visit http://plastic.sourceforge.net/


VO Explorer


VO Explorer


The Task Runner


Client-side Science Scripting

• Python - easy to learn high level programming language

import sys, xmlrpclib, os, time

prefix = file(os.path.expanduser("~/.astrogrid-desktop")).next().rstrip()

acr = xmlrpclib.Server(prefix+"xmlrpc")

ag = acr.astrogrid

userInfo = ag.community.getUserInformation()


Client Side Workflow - Taverna


Client Side Workflow - Taverna


AstroGrid is not…

• True Grid Computing– Could be loosely referred to as a data grid– Efforts are being made to get it working with true

computational grids

• A solution to the entire Virtual Observatory– Focus has been on infrastructure

• The only way for astronomers to access online data


But it….

• Is probably the most advanced of the Virtual Observatories

• Does provide an easy/heterogenous method for astronomers to access their data

• Will continue to be improved upon by the dedicated team of programmers as members of AstroGrid and other astronomers increasingly use the software and provide critical user feedback

Documents

AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick