Science Overview Architectural Status ( k) Nicholas Walton IoA, Cambridge N A Walton: Science Overview: AstroGrid Collaboration

Science Overview Architectural Status

(http://www.astrogrid.ac.uk)

Nicholas WaltonIoA, Cambridge

N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p1 Printed: 12/12/01

Multi-wavelength showing the jet in M87: from top to bottom – Chandra X-ray, HST optical, Gemini mid-IR, VLA radio

credits: “NASA / Chandra X-ray Observatory / Herman Marshall (MIT)”, “NASA/HST/Eric Perlman (UMBC), “Gemini Observatory/OSCIR”, “VLA/NSF/Eric Perlman (UMBC)/Fang Zhou, Biretta (STScI)/F Owen (NRA)”

Drivers: The Sociology of Astronomy Continuing Collectivization

Facility class (common user) instruments Central development of supporting s/w (e.g. Iraf) Calibrated archives and access tools (e.g. IPAC) Information services (e.g. ADS, NED, astro-ph) Consortium projects (e.g. MACHO, SLOAN, VISTA)

Evolving Developments Inter-operable archives, joint queries (e.g. MAST) Data mining (exploration and analysis tools) Information discovery tools


Drivers: Technological Change I The Growth of Data

Significant increase in number and size of telescopes In the optical: ESO's 4x8-m VLT, Gemini's 2x8-m In the x-ray: XMM-Newton, Chandra In the mm: ALMA

Significant increase in size and multiplex capabilities of associated instrumentation and detectors, e.g.:

In the optical: VISTA will have a Gpixel array In the radio: e-Merlin with data rates of 320 Gbps will

generate


Drivers: Technological Change II The Growth of Data Archives

Many observatories hold multi-TB archives New initiatives set-up to support new observing

capabilities (e.g. TeraPix) In the optical, need to ingress all-sky survey's

Whole sky at 0.1 arcsec/pix is 100TB

Increasing Importance of Archival Data Time on expensive facilities (e.g. HST) only awarded

if the archive has been searched before hand This trend is continuing, driving observatory and user

demand for more accessible archival dataN A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p4 Printed: 12/12/01

Empowering Science Driven Observational Astronomy

Break down the 'wavelength' barriers e.g. Greater focus on science driven proposals

encouraging the use of data sets from across a wide range of wavelengths

Increase Coordination across countries e.g. The UK joins ESO: improved knowledge transfer

Increase Access e.g. East European countries may not be able to fund

a major new telescope but can contribute to, and access 'Virtual Observatories'


Linking the near and distant Universe Comparison of rest frame samples to study evolution of

galaxies Combination of UV, optical and IR datasets

Creating the 'Digital Sky' Temporal data measures motions in the Galactic centre,

probes the creation of our Galaxy The search for extra-solar planets

The planet-transit technique using federated survey data for millions of stars

Astronomical Drivers: Enabling New Science


SuperCOSMOS (UK: now till 2002) Based on Schmidt plates - Science database ~2TB

Sloan Digital Sky Survey (US: now till 2005) Dedicated CCD survey telescope: Science database ~10TB

UKIDSS (UK: from 2003) IR camera on 4-m UKIRT: Science database ~30TB

VISTA (UK: from 2005) IR camera on 4-m VISTA: Science database ~300TB

LSST (US: from 2007-8?) Dedicated ~8-m telescope: All sky/few nights:~5000TB/yr

Astronomical Drivers: New Era of Surveys


Investigating the progenitors of sources that show variability Dark matter revealed by microlensing events Planets revealed by stellar variability Formation of neutron stars revealed by GRB's Death of massive stars revealed by Type II SN

Astronomical Drivers: Pre-Discovery Mining

The progenitor of SN1999gi is <9 M: found from mining pre-discovery HST images.(Smartt et al, 2001, ApJ, 556, L29)


Huge data sets open up the possibility to find rare objects Those that are hard to find:

Brown dwarfs, have unique red colours Those that are intrinsically rare

High-z quasars, stand out due to suppression of their blue colour by the Ly- forest

Astronomical Drivers: Rare Objects

Hi-z QSO's found from SDSS multi-colour data: the shaded area is domain of QSO's, solid line is track for increasing z (Fan et al, 2001, ApJ, 121, 31)


Large datasets open the possibility to discover new objects Those that have been missed before because

they are extremely rare or short-lived Those that have previously been

misclassified, revealed as outliers in new parameter space correlations

Astronomical Drivers: New Objects

DPOSS group, during searches for high-z quasars, have discovered peculiar objects, this one perhapsa BAL QSO (Djorgovski et al, 2001,PASP, 225, 52)


Astronomical Drivers: Cosmology thru SN Ia


Example SCP datafrom Perlmutter, 2001

snap.lbl.gov/pubdocs/SAUL.pdf

AstroGrid will enable rapid typing of search fields via galaxy photo-z'sin support of UK/EU groups, e.g. EC RTN SN network.Issue: large scale image deconvolution matching candidates to z

Astronomical Drivers: The X-ray Universe


AstroGrid will enhance the ability toclassify new X-ray objects and speedobject sample creation Issue: federating large scale multi-λ data

The ELIAS team is using Chandra, XMM, ISO, SCUBA, optical imagingand VLA data to study a number of fields: seewww.roe.ac.uk/~omar/survey.html

Astronomical Drivers: WD's and Dark Matter


WD0346+246, moving at 1.3''/year,was discovered by Hambly et al.,(2000, Nature, 403, 57). It is cool,but has blue colours in the IR due tocollision absorption by H

2

AstroGrid products will enable the federation of multi-epoch data to increase the number of White Dwarf's known. With IRdata cool objects will be foundimpacting on the Dark Matter problem. Issue: TB's of multi-epoch, multi-colour data, finding moving objects

Astronomical Drivers: Solar Complexity


Solar-B will be launched in 2005, aimed at studying the interaction between the Sun's magnetic fieldand its corona

Solar-B will be supportedby complementary space/ground data.

Science often targets events with many multi-λobservations required

Image from Marshall SpaceFlight Centre, NASA

science.msfc.nasa.gov/ssl/pad/solar/Beyond_Solar-B.htm

AstroGrid: framework for applications

Enable future development and deployment of AstroGrid and External (e.g NVO) tools:

A NVO prototype of an automated discovery tool for arcs developedby A Szalay

Automated detection of outliersin SDSS two colour data(Connolly et al, 2001, AJ in press)


AstroGrid: a Typical Challenge

• Problems in matching multi-λ survey data: Differences in angular resolution, s/n ratios, backgrounds, etc (Djorgovski et al, 2001, astro-ph/0108346)



Astronomical e-Science Initiatives

AstroGrid www.astrogrid.ac.uk

European Grid of Solar Observations www.mssl.ucl.ac.uk/gri/egso

Astrophysical Virtual Observatory www.eso.org/avoAstroVirtel ecf.hq.eso.org/astrovirtel

National Virtual Observatory www.us-vo.org

The Digital Sky www.cacr.caltech.edu/SDA/digital_sky.html

The Virtual Sky Project virtualsky.org

Sloan Digital Sky Survey: SkyServer skyserver.fnal.gov/en

Skyview skyview.gsfc.nasa.gov

Rapid development of science requirements Emphasis on user input via consultation

Development of 'use cases' Example: Formation of large scale structure

Construct unbiased cluster of galaxies sample over z to test various cosmological models of galaxy formation

Need to operate on large data sets, constructing simulated surveys to test for selection effects

Generate predictions of observed sample properties and compare with observed samples

Development of 'White Paper' developing AstroGrid's role for Phase-B and in the context of larger VO picture

Towards Phase-B: The Science Case


Developing the Science Requirements Recognising the specific need to support UK science and

associated key datasets was the backdrop to the genesis of the AstroGrid program

Focus on community involvement

Development of a few scenarios for AstroGrid, highlighting there capabilities and application to science areas

Invite comment from community Interaction channels will be via

The PS visiting astronomy groups

Presentations, both face to face and on-line Development of a suitable questionnaire

Organise 2nd AstroGrid Workshop: Mar/Apr 2002.


Use Case Development: practical req's


In addition to development of the science requirements, generic use cases are being developed, e.g.: Log on remotely, operate remotely Donate a small catalogue to the grid Fix the WCS in a FITS image Find observations via instrument footprint Synthesis colour data for a set of objects Matching objects by position with errors

As the science requirements are formalised, use cases will be derived from them

A Baseline Architecture for AstroGrid


Architecture must address Top Level Design Issues:

Ease of use, emphasising a uniform appearance Functionality Speed of Response Relevance Cost Reliability Scalabilty Outreach Level of data-provider buy in

Architecture: Quality Assurance


AstroGrid federation capabilities will characterise quality parameters, e.g.: Positional accuracy →reference frames Photometric accuracy →std colour systems Completeness →weight maps Temporal resolution Wavelength resolution and coverage Photometric system Image distrotions Artifacts Catalogues →criteria used to ID objects, etc.

Architecture: Data Inclusion Model


AstroGrid will potentially enable any data sets to be accessed through AstroGrid services.

However the data must be adequately described.

Data How

any anyrestricted restricted

AstroGrid any restricted (described)

Unified Modelling Language (UML)

representation of the query architecture


Architecture Overview


Major elements in the query form a circular flow:

> a command flow involving user initiation> an inner 'grid' loop with bulk data flows

- Client s/w WP-A3 <=======<- Query Engine WP-A2 | |- Resource Management WP-A2 <====<====< |- Resource Mapping/index WP-A2 | | |- Data Harvesting WP-A2 | | |- Data Services WP-A4 | | |- Results Management WP-A4 | | |- Data Mining WP-A4 >====>====> |- Visualisation WP-A3 |=======^

Noted above are WP areas leading expansion of the architecture inthat area

Architecture Overview: UML view


Detailing the Architecture


The Data flow diagrams (DFD's) show a functional decomposition

They do not say how functions are distributed across hardware or software

Grid 'layer' diagrams are developed later when the design has been matched to the analysis

The DFD's emphasise the flow of data (s/w distribution), the transformations of data (s/w re-use) and the nature of data between transformations (interoperability)

Flows described in a Data Dictionary Transforms described in Class-Responsibility-

Collaboration (CRC) notes


UML representation of the 'query' process

Diagram to be read with: CRC Notes Data Dictionary

UML representation of the 'Harvest' process


Diagram to be read with: CRC Notes Data Dictionary

Extract from the Data Dictionary


Query A list of constraints on data to be extracted from the Grid

Authorisation Something that indicates that a user is allowed to use a

resource up to some certain level

Table meta-data Describe a table of data sufficiently that the table can be

understood by Grid s/w and transformed as necessary

Formal catalogue-extract Tabular data in AstroGrid favoured form. The catalogue

extracts need to be merged and transformed. A rich format such as XML is required.

Extract from the 'query' CRC


Query-form display Displays forms to capture queries, passes contents of form to

Grid, passes user's credentials to the Grid.

Query scheduler Determines whether a query can be run immediately; whether

is must be run in batch; whether it needs prior approval for use in resources.

Resource authoriser Determines whether a given user is allowed to use a given

resource at a particular level. Probably needs to be built from scratch

The NVO Model: A Comparison


NVO layer model, figure taken fromnvo-proj.pdf at www.us-vo.org

Usual 'Grid' fabric, resource& collective layers added to with astro-specfic user &connectivity layers

NVO focussed on (mainly)optical and near-IR large scale survey data sets.

AstroGrid is a major new UK funded e-science initiative

In partnership with EU centres it will play a lead role in the realisation of a European Virtual Observatory

AstroGrid is poised to significantly enhance the opportunities of the UK astronomical research community

Concluding Remarks


Documents

Science Overview Architectural Status ( k) Nicholas Walton IoA, Cambridge N A Walton: Science Overview: AstroGrid Collaboration